2024-03-22T01:52:07.445 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-22T01:52:07.445 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780 -- /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/orig.config.yaml 2024-03-22T01:52:07.508 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T01:52:07.605 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780 branch: wip-batrick-testing-20240321.172515 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7615780' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240321.172515 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240321.172515 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 316 sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e sleep_before_teardown: 0 subset: 420/512 suite: fs suite_branch: wip-batrick-testing-20240321.172515 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABBEJJ12xVeU9VVaU7NVr9a5HHvBVgeLXII/Azl/ExMNqZ9vKDXvj6JO64DOTTW2+8Q/3XW1ZbVtXg9Jq8DBy0= smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcvtlWFYaAxAV7dno+NRUQgPrLuxHpRzJclzqb8XzmZ47IN8waAaMBXKI6oJCaTkx5G96PlyI5zFOO/ure4baM= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr+YhhK5n9sH0EXt+XDMAMa+El1iIL7sgm3Fd58rBkOpLPMTCoB97q5tRyIZsQbSzCcJFqVOC3AuLbsggY1gk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_01:06:30 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-22T01:52:07.605 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa; will attempt to use it 2024-03-22T01:52:07.606 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks 2024-03-22T01:52:07.606 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-22T01:52:07.608 INFO:teuthology.task.internal:Checking packages... 2024-03-22T01:52:07.632 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e' 2024-03-22T01:52:07.632 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T01:52:07.632 INFO:teuthology.packaging:ref: None 2024-03-22T01:52:07.632 INFO:teuthology.packaging:tag: None 2024-03-22T01:52:07.632 INFO:teuthology.packaging:branch: wip-batrick-testing-20240321.172515 2024-03-22T01:52:07.632 INFO:teuthology.packaging:sha1: c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T01:52:07.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240321.172515 2024-03-22T01:52:07.773 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2344.gc3a04c0e 2024-03-22T01:52:07.776 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-22T01:52:07.782 INFO:teuthology.task.internal:no buildpackages task found 2024-03-22T01:52:07.782 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-22T01:52:07.803 INFO:teuthology.task.internal:Saving configuration 2024-03-22T01:52:07.823 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-22T01:52:07.833 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T01:52:07.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.866723', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABBEJJ12xVeU9VVaU7NVr9a5HHvBVgeLXII/Azl/ExMNqZ9vKDXvj6JO64DOTTW2+8Q/3XW1ZbVtXg9Jq8DBy0='} 2024-03-22T01:52:07.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.865487', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcvtlWFYaAxAV7dno+NRUQgPrLuxHpRzJclzqb8XzmZ47IN8waAaMBXKI6oJCaTkx5G96PlyI5zFOO/ure4baM='} 2024-03-22T01:52:07.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.864015', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr+YhhK5n9sH0EXt+XDMAMa+El1iIL7sgm3Fd58rBkOpLPMTCoB97q5tRyIZsQbSzCcJFqVOC3AuLbsggY1gk='} 2024-03-22T01:52:07.902 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-22T01:52:07.909 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-22T01:52:07.909 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-22T01:52:07.909 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-22T01:52:07.909 INFO:teuthology.run_tasks:Running task console_log... 2024-03-22T01:52:08.009 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3d2c890160>, signals=[15]) 2024-03-22T01:52:08.009 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-22T01:52:08.015 INFO:teuthology.task.internal:Opening connections... 2024-03-22T01:52:08.015 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T01:52:08.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:52:08.123 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-03-22T01:52:08.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:52:08.217 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-03-22T01:52:08.218 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:52:08.311 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-22T01:52:08.318 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-03-22T01:52:08.336 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-03-22T01:52:08.336 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-03-22T01:52:08.395 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:52:08.396 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:52:08.397 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:52:08.397 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:52:08.397 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:52:08.397 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-03-22T01:52:08.424 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-03-22T01:52:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-03-22T01:52:08.442 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-03-22T01:52:08.500 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:52:08.501 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:52:08.501 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-03-22T01:52:08.524 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-03-22T01:52:08.541 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-03-22T01:52:08.541 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-03-22T01:52:08.598 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-03-22T01:52:08.599 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T01:52:08.600 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T01:52:08.600 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T01:52:08.600 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-03-22T01:52:08.624 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-22T01:52:08.634 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-22T01:52:08.644 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-22T01:52:08.645 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:52:08.647 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:52:08.650 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-03-22T01:52:08.666 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-22T01:52:08.672 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-22T01:52:08.672 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:52:08.705 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:52:08.707 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-03-22T01:52:08.818 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-22T01:52:08.892 INFO:teuthology.run_tasks:Running task kernel... 2024-03-22T01:52:08.907 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-22T01:52:08.907 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-22T01:52:08.907 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-22T01:52:08.907 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:52:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:52:08.925 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-22T01:52:08.983 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-03-22T01:52:08.983 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-361.el9.x86_64 2024-03-22T01:52:08.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-03-22T01:52:11.080 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-22T01:52:12.522 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-22T01:52:13.554 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-22T01:52:29.629 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:14 2024-03-22T01:52:33.771 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-22T01:52:35.398 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-22T01:52:36.809 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-22T01:52:41.873 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-22T01:52:44.097 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T01:52:44.097 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T01:52:44.200 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-22T01:52:44.211 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-22T01:52:44.211 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:52:44.212 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 76 M 2024-03-22T01:52:44.213 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 123 M 2024-03-22T01:52:44.213 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-22T01:52:44.800 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 99 kB/s | 44 kB 00:00 2024-03-22T01:52:46.158 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.2 MB/s | 15 MB 00:01 2024-03-22T01:52:46.525 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-22T01:52:47.134 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-22T01:52:47.137 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-22T01:52:47.137 INFO:teuthology.orchestra.run.smithi028.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-22T01:52:47.830 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-22T01:52:48.256 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-22T01:52:48.256 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-22T01:52:49.328 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-22T01:52:49.328 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-22T01:52:51.950 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-22T01:52:52.572 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T01:52:52.659 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:52:55.542 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:52:55.678 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:53:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:53:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:36.713 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:36.713 INFO:teuthology.orchestra.run.smithi028.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T01:53:36.713 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:53:36.758 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T01:53:37.168 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T01:53:37.169 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T01:53:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T01:53:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:53:37.576 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-22T01:53:37.576 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T01:53:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T01:53:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T01:53:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T01:53:37.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:53:37.577 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-22T01:53:37.900 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T01:53:38.000 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T01:53:38.000 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T01:53:38.000 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T01:53:38.000 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T01:53:38.000 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T01:53:38.001 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-22T01:53:38.001 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T01:53:38.001 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T01:53:38.084 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T01:53:38.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T01:53:38.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T01:53:38.133 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T01:53:38.134 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T01:53:38.135 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T01:53:38.135 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-03-22T01:53:38.135 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T01:53:38.135 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T01:53:38.136 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T01:53:38.136 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-430.el9.x86_64 2024-03-22T01:53:38.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T01:53:38.697 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-03-22T01:53:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T01:53:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-03-22T01:53:40.053 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-03-22T01:53:40.069 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.mdEzvD66tJ 2024-03-22T01:53:40.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mdEzvD66tJ 2024-03-22T01:53:40.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.mdEzvD66tJ 2024-03-22T01:53:40.275 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.mdEzvD66tJ is 6KB 2024-03-22T01:53:40.330 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.mdEzvD66tJ 2024-03-22T01:53:40.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T01:53:40.415 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T01:53:40.415 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T01:53:40.415 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T01:53:40.415 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T01:53:40.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T01:53:40.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-03-22T01:54:10.581 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-22T01:54:10.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-22T01:54:10.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:54:29.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-03-22T01:54:38.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-22T01:54:38.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T01:55:10.008 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-22T01:55:10.461 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-22T01:55:10.461 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-22T01:55:10.461 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-03-22T01:55:10.475 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-430.el9.x86_64 2024-03-22T01:55:10.475 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T01:55:10.475 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T01:55:10.475 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T01:55:11.476 DEBUG:teuthology.parallel:result is None 2024-03-22T01:55:11.477 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-22T01:55:11.489 INFO:teuthology.task.internal:Creating test directory... 2024-03-22T01:55:11.489 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:55:11.492 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:55:11.494 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T01:55:11.512 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-22T01:55:11.623 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-22T01:55:11.732 INFO:teuthology.task.internal:Creating archive directory... 2024-03-22T01:55:11.733 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:55:11.735 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:55:11.737 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T01:55:12.048 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-22T01:55:12.058 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-22T01:55:12.059 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:55:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:55:12.075 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T01:55:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:55:12.092 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:55:12.094 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:55:12.118 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T01:55:12.166 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.172 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.200 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.202 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.229 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.254 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T01:55:12.256 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-22T01:55:12.264 INFO:teuthology.task.internal:Configuring sudo... 2024-03-22T01:55:12.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:55:12.266 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:55:12.269 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T01:55:12.299 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-22T01:55:12.318 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-22T01:55:12.318 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:55:12.338 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:55:12.340 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T01:55:12.356 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:12.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:12.487 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:12.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:12.618 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T01:55:12.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:55:12.690 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:12.729 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:12.787 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:12.872 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:12.916 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T01:55:12.916 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:55:12.985 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:13.024 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T01:55:13.087 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:13.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T01:55:13.220 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T01:55:13.220 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T01:55:13.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-03-22T01:55:13.293 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-03-22T01:55:13.296 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-03-22T01:55:13.337 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:55:13.342 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:55:13.380 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T01:55:13.790 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-22T01:55:13.798 INFO:teuthology.task.internal:Starting timer... 2024-03-22T01:55:13.799 INFO:teuthology.run_tasks:Running task pcp... 2024-03-22T01:55:13.913 INFO:teuthology.run_tasks:Running task selinux... 2024-03-22T01:55:13.966 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-22T01:55:13.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-03-22T01:55:14.017 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-03-22T01:55:14.019 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-03-22T01:55:14.074 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-03-22T01:55:14.076 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-03-22T01:55:14.137 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-03-22T01:55:14.139 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-22T01:55:14.139 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-03-22T01:55:14.163 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-03-22T01:55:14.164 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-03-22T01:55:14.190 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-03-22T01:55:14.190 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-03-22T01:55:14.210 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-03-22T01:55:14.210 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi083.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2024-03-22T01:55:14.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:55:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:55:14.238 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:55:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:55:14.266 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T01:55:14.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T01:55:14.295 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-22T01:55:14.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:55:14.416 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:55:14.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-03-22T01:55:14.486 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-22T01:55:14.495 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-22T01:55:14.501 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-22T01:55:14.648 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-22T01:55:14.664 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-22T01:55:14.666 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi083.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-22T02:00:05.896 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi083.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2024-03-22T02:00:05.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-22T02:00:05.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:00:05.989 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-03-22T02:00:06.066 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-03-22T02:00:06.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-22T02:00:06.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:00:06.156 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-03-22T02:00:06.224 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-03-22T02:00:06.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-22T02:00:06.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T02:00:06.316 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-22T02:00:06.386 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-22T02:00:06.386 INFO:teuthology.run_tasks:Running task clock... 2024-03-22T02:00:06.397 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-22T02:00:06.397 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T02:00:06.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:00:06.400 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T02:00:06.400 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:00:06.402 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T02:00:06.402 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T02:00:06.430 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T02:00:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T02:00:06.446 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T02:00:06.448 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T02:00:06.460 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T02:00:06.475 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-03-22T02:00:06.478 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T02:00:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-03-22T02:00:06.488 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-03-22T02:00:06.492 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-03-22T02:00:06.505 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T02:00:06.508 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-03-22T02:00:06.509 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T02:00:06.521 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T02:00:06.522 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-03-22T02:00:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T02:00:06.538 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T02:00:06.554 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T02:00:06.565 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.569 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:00:06.582 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:00:06.582 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-22T02:00:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.583 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.598 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.602 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T02:00:06.602 INFO:teuthology.run_tasks:Running task install... 2024-03-22T02:00:06.610 DEBUG:teuthology.task.install:project ceph 2024-03-22T02:00:06.611 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'}} 2024-03-22T02:00:06.611 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'} 2024-03-22T02:00:06.611 INFO:teuthology.task.install:Using flavor: default 2024-03-22T02:00:06.619 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-22T02:00:06.619 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-22T02:00:06.619 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T02:00:06.620 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:00:06.622 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T02:00:06.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:00:06.623 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'tag': None, 'wait_for_package': False} 2024-03-22T02:00:06.623 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:00:06.766 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T02:00:06.766 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T02:00:06.771 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T02:00:06.771 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T02:00:06.852 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/ 2024-03-22T02:00:06.852 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2344.gc3a04c0e 2024-03-22T02:00:06.876 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T02:00:06.876 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:00:06.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T02:00:06.904 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T02:00:06.904 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T02:00:06.945 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T02:00:06.945 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:00:06.945 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T02:00:06.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T02:00:06.974 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T02:00:06.974 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T02:00:07.020 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-22T02:00:07.021 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:00:07.021 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-22T02:00:07.042 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T02:00:07.050 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-22T02:00:07.050 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-22T02:00:07.051 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T02:00:07.117 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-03-22T02:00:07.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-03-22T02:00:07.120 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-22T02:00:07.122 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T02:00:07.188 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-03-22T02:00:07.190 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-03-22T02:00:07.200 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-22T02:00:07.267 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-03-22T02:00:07.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-03-22T02:00:07.370 INFO:teuthology.orchestra.run.smithi028.stdout:74 files removed 2024-03-22T02:00:07.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T02:00:07.423 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-03-22T02:00:07.450 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T02:00:07.502 INFO:teuthology.orchestra.run.smithi083.stdout:74 files removed 2024-03-22T02:00:07.529 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-22T02:00:08.120 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 200 kB/s | 83 kB 00:00 2024-03-22T02:00:08.143 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-03-22T02:00:08.172 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 237 kB/s | 83 kB 00:00 2024-03-22T02:00:08.477 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 39 kB/s | 12 kB 00:00 2024-03-22T02:00:08.548 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-03-22T02:00:08.562 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-03-22T02:00:08.828 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-22T02:00:08.890 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-22T02:00:08.938 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-22T02:00:09.392 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-22T02:00:09.397 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-22T02:00:09.790 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-22T02:00:10.784 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-22T02:00:10.861 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-22T02:00:11.257 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-22T02:00:11.742 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-22T02:00:12.048 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-22T02:00:12.529 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-03-22T02:00:13.749 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-03-22T02:00:15.485 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-03-22T02:00:19.427 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-22T02:00:20.978 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-22T02:00:22.105 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-22T02:00:26.250 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 707 kB/s | 6.1 MB 00:08 2024-03-22T02:00:27.055 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-22T02:00:28.150 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-22T02:00:29.374 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:29.375 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:29.386 INFO:teuthology.orchestra.run.smithi083.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T02:00:29.387 INFO:teuthology.orchestra.run.smithi083.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T02:00:29.388 INFO:teuthology.orchestra.run.smithi083.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T02:00:29.389 INFO:teuthology.orchestra.run.smithi083.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:29.391 INFO:teuthology.orchestra.run.smithi083.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:29.393 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T02:00:29.394 INFO:teuthology.orchestra.run.smithi083.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T02:00:29.473 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-03-22T02:00:29.493 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T02:00:29.494 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T02:00:29.495 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T02:00:29.496 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T02:00:29.497 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T02:00:29.498 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T02:00:29.499 INFO:teuthology.orchestra.run.smithi083.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T02:00:29.500 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T02:00:29.501 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T02:00:29.502 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T02:00:29.503 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T02:00:29.504 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T02:00:29.505 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T02:00:29.506 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T02:00:29.507 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T02:00:29.508 INFO:teuthology.orchestra.run.smithi083.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:============================================================================================ 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:Install 150 Packages 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 24 Packages 2024-03-22T02:00:29.509 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:00:29.510 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 226 M 2024-03-22T02:00:29.510 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-03-22T02:00:30.189 INFO:teuthology.orchestra.run.smithi083.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-03-22T02:00:30.465 INFO:teuthology.orchestra.run.smithi083.stdout:(2/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 3.2 MB/s | 890 kB 00:00 2024-03-22T02:00:30.531 INFO:teuthology.orchestra.run.smithi083.stdout:(3/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-22T02:00:30.548 INFO:teuthology.orchestra.run.smithi083.stdout:(4/174): ceph-immutable-object-cache-19.0.0-234 1.7 MB/s | 145 kB 00:00 2024-03-22T02:00:30.690 INFO:teuthology.orchestra.run.smithi083.stdout:(5/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 15 MB/s | 2.4 MB 00:00 2024-03-22T02:00:30.818 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 21 MB 00:02 2024-03-22T02:00:30.924 INFO:teuthology.orchestra.run.smithi083.stdout:(6/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 4.0 MB/s | 1.5 MB 00:00 2024-03-22T02:00:30.999 INFO:teuthology.orchestra.run.smithi083.stdout:(7/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x86 15 MB/s | 4.7 MB 00:00 2024-03-22T02:00:31.624 INFO:teuthology.orchestra.run.smithi083.stdout:(8/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el9 16 MB/s | 9.8 MB 00:00 2024-03-22T02:00:31.666 INFO:teuthology.orchestra.run.smithi083.stdout:(9/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el9 602 kB/s | 25 kB 00:00 2024-03-22T02:00:33.175 INFO:teuthology.orchestra.run.smithi083.stdout:(10/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x8 7.5 MB/s | 17 MB 00:02 2024-03-22T02:00:33.225 INFO:teuthology.orchestra.run.smithi083.stdout:(11/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 4.4 MB/s | 222 kB 00:00 2024-03-22T02:00:33.300 INFO:teuthology.orchestra.run.smithi083.stdout:(12/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 360 kB/s | 27 kB 00:00 2024-03-22T02:00:33.367 INFO:teuthology.orchestra.run.smithi083.stdout:(13/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 476 kB/s | 32 kB 00:00 2024-03-22T02:00:33.526 INFO:teuthology.orchestra.run.smithi083.stdout:(14/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 4.6 MB/s | 747 kB 00:00 2024-03-22T02:00:33.610 INFO:teuthology.orchestra.run.smithi083.stdout:(15/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 1.9 MB/s | 163 kB 00:00 2024-03-22T02:00:33.676 INFO:teuthology.orchestra.run.smithi083.stdout:(16/174): librados-devel-19.0.0-2344.gc3a04c0e. 1.9 MB/s | 127 kB 00:00 2024-03-22T02:00:33.769 INFO:teuthology.orchestra.run.smithi083.stdout:(17/174): libradosstriper1-19.0.0-2344.gc3a04c0 5.3 MB/s | 501 kB 00:00 2024-03-22T02:00:34.186 INFO:teuthology.orchestra.run.smithi083.stdout:(18/174): ceph-common-19.0.0-2344.gc3a04c0e.el9 4.9 MB/s | 21 MB 00:04 2024-03-22T02:00:34.269 INFO:teuthology.orchestra.run.smithi083.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 543 kB/s | 45 kB 00:00 2024-03-22T02:00:34.386 INFO:teuthology.orchestra.run.smithi083.stdout:(20/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 8.3 MB/s | 5.1 MB 00:00 2024-03-22T02:00:34.411 INFO:teuthology.orchestra.run.smithi083.stdout:(21/174): python3-ceph-common-19.0.0-2344.gc3a0 928 kB/s | 131 kB 00:00 2024-03-22T02:00:34.453 INFO:teuthology.orchestra.run.smithi083.stdout:(22/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 2.4 MB/s | 161 kB 00:00 2024-03-22T02:00:34.904 INFO:teuthology.orchestra.run.smithi083.stdout:(23/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 15 MB/s | 48 MB 00:03 2024-03-22T02:00:34.929 INFO:teuthology.orchestra.run.smithi083.stdout:(24/174): python3-rados-19.0.0-2344.gc3a04c0e.e 621 kB/s | 321 kB 00:00 2024-03-22T02:00:34.954 INFO:teuthology.orchestra.run.smithi083.stdout:(25/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 2.0 MB/s | 99 kB 00:00 2024-03-22T02:00:34.979 INFO:teuthology.orchestra.run.smithi083.stdout:(26/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 567 kB/s | 298 kB 00:00 2024-03-22T02:00:35.005 INFO:teuthology.orchestra.run.smithi083.stdout:(27/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-22T02:00:35.080 INFO:teuthology.orchestra.run.smithi083.stdout:(28/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-22T02:00:35.105 INFO:teuthology.orchestra.run.smithi083.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2344.g 338 kB/s | 34 kB 00:00 2024-03-22T02:00:35.130 INFO:teuthology.orchestra.run.smithi083.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 2.7 MB/s | 140 kB 00:00 2024-03-22T02:00:35.189 INFO:teuthology.orchestra.run.smithi083.stdout:(31/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 13 MB/s | 3.1 MB 00:00 2024-03-22T02:00:35.264 INFO:teuthology.orchestra.run.smithi083.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2344.gc3 3.2 MB/s | 248 kB 00:00 2024-03-22T02:00:35.323 INFO:teuthology.orchestra.run.smithi083.stdout:(33/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 847 kB/s | 49 kB 00:00 2024-03-22T02:00:35.382 INFO:teuthology.orchestra.run.smithi083.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2344.gc 282 kB/s | 16 kB 00:00 2024-03-22T02:00:35.448 INFO:teuthology.orchestra.run.smithi083.stdout:(35/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 3.9 MB/s | 264 kB 00:00 2024-03-22T02:00:35.532 INFO:teuthology.orchestra.run.smithi083.stdout:(36/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 8.8 MB/s | 750 kB 00:00 2024-03-22T02:00:35.590 INFO:teuthology.orchestra.run.smithi083.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 314 kB/s | 18 kB 00:00 2024-03-22T02:00:35.649 INFO:teuthology.orchestra.run.smithi083.stdout:(38/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 6.5 MB/s | 3.5 MB 00:00 2024-03-22T02:00:35.766 INFO:teuthology.orchestra.run.smithi083.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-22T02:00:35.786 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 868 kB/s | 19 MB 00:22 2024-03-22T02:00:35.867 INFO:teuthology.orchestra.run.smithi083.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-22T02:00:35.892 INFO:teuthology.orchestra.run.smithi083.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-22T02:00:35.917 INFO:teuthology.orchestra.run.smithi083.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 307 kB 00:00 2024-03-22T02:00:35.951 INFO:teuthology.orchestra.run.smithi083.stdout:(43/174): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-03-22T02:00:35.967 INFO:teuthology.orchestra.run.smithi083.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 425 kB/s | 32 kB 00:00 2024-03-22T02:00:36.002 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-22T02:00:36.084 INFO:teuthology.orchestra.run.smithi083.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-03-22T02:00:36.110 INFO:teuthology.orchestra.run.smithi083.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 122 kB/s | 17 kB 00:00 2024-03-22T02:00:36.136 INFO:teuthology.orchestra.run.smithi083.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 10 kB 00:00 2024-03-22T02:00:36.160 INFO:teuthology.orchestra.run.smithi083.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-22T02:00:36.186 INFO:teuthology.orchestra.run.smithi083.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-22T02:00:36.219 INFO:teuthology.orchestra.run.smithi083.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-22T02:00:36.244 INFO:teuthology.orchestra.run.smithi083.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.8 MB/s | 578 kB 00:00 2024-03-22T02:00:36.269 INFO:teuthology.orchestra.run.smithi083.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 46 kB 00:00 2024-03-22T02:00:36.295 INFO:teuthology.orchestra.run.smithi083.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-22T02:00:36.320 INFO:teuthology.orchestra.run.smithi083.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-22T02:00:36.345 INFO:teuthology.orchestra.run.smithi083.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 690 kB/s | 34 kB 00:00 2024-03-22T02:00:36.370 INFO:teuthology.orchestra.run.smithi083.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-22T02:00:36.479 INFO:teuthology.orchestra.run.smithi083.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-22T02:00:36.521 INFO:teuthology.orchestra.run.smithi083.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 8.8 MB/s | 1.5 MB 00:00 2024-03-22T02:00:36.546 INFO:teuthology.orchestra.run.smithi083.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.6 MB/s | 811 kB 00:00 2024-03-22T02:00:36.571 INFO:teuthology.orchestra.run.smithi083.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-03-22T02:00:36.613 INFO:teuthology.orchestra.run.smithi083.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-22T02:00:36.764 INFO:teuthology.orchestra.run.smithi083.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-22T02:00:36.814 INFO:teuthology.orchestra.run.smithi083.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 320 kB/s | 78 kB 00:00 2024-03-22T02:00:36.923 INFO:teuthology.orchestra.run.smithi083.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-22T02:00:36.948 INFO:teuthology.orchestra.run.smithi083.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 135 kB/s | 24 kB 00:00 2024-03-22T02:00:36.973 INFO:teuthology.orchestra.run.smithi083.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 116 kB/s | 18 kB 00:00 2024-03-22T02:00:36.998 INFO:teuthology.orchestra.run.smithi083.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2024-03-22T02:00:37.023 INFO:teuthology.orchestra.run.smithi083.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-22T02:00:37.147 INFO:teuthology.orchestra.run.smithi083.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-22T02:00:37.173 INFO:teuthology.orchestra.run.smithi083.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-03-22T02:00:37.198 INFO:teuthology.orchestra.run.smithi083.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-22T02:00:37.223 INFO:teuthology.orchestra.run.smithi083.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-22T02:00:37.248 INFO:teuthology.orchestra.run.smithi083.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-22T02:00:37.274 INFO:teuthology.orchestra.run.smithi083.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-22T02:00:37.299 INFO:teuthology.orchestra.run.smithi083.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-22T02:00:37.324 INFO:teuthology.orchestra.run.smithi083.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-03-22T02:00:37.358 INFO:teuthology.orchestra.run.smithi083.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-22T02:00:37.383 INFO:teuthology.orchestra.run.smithi083.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 513 kB/s | 43 kB 00:00 2024-03-22T02:00:37.808 INFO:teuthology.orchestra.run.smithi083.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-22T02:00:37.833 INFO:teuthology.orchestra.run.smithi083.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 86 kB/s | 39 kB 00:00 2024-03-22T02:00:37.859 INFO:teuthology.orchestra.run.smithi083.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 87 kB/s | 41 kB 00:00 2024-03-22T02:00:37.884 INFO:teuthology.orchestra.run.smithi083.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-22T02:00:37.909 INFO:teuthology.orchestra.run.smithi083.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-22T02:00:37.934 INFO:teuthology.orchestra.run.smithi083.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 838 kB/s | 63 kB 00:00 2024-03-22T02:00:37.960 INFO:teuthology.orchestra.run.smithi083.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 588 kB/s | 44 kB 00:00 2024-03-22T02:00:37.985 INFO:teuthology.orchestra.run.smithi083.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-22T02:00:38.010 INFO:teuthology.orchestra.run.smithi083.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-22T02:00:38.044 INFO:teuthology.orchestra.run.smithi083.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-22T02:00:38.061 INFO:teuthology.orchestra.run.smithi083.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-22T02:00:38.086 INFO:teuthology.orchestra.run.smithi083.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-22T02:00:38.111 INFO:teuthology.orchestra.run.smithi083.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.7 MB/s | 187 kB 00:00 2024-03-22T02:00:38.458 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:38.459 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:38.472 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T02:00:38.473 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T02:00:38.474 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T02:00:38.475 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:38.477 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:38.480 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T02:00:38.481 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T02:00:38.503 INFO:teuthology.orchestra.run.smithi083.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 499 kB/s | 195 kB 00:00 2024-03-22T02:00:38.554 INFO:teuthology.orchestra.run.smithi083.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-22T02:00:38.565 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-03-22T02:00:38.578 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-22T02:00:38.578 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-03-22T02:00:38.578 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-22T02:00:38.578 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-03-22T02:00:38.578 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T02:00:38.579 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T02:00:38.580 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-03-22T02:00:38.581 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T02:00:38.582 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-03-22T02:00:38.583 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T02:00:38.584 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T02:00:38.585 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T02:00:38.586 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T02:00:38.587 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T02:00:38.588 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T02:00:38.589 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T02:00:38.590 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T02:00:38.591 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T02:00:38.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:Install 150 Packages 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 24 Packages 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 226 M 2024-03-22T02:00:38.593 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-03-22T02:00:38.604 INFO:teuthology.orchestra.run.smithi083.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-03-22T02:00:38.721 INFO:teuthology.orchestra.run.smithi083.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 129 kB/s | 21 kB 00:00 2024-03-22T02:00:38.762 INFO:teuthology.orchestra.run.smithi083.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 161 kB/s | 25 kB 00:00 2024-03-22T02:00:38.938 INFO:teuthology.orchestra.run.smithi083.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-22T02:00:39.063 INFO:teuthology.orchestra.run.smithi083.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-22T02:00:39.164 INFO:teuthology.orchestra.run.smithi083.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.1 MB/s | 216 kB 00:00 2024-03-22T02:00:39.214 INFO:teuthology.orchestra.run.smithi083.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 700 kB/s | 35 kB 00:00 2024-03-22T02:00:39.264 INFO:teuthology.orchestra.run.smithi083.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 892 kB/s | 44 kB 00:00 2024-03-22T02:00:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-22T02:00:39.314 INFO:teuthology.orchestra.run.smithi083.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-22T02:00:39.487 INFO:teuthology.orchestra.run.smithi028.stdout:(2/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 4.5 MB/s | 890 kB 00:00 2024-03-22T02:00:39.531 INFO:teuthology.orchestra.run.smithi083.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-22T02:00:39.553 INFO:teuthology.orchestra.run.smithi028.stdout:(3/174): ceph-immutable-object-cache-19.0.0-234 2.1 MB/s | 145 kB 00:00 2024-03-22T02:00:39.787 INFO:teuthology.orchestra.run.smithi028.stdout:(4/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 10 MB/s | 2.4 MB 00:00 2024-03-22T02:00:39.856 INFO:teuthology.orchestra.run.smithi083.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.6 MB/s | 4.1 MB 00:01 2024-03-22T02:00:39.954 INFO:teuthology.orchestra.run.smithi028.stdout:(5/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-22T02:00:40.446 INFO:teuthology.orchestra.run.smithi028.stdout:(6/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x86 9.6 MB/s | 4.7 MB 00:00 2024-03-22T02:00:40.478 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-22T02:00:40.738 INFO:teuthology.orchestra.run.smithi028.stdout:(7/174): ceph-common-19.0.0-2344.gc3a04c0e.el9. 13 MB/s | 21 MB 00:01 2024-03-22T02:00:41.315 INFO:teuthology.orchestra.run.smithi083.stdout:(105/174): libgfortran-11.4.1-3.el9.x86_64.rpm 249 kB/s | 803 kB 00:03 2024-03-22T02:00:41.588 INFO:teuthology.orchestra.run.smithi028.stdout:(8/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el9 12 MB/s | 9.8 MB 00:00 2024-03-22T02:00:41.688 INFO:teuthology.orchestra.run.smithi028.stdout:(9/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-03-22T02:00:41.705 INFO:teuthology.orchestra.run.smithi028.stdout:(10/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el 215 kB/s | 25 kB 00:00 2024-03-22T02:00:41.872 INFO:teuthology.orchestra.run.smithi028.stdout:(11/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x8 12 MB/s | 17 MB 00:01 2024-03-22T02:00:41.898 INFO:teuthology.orchestra.run.smithi028.stdout:(12/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 1.1 MB/s | 222 kB 00:00 2024-03-22T02:00:41.914 INFO:teuthology.orchestra.run.smithi028.stdout:(13/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 649 kB/s | 27 kB 00:00 2024-03-22T02:00:41.931 INFO:teuthology.orchestra.run.smithi028.stdout:(14/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 944 kB/s | 32 kB 00:00 2024-03-22T02:00:41.998 INFO:teuthology.orchestra.run.smithi028.stdout:(15/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 8.8 MB/s | 747 kB 00:00 2024-03-22T02:00:42.023 INFO:teuthology.orchestra.run.smithi028.stdout:(16/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 1.7 MB/s | 163 kB 00:00 2024-03-22T02:00:42.040 INFO:teuthology.orchestra.run.smithi028.stdout:(17/174): librados-devel-19.0.0-2344.gc3a04c0e. 2.9 MB/s | 127 kB 00:00 2024-03-22T02:00:42.093 INFO:teuthology.orchestra.run.smithi028.stdout:(18/174): libradosstriper1-19.0.0-2344.gc3a04c0 7.1 MB/s | 501 kB 00:00 2024-03-22T02:00:42.168 INFO:teuthology.orchestra.run.smithi028.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 602 kB/s | 45 kB 00:00 2024-03-22T02:00:42.235 INFO:teuthology.orchestra.run.smithi028.stdout:(20/174): python3-ceph-common-19.0.0-2344.gc3a0 1.9 MB/s | 131 kB 00:00 2024-03-22T02:00:42.294 INFO:teuthology.orchestra.run.smithi028.stdout:(21/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 2.7 MB/s | 161 kB 00:00 2024-03-22T02:00:42.304 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-22T02:00:42.377 INFO:teuthology.orchestra.run.smithi028.stdout:(22/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 15 MB/s | 5.1 MB 00:00 2024-03-22T02:00:42.402 INFO:teuthology.orchestra.run.smithi028.stdout:(23/174): python3-rados-19.0.0-2344.gc3a04c0e.e 2.9 MB/s | 321 kB 00:00 2024-03-22T02:00:42.427 INFO:teuthology.orchestra.run.smithi028.stdout:(24/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 5.8 MB/s | 298 kB 00:00 2024-03-22T02:00:42.453 INFO:teuthology.orchestra.run.smithi028.stdout:(25/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 1.9 MB/s | 99 kB 00:00 2024-03-22T02:00:42.486 INFO:teuthology.orchestra.run.smithi028.stdout:(26/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-22T02:00:42.548 INFO:teuthology.orchestra.run.smithi083.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 204 kB/s | 548 kB 00:02 2024-03-22T02:00:42.578 INFO:teuthology.orchestra.run.smithi028.stdout:(27/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-22T02:00:42.662 INFO:teuthology.orchestra.run.smithi028.stdout:(28/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 15 MB/s | 3.1 MB 00:00 2024-03-22T02:00:42.687 INFO:teuthology.orchestra.run.smithi028.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2344.g 312 kB/s | 34 kB 00:00 2024-03-22T02:00:42.704 INFO:teuthology.orchestra.run.smithi028.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 3.3 MB/s | 140 kB 00:00 2024-03-22T02:00:43.046 INFO:teuthology.orchestra.run.smithi028.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 9.8 MB/s | 3.5 MB 00:00 2024-03-22T02:00:43.122 INFO:teuthology.orchestra.run.smithi028.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2344.gc3 3.2 MB/s | 248 kB 00:00 2024-03-22T02:00:43.189 INFO:teuthology.orchestra.run.smithi028.stdout:(33/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 742 kB/s | 49 kB 00:00 2024-03-22T02:00:43.281 INFO:teuthology.orchestra.run.smithi028.stdout:(34/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-22T02:00:43.297 INFO:teuthology.orchestra.run.smithi028.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2344.gc 152 kB/s | 16 kB 00:00 2024-03-22T02:00:43.331 INFO:teuthology.orchestra.run.smithi028.stdout:(36/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 5.2 MB/s | 264 kB 00:00 2024-03-22T02:00:43.365 INFO:teuthology.orchestra.run.smithi028.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 547 kB/s | 18 kB 00:00 2024-03-22T02:00:43.398 INFO:teuthology.orchestra.run.smithi028.stdout:(38/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 7.3 MB/s | 750 kB 00:00 2024-03-22T02:00:43.815 INFO:teuthology.orchestra.run.smithi028.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.4 MB/s | 662 kB 00:00 2024-03-22T02:00:43.890 INFO:teuthology.orchestra.run.smithi028.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-03-22T02:00:43.924 INFO:teuthology.orchestra.run.smithi083.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 118 kB/s | 309 kB 00:02 2024-03-22T02:00:43.957 INFO:teuthology.orchestra.run.smithi083.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 195 kB/s | 863 kB 00:04 2024-03-22T02:00:44.007 INFO:teuthology.orchestra.run.smithi083.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 13 kB/s | 19 kB 00:01 2024-03-22T02:00:44.041 INFO:teuthology.orchestra.run.smithi028.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 920 kB 00:00 2024-03-22T02:00:44.074 INFO:teuthology.orchestra.run.smithi028.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 1.6 MB/s | 307 kB 00:00 2024-03-22T02:00:44.108 INFO:teuthology.orchestra.run.smithi028.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 473 kB/s | 32 kB 00:00 2024-03-22T02:00:44.441 INFO:teuthology.orchestra.run.smithi028.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 51 kB/s | 17 kB 00:00 2024-03-22T02:00:44.667 INFO:teuthology.orchestra.run.smithi028.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 45 kB/s | 10 kB 00:00 2024-03-22T02:00:44.709 INFO:teuthology.orchestra.run.smithi028.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.7 MB/s | 3.0 MB 00:00 2024-03-22T02:00:44.750 INFO:teuthology.orchestra.run.smithi028.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-22T02:00:44.784 INFO:teuthology.orchestra.run.smithi028.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-03-22T02:00:44.935 INFO:teuthology.orchestra.run.smithi028.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-03-22T02:00:45.026 INFO:teuthology.orchestra.run.smithi028.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 2.3 MB/s | 578 kB 00:00 2024-03-22T02:00:45.052 INFO:teuthology.orchestra.run.smithi028.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 399 kB/s | 46 kB 00:00 2024-03-22T02:00:45.118 INFO:teuthology.orchestra.run.smithi028.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-22T02:00:45.144 INFO:teuthology.orchestra.run.smithi028.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-22T02:00:45.186 INFO:teuthology.orchestra.run.smithi028.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-22T02:00:45.203 INFO:teuthology.orchestra.run.smithi028.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 584 kB/s | 34 kB 00:00 2024-03-22T02:00:45.536 INFO:teuthology.orchestra.run.smithi028.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 4.6 MB/s | 1.5 MB 00:00 2024-03-22T02:00:45.612 INFO:teuthology.orchestra.run.smithi028.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-03-22T02:00:45.653 INFO:teuthology.orchestra.run.smithi028.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 6.9 MB/s | 811 kB 00:00 2024-03-22T02:00:45.679 INFO:teuthology.orchestra.run.smithi028.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-22T02:00:45.863 INFO:teuthology.orchestra.run.smithi028.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-22T02:00:45.929 INFO:teuthology.orchestra.run.smithi028.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 21 MB/s | 5.7 MB 00:00 2024-03-22T02:00:45.954 INFO:teuthology.orchestra.run.smithi028.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 862 kB/s | 78 kB 00:00 2024-03-22T02:00:46.113 INFO:teuthology.orchestra.run.smithi028.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-22T02:00:46.138 INFO:teuthology.orchestra.run.smithi028.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 134 kB/s | 24 kB 00:00 2024-03-22T02:00:46.164 INFO:teuthology.orchestra.run.smithi028.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-22T02:00:46.189 INFO:teuthology.orchestra.run.smithi028.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-22T02:00:46.214 INFO:teuthology.orchestra.run.smithi028.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-22T02:00:46.331 INFO:teuthology.orchestra.run.smithi028.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-22T02:00:46.356 INFO:teuthology.orchestra.run.smithi028.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-22T02:00:46.382 INFO:teuthology.orchestra.run.smithi028.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-22T02:00:46.407 INFO:teuthology.orchestra.run.smithi028.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-22T02:00:46.432 INFO:teuthology.orchestra.run.smithi028.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-22T02:00:46.457 INFO:teuthology.orchestra.run.smithi028.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-22T02:00:46.483 INFO:teuthology.orchestra.run.smithi028.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-22T02:00:46.508 INFO:teuthology.orchestra.run.smithi028.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-22T02:00:46.533 INFO:teuthology.orchestra.run.smithi028.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-03-22T02:00:46.558 INFO:teuthology.orchestra.run.smithi028.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 854 kB/s | 43 kB 00:00 2024-03-22T02:00:46.717 INFO:teuthology.orchestra.run.smithi028.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 247 kB/s | 39 kB 00:00 2024-03-22T02:00:46.909 INFO:teuthology.orchestra.run.smithi028.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-22T02:00:46.934 INFO:teuthology.orchestra.run.smithi028.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 191 kB/s | 41 kB 00:00 2024-03-22T02:00:46.960 INFO:teuthology.orchestra.run.smithi028.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-22T02:00:46.985 INFO:teuthology.orchestra.run.smithi028.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-22T02:00:47.010 INFO:teuthology.orchestra.run.smithi028.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-22T02:00:47.036 INFO:teuthology.orchestra.run.smithi028.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-22T02:00:47.052 INFO:teuthology.orchestra.run.smithi028.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-22T02:00:47.078 INFO:teuthology.orchestra.run.smithi028.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-22T02:00:47.120 INFO:teuthology.orchestra.run.smithi028.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-22T02:00:47.145 INFO:teuthology.orchestra.run.smithi028.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-22T02:00:47.170 INFO:teuthology.orchestra.run.smithi028.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-22T02:00:47.195 INFO:teuthology.orchestra.run.smithi028.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-22T02:00:47.263 INFO:teuthology.orchestra.run.smithi028.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-22T02:00:47.321 INFO:teuthology.orchestra.run.smithi028.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-03-22T02:00:47.363 INFO:teuthology.orchestra.run.smithi028.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-22T02:00:47.406 INFO:teuthology.orchestra.run.smithi028.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-22T02:00:48.030 INFO:teuthology.orchestra.run.smithi028.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 32 kB/s | 21 kB 00:00 2024-03-22T02:00:48.047 INFO:teuthology.orchestra.run.smithi028.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 40 kB/s | 25 kB 00:00 2024-03-22T02:00:48.274 INFO:teuthology.orchestra.run.smithi083.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 6.0 kB/s | 26 kB 00:04 2024-03-22T02:00:48.557 INFO:teuthology.orchestra.run.smithi083.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 11 kB/s | 49 kB 00:04 2024-03-22T02:00:48.972 INFO:teuthology.orchestra.run.smithi028.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 265 kB/s | 245 kB 00:00 2024-03-22T02:00:49.889 INFO:teuthology.orchestra.run.smithi028.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 270 kB/s | 248 kB 00:00 2024-03-22T02:00:50.348 INFO:teuthology.orchestra.run.smithi028.stdout:(99/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 5.5 MB/s | 48 MB 00:08 2024-03-22T02:00:50.990 INFO:teuthology.orchestra.run.smithi028.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 54 kB/s | 35 kB 00:00 2024-03-22T02:00:51.123 INFO:teuthology.orchestra.run.smithi028.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 176 kB/s | 216 kB 00:01 2024-03-22T02:00:51.624 INFO:teuthology.orchestra.run.smithi028.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 70 kB/s | 44 kB 00:00 2024-03-22T02:00:51.674 INFO:teuthology.orchestra.run.smithi028.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 450 kB/s | 22 kB 00:00 2024-03-22T02:00:51.782 INFO:teuthology.orchestra.run.smithi028.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.8 MB/s | 863 kB 00:00 2024-03-22T02:00:51.824 INFO:teuthology.orchestra.run.smithi028.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 134 kB/s | 94 kB 00:00 2024-03-22T02:00:51.879 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 21 MB 00:09 2024-03-22T02:00:51.983 INFO:teuthology.orchestra.run.smithi028.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.9 MB/s | 309 kB 00:00 2024-03-22T02:00:52.016 INFO:teuthology.orchestra.run.smithi028.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-22T02:00:52.042 INFO:teuthology.orchestra.run.smithi028.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 331 kB/s | 19 kB 00:00 2024-03-22T02:00:52.092 INFO:teuthology.orchestra.run.smithi028.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-22T02:00:52.142 INFO:teuthology.orchestra.run.smithi028.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 973 kB/s | 49 kB 00:00 2024-03-22T02:00:52.201 INFO:teuthology.orchestra.run.smithi028.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-22T02:00:52.251 INFO:teuthology.orchestra.run.smithi028.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-22T02:00:52.327 INFO:teuthology.orchestra.run.smithi028.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-22T02:00:52.377 INFO:teuthology.orchestra.run.smithi028.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-03-22T02:00:52.419 INFO:teuthology.orchestra.run.smithi028.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-03-22T02:00:52.461 INFO:teuthology.orchestra.run.smithi028.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 776 kB/s | 32 kB 00:00 2024-03-22T02:00:52.503 INFO:teuthology.orchestra.run.smithi028.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 361 kB/s | 15 kB 00:00 2024-03-22T02:00:52.553 INFO:teuthology.orchestra.run.smithi028.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-03-22T02:00:52.603 INFO:teuthology.orchestra.run.smithi028.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-22T02:00:52.653 INFO:teuthology.orchestra.run.smithi028.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-22T02:00:52.695 INFO:teuthology.orchestra.run.smithi028.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 846 kB/s | 35 kB 00:00 2024-03-22T02:00:52.737 INFO:teuthology.orchestra.run.smithi028.stdout:(122/174): python3-google-auth-2.28.2-1.el9.noa 4.8 MB/s | 203 kB 00:00 2024-03-22T02:00:52.846 INFO:teuthology.orchestra.run.smithi028.stdout:(123/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-03-22T02:00:52.904 INFO:teuthology.orchestra.run.smithi028.stdout:(124/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-22T02:00:52.930 INFO:teuthology.orchestra.run.smithi028.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-22T02:00:52.946 INFO:teuthology.orchestra.run.smithi028.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-03-22T02:00:52.964 INFO:teuthology.orchestra.run.smithi028.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 530 kB/s | 18 kB 00:00 2024-03-22T02:00:53.105 INFO:teuthology.orchestra.run.smithi028.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 146 kB/s | 23 kB 00:00 2024-03-22T02:00:53.131 INFO:teuthology.orchestra.run.smithi028.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 116 kB/s | 19 kB 00:00 2024-03-22T02:00:53.156 INFO:teuthology.orchestra.run.smithi028.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-22T02:00:53.198 INFO:teuthology.orchestra.run.smithi028.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-22T02:00:53.240 INFO:teuthology.orchestra.run.smithi028.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.9 MB/s | 79 kB 00:00 2024-03-22T02:00:53.273 INFO:teuthology.orchestra.run.smithi028.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-03-22T02:00:53.299 INFO:teuthology.orchestra.run.smithi028.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 988 kB/s | 58 kB 00:00 2024-03-22T02:00:53.341 INFO:teuthology.orchestra.run.smithi028.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-22T02:00:53.366 INFO:teuthology.orchestra.run.smithi028.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-22T02:00:53.391 INFO:teuthology.orchestra.run.smithi083.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 13 kB/s | 67 kB 00:05 2024-03-22T02:00:53.391 INFO:teuthology.orchestra.run.smithi028.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-22T02:00:53.416 INFO:teuthology.orchestra.run.smithi028.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-03-22T02:00:53.434 INFO:teuthology.orchestra.run.smithi028.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 736 kB/s | 31 kB 00:00 2024-03-22T02:00:53.458 INFO:teuthology.orchestra.run.smithi028.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-03-22T02:00:53.476 INFO:teuthology.orchestra.run.smithi028.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-22T02:00:53.501 INFO:teuthology.orchestra.run.smithi028.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 846 kB/s | 36 kB 00:00 2024-03-22T02:00:53.518 INFO:teuthology.orchestra.run.smithi028.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.2 MB/s | 51 kB 00:00 2024-03-22T02:00:53.543 INFO:teuthology.orchestra.run.smithi028.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-22T02:00:53.569 INFO:teuthology.orchestra.run.smithi028.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-22T02:00:53.594 INFO:teuthology.orchestra.run.smithi028.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-03-22T02:00:53.619 INFO:teuthology.orchestra.run.smithi028.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-03-22T02:00:53.644 INFO:teuthology.orchestra.run.smithi028.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-22T02:00:53.786 INFO:teuthology.orchestra.run.smithi028.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-22T02:00:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:(150/174): openssl-devel-3.0.7-27.el9.x86_64.rp 718 kB/s | 4.1 MB 00:05 2024-03-22T02:00:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-22T02:00:54.012 INFO:teuthology.orchestra.run.smithi028.stdout:(152/174): librados2-19.0.0-2344.gc3a04c0e.el9. 9.2 MB/s | 3.4 MB 00:00 2024-03-22T02:00:54.141 INFO:teuthology.orchestra.run.smithi083.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 27 kB/s | 151 kB 00:05 2024-03-22T02:00:54.145 INFO:teuthology.orchestra.run.smithi028.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-22T02:00:54.187 INFO:teuthology.orchestra.run.smithi028.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 10 MB/s | 1.8 MB 00:00 2024-03-22T02:00:54.237 INFO:teuthology.orchestra.run.smithi028.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 459 kB/s | 42 kB 00:00 2024-03-22T02:00:54.288 INFO:teuthology.orchestra.run.smithi028.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-03-22T02:00:54.313 INFO:teuthology.orchestra.run.smithi028.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 947 kB/s | 71 kB 00:00 2024-03-22T02:00:54.355 INFO:teuthology.orchestra.run.smithi028.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 544 kB/s | 36 kB 00:00 2024-03-22T02:00:54.430 INFO:teuthology.orchestra.run.smithi028.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-22T02:00:54.464 INFO:teuthology.orchestra.run.smithi028.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-22T02:00:54.514 INFO:teuthology.orchestra.run.smithi028.stdout:(161/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 4.3 MB/s | 3.1 MB 00:00 2024-03-22T02:00:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-03-22T02:00:54.573 INFO:teuthology.orchestra.run.smithi028.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 995 kB/s | 108 kB 00:00 2024-03-22T02:00:54.599 INFO:teuthology.orchestra.run.smithi028.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-22T02:00:54.632 INFO:teuthology.orchestra.run.smithi028.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-22T02:00:54.657 INFO:teuthology.orchestra.run.smithi028.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 764 kB/s | 64 kB 00:00 2024-03-22T02:00:54.682 INFO:teuthology.orchestra.run.smithi028.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 343 kB/s | 29 kB 00:00 2024-03-22T02:00:54.774 INFO:teuthology.orchestra.run.smithi028.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-22T02:00:54.866 INFO:teuthology.orchestra.run.smithi028.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-03-22T02:00:54.908 INFO:teuthology.orchestra.run.smithi028.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-03-22T02:00:54.942 INFO:teuthology.orchestra.run.smithi028.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-03-22T02:00:54.967 INFO:teuthology.orchestra.run.smithi028.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 310 kB/s | 31 kB 00:00 2024-03-22T02:00:55.050 INFO:teuthology.orchestra.run.smithi028.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.6 MB/s | 1.1 MB 00:00 2024-03-22T02:00:55.634 INFO:teuthology.orchestra.run.smithi028.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 107 kB/s | 74 kB 00:00 2024-03-22T02:00:55.641 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:00:55.641 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 226 MB 00:17 2024-03-22T02:00:56.098 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-03-22T02:00:56.396 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-03-22T02:00:56.396 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-03-22T02:00:56.837 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-22T02:00:58.543 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-03-22T02:00:58.543 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-03-22T02:00:59.152 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:59.153 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-22T02:00:59.167 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-22T02:00:59.168 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-22T02:00:59.169 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-22T02:00:59.170 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:59.172 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-22T02:00:59.176 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-22T02:00:59.177 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-22T02:00:59.256 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-22T02:00:59.269 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 6.6 k 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.4 M 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 890 k 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 145 k 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 1.5 M 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 140 k 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 3.5 M 2024-03-22T02:00:59.270 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 7.4 M 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 49 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 9.8 M 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 48 M 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 264 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 750 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 222 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 27 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 18 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-22T02:00:59.271 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 32 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 747 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 127 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 161 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 321 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 298 k 2024-03-22T02:00:59.272 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 99 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 85 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 170 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-22T02:00:59.273 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.4 M 2024-03-22T02:00:59.274 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 3.1 M 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-22T02:00:59.275 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 21 M 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 34 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 2.4 M 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 248 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 4.7 M 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 17 M 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2344.gc3a04c0e.el9 ceph-noarch 16 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 25 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-22T02:00:59.276 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 163 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 501 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-22T02:00:59.277 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 5.1 M 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-22T02:00:59.278 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 45 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-2344.gc3a04c0e.el9 ceph 131 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-22T02:00:59.279 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-22T02:00:59.280 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-22T02:00:59.281 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-22T02:00:59.282 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-22T02:00:59.283 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout:Install 150 Packages 2024-03-22T02:00:59.284 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 24 Packages 2024-03-22T02:00:59.285 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:00:59.285 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 226 M 2024-03-22T02:00:59.285 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-22T02:01:00.205 INFO:teuthology.orchestra.run.smithi120.stdout:(1/174): ceph-19.0.0-2344.gc3a04c0e.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-22T02:01:00.389 INFO:teuthology.orchestra.run.smithi120.stdout:(2/174): ceph-fuse-19.0.0-2344.gc3a04c0e.el9.x8 4.7 MB/s | 890 kB 00:00 2024-03-22T02:01:00.439 INFO:teuthology.orchestra.run.smithi120.stdout:(3/174): ceph-immutable-object-cache-19.0.0-234 2.8 MB/s | 145 kB 00:00 2024-03-22T02:01:00.689 INFO:teuthology.orchestra.run.smithi120.stdout:(4/174): ceph-mds-19.0.0-2344.gc3a04c0e.el9.x86 9.4 MB/s | 2.4 MB 00:00 2024-03-22T02:01:00.849 INFO:teuthology.orchestra.run.smithi083.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 267 kB/s | 4.4 MB 00:16 2024-03-22T02:01:00.864 INFO:teuthology.orchestra.run.smithi120.stdout:(5/174): ceph-mgr-19.0.0-2344.gc3a04c0e.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-22T02:01:00.940 INFO:teuthology.orchestra.run.smithi120.stdout:(6/174): ceph-base-19.0.0-2344.gc3a04c0e.el9.x8 6.0 MB/s | 5.4 MB 00:00 2024-03-22T02:01:01.365 INFO:teuthology.orchestra.run.smithi120.stdout:(7/174): ceph-mon-19.0.0-2344.gc3a04c0e.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-03-22T02:01:02.399 INFO:teuthology.orchestra.run.smithi120.stdout:(8/174): ceph-common-19.0.0-2344.gc3a04c0e.el9. 8.7 MB/s | 21 MB 00:02 2024-03-22T02:01:02.516 INFO:teuthology.orchestra.run.smithi120.stdout:(9/174): ceph-radosgw-19.0.0-2344.gc3a04c0e.el9 8.6 MB/s | 9.8 MB 00:01 2024-03-22T02:01:02.533 INFO:teuthology.orchestra.run.smithi083.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 65 kB/s | 548 kB 00:08 2024-03-22T02:01:02.541 INFO:teuthology.orchestra.run.smithi120.stdout:(10/174): ceph-selinux-19.0.0-2344.gc3a04c0e.el 178 kB/s | 25 kB 00:00 2024-03-22T02:01:02.616 INFO:teuthology.orchestra.run.smithi120.stdout:(11/174): cephfs-mirror-19.0.0-2344.gc3a04c0e.e 2.9 MB/s | 222 kB 00:00 2024-03-22T02:01:02.691 INFO:teuthology.orchestra.run.smithi120.stdout:(12/174): cephfs-shell-19.0.0-2344.gc3a04c0e.el 361 kB/s | 27 kB 00:00 2024-03-22T02:01:02.775 INFO:teuthology.orchestra.run.smithi120.stdout:(13/174): libcephfs-devel-19.0.0-2344.gc3a04c0e 379 kB/s | 32 kB 00:00 2024-03-22T02:01:02.934 INFO:teuthology.orchestra.run.smithi120.stdout:(14/174): libcephfs2-19.0.0-2344.gc3a04c0e.el9. 4.6 MB/s | 747 kB 00:00 2024-03-22T02:01:03.017 INFO:teuthology.orchestra.run.smithi120.stdout:(15/174): libcephsqlite-19.0.0-2344.gc3a04c0e.e 1.9 MB/s | 163 kB 00:00 2024-03-22T02:01:03.101 INFO:teuthology.orchestra.run.smithi120.stdout:(16/174): librados-devel-19.0.0-2344.gc3a04c0e. 1.5 MB/s | 127 kB 00:00 2024-03-22T02:01:03.209 INFO:teuthology.orchestra.run.smithi120.stdout:(17/174): libradosstriper1-19.0.0-2344.gc3a04c0 4.5 MB/s | 501 kB 00:00 2024-03-22T02:01:03.382 INFO:teuthology.orchestra.run.smithi083.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 84 kB/s | 838 kB 00:09 2024-03-22T02:01:03.385 INFO:teuthology.orchestra.run.smithi120.stdout:(18/174): ceph-osd-19.0.0-2344.gc3a04c0e.el9.x8 6.9 MB/s | 17 MB 00:02 2024-03-22T02:01:03.477 INFO:teuthology.orchestra.run.smithi120.stdout:(19/174): python3-ceph-argparse-19.0.0-2344.gc3 494 kB/s | 45 kB 00:00 2024-03-22T02:01:03.569 INFO:teuthology.orchestra.run.smithi120.stdout:(20/174): python3-ceph-common-19.0.0-2344.gc3a0 1.4 MB/s | 131 kB 00:00 2024-03-22T02:01:03.652 INFO:teuthology.orchestra.run.smithi120.stdout:(21/174): librgw2-19.0.0-2344.gc3a04c0e.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-22T02:01:03.677 INFO:teuthology.orchestra.run.smithi120.stdout:(22/174): python3-cephfs-19.0.0-2344.gc3a04c0e. 1.5 MB/s | 161 kB 00:00 2024-03-22T02:01:03.728 INFO:teuthology.orchestra.run.smithi120.stdout:(23/174): python3-rados-19.0.0-2344.gc3a04c0e.e 4.2 MB/s | 321 kB 00:00 2024-03-22T02:01:03.732 INFO:teuthology.orchestra.run.smithi083.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 15 kB/s | 43 kB 00:02 2024-03-22T02:01:03.761 INFO:teuthology.orchestra.run.smithi120.stdout:(24/174): python3-rbd-19.0.0-2344.gc3a04c0e.el9 3.5 MB/s | 298 kB 00:00 2024-03-22T02:01:03.795 INFO:teuthology.orchestra.run.smithi120.stdout:(25/174): python3-rgw-19.0.0-2344.gc3a04c0e.el9 1.5 MB/s | 99 kB 00:00 2024-03-22T02:01:03.825 INFO:teuthology.orchestra.run.smithi083.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 25 kB/s | 32 kB 00:01 2024-03-22T02:01:03.837 INFO:teuthology.orchestra.run.smithi120.stdout:(26/174): rbd-fuse-19.0.0-2344.gc3a04c0e.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-22T02:01:03.925 INFO:teuthology.orchestra.run.smithi083.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 28 kB/s | 15 kB 00:00 2024-03-22T02:01:03.937 INFO:teuthology.orchestra.run.smithi120.stdout:(27/174): rbd-nbd-19.0.0-2344.gc3a04c0e.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-22T02:01:04.062 INFO:teuthology.orchestra.run.smithi120.stdout:(28/174): rbd-mirror-19.0.0-2344.gc3a04c0e.el9. 12 MB/s | 3.1 MB 00:00 2024-03-22T02:01:04.088 INFO:teuthology.orchestra.run.smithi120.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2344.g 226 kB/s | 34 kB 00:00 2024-03-22T02:01:04.130 INFO:teuthology.orchestra.run.smithi120.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2344.gc3a04c0 2.0 MB/s | 140 kB 00:00 2024-03-22T02:01:04.284 INFO:teuthology.orchestra.run.smithi083.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 312 kB/s | 172 kB 00:00 2024-03-22T02:01:04.709 INFO:teuthology.orchestra.run.smithi083.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 405 kB/s | 358 kB 00:00 2024-03-22T02:01:04.713 INFO:teuthology.orchestra.run.smithi120.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2344.gc3a04 5.6 MB/s | 3.5 MB 00:00 2024-03-22T02:01:04.805 INFO:teuthology.orchestra.run.smithi120.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-22T02:01:04.830 INFO:teuthology.orchestra.run.smithi120.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2344.gc3 2.1 MB/s | 248 kB 00:00 2024-03-22T02:01:04.906 INFO:teuthology.orchestra.run.smithi120.stdout:(34/174): ceph-mgr-rook-19.0.0-2344.gc3a04c0e.e 493 kB/s | 49 kB 00:00 2024-03-22T02:01:04.931 INFO:teuthology.orchestra.run.smithi120.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2344.gc 164 kB/s | 16 kB 00:00 2024-03-22T02:01:05.018 INFO:teuthology.orchestra.run.smithi083.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 277 kB/s | 303 kB 00:01 2024-03-22T02:01:05.023 INFO:teuthology.orchestra.run.smithi120.stdout:(36/174): ceph-volume-19.0.0-2344.gc3a04c0e.el9 2.2 MB/s | 264 kB 00:00 2024-03-22T02:01:05.044 INFO:teuthology.orchestra.run.smithi083.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 46 kB/s | 35 kB 00:00 2024-03-22T02:01:05.140 INFO:teuthology.orchestra.run.smithi120.stdout:(37/174): cephfs-top-19.0.0-2344.gc3a04c0e.el9. 156 kB/s | 18 kB 00:00 2024-03-22T02:01:05.165 INFO:teuthology.orchestra.run.smithi120.stdout:(38/174): cephadm-19.0.0-2344.gc3a04c0e.el9.noa 3.1 MB/s | 750 kB 00:00 2024-03-22T02:01:05.262 INFO:teuthology.orchestra.run.smithi083.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 367 kB/s | 203 kB 00:00 2024-03-22T02:01:05.465 INFO:teuthology.orchestra.run.smithi120.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 662 kB 00:00 2024-03-22T02:01:05.532 INFO:teuthology.orchestra.run.smithi120.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 920 kB 00:00 2024-03-22T02:01:05.557 INFO:teuthology.orchestra.run.smithi120.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-03-22T02:01:05.599 INFO:teuthology.orchestra.run.smithi120.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-22T02:01:05.625 INFO:teuthology.orchestra.run.smithi120.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 472 kB/s | 32 kB 00:00 2024-03-22T02:01:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-03-22T02:01:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T02:01:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T02:01:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T02:01:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T02:01:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T02:01:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T02:01:06.084 INFO:teuthology.orchestra.run.smithi120.stdout:(44/174): ceph-test-19.0.0-2344.gc3a04c0e.el9.x 13 MB/s | 48 MB 00:03 2024-03-22T02:01:06.109 INFO:teuthology.orchestra.run.smithi120.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 17 kB 00:00 2024-03-22T02:01:06.192 INFO:teuthology.orchestra.run.smithi120.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.0 MB/s | 3.0 MB 00:00 2024-03-22T02:01:06.217 INFO:teuthology.orchestra.run.smithi120.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 76 kB/s | 10 kB 00:00 2024-03-22T02:01:06.242 INFO:teuthology.orchestra.run.smithi120.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-22T02:01:06.262 INFO:teuthology.orchestra.run.smithi083.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 11 kB/s | 11 kB 00:00 2024-03-22T02:01:06.276 INFO:teuthology.orchestra.run.smithi120.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 766 kB/s | 44 kB 00:00 2024-03-22T02:01:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T02:01:06.301 INFO:teuthology.orchestra.run.smithi120.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-22T02:01:06.335 INFO:teuthology.orchestra.run.smithi120.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-22T02:01:06.360 INFO:teuthology.orchestra.run.smithi120.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 561 kB/s | 46 kB 00:00 2024-03-22T02:01:06.385 INFO:teuthology.orchestra.run.smithi120.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-22T02:01:06.411 INFO:teuthology.orchestra.run.smithi120.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-22T02:01:06.436 INFO:teuthology.orchestra.run.smithi120.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 681 kB/s | 34 kB 00:00 2024-03-22T02:01:06.462 INFO:teuthology.orchestra.run.smithi120.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-22T02:01:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T02:01:06.679 INFO:teuthology.orchestra.run.smithi120.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-03-22T02:01:06.721 INFO:teuthology.orchestra.run.smithi120.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 5.4 MB/s | 1.5 MB 00:00 2024-03-22T02:01:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:06.754 INFO:teuthology.orchestra.run.smithi120.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 2.7 MB/s | 811 kB 00:00 2024-03-22T02:01:06.779 INFO:teuthology.orchestra.run.smithi120.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-22T02:01:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:06.863 INFO:teuthology.orchestra.run.smithi120.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-03-22T02:01:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:06.971 INFO:teuthology.orchestra.run.smithi120.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-03-22T02:01:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:01:06.996 INFO:teuthology.orchestra.run.smithi120.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 359 kB/s | 78 kB 00:00 2024-03-22T02:01:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T02:01:07.098 INFO:teuthology.orchestra.run.smithi120.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-03-22T02:01:07.122 INFO:teuthology.orchestra.run.smithi120.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 166 kB/s | 24 kB 00:00 2024-03-22T02:01:07.147 INFO:teuthology.orchestra.run.smithi120.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 122 kB/s | 18 kB 00:00 2024-03-22T02:01:07.173 INFO:teuthology.orchestra.run.smithi120.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-22T02:01:07.198 INFO:teuthology.orchestra.run.smithi120.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-22T02:01:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T02:01:07.271 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T02:01:07.281 INFO:teuthology.orchestra.run.smithi120.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-22T02:01:07.296 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:07.307 INFO:teuthology.orchestra.run.smithi120.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-22T02:01:07.340 INFO:teuthology.orchestra.run.smithi120.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-22T02:01:07.354 INFO:teuthology.orchestra.run.smithi083.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 63 kB/s | 144 kB 00:02 2024-03-22T02:01:07.365 INFO:teuthology.orchestra.run.smithi120.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 828 kB/s | 69 kB 00:00 2024-03-22T02:01:07.391 INFO:teuthology.orchestra.run.smithi120.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-22T02:01:07.417 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:07.424 INFO:teuthology.orchestra.run.smithi120.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-22T02:01:07.445 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:07.449 INFO:teuthology.orchestra.run.smithi120.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-22T02:01:07.475 INFO:teuthology.orchestra.run.smithi120.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-22T02:01:07.500 INFO:teuthology.orchestra.run.smithi120.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-22T02:01:07.519 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:07.525 INFO:teuthology.orchestra.run.smithi120.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 571 kB/s | 43 kB 00:00 2024-03-22T02:01:07.550 INFO:teuthology.orchestra.run.smithi120.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 778 kB/s | 39 kB 00:00 2024-03-22T02:01:07.558 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T02:01:07.580 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:07.596 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:07.709 INFO:teuthology.orchestra.run.smithi120.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 225 kB/s | 41 kB 00:00 2024-03-22T02:01:07.734 INFO:teuthology.orchestra.run.smithi120.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-03-22T02:01:07.762 INFO:teuthology.orchestra.run.smithi083.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 12 kB/s | 18 kB 00:01 2024-03-22T02:01:07.805 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:07.882 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:07.943 INFO:teuthology.orchestra.run.smithi120.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-22T02:01:07.968 INFO:teuthology.orchestra.run.smithi120.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 221 kB/s | 56 kB 00:00 2024-03-22T02:01:07.993 INFO:teuthology.orchestra.run.smithi120.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 244 kB/s | 63 kB 00:00 2024-03-22T02:01:08.010 INFO:teuthology.orchestra.run.smithi120.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 664 kB/s | 44 kB 00:00 2024-03-22T02:01:08.035 INFO:teuthology.orchestra.run.smithi120.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-22T02:01:08.061 INFO:teuthology.orchestra.run.smithi120.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-22T02:01:08.094 INFO:teuthology.orchestra.run.smithi120.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-22T02:01:08.119 INFO:teuthology.orchestra.run.smithi120.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-22T02:01:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T02:01:08.153 INFO:teuthology.orchestra.run.smithi120.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-22T02:01:08.178 INFO:teuthology.orchestra.run.smithi120.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-22T02:01:08.295 INFO:teuthology.orchestra.run.smithi120.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-22T02:01:08.320 INFO:teuthology.orchestra.run.smithi120.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-22T02:01:08.323 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T02:01:08.347 INFO:teuthology.orchestra.run.smithi120.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-22T02:01:08.371 INFO:teuthology.orchestra.run.smithi083.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 622 kB/s | 2.0 MB 00:03 2024-03-22T02:01:08.379 INFO:teuthology.orchestra.run.smithi120.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-03-22T02:01:08.405 INFO:teuthology.orchestra.run.smithi120.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 257 kB/s | 21 kB 00:00 2024-03-22T02:01:08.430 INFO:teuthology.orchestra.run.smithi120.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 304 kB/s | 25 kB 00:00 2024-03-22T02:01:08.438 INFO:teuthology.orchestra.run.smithi083.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 21 kB/s | 23 kB 00:01 2024-03-22T02:01:08.445 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:08.505 INFO:teuthology.orchestra.run.smithi120.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-22T02:01:08.505 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:08.530 INFO:teuthology.orchestra.run.smithi083.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 25 kB/s | 19 kB 00:00 2024-03-22T02:01:08.539 INFO:teuthology.orchestra.run.smithi120.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.2 MB/s | 248 kB 00:00 2024-03-22T02:01:08.597 INFO:teuthology.orchestra.run.smithi083.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 87 kB/s | 20 kB 00:00 2024-03-22T02:01:08.601 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:08.648 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:08.655 INFO:teuthology.orchestra.run.smithi120.stdout:(100/174): openssl-devel-3.0.7-27.el9.x86_64.rp 15 MB/s | 4.1 MB 00:00 2024-03-22T02:01:08.681 INFO:teuthology.orchestra.run.smithi120.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.2 MB/s | 216 kB 00:00 2024-03-22T02:01:08.823 INFO:teuthology.orchestra.run.smithi120.stdout:(102/174): qatlib-service-23.11.0-1.el9.x86_64. 123 kB/s | 35 kB 00:00 2024-03-22T02:01:08.868 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:08.873 INFO:teuthology.orchestra.run.smithi120.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 232 kB/s | 44 kB 00:00 2024-03-22T02:01:08.906 INFO:teuthology.orchestra.run.smithi120.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 416 kB/s | 94 kB 00:00 2024-03-22T02:01:08.981 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T02:01:09.065 INFO:teuthology.orchestra.run.smithi120.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-22T02:01:09.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T02:01:09.119 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T02:01:09.146 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T02:01:09.182 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T02:01:09.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T02:01:09.318 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T02:01:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T02:01:09.423 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T02:01:09.464 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T02:01:09.508 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T02:01:09.540 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T02:01:09.568 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T02:01:09.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T02:01:09.672 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T02:01:09.716 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T02:01:09.744 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T02:01:09.816 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T02:01:09.859 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T02:01:09.936 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T02:01:10.073 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T02:01:10.129 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T02:01:10.140 INFO:teuthology.orchestra.run.smithi120.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 287 kB/s | 309 kB 00:01 2024-03-22T02:01:10.173 INFO:teuthology.orchestra.run.smithi120.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 433 kB/s | 548 kB 00:01 2024-03-22T02:01:10.198 INFO:teuthology.orchestra.run.smithi120.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 651 kB/s | 863 kB 00:01 2024-03-22T02:01:10.224 INFO:teuthology.orchestra.run.smithi120.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 234 kB/s | 19 kB 00:00 2024-03-22T02:01:10.324 INFO:teuthology.orchestra.run.smithi120.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-03-22T02:01:10.416 INFO:teuthology.orchestra.run.smithi120.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 737 kB/s | 67 kB 00:00 2024-03-22T02:01:10.566 INFO:teuthology.orchestra.run.smithi120.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-22T02:01:10.583 INFO:teuthology.orchestra.run.smithi120.stdout:(113/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 67 kB/s | 26 kB 00:00 2024-03-22T02:01:10.608 INFO:teuthology.orchestra.run.smithi120.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 787 kB/s | 151 kB 00:00 2024-03-22T02:01:10.682 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T02:01:10.683 INFO:teuthology.orchestra.run.smithi120.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-22T02:01:10.709 INFO:teuthology.orchestra.run.smithi120.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-03-22T02:01:10.766 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T02:01:10.767 INFO:teuthology.orchestra.run.smithi120.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-03-22T02:01:10.793 INFO:teuthology.orchestra.run.smithi120.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-03-22T02:01:10.851 INFO:teuthology.orchestra.run.smithi120.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-22T02:01:10.861 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T02:01:10.910 INFO:teuthology.orchestra.run.smithi120.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-22T02:01:10.917 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:10.925 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:10.935 INFO:teuthology.orchestra.run.smithi120.stdout:(121/174): python3-asyncssh-2.13.2-5.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-03-22T02:01:10.958 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T02:01:10.960 INFO:teuthology.orchestra.run.smithi120.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-22T02:01:10.994 INFO:teuthology.orchestra.run.smithi120.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-03-22T02:01:11.015 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T02:01:11.036 INFO:teuthology.orchestra.run.smithi120.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-22T02:01:11.053 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T02:01:11.083 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T02:01:11.111 INFO:teuthology.orchestra.run.smithi120.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-22T02:01:11.136 INFO:teuthology.orchestra.run.smithi120.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-03-22T02:01:11.161 INFO:teuthology.orchestra.run.smithi120.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 863 kB/s | 144 kB 00:00 2024-03-22T02:01:11.187 INFO:teuthology.orchestra.run.smithi120.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 235 kB/s | 18 kB 00:00 2024-03-22T02:01:11.220 INFO:teuthology.orchestra.run.smithi120.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-03-22T02:01:11.256 INFO:teuthology.orchestra.run.smithi120.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 206 kB/s | 19 kB 00:00 2024-03-22T02:01:11.281 INFO:teuthology.orchestra.run.smithi120.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 208 kB/s | 20 kB 00:00 2024-03-22T02:01:11.357 INFO:teuthology.orchestra.run.smithi120.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-03-22T02:01:11.377 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T02:01:11.382 INFO:teuthology.orchestra.run.smithi120.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-03-22T02:01:11.432 INFO:teuthology.orchestra.run.smithi120.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 4.8 MB/s | 1.0 MB 00:00 2024-03-22T02:01:11.457 INFO:teuthology.orchestra.run.smithi120.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 573 kB/s | 58 kB 00:00 2024-03-22T02:01:11.491 INFO:teuthology.orchestra.run.smithi120.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-22T02:01:11.541 INFO:teuthology.orchestra.run.smithi120.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 151 kB/s | 16 kB 00:00 2024-03-22T02:01:11.567 INFO:teuthology.orchestra.run.smithi120.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 825 kB/s | 90 kB 00:00 2024-03-22T02:01:11.568 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T02:01:11.594 INFO:teuthology.orchestra.run.smithi120.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-03-22T02:01:11.634 INFO:teuthology.orchestra.run.smithi120.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-03-22T02:01:11.668 INFO:teuthology.orchestra.run.smithi120.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-22T02:01:11.668 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T02:01:11.693 INFO:teuthology.orchestra.run.smithi120.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-22T02:01:11.712 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T02:01:11.726 INFO:teuthology.orchestra.run.smithi120.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-22T02:01:11.752 INFO:teuthology.orchestra.run.smithi120.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-22T02:01:11.777 INFO:teuthology.orchestra.run.smithi120.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-22T02:01:11.811 INFO:teuthology.orchestra.run.smithi120.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-03-22T02:01:11.845 INFO:teuthology.orchestra.run.smithi120.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 298 kB/s | 20 kB 00:00 2024-03-22T02:01:11.849 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T02:01:11.878 INFO:teuthology.orchestra.run.smithi120.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-22T02:01:11.900 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T02:01:11.912 INFO:teuthology.orchestra.run.smithi120.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-22T02:01:11.956 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T02:01:12.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T02:01:12.076 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T02:01:12.078 INFO:teuthology.orchestra.run.smithi120.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-03-22T02:01:12.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T02:01:12.129 INFO:teuthology.orchestra.run.smithi120.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-22T02:01:12.159 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T02:01:12.194 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T02:01:12.204 INFO:teuthology.orchestra.run.smithi120.stdout:(152/174): librados2-19.0.0-2344.gc3a04c0e.el9. 10 MB/s | 3.4 MB 00:00 2024-03-22T02:01:12.238 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T02:01:12.280 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T02:01:12.355 INFO:teuthology.orchestra.run.smithi120.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 10 MB/s | 2.3 MB 00:00 2024-03-22T02:01:12.396 INFO:teuthology.orchestra.run.smithi120.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 9.6 MB/s | 1.8 MB 00:00 2024-03-22T02:01:12.422 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T02:01:12.422 INFO:teuthology.orchestra.run.smithi120.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 631 kB/s | 42 kB 00:00 2024-03-22T02:01:12.447 INFO:teuthology.orchestra.run.smithi120.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-03-22T02:01:12.477 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T02:01:12.481 INFO:teuthology.orchestra.run.smithi120.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-22T02:01:12.506 INFO:teuthology.orchestra.run.smithi120.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 620 kB/s | 36 kB 00:00 2024-03-22T02:01:12.507 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T02:01:12.532 INFO:teuthology.orchestra.run.smithi120.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-22T02:01:12.548 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T02:01:12.563 INFO:teuthology.orchestra.run.smithi083.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 11 kB/s | 46 kB 00:04 2024-03-22T02:01:12.574 INFO:teuthology.orchestra.run.smithi120.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-22T02:01:12.599 INFO:teuthology.orchestra.run.smithi120.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 345 kB/s | 23 kB 00:00 2024-03-22T02:01:12.616 INFO:teuthology.orchestra.run.smithi120.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-22T02:01:12.641 INFO:teuthology.orchestra.run.smithi120.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.6 MB/s | 155 kB 00:00 2024-03-22T02:01:12.645 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T02:01:12.663 INFO:teuthology.orchestra.run.smithi083.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 247 kB/s | 1.0 MB 00:04 2024-03-22T02:01:12.666 INFO:teuthology.orchestra.run.smithi120.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-22T02:01:12.688 INFO:teuthology.orchestra.run.smithi083.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 19 kB/s | 79 kB 00:04 2024-03-22T02:01:12.690 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T02:01:12.692 INFO:teuthology.orchestra.run.smithi120.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-22T02:01:12.717 INFO:teuthology.orchestra.run.smithi120.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-22T02:01:12.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T02:01:12.730 INFO:teuthology.orchestra.run.smithi083.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 345 kB/s | 58 kB 00:00 2024-03-22T02:01:12.755 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T02:01:12.759 INFO:teuthology.orchestra.run.smithi120.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-03-22T02:01:12.780 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T02:01:12.818 INFO:teuthology.orchestra.run.smithi120.stdout:(168/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 3.5 MB/s | 3.1 MB 00:00 2024-03-22T02:01:12.866 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T02:01:12.877 INFO:teuthology.orchestra.run.smithi120.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 14 MB/s | 2.2 MB 00:00 2024-03-22T02:01:12.927 INFO:teuthology.orchestra.run.smithi120.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 14 MB/s | 2.3 MB 00:00 2024-03-22T02:01:12.939 INFO:teuthology.orchestra.run.smithi083.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 989 kB/s | 272 kB 00:00 2024-03-22T02:01:12.952 INFO:teuthology.orchestra.run.smithi120.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 415 kB/s | 31 kB 00:00 2024-03-22T02:01:12.964 INFO:teuthology.orchestra.run.smithi083.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 60 kB/s | 16 kB 00:00 2024-03-22T02:01:12.994 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T02:01:12.994 INFO:teuthology.orchestra.run.smithi120.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-03-22T02:01:13.019 INFO:teuthology.orchestra.run.smithi120.stdout:(173/174): util-linux-core-2.37.4-18.el9.x86_64 2.3 MB/s | 465 kB 00:00 2024-03-22T02:01:13.034 INFO:teuthology.orchestra.run.smithi083.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 299 kB/s | 90 kB 00:00 2024-03-22T02:01:13.053 INFO:teuthology.orchestra.run.smithi120.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 742 kB/s | 74 kB 00:00 2024-03-22T02:01:13.060 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:01:13.060 INFO:teuthology.orchestra.run.smithi120.stdout:Total 16 MB/s | 226 MB 00:13 2024-03-22T02:01:13.103 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T02:01:13.147 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T02:01:13.205 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T02:01:13.293 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T02:01:13.485 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T02:01:13.490 INFO:teuthology.orchestra.run.smithi083.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 51 kB/s | 28 kB 00:00 2024-03-22T02:01:13.518 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-22T02:01:13.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T02:01:13.548 INFO:teuthology.orchestra.run.smithi083.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 53 kB/s | 31 kB 00:00 2024-03-22T02:01:13.677 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T02:01:13.721 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T02:01:13.755 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T02:01:13.786 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T02:01:13.791 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-22T02:01:13.792 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-22T02:01:13.814 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T02:01:13.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T02:01:13.887 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T02:01:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T02:01:13.932 INFO:teuthology.orchestra.run.smithi083.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 209 kB/s | 188 kB 00:00 2024-03-22T02:01:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T02:01:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T02:01:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T02:01:14.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T02:01:14.133 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T02:01:14.168 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T02:01:14.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T02:01:14.350 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T02:01:14.357 INFO:teuthology.orchestra.run.smithi083.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 68 kB/s | 59 kB 00:00 2024-03-22T02:01:14.399 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T02:01:14.430 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T02:01:14.486 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T02:01:14.587 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T02:01:14.637 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T02:01:14.682 INFO:teuthology.orchestra.run.smithi083.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 32 kB/s | 36 kB 00:01 2024-03-22T02:01:14.975 INFO:teuthology.orchestra.run.smithi083.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 49 kB/s | 51 kB 00:01 2024-03-22T02:01:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T02:01:15.450 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T02:01:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T02:01:15.521 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-22T02:01:15.521 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-22T02:01:15.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T02:01:15.623 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T02:01:15.660 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T02:01:15.844 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T02:01:15.882 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T02:01:15.962 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T02:01:16.068 INFO:teuthology.orchestra.run.smithi083.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 134 kB/s | 230 kB 00:01 2024-03-22T02:01:16.436 INFO:teuthology.orchestra.run.smithi083.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 51 kB/s | 90 kB 00:01 2024-03-22T02:01:16.642 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T02:01:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T02:01:17.411 INFO:teuthology.orchestra.run.smithi083.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 175 kB/s | 427 kB 00:02 2024-03-22T02:01:17.436 INFO:teuthology.orchestra.run.smithi083.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 15 kB/s | 20 kB 00:01 2024-03-22T02:01:17.611 INFO:teuthology.orchestra.run.smithi083.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 163 kB/s | 192 kB 00:01 2024-03-22T02:01:17.736 INFO:teuthology.orchestra.run.smithi083.stdout:(150/174): librados2-19.0.0-2344.gc3a04c0e.el9. 11 MB/s | 3.4 MB 00:00 2024-03-22T02:01:17.811 INFO:teuthology.orchestra.run.smithi083.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-03-22T02:01:17.945 INFO:teuthology.orchestra.run.smithi083.stdout:(152/174): librbd1-19.0.0-2344.gc3a04c0e.el9.x8 9.4 MB/s | 3.1 MB 00:00 2024-03-22T02:01:18.061 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T02:01:18.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T02:01:18.170 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T02:01:18.201 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T02:01:18.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T02:01:18.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T02:01:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:18.705 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T02:01:18.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T02:01:19.227 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T02:01:19.261 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:19.316 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:19.338 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T02:01:21.606 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:21.642 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:22.218 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:22.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:01:22.262 INFO:teuthology.orchestra.run.smithi083.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 334 kB/s | 1.6 MB 00:04 2024-03-22T02:01:22.347 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:01:22.348 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T02:01:22.348 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T02:01:22.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:22.388 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:01:22.397 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-22T02:01:22.498 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T02:01:22.533 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T02:01:22.572 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:22.581 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:22.621 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T02:01:22.674 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T02:01:22.709 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T02:01:22.734 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T02:01:22.776 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T02:01:22.945 INFO:teuthology.orchestra.run.smithi083.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 62 kB/s | 42 kB 00:00 2024-03-22T02:01:23.097 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T02:01:23.144 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:23.265 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:23.295 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:23.387 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:23.416 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:01:23.458 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T02:01:23.630 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T02:01:23.681 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T02:01:23.709 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:23.796 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:23.818 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:23.874 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:23.912 INFO:teuthology.orchestra.run.smithi083.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 257 kB/s | 248 kB 00:00 2024-03-22T02:01:23.914 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T02:01:23.937 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:23.952 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:24.137 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:24.154 INFO:teuthology.orchestra.run.smithi083.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 295 kB/s | 71 kB 00:00 2024-03-22T02:01:24.168 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:24.208 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T02:01:24.313 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T02:01:24.338 INFO:teuthology.orchestra.run.smithi083.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 294 kB/s | 1.8 MB 00:06 2024-03-22T02:01:24.404 INFO:teuthology.orchestra.run.smithi083.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 146 kB/s | 36 kB 00:00 2024-03-22T02:01:24.411 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:24.455 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:24.526 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:24.573 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:24.784 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:24.894 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T02:01:24.987 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T02:01:25.027 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T02:01:25.046 INFO:teuthology.orchestra.run.smithi083.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 326 kB/s | 2.3 MB 00:07 2024-03-22T02:01:25.063 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T02:01:25.106 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T02:01:25.204 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T02:01:25.246 INFO:teuthology.orchestra.run.smithi083.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 115 kB/s | 23 kB 00:00 2024-03-22T02:01:25.259 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T02:01:25.298 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T02:01:25.305 INFO:teuthology.orchestra.run.smithi083.stdout:(161/174): elfutils-libelf-0.190-2.el9.x86_64.r 199 kB/s | 192 kB 00:00 2024-03-22T02:01:25.362 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T02:01:25.397 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T02:01:25.441 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T02:01:25.474 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T02:01:25.502 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T02:01:25.570 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T02:01:25.608 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T02:01:25.642 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T02:01:25.671 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T02:01:25.738 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T02:01:25.764 INFO:teuthology.orchestra.run.smithi083.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 210 kB/s | 108 kB 00:00 2024-03-22T02:01:25.785 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T02:01:25.857 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T02:01:25.991 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T02:01:26.054 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T02:01:26.106 INFO:teuthology.orchestra.run.smithi083.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 194 kB/s | 155 kB 00:00 2024-03-22T02:01:26.464 INFO:teuthology.orchestra.run.smithi083.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 194 kB/s | 136 kB 00:00 2024-03-22T02:01:26.489 INFO:teuthology.orchestra.run.smithi083.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 166 kB/s | 64 kB 00:00 2024-03-22T02:01:26.572 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T02:01:26.606 INFO:teuthology.orchestra.run.smithi083.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 202 kB/s | 29 kB 00:00 2024-03-22T02:01:26.650 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T02:01:26.731 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T02:01:26.771 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:26.778 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:26.810 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T02:01:26.851 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T02:01:26.889 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T02:01:26.919 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T02:01:26.931 INFO:teuthology.orchestra.run.smithi083.stdout:(167/174): elfutils-libs-0.190-2.el9.x86_64.rpm 101 kB/s | 256 kB 00:02 2024-03-22T02:01:27.162 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T02:01:27.350 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T02:01:27.406 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T02:01:27.450 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T02:01:27.588 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T02:01:27.637 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T02:01:27.690 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T02:01:27.755 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T02:01:27.787 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T02:01:27.824 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T02:01:27.870 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T02:01:27.907 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T02:01:27.948 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T02:01:27.991 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T02:01:28.123 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T02:01:28.239 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T02:01:28.403 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T02:01:28.445 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T02:01:28.542 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T02:01:28.578 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T02:01:28.618 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T02:01:28.643 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T02:01:28.668 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T02:01:28.742 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T02:01:28.876 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T02:01:28.974 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T02:01:29.010 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T02:01:29.058 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T02:01:29.114 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T02:01:29.296 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T02:01:29.340 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T02:01:29.466 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T02:01:29.505 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T02:01:29.536 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T02:01:29.568 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T02:01:29.596 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T02:01:29.639 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T02:01:29.678 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T02:01:29.718 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T02:01:29.748 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T02:01:29.824 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T02:01:29.858 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T02:01:29.884 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T02:01:29.925 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T02:01:29.960 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T02:01:30.050 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T02:01:30.139 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T02:01:30.182 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T02:01:30.198 INFO:teuthology.orchestra.run.smithi083.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 334 kB/s | 1.2 MB 00:03 2024-03-22T02:01:30.214 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T02:01:30.262 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T02:01:30.359 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T02:01:30.416 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T02:01:31.136 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T02:01:31.180 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T02:01:31.216 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T02:01:31.358 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T02:01:31.635 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T02:01:31.740 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T02:01:31.919 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T02:01:31.946 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T02:01:32.024 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T02:01:32.223 INFO:teuthology.orchestra.run.smithi083.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 230 kB/s | 465 kB 00:02 2024-03-22T02:01:32.432 INFO:teuthology.orchestra.run.smithi083.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 149 kB/s | 31 kB 00:00 2024-03-22T02:01:32.659 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T02:01:32.776 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T02:01:33.690 INFO:teuthology.orchestra.run.smithi083.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 312 kB/s | 2.2 MB 00:07 2024-03-22T02:01:33.857 INFO:teuthology.orchestra.run.smithi083.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 446 kB/s | 74 kB 00:00 2024-03-22T02:01:33.983 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T02:01:34.074 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T02:01:34.105 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T02:01:34.136 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T02:01:34.163 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T02:01:34.476 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T02:01:34.502 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:34.536 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:01:34.536 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-03-22T02:01:34.537 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:34.593 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:34.624 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T02:01:34.660 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T02:01:34.698 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:01:34.712 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:01:34.712 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:34.712 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T02:01:34.712 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:35.120 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T02:01:35.156 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:35.209 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:35.232 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T02:01:35.907 INFO:teuthology.orchestra.run.smithi083.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 260 kB/s | 2.3 MB 00:08 2024-03-22T02:01:36.260 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:01:36.285 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:36.519 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:01:36.541 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:01:36.541 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:36.542 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T02:01:36.542 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:01:36.542 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:01:36.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:36.774 INFO:teuthology.orchestra.run.smithi083.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 253 kB/s | 1.1 MB 00:04 2024-03-22T02:01:36.782 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:01:36.782 INFO:teuthology.orchestra.run.smithi083.stdout:Total 3.4 MB/s | 226 MB 01:07 2024-03-22T02:01:37.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:01:37.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:37.130 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T02:01:37.228 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-03-22T02:01:37.410 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:37.447 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:37.497 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-03-22T02:01:37.497 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-03-22T02:01:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:01:37.578 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:01:37.611 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T02:01:37.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T02:01:37.701 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:01:37.993 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:01:38.034 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:01:38.115 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:01:38.116 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T02:01:38.116 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T02:01:38.116 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:38.144 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:01:38.798 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:01:38.836 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:01:39.223 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-03-22T02:01:39.223 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-03-22T02:01:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:01:39.878 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:01:39.985 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:01:40.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T02:01:40.188 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:01:40.210 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:01:40.211 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:40.211 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T02:01:40.211 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:01:40.211 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:01:40.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:40.256 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:01:40.266 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:01:40.317 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T02:01:41.256 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T02:01:41.282 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:01:41.304 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:01:41.304 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:41.305 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T02:01:41.305 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:01:41.305 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:01:41.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:41.358 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:01:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:01:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:41.380 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T02:01:41.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:01:41.447 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:01:41.447 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:41.448 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T02:01:41.448 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:01:41.448 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:01:41.448 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:41.776 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:01:41.798 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:46.000 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-03-22T02:01:46.120 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-22T02:01:46.153 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-22T02:01:46.185 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:46.193 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-22T02:01:46.234 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-22T02:01:46.296 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-22T02:01:46.329 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-22T02:01:46.364 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-22T02:01:46.406 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-22T02:01:46.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:01:46.564 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T02:01:46.593 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T02:01:46.620 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T02:01:46.724 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-22T02:01:46.782 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:46.810 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T02:01:46.885 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:46.924 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:46.932 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T02:01:46.976 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T02:01:47.007 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:47.032 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T02:01:47.036 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:01:47.078 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-22T02:01:47.101 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T02:01:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T02:01:47.169 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T02:01:47.209 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T02:01:47.239 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T02:01:47.253 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-22T02:01:47.301 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-22T02:01:47.310 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T02:01:47.321 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:47.369 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T02:01:47.404 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T02:01:47.424 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:47.449 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:47.511 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 17/198 2024-03-22T02:01:47.551 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-22T02:01:47.574 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:47.588 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-22T02:01:47.731 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:47.782 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-22T02:01:47.811 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-22T02:01:47.907 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-22T02:01:47.921 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:01:47.960 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:01:47.979 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:01:48.005 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:48.066 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-22T02:01:48.125 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:48.200 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:48.412 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-22T02:01:48.521 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-22T02:01:48.608 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-22T02:01:48.647 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-22T02:01:48.674 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-22T02:01:48.709 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-22T02:01:48.783 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-22T02:01:48.837 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-22T02:01:48.864 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T02:01:48.875 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-22T02:01:48.940 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-22T02:01:48.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T02:01:48.983 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-22T02:01:49.012 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T02:01:49.026 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-22T02:01:49.059 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-22T02:01:49.087 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-22T02:01:49.100 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T02:01:49.132 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T02:01:49.157 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T02:01:49.157 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-22T02:01:49.182 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T02:01:49.192 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 39/198 2024-03-22T02:01:49.209 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T02:01:49.209 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:01:49.235 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 40/198 2024-03-22T02:01:49.264 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 41/198 2024-03-22T02:01:49.297 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:01:49.326 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T02:01:49.326 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:01:49.333 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-22T02:01:49.379 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-22T02:01:49.411 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:01:49.444 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T02:01:49.452 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-22T02:01:49.470 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T02:01:49.496 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T02:01:49.520 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T02:01:49.561 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T02:01:49.584 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-22T02:01:49.601 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T02:01:49.633 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T02:01:49.639 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-22T02:01:49.658 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T02:01:49.684 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T02:01:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:01:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T02:01:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:01:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:01:49.920 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:01:49.920 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:01:49.986 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:01:49.990 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-03-22T02:01:49.991 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:50.011 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:01:50.043 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T02:01:50.062 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T02:01:50.096 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T02:01:50.155 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-22T02:01:50.162 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:01:50.166 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:01:50.177 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:01:50.177 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:50.177 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T02:01:50.177 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:50.235 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-22T02:01:50.313 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-22T02:01:50.380 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:50.387 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-22T02:01:50.420 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-22T02:01:50.460 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-22T02:01:50.498 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 53/198 2024-03-22T02:01:50.558 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-22T02:01:50.774 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-22T02:01:50.936 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-22T02:01:51.015 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-22T02:01:51.058 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-22T02:01:51.196 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-22T02:01:51.236 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-22T02:01:51.291 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-22T02:01:51.364 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-22T02:01:51.403 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-22T02:01:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-22T02:01:51.487 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-22T02:01:51.523 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-22T02:01:51.566 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-22T02:01:51.608 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-22T02:01:51.665 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:01:51.691 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:51.741 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-22T02:01:51.780 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-22T02:01:51.813 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-22T02:01:51.888 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-22T02:01:51.916 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:01:51.938 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:01:51.938 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:51.939 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T02:01:51.939 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:01:51.939 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:01:51.939 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:51.982 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-22T02:01:52.022 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-22T02:01:52.062 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-22T02:01:52.088 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-22T02:01:52.113 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-22T02:01:52.185 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-22T02:01:52.321 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-22T02:01:52.375 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:01:52.397 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:01:52.397 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:52.397 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T02:01:52.398 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:01:52.398 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:01:52.398 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:52.420 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-22T02:01:52.462 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-22T02:01:52.494 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T02:01:52.511 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-22T02:01:52.574 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-22T02:01:52.758 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-22T02:01:52.800 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-22T02:01:52.845 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:01:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:01:52.907 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T02:01:52.927 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-22T02:01:52.967 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T02:01:52.973 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-22T02:01:52.991 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:01:53.005 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-22T02:01:53.036 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-22T02:01:53.064 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-22T02:01:53.115 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-22T02:01:53.155 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-22T02:01:53.186 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-22T02:01:53.216 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-22T02:01:53.292 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-22T02:01:53.335 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-22T02:01:53.368 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-22T02:01:53.409 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-22T02:01:53.444 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-22T02:01:53.532 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-22T02:01:53.615 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-22T02:01:53.658 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-22T02:01:53.698 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-22T02:01:53.745 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-22T02:01:53.843 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-22T02:01:53.893 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-22T02:01:54.020 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:01:54.059 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:01:54.609 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-22T02:01:54.655 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-22T02:01:54.689 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-22T02:01:54.737 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-22T02:01:54.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-22T02:01:54.882 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-22T02:01:55.025 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:01:55.051 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:01:55.060 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-22T02:01:55.088 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-22T02:01:55.153 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:01:55.202 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-22T02:01:55.315 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T02:01:55.345 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:01:55.368 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:55.413 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:01:55.424 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:01:55.483 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T02:01:55.485 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T02:01:55.486 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T02:01:55.487 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T02:01:55.488 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T02:01:55.489 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T02:01:55.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T02:01:55.491 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T02:01:55.492 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T02:01:55.493 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T02:01:55.494 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T02:01:55.495 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T02:01:55.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T02:01:55.497 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T02:01:55.498 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T02:01:55.499 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T02:01:55.500 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:01:55.501 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T02:01:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T02:01:55.502 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T02:01:56.021 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-22T02:01:56.135 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-22T02:01:56.350 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T02:01:56.382 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:01:56.404 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:56.449 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:01:56.471 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:01:56.471 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:56.471 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T02:01:56.471 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:56.533 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:01:56.555 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:56.851 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:01:56.873 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:01:57.349 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-22T02:01:57.431 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-22T02:01:57.462 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-22T02:01:57.493 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-22T02:01:57.520 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-22T02:01:57.832 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-22T02:01:57.867 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:57.956 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 124/198 2024-03-22T02:01:57.988 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 125/198 2024-03-22T02:01:58.025 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-22T02:01:58.240 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T02:01:58.241 INFO:teuthology.orchestra.run.smithi028.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.242 INFO:teuthology.orchestra.run.smithi028.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.243 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: jq-1.6-16.el9.x86_64 2024-03-22T02:01:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T02:01:58.246 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T02:01:58.247 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T02:01:58.248 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T02:01:58.249 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T02:01:58.250 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T02:01:58.251 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T02:01:58.252 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T02:01:58.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T02:01:58.254 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T02:01:58.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:01:58.256 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-03-22T02:01:58.486 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-22T02:01:58.512 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:58.582 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 128/198 2024-03-22T02:01:58.605 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 129/198 2024-03-22T02:01:58.729 DEBUG:teuthology.parallel:result is None 2024-03-22T02:02:00.767 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:02:00.802 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:02:01.331 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:02:01.338 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 130/198 2024-03-22T02:02:01.380 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:02:01.381 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T02:02:01.411 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T02:02:01.438 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T02:02:01.461 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 131/198 2024-03-22T02:02:01.462 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-22T02:02:01.462 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-22T02:02:01.462 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:01.489 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:02:01.618 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T02:02:01.730 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T02:02:01.779 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T02:02:01.818 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T02:02:01.871 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T02:02:01.905 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T02:02:01.947 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T02:02:01.987 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T02:02:02.017 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T02:02:02.072 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T02:02:02.133 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T02:02:02.166 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T02:02:02.812 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:02.848 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:02.868 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:03.685 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T02:02:03.769 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T02:02:03.809 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T02:02:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T02:02:03.931 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T02:02:03.956 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T02:02:03.982 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T02:02:04.008 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T02:02:04.008 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:02:04.096 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:02:04.134 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T02:02:04.134 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:02:04.211 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:02:04.237 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T02:02:04.270 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T02:02:04.296 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T02:02:04.320 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T02:02:04.354 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T02:02:04.393 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T02:02:04.426 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T02:02:04.451 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T02:02:04.476 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T02:02:04.502 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:02:04.527 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T02:02:04.533 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:04.533 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:04.714 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:04.714 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:02:04.797 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:02:04.821 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:02:04.854 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T02:02:04.881 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T02:02:04.906 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T02:02:04.979 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T02:02:10.607 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T02:02:10.608 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T02:02:10.609 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T02:02:10.626 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T02:02:10.627 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T02:02:10.628 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T02:02:10.629 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T02:02:10.630 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T02:02:10.631 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T02:02:10.632 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T02:02:10.632 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T02:02:10.632 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T02:02:10.632 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T02:02:10.632 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T02:02:10.633 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T02:02:10.634 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T02:02:10.635 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T02:02:10.636 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T02:02:10.637 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T02:02:10.638 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T02:02:10.639 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T02:02:10.640 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 132/198 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-03-22T02:02:13.366 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-03-22T02:02:13.367 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-03-22T02:02:13.367 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-03-22T02:02:13.367 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:13.546 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:02:13.561 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 133/198 2024-03-22T02:02:13.561 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:13.562 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-22T02:02:13.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T02:02:13.669 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.670 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.671 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.672 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T02:02:13.673 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.674 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T02:02:13.675 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T02:02:13.676 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.677 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T02:02:13.678 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T02:02:13.679 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T02:02:13.680 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T02:02:13.681 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:13.682 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:02:13.683 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-22T02:02:14.183 DEBUG:teuthology.parallel:result is None 2024-03-22T02:02:15.016 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:02:15.041 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 134/198 2024-03-22T02:02:15.041 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:15.041 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-22T02:02:15.041 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:02:15.042 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-22T02:02:15.042 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:15.267 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 135/198 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-22T02:02:15.289 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:15.734 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:02:15.756 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 136/198 2024-03-22T02:02:15.756 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:15.756 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-22T02:02:15.756 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:02:15.756 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-22T02:02:15.757 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:15.844 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-22T02:02:16.162 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:02:16.193 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 138/198 2024-03-22T02:02:16.226 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 139/198 2024-03-22T02:02:16.285 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 140/198 2024-03-22T02:02:16.316 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:02:17.345 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 141/198 2024-03-22T02:02:17.384 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:02:18.346 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 142/198 2024-03-22T02:02:18.378 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:02:18.477 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 143/198 2024-03-22T02:02:18.632 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 144/198 2024-03-22T02:02:18.664 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:02:18.686 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 145/198 2024-03-22T02:02:18.687 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:18.687 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-22T02:02:18.687 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:02:18.687 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-22T02:02:18.687 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:18.731 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:02:18.742 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 146/198 2024-03-22T02:02:18.787 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 147/198 2024-03-22T02:02:19.664 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-22T02:02:19.692 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:02:19.714 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 149/198 2024-03-22T02:02:19.714 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:19.714 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-22T02:02:19.714 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:02:19.714 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-22T02:02:19.715 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:19.759 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:02:19.780 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 150/198 2024-03-22T02:02:19.780 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:19.780 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-22T02:02:19.780 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:19.835 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-22T02:02:19.857 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:20.160 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:02:20.182 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 152/198 2024-03-22T02:02:20.182 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-22T02:02:20.182 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-22T02:02:20.183 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:02:20.183 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-22T02:02:20.183 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:24.615 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:02:24.656 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 154/198 2024-03-22T02:02:24.686 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 155/198 2024-03-22T02:02:24.714 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-22T02:02:24.892 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 157/198 2024-03-22T02:02:25.005 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-22T02:02:25.045 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 159/198 2024-03-22T02:02:25.084 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-22T02:02:25.129 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-22T02:02:25.164 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-22T02:02:25.196 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 163/198 2024-03-22T02:02:25.228 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-22T02:02:25.258 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-22T02:02:25.304 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-22T02:02:25.357 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-22T02:02:25.382 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-22T02:02:25.965 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:26.005 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:26.025 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-22T02:02:26.831 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 170/198 2024-03-22T02:02:26.909 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-22T02:02:26.949 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-22T02:02:27.030 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-22T02:02:27.062 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-22T02:02:27.087 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-22T02:02:27.113 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-22T02:02:27.139 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-22T02:02:27.140 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:02:27.227 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-22T02:02:27.257 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-22T02:02:27.257 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:02:27.342 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-22T02:02:27.368 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-22T02:02:27.418 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-22T02:02:27.443 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-22T02:02:27.468 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-22T02:02:27.501 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-22T02:02:27.541 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-22T02:02:27.573 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-22T02:02:27.598 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-22T02:02:27.624 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-22T02:02:27.649 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:02:27.674 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-22T02:02:27.680 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:27.680 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:27.877 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-22T02:02:27.878 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:02:27.960 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-22T02:02:27.985 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:02:28.017 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-22T02:02:28.045 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-22T02:02:28.069 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-22T02:02:28.133 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:02:33.654 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 1/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 3/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 4/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2344.gc3a04 5/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 6/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 7/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 8/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 9/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 10/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 11/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 12/198 2024-03-22T02:02:33.655 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 13/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 14/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_ 15/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 16/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 17/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_6 18/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86 19/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 20/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el 21/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9. 22/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_6 23/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 24/198 2024-03-22T02:02:33.656 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 25/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 26/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 27/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 28/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 29/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e. 30/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noa 31/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.n 32/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a 33/198 2024-03-22T02:02:33.657 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el 34/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 35/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.e 36/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 37/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 38/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 39/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-22T02:02:33.658 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-22T02:02:33.659 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-22T02:02:33.660 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-22T02:02:33.661 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-22T02:02:33.662 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-22T02:02:33.663 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-22T02:02:33.664 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-22T02:02:33.665 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-22T02:02:33.666 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 151/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-22T02:02:33.667 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 153/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-22T02:02:33.668 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-22T02:02:33.669 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-22T02:02:33.670 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-22T02:02:33.671 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:36.627 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: acl-2.3.1-4.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.628 INFO:teuthology.orchestra.run.smithi083.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: automake-1.16.2-8.el9.noarch 2024-03-22T02:02:36.629 INFO:teuthology.orchestra.run.smithi083.stdout: bison-3.7.4-5.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.630 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.631 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-shell-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: cephfs-top-2:19.0.0-2344.gc3a04c0e.el9.noarch 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: flex-2.6.4-9.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: jq-1.6-16.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-22T02:02:36.632 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-22T02:02:36.633 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-22T02:02:36.634 INFO:teuthology.orchestra.run.smithi083.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-13.11-1.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-22T02:02:36.635 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-22T02:02:36.636 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-22T02:02:36.637 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-22T02:02:36.638 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-22T02:02:36.639 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.640 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.0.0-2344.gc3a04c0e.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-3.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-22T02:02:36.641 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-03-22T02:02:36.642 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-03-22T02:02:37.086 DEBUG:teuthology.parallel:result is None 2024-03-22T02:02:37.087 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:37.230 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T02:02:37.256 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.257 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.257 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T02:02:37.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:37.483 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T02:02:37.504 INFO:teuthology.orchestra.run.smithi083.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.504 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.504 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T02:02:37.506 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:37.648 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-22T02:02:37.674 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.674 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2344.gc3a04c0e.el9 2024-03-22T02:02:37.674 INFO:teuthology.task.install:The correct ceph version 19.0.0-2344.gc3a04c0e is installed. 2024-03-22T02:02:37.676 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-22T02:02:37.676 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:02:37.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T02:02:37.711 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:02:37.711 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T02:02:37.741 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:02:37.741 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-22T02:02:37.776 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-22T02:02:37.776 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:02:37.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T02:02:37.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T02:02:37.878 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:02:37.878 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T02:02:37.906 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T02:02:37.972 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:02:37.972 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-03-22T02:02:38.005 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-22T02:02:38.074 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-22T02:02:38.074 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:02:38.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T02:02:38.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T02:02:38.170 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:02:38.170 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T02:02:38.197 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T02:02:38.264 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:02:38.264 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-22T02:02:38.294 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-22T02:02:38.365 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-22T02:02:38.365 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:02:38.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T02:02:38.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T02:02:38.458 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:02:38.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T02:02:38.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T02:02:38.552 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:02:38.552 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-03-22T02:02:38.585 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-22T02:02:38.657 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-22T02:02:38.761 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-22T02:02:38.761 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:38.762 INFO:tasks.cephadm:Cluster fsid is 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:02:38.762 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-22T02:02:38.762 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.83', 'mon.c': '172.21.15.120'} 2024-03-22T02:02:38.762 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-03-22T02:02:38.762 INFO:tasks.cephadm:First mgr is x 2024-03-22T02:02:38.762 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-22T02:02:38.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-03-22T02:02:38.790 DEBUG:teuthology.orchestra.run.smithi083:> sudo hostname $(hostname -s) 2024-03-22T02:02:38.818 DEBUG:teuthology.orchestra.run.smithi120:> sudo hostname $(hostname -s) 2024-03-22T02:02:38.849 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-22T02:02:38.850 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:39.074 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77857/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2344-gc3a04c0e', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2344.gc3a04c0e'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-21 21:00:27.602277', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240321.172515', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-22T02:02:39.216 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-batrick-testing-20240321.172515, sha1 c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:02:39.217 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm 2024-03-22T02:02:39.218 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:39.724 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 02:02 /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:39.724 DEBUG:teuthology.orchestra.run.smithi083:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:40.509 INFO:teuthology.orchestra.run.smithi083.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 02:02 /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:40.509 DEBUG:teuthology.orchestra.run.smithi120:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240321.172515/c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:41.299 INFO:teuthology.orchestra.run.smithi120.stdout:-rw-r--r--. 1 ubuntu ubuntu 769354 Mar 22 02:02 /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:41.299 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:41.318 DEBUG:teuthology.orchestra.run.smithi083:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:41.336 DEBUG:teuthology.orchestra.run.smithi120:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T02:02:41.371 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e on all hosts... 2024-03-22T02:02:41.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T02:02:41.374 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T02:02:41.378 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e pull 2024-03-22T02:02:42.142 INFO:teuthology.orchestra.run.smithi120.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T02:02:42.208 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T02:02:42.317 INFO:teuthology.orchestra.run.smithi083.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T02:03:11.410 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout: "repo_digests": [ 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout: ] 2024-03-22T02:03:11.411 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout: "repo_digests": [ 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-03-22T02:03:14.288 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0", 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0" 2024-03-22T02:03:14.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:03:14.438 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:03:14.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-03-22T02:03:14.504 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph 2024-03-22T02:03:14.539 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2024-03-22T02:03:14.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-03-22T02:03:14.597 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /etc/ceph 2024-03-22T02:03:14.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /etc/ceph 2024-03-22T02:03:14.653 INFO:tasks.cephadm:Writing seed config... 2024-03-22T02:03:14.654 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-22T02:03:14.655 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-03-22T02:03:14.656 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-22T02:03:14.657 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-22T02:03:14.658 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-22T02:03:14.659 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:03:14.659 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-22T02:03:14.674 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 426c3602-e7f0-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-22T02:03:14.674 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service 2024-03-22T02:03:14.716 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x.service 2024-03-22T02:03:14.758 INFO:tasks.cephadm:Bootstrapping... 2024-03-22T02:03:14.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e -v bootstrap --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:03:14.955 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-03-22T02:03:14.955 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e', '-v', 'bootstrap', '--fsid', '426c3602-e7f0-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-03-22T02:03:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-22T02:03:14.956 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-03-22T02:03:14.980 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T02:03:14.980 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-03-22T02:03:14.980 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-03-22T02:03:14.987 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T02:03:14.987 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T02:03:14.993 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T02:03:14.993 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:03:14.999 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-03-22T02:03:15.005 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-03-22T02:03:15.005 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-03-22T02:03:15.005 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-03-22T02:03:15.028 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-22T02:03:15.028 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-22T02:03:15.029 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-03-22T02:03:15.029 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-03-22T02:03:15.034 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-22T02:03:15.035 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T02:03:15.040 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-22T02:03:15.040 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:03:15.046 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-03-22T02:03:15.052 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-03-22T02:03:15.052 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-03-22T02:03:15.052 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-03-22T02:03:15.052 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:15.052 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 140659119424464 on /run/cephadm/426c3602-e7f0-11ee-95cd-87774f69a715.lock 2024-03-22T02:03:15.053 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 140659119424464 acquired on /run/cephadm/426c3602-e7f0-11ee-95cd-87774f69a715.lock 2024-03-22T02:03:15.053 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-03-22T02:03:15.053 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-03-22T02:03:15.053 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-03-22T02:03:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-03-22T02:03:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-03-22T02:03:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-22T02:03:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-22T02:03:15.059 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-22T02:03:15.061 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-22T02:03:15.061 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-22T02:03:15.062 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T02:03:15.062 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-22T02:03:15.062 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-03-22T02:03:15.062 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-22T02:03:15.062 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-03-22T02:03:15.063 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-03-22T02:03:15.063 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-22T02:03:15.063 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-22T02:03:15.064 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T02:03:15.556 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 2024-03-22T02:03:15.556 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e... 2024-03-22T02:03:15.556 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-03-22T02:03:15.557 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:05236af4953a5298b5dfb26dfe4c3e6900902214dbe03ae2042572624f92ee73 2024-03-22T02:03:15.557 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-22T02:03:15.557 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 2024-03-22T02:03:15.557 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-22T02:03:16.502 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev) 2024-03-22T02:03:16.502 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev) 2024-03-22T02:03:16.503 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-03-22T02:03:17.415 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-03-22T02:03:17.415 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-03-22T02:03:18.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDl5vxlwQbBKxAAO4Ybg6GotCjxOAz+/hUMiA== 2024-03-22T02:03:19.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDm5vxlNon+JxAALI/3j5193t20cXatrcfmVA== 2024-03-22T02:03:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDn5vxl4VVbIhAACurM1ROcM1La9uWpGj0t6A== 2024-03-22T02:03:20.121 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:21.211 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T02:03:21.212 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:03:21.212 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-03-22T02:03:22.131 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-03-22T02:03:22.332 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-22T02:03:22.474 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-22T02:03:22.647 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target. 2024-03-22T02:03:22.647 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target. 2024-03-22T02:03:22.877 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a 2024-03-22T02:03:22.877 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-22T02:03:23.036 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target.wants/ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715@.service. 2024-03-22T02:03:23.570 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:03:23.571 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-22T02:03:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:03:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:03:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-22T02:03:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T02:03:23.578 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-03-22T02:03:23.578 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-03-22T02:03:24.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-03-22T02:03:24.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:24.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-22T02:03:24.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:24.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.561398s) 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-03-22T02:03:24.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.018+0000 7fe196237700 1 Processor -- start 2024-03-22T02:03:24.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.018+0000 7fe196237700 1 -- start start 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe196237700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe196237700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe19007c810 con 0x7fe19007ddd0 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe18f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe18f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36132/0 (socket says 172.21.15.28:36132) 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe18f7fe700 1 -- 172.21.15.28:0/1107304001 learned_addr learned my addr 172.21.15.28:0/1107304001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:24.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.019+0000 7fe18f7fe700 1 -- 172.21.15.28:0/1107304001 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19007c990 con 0x7fe19007ddd0 2024-03-22T02:03:24.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe18f7fe700 1 --2- 172.21.15.28:0/1107304001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe1780098a0 tx=0x7fe178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27e9c8b9ab4b9e5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:24.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe18e7fc700 1 -- 172.21.15.28:0/1107304001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe178004160 con 0x7fe19007ddd0 2024-03-22T02:03:24.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe18e7fc700 1 -- 172.21.15.28:0/1107304001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe178004300 con 0x7fe19007ddd0 2024-03-22T02:03:24.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe18e7fc700 1 -- 172.21.15.28:0/1107304001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1780036a0 con 0x7fe19007ddd0 2024-03-22T02:03:24.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 -- 172.21.15.28:0/1107304001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 msgr2=0x7fe19007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 --2- 172.21.15.28:0/1107304001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe1780098a0 tx=0x7fe178009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 -- 172.21.15.28:0/1107304001 shutdown_connections 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 --2- 172.21.15.28:0/1107304001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19007c2d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 -- 172.21.15.28:0/1107304001 >> 172.21.15.28:0/1107304001 conn(0x7fe190103770 msgr2=0x7fe190105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 -- 172.21.15.28:0/1107304001 shutdown_connections 2024-03-22T02:03:24.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.020+0000 7fe196237700 1 -- 172.21.15.28:0/1107304001 wait complete. 2024-03-22T02:03:24.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe196237700 1 Processor -- start 2024-03-22T02:03:24.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe196237700 1 -- start start 2024-03-22T02:03:24.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe196237700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:24.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe196237700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe19007cec0 con 0x7fe19007ddd0 2024-03-22T02:03:24.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe18f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe18f7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36142/0 (socket says 172.21.15.28:36142) 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.021+0000 7fe18f7fe700 1 -- 172.21.15.28:0/2706987947 learned_addr learned my addr 172.21.15.28:0/2706987947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18f7fe700 1 -- 172.21.15.28:0/2706987947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019def0 con 0x7fe19007ddd0 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18f7fe700 1 --2- 172.21.15.28:0/2706987947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe178000c00 tx=0x7fe178004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe178003db0 con 0x7fe19007ddd0 2024-03-22T02:03:24.670 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe178011630 con 0x7fe19007ddd0 2024-03-22T02:03:24.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019e180 con 0x7fe19007ddd0 2024-03-22T02:03:24.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe178011950 con 0x7fe19007ddd0 2024-03-22T02:03:24.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1901a0e70 con 0x7fe19007ddd0 2024-03-22T02:03:24.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe178011af0 con 0x7fe19007ddd0 2024-03-22T02:03:24.671 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.022+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe17801c450 con 0x7fe19007ddd0 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.023+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe190063cb0 con 0x7fe19007ddd0 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.025+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe178032020 con 0x7fe19007ddd0 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.098+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fe19002def0 con 0x7fe19007ddd0 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.098+0000 7fe18cff9700 1 -- 172.21.15.28:0/2706987947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fe178004260 con 0x7fe19007ddd0 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 msgr2=0x7fe19019d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:24.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 --2- 172.21.15.28:0/2706987947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe178000c00 tx=0x7fe178004750 comp rx=0 tx=0).stop 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 shutdown_connections 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 --2- 172.21.15.28:0/2706987947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19007ddd0 0x7fe19019d9b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 >> 172.21.15.28:0/2706987947 conn(0x7fe190103770 msgr2=0x7fe190103b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 shutdown_connections 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:24.099+0000 7fe196237700 1 -- 172.21.15.28:0/2706987947 wait complete. 2024-03-22T02:03:24.673 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-03-22T02:03:24.674 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-03-22T02:03:25.811 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:25.812 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d6714e700 1 Processor -- start 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d6714e700 1 -- start start 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d6714e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d6714e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6007be40 con 0x7f7d60108bb0 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d64eea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d64eea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36154/0 (socket says 172.21.15.28:36154) 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.119+0000 7f7d64eea700 1 -- 172.21.15.28:0/3748392578 learned_addr learned my addr 172.21.15.28:0/3748392578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:25.813 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d64eea700 1 -- 172.21.15.28:0/3748392578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6010b4e0 con 0x7f7d60108bb0 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d64eea700 1 --2- 172.21.15.28:0/3748392578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7d540098a0 tx=0x7f7d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bab1291c9f4d4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d5f7fe700 1 -- 172.21.15.28:0/3748392578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d54004160 con 0x7f7d60108bb0 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d5f7fe700 1 -- 172.21.15.28:0/3748392578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d54004300 con 0x7f7d60108bb0 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d5f7fe700 1 -- 172.21.15.28:0/3748392578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d540036a0 con 0x7f7d60108bb0 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d6714e700 1 -- 172.21.15.28:0/3748392578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 msgr2=0x7f7d6010afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d6714e700 1 --2- 172.21.15.28:0/3748392578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7d540098a0 tx=0x7f7d54009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d6714e700 1 -- 172.21.15.28:0/3748392578 shutdown_connections 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d6714e700 1 --2- 172.21.15.28:0/3748392578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6010afa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.120+0000 7f7d6714e700 1 -- 172.21.15.28:0/3748392578 >> 172.21.15.28:0/3748392578 conn(0x7f7d601025a0 msgr2=0x7f7d60104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 -- 172.21.15.28:0/3748392578 shutdown_connections 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 -- 172.21.15.28:0/3748392578 wait complete. 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 Processor -- start 2024-03-22T02:03:25.814 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 -- start start 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d6714e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6010b950 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d64eea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d64eea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36166/0 (socket says 172.21.15.28:36166) 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.121+0000 7f7d64eea700 1 -- 172.21.15.28:0/2270122136 learned_addr learned my addr 172.21.15.28:0/2270122136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d64eea700 1 -- 172.21.15.28:0/2270122136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6019bd40 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d64eea700 1 --2- 172.21.15.28:0/2270122136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d54004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d54003db0 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d54011630 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6019c030 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d54011950 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6019ecc0 con 0x7f7d60108bb0 2024-03-22T02:03:25.815 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7d5401a460 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.122+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7d5401b780 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.123+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d60063cb0 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.125+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7d5401ba20 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.196+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f7d6002dfe0 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.199+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f7d5401bc00 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 msgr2=0x7f7d6019b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 --2- 172.21.15.28:0/2270122136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7d54000c00 tx=0x7f7d54004750 comp rx=0 tx=0).stop 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d5dffb700 1 -- 172.21.15.28:0/2270122136 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7d5401bde0 con 0x7f7d60108bb0 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 shutdown_connections 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 --2- 172.21.15.28:0/2270122136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d60108bb0 0x7f7d6019b800 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 >> 172.21.15.28:0/2270122136 conn(0x7f7d601025a0 msgr2=0x7f7d601048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:25.816 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 shutdown_connections 2024-03-22T02:03:25.817 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:25.200+0000 7f7d6714e700 1 -- 172.21.15.28:0/2270122136 wait complete. 2024-03-22T02:03:25.817 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f8e440700 1 Processor -- start 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f8e440700 1 -- start start 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f8e440700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f8e440700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f881079f0 con 0x7f9f88107000 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36182/0 (socket says 172.21.15.28:36182) 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.222+0000 7f9f87fff700 1 -- 172.21.15.28:0/3411504503 learned_addr learned my addr 172.21.15.28:0/3411504503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.223+0000 7f9f87fff700 1 -- 172.21.15.28:0/3411504503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f88108200 con 0x7f9f88107000 2024-03-22T02:03:26.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.223+0000 7f9f87fff700 1 --2- 172.21.15.28:0/3411504503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f780098a0 tx=0x7f9f78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa800daff03b5cc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.223+0000 7f9f86ffd700 1 -- 172.21.15.28:0/3411504503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f78004160 con 0x7f9f88107000 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.223+0000 7f9f86ffd700 1 -- 172.21.15.28:0/3411504503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f78004300 con 0x7f9f88107000 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.223+0000 7f9f86ffd700 1 -- 172.21.15.28:0/3411504503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f78011b50 con 0x7f9f88107000 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- 172.21.15.28:0/3411504503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 msgr2=0x7f9f88107420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 --2- 172.21.15.28:0/3411504503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9f780098a0 tx=0x7f9f78009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- 172.21.15.28:0/3411504503 shutdown_connections 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 --2- 172.21.15.28:0/3411504503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88107420 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- 172.21.15.28:0/3411504503 >> 172.21.15.28:0/3411504503 conn(0x7f9f88102580 msgr2=0x7f9f881049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- 172.21.15.28:0/3411504503 shutdown_connections 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- 172.21.15.28:0/3411504503 wait complete. 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 Processor -- start 2024-03-22T02:03:26.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.224+0000 7f9f8e440700 1 -- start start 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f8e440700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f8e440700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8807af80 con 0x7f9f88107000 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36190/0 (socket says 172.21.15.28:36190) 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f87fff700 1 -- 172.21.15.28:0/1246930347 learned_addr learned my addr 172.21.15.28:0/1246930347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f87fff700 1 -- 172.21.15.28:0/1246930347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8819a400 con 0x7f9f88107000 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f87fff700 1 --2- 172.21.15.28:0/1246930347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f78011670 tx=0x7f9f78011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.225+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f780039c0 con 0x7f9f88107000 2024-03-22T02:03:26.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8819a690 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9f78011e50 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9f7801a5c0 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8819d380 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9f7801ac50 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.226+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9f78019700 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.227+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f88063cb0 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.231+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9f78003b60 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.300+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f9f88193420 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.301+0000 7f9f857fa700 1 -- 172.21.15.28:0/1246930347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f9f780199a0 con 0x7f9f88107000 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.301+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 msgr2=0x7f9f88199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.301+0000 7f9f8e440700 1 --2- 172.21.15.28:0/1246930347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9f78011670 tx=0x7f9f78011b60 comp rx=0 tx=0).stop 2024-03-22T02:03:26.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.302+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 shutdown_connections 2024-03-22T02:03:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.302+0000 7f9f8e440700 1 --2- 172.21.15.28:0/1246930347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f88107000 0x7f9f88199ec0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.302+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 >> 172.21.15.28:0/1246930347 conn(0x7f9f88102580 msgr2=0x7f9f881904e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.302+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 shutdown_connections 2024-03-22T02:03:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:26.302+0000 7f9f8e440700 1 -- 172.21.15.28:0/1246930347 wait complete. 2024-03-22T02:03:26.829 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-03-22T02:03:27.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:26 smithi028 systemd[1]: Stopping Ceph mon.a for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:03:27.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31523]: 2024-03-22T02:03:27.091+0000 7f4501c06700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:03:27.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31523]: 2024-03-22T02:03:27.091+0000 7f4501c06700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-22T02:03:27.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 podman[31784]: 2024-03-22 02:03:27.112458829 +0000 UTC m=+0.119736320 container died c7a8a41f1cf5bfdc361f1d3b9180a5da176f1e36dacc509ea21bc59ca9a3baab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-22T02:03:27.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 podman[31784]: 2024-03-22 02:03:27.394783215 +0000 UTC m=+0.402060713 container cleanup c7a8a41f1cf5bfdc361f1d3b9180a5da176f1e36dacc509ea21bc59ca9a3baab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T02:03:27.394 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 bash[31784]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a 2024-03-22T02:03:27.721 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 podman[31798]: 2024-03-22 02:03:27.612164371 +0000 UTC m=+0.499553853 container remove c7a8a41f1cf5bfdc361f1d3b9180a5da176f1e36dacc509ea21bc59ca9a3baab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-22T02:03:27.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 systemd[1]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-22T02:03:27.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 systemd[1]: Stopped Ceph mon.a for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:03:27.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 systemd[1]: Starting Ceph mon.a for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:03:27.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 podman[31897]: 2024-03-22 02:03:27.913019903 +0000 UTC m=+0.085635148 container create 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T02:03:27.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:27 smithi028 podman[31897]: 2024-03-22 02:03:27.845233235 +0000 UTC m=+0.017848447 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:03:28.161 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 podman[31897]: 2024-03-22 02:03:28.056486115 +0000 UTC m=+0.229101317 container init 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 podman[31897]: 2024-03-22 02:03:28.059251184 +0000 UTC m=+0.231866384 container start 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: pidfile_write: ignore empty --pid-file 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: load: jerasure load: lrc 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: RocksDB version: 7.9.2 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Git sha 0 2024-03-22T02:03:28.249 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: DB SUMMARY 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: DB Session ID: WYAM3ZWTL4NBKUKPLMXF 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105764 ; 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T02:03:28.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:03:28.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T02:03:28.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.env: 0x559f823e5c60 2024-03-22T02:03:28.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T02:03:28.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.info_log: 0x559f83331ae0 2024-03-22T02:03:28.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:03:28.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.statistics: (nil) 2024-03-22T02:03:28.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.use_fsync: 0 2024-03-22T02:03:28.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:03:28.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.db_log_dir: 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.wal_dir: 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:03:28.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.write_buffer_manager: 0x559f833021e0 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:03:28.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.unordered_write: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.row_cache: None 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.wal_filter: None 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:03:28.255 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.wal_compression: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T02:03:28.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_open_files: -1 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:03:28.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Compression algorithms supported: 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kZSTD supported: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:03:28.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.merge_operator: 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_filter: None 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559f83331be0) 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:03:28.259 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x559f83335090 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-03-22T02:03:28.260 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-03-22T02:03:28.261 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression: NoCompression 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:03:28.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.num_levels: 7 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:03:28.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:03:28.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T02:03:28.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:03:28.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.ttl: 2592000 2024-03-22T02:03:28.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T02:03:28.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cf3955b3-5409-4ea7-beb4-6d2d74a8956d 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073008087686, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073008088626, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100821, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98736, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14328, "raw_average_key_size": 49, "raw_value_size": 90951, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711073008, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cf3955b3-5409-4ea7-beb4-6d2d74a8956d", "db_session_id": "WYAM3ZWTL4NBKUKPLMXF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073008088709, "job": 1, "event": "recovery_finished"} 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559f83424000 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: DB pointer 0x559f8340e000 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T02:03:28.269 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 100.29 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T02:03:28.270 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 100.29 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 164.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 164.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:03:28.271 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 32.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 32.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x559f83335090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???) e1 preinit fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).mds e1 new map 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).mds e1 print_map 2024-03-22T02:03:28.272 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 bash[31897]: 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: monmap epoch 1 2024-03-22T02:03:28.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: last_changed 2024-03-22T02:03:20.403582+0000 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: min_mon_release 19 (squid) 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: election_strategy: 1 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: fsmap 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 ceph-mon[31916]: mgrmap e1: no daemons active 2024-03-22T02:03:28.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:28 smithi028 systemd[1]: Started Ceph mon.a for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:03:29.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.588+0000 7fe21c796700 1 Processor -- start 2024-03-22T02:03:29.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.588+0000 7fe21c796700 1 -- start start 2024-03-22T02:03:29.236 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.589+0000 7fe21c796700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.589+0000 7fe21c796700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe21407be20 con 0x7fe214108be0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.589+0000 7fe21a532700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.589+0000 7fe21a532700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36194/0 (socket says 172.21.15.28:36194) 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.589+0000 7fe21a532700 1 -- 172.21.15.28:0/1296009921 learned_addr learned my addr 172.21.15.28:0/1296009921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe21a532700 1 -- 172.21.15.28:0/1296009921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21410b510 con 0x7fe214108be0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe21a532700 1 --2- 172.21.15.28:0/1296009921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe2100098a0 tx=0x7fe210009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7397db9a1f7b160 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe219530700 1 -- 172.21.15.28:0/1296009921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe210004160 con 0x7fe214108be0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe219530700 1 -- 172.21.15.28:0/1296009921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe210004300 con 0x7fe214108be0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe219530700 1 -- 172.21.15.28:0/1296009921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe210011b50 con 0x7fe214108be0 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe21c796700 1 -- 172.21.15.28:0/1296009921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 msgr2=0x7fe21410afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.590+0000 7fe21c796700 1 --2- 172.21.15.28:0/1296009921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe2100098a0 tx=0x7fe210009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- 172.21.15.28:0/1296009921 shutdown_connections 2024-03-22T02:03:29.237 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 --2- 172.21.15.28:0/1296009921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21410afd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- 172.21.15.28:0/1296009921 >> 172.21.15.28:0/1296009921 conn(0x7fe2141025d0 msgr2=0x7fe214104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- 172.21.15.28:0/1296009921 shutdown_connections 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- 172.21.15.28:0/1296009921 wait complete. 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 Processor -- start 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- start start 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.591+0000 7fe21c796700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe21410c530 con 0x7fe214108be0 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21a532700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21a532700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36204/0 (socket says 172.21.15.28:36204) 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21a532700 1 -- 172.21.15.28:0/3513127149 learned_addr learned my addr 172.21.15.28:0/3513127149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21a532700 1 -- 172.21.15.28:0/3513127149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21419c5c0 con 0x7fe214108be0 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21a532700 1 --2- 172.21.15.28:0/3513127149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe210011b60 tx=0x7fe210011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:29.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe210003b20 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe210003cc0 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21419c850 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe210024840 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21419f540 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.592+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe210022020 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.593+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe210011e00 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.593+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe214063cb0 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.596+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe210043dd0 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.665+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fe21419f710 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.666+0000 7fe20b7fe700 1 -- 172.21.15.28:0/3513127149 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fe210032030 con 0x7fe214108be0 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 msgr2=0x7fe21419c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 --2- 172.21.15.28:0/3513127149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe210011b60 tx=0x7fe210011c40 comp rx=0 tx=0).stop 2024-03-22T02:03:29.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 shutdown_connections 2024-03-22T02:03:29.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 --2- 172.21.15.28:0/3513127149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe214108be0 0x7fe21419c080 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:29.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 >> 172.21.15.28:0/3513127149 conn(0x7fe2141025d0 msgr2=0x7fe2141030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:29.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 shutdown_connections 2024-03-22T02:03:29.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:28.667+0000 7fe21c796700 1 -- 172.21.15.28:0/3513127149 wait complete. 2024-03-22T02:03:29.240 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-22T02:03:29.261 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:03:29.261 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-03-22T02:03:29.262 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-22T02:03:29.262 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-22T02:03:29.524 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x 2024-03-22T02:03:29.525 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-22T02:03:29.671 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715.target.wants/ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-426c3602-e7f0-11ee-95cd-87774f69a715@.service. 2024-03-22T02:03:29.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:29 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3513127149' entity='client.admin' 2024-03-22T02:03:29.821 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:29 smithi028 systemd[1]: Starting Ceph mgr.x for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:03:30.176 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:29 smithi028 podman[32131]: 2024-03-22 02:03:29.804649781 +0000 UTC m=+0.017843863 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:03:30.177 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:29 smithi028 podman[32131]: 2024-03-22 02:03:29.922849561 +0000 UTC m=+0.136043638 container create 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-22T02:03:30.177 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 podman[32131]: 2024-03-22 02:03:30.102190814 +0000 UTC m=+0.315384901 container init 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2) 2024-03-22T02:03:30.177 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 podman[32131]: 2024-03-22 02:03:30.104612549 +0000 UTC m=+0.317806627 container start 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T02:03:30.189 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:03:30.190 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-22T02:03:30.197 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:03:30.197 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:03:30.197 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-22T02:03:30.197 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-03-22T02:03:30.205 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:03:30.205 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-22T02:03:30.212 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:03:30.213 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:03:30.213 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-22T02:03:30.213 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-22T02:03:30.213 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-03-22T02:03:30.213 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-03-22T02:03:30.504 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 bash[32131]: 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd 2024-03-22T02:03:30.505 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 systemd[1]: Started Ceph mgr.x for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:03:30.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2997833495' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:03:30.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:30.679+0000 7f8b3b0b1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T02:03:30.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:30.780+0000 7f8b3b0b1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T02:03:31.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:03:31.238 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:03:31.239 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:03:31.240 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:03:31.241 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:03:23.539550+0000", 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.605+0000 7fc9bbab1700 1 Processor -- start 2024-03-22T02:03:31.242 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.605+0000 7fc9bbab1700 1 -- start start 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.605+0000 7fc9bbab1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.605+0000 7fc9bbab1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b41079f0 con 0x7fc9b4107000 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b984d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b984d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36214/0 (socket says 172.21.15.28:36214) 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b984d700 1 -- 172.21.15.28:0/1799919250 learned_addr learned my addr 172.21.15.28:0/1799919250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b984d700 1 -- 172.21.15.28:0/1799919250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b4108200 con 0x7fc9b4107000 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b984d700 1 --2- 172.21.15.28:0/1799919250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40098a0 tx=0x7fc9a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33372aa188c3af04 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b884b700 1 -- 172.21.15.28:0/1799919250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a4004160 con 0x7fc9b4107000 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b884b700 1 -- 172.21.15.28:0/1799919250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc9a4004300 con 0x7fc9b4107000 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.606+0000 7fc9b884b700 1 -- 172.21.15.28:0/1799919250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a4011b50 con 0x7fc9b4107000 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 -- 172.21.15.28:0/1799919250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 msgr2=0x7fc9b4107420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:31.243 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 --2- 172.21.15.28:0/1799919250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc9a40098a0 tx=0x7fc9a4009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 -- 172.21.15.28:0/1799919250 shutdown_connections 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 --2- 172.21.15.28:0/1799919250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4107420 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 -- 172.21.15.28:0/1799919250 >> 172.21.15.28:0/1799919250 conn(0x7fc9b4102580 msgr2=0x7fc9b41049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 -- 172.21.15.28:0/1799919250 shutdown_connections 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.607+0000 7fc9bbab1700 1 -- 172.21.15.28:0/1799919250 wait complete. 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9bbab1700 1 Processor -- start 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9bbab1700 1 -- start start 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9bbab1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9bbab1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b410b5b0 con 0x7fc9b4107000 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9b984d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9b984d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36216/0 (socket says 172.21.15.28:36216) 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.608+0000 7fc9b984d700 1 -- 172.21.15.28:0/2997833495 learned_addr learned my addr 172.21.15.28:0/2997833495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:31.244 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9b984d700 1 -- 172.21.15.28:0/2997833495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b41926a0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9b984d700 1 --2- 172.21.15.28:0/2997833495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000c00 tx=0x7fc9a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a4003b20 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b41908e0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc9a4003cc0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a4024910 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b4190dc0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc9a4022020 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.609+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc9a4011e00 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.610+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b4063cb0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.612+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc9a4042dd0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.686+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc9b4195eb0 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.686+0000 7fc9aaffd700 1 -- 172.21.15.28:0/2997833495 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc9a4031020 con 0x7fc9b4107000 2024-03-22T02:03:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 msgr2=0x7fc9b4192160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 --2- 172.21.15.28:0/2997833495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4000c00 tx=0x7fc9a4011c10 comp rx=0 tx=0).stop 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 shutdown_connections 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 --2- 172.21.15.28:0/2997833495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b4107000 0x7fc9b4192160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 >> 172.21.15.28:0/2997833495 conn(0x7fc9b4102580 msgr2=0x7fc9b4103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 shutdown_connections 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:30.687+0000 7fc9bbab1700 1 -- 172.21.15.28:0/2997833495 wait complete. 2024-03-22T02:03:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-03-22T02:03:31.784 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:31.449+0000 7f8b3b0b1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T02:03:31.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:31.544+0000 7f8b3b0b1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T02:03:31.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:31.686+0000 7f8b3b0b1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T02:03:31.785 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:31.784+0000 7f8b3b0b1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T02:03:32.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:31.869+0000 7f8b3b0b1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T02:03:32.433 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:32.137+0000 7f8b3b0b1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T02:03:32.434 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:32.249+0000 7f8b3b0b1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T02:03:32.434 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:32.337+0000 7f8b3b0b1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T02:03:32.434 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:32.433+0000 7f8b3b0b1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T02:03:32.821 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:32.611+0000 7f8b3b0b1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T02:03:34.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:33 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3373726662' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:03:34.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:33.629+0000 7f8b3b0b1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T02:03:34.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:33.771+0000 7f8b3b0b1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T02:03:34.322 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:34.322 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:03:34.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T02:03:34.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:03:34.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:03:34.326 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:03:34.327 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:34.328 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:03:23.539550+0000", 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.609+0000 7f7aa2195700 1 Processor -- start 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.609+0000 7f7aa2195700 1 -- start start 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7aa2195700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7aa2195700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c07bd80 con 0x7f7a9c108c90 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:34.329 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36124/0 (socket says 172.21.15.28:36124) 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9b7fe700 1 -- 172.21.15.28:0/3646487693 learned_addr learned my addr 172.21.15.28:0/3646487693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9b7fe700 1 -- 172.21.15.28:0/3646487693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c10b5c0 con 0x7f7a9c108c90 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9b7fe700 1 --2- 172.21.15.28:0/3646487693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a840098a0 tx=0x7f7a84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c6a5e9d056643 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9a7fc700 1 -- 172.21.15.28:0/3646487693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a84004160 con 0x7f7a9c108c90 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9a7fc700 1 -- 172.21.15.28:0/3646487693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a84004300 con 0x7f7a9c108c90 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.610+0000 7f7a9a7fc700 1 -- 172.21.15.28:0/3646487693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a84011b50 con 0x7f7a9c108c90 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 -- 172.21.15.28:0/3646487693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 msgr2=0x7f7a9c10b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 --2- 172.21.15.28:0/3646487693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7a840098a0 tx=0x7f7a84009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:34.330 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 -- 172.21.15.28:0/3646487693 shutdown_connections 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 --2- 172.21.15.28:0/3646487693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c10b080 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 -- 172.21.15.28:0/3646487693 >> 172.21.15.28:0/3646487693 conn(0x7f7a9c1025c0 msgr2=0x7f7a9c104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 -- 172.21.15.28:0/3646487693 shutdown_connections 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.611+0000 7f7aa2195700 1 -- 172.21.15.28:0/3646487693 wait complete. 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7aa2195700 1 Processor -- start 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7aa2195700 1 -- start start 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7aa2195700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7aa2195700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9c10c520 con 0x7f7a9c108c90 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7a9b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:34.331 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7a9b7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36136/0 (socket says 172.21.15.28:36136) 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.612+0000 7f7a9b7fe700 1 -- 172.21.15.28:0/3373726662 learned_addr learned my addr 172.21.15.28:0/3373726662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a9b7fe700 1 -- 172.21.15.28:0/3373726662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a9c108a50 con 0x7f7a9c108c90 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a9b7fe700 1 --2- 172.21.15.28:0/3373726662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a84011670 tx=0x7f7a84011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a840039c0 con 0x7f7a9c108c90 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7a84011e50 con 0x7f7a9c108c90 2024-03-22T02:03:34.332 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a9c106c40 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a8401a5c0 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a9c107180 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7a8401ac50 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.613+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7a84019700 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.614+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a9c063cb0 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.616+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7a84003b60 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.690+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7a9c1076b0 con 0x7f7a9c108c90 2024-03-22T02:03:34.333 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.691+0000 7f7a98ff9700 1 -- 172.21.15.28:0/3373726662 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7a840199a0 con 0x7f7a9c108c90 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.691+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 msgr2=0x7f7a9c108510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.691+0000 7f7aa2195700 1 --2- 172.21.15.28:0/3373726662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a84011670 tx=0x7f7a84011b60 comp rx=0 tx=0).stop 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.692+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 shutdown_connections 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.692+0000 7f7aa2195700 1 --2- 172.21.15.28:0/3373726662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a9c108c90 0x7f7a9c108510 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.692+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 >> 172.21.15.28:0/3373726662 conn(0x7f7a9c1025c0 msgr2=0x7f7a9c1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.692+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 shutdown_connections 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:33.692+0000 7f7aa2195700 1 -- 172.21.15.28:0/3373726662 wait complete. 2024-03-22T02:03:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-03-22T02:03:34.406 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:34.112+0000 7f8b3b0b1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T02:03:34.406 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:34.206+0000 7f8b3b0b1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T02:03:34.407 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:34.406+0000 7f8b3b0b1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:03:34.821 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:34.596+0000 7f8b3b0b1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T02:03:35.145 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:34.885+0000 7f8b3b0b1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:03:35.145 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:35.145+0000 7f8b3b0b1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T02:03:35.571 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:35.387+0000 7f8b3b0b1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T02:03:36.528 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:36.219+0000 7f8b3b0b1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T02:03:37.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:36 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2870428369' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:03:37.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:36.818+0000 7f8b3b0b1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T02:03:37.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:36.923+0000 7f8b3b0b1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T02:03:37.374 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:37.375 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:03:37.376 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:03:37.377 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:03:37.378 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:03:23.539550+0000", 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:37.379 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.731+0000 7f05205cc700 1 Processor -- start 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.731+0000 7f05205cc700 1 -- start start 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f05205cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f05205cc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f051807c7d0 con 0x7f051807dd90 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f051e368700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f051e368700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36144/0 (socket says 172.21.15.28:36144) 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f051e368700 1 -- 172.21.15.28:0/1860569082 learned_addr learned my addr 172.21.15.28:0/1860569082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f051e368700 1 -- 172.21.15.28:0/1860569082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051807c950 con 0x7f051807dd90 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.732+0000 7f051e368700 1 --2- 172.21.15.28:0/1860569082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f050c0098a0 tx=0x7f050c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a03815db8368557c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f051d366700 1 -- 172.21.15.28:0/1860569082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f050c004160 con 0x7f051807dd90 2024-03-22T02:03:37.380 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f051d366700 1 -- 172.21.15.28:0/1860569082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f050c004300 con 0x7f051807dd90 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f051d366700 1 -- 172.21.15.28:0/1860569082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f050c011b50 con 0x7f051807dd90 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 -- 172.21.15.28:0/1860569082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 msgr2=0x7f051807c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 --2- 172.21.15.28:0/1860569082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f050c0098a0 tx=0x7f050c009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 -- 172.21.15.28:0/1860569082 shutdown_connections 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 --2- 172.21.15.28:0/1860569082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051807c290 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 -- 172.21.15.28:0/1860569082 >> 172.21.15.28:0/1860569082 conn(0x7f0518103720 msgr2=0x7f0518105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.733+0000 7f05205cc700 1 -- 172.21.15.28:0/1860569082 shutdown_connections 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f05205cc700 1 -- 172.21.15.28:0/1860569082 wait complete. 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f05205cc700 1 Processor -- start 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f05205cc700 1 -- start start 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f05205cc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f05205cc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f051810e6f0 con 0x7f051807dd90 2024-03-22T02:03:37.381 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f051e368700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.734+0000 7f051e368700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36146/0 (socket says 172.21.15.28:36146) 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f051e368700 1 -- 172.21.15.28:0/2870428369 learned_addr learned my addr 172.21.15.28:0/2870428369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f051e368700 1 -- 172.21.15.28:0/2870428369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f051819e7c0 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f051e368700 1 --2- 172.21.15.28:0/2870428369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f050c011670 tx=0x7f050c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f050c0039c0 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f050c011e50 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f050c01a5c0 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f051819ea50 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05181a1740 con 0x7f051807dd90 2024-03-22T02:03:37.382 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f050c01ac00 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.735+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f050c019a90 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.736+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0518063cb0 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.738+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f050c003b60 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.812+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f05181a1910 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.813+0000 7f050b7fe700 1 -- 172.21.15.28:0/2870428369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f050c019d30 con 0x7f051807dd90 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 msgr2=0x7f051819e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 --2- 172.21.15.28:0/2870428369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f050c011670 tx=0x7f050c011b60 comp rx=0 tx=0).stop 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 shutdown_connections 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 --2- 172.21.15.28:0/2870428369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051807dd90 0x7f051819e280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 >> 172.21.15.28:0/2870428369 conn(0x7f0518103720 msgr2=0x7f0518104290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 shutdown_connections 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:36.814+0000 7f05205cc700 1 -- 172.21.15.28:0/2870428369 wait complete. 2024-03-22T02:03:37.383 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-03-22T02:03:37.571 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:37.148+0000 7f8b3b0b1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T02:03:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: Activating manager daemon x 2024-03-22T02:03:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: mgrmap e2: x(active, starting, since 0.00309969s) 2024-03-22T02:03:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: Manager daemon x is now available 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' 2024-03-22T02:03:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:37 smithi028 ceph-mon[31916]: from='mgr.14100 172.21.15.28:0/3495204857' entity='mgr.x' 2024-03-22T02:03:39.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:39 smithi028 ceph-mon[31916]: mgrmap e3: x(active, since 1.00715s) 2024-03-22T02:03:41.473 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:41 smithi028 ceph-mon[31916]: mgrmap e4: x(active, since 3s) 2024-03-22T02:03:43.002 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:42 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/254317483' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T02:03:48.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-03-22T02:03:48.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-22T02:03:48.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-22T02:03:48.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-22T02:03:48.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-03-22T02:03:23.539550+0000", 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:03:48.633 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4b052d700 1 Processor -- start 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4b052d700 1 -- start start 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4b052d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4b052d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a8108b80 con 0x7fe4a8108190 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4ae2c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4ae2c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59698/0 (socket says 172.21.15.28:59698) 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4ae2c9700 1 -- 172.21.15.28:0/3524319564 learned_addr learned my addr 172.21.15.28:0/3524319564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.430+0000 7fe4ae2c9700 1 -- 172.21.15.28:0/3524319564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a81092f0 con 0x7fe4a8108190 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4ae2c9700 1 --2- 172.21.15.28:0/3524319564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5bb0b531ee07cc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:48.634 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4ad2c7700 1 -- 172.21.15.28:0/3524319564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a4004160 con 0x7fe4a8108190 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4ad2c7700 1 -- 172.21.15.28:0/3524319564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe4a4004300 con 0x7fe4a8108190 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4ad2c7700 1 -- 172.21.15.28:0/3524319564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a4011b50 con 0x7fe4a8108190 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4b052d700 1 -- 172.21.15.28:0/3524319564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 msgr2=0x7fe4a81085b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4b052d700 1 --2- 172.21.15.28:0/3524319564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40098a0 tx=0x7fe4a4009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4b052d700 1 -- 172.21.15.28:0/3524319564 shutdown_connections 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4b052d700 1 --2- 172.21.15.28:0/3524319564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a81085b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.431+0000 7fe4b052d700 1 -- 172.21.15.28:0/3524319564 >> 172.21.15.28:0/3524319564 conn(0x7fe4a8103750 msgr2=0x7fe4a8105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 -- 172.21.15.28:0/3524319564 shutdown_connections 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 -- 172.21.15.28:0/3524319564 wait complete. 2024-03-22T02:03:48.635 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 Processor -- start 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 -- start start 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4b052d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4a810c6a0 con 0x7fe4a8108190 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4ae2c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4ae2c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59710/0 (socket says 172.21.15.28:59710) 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.432+0000 7fe4ae2c9700 1 -- 172.21.15.28:0/254317483 learned_addr learned my addr 172.21.15.28:0/254317483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe4ae2c9700 1 -- 172.21.15.28:0/254317483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4a807e060 con 0x7fe4a8108190 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe4ae2c9700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4011670 tx=0x7fe4a4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a40039c0 con 0x7fe4a8108190 2024-03-22T02:03:48.636 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe4a4011e50 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe4a401a5c0 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4a807c250 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.433+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4a807c790 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.434+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7fe4a4024430 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.434+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4a8063cb0 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.434+0000 7fe49f7fe700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 0x7fe49403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.434+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe4a404d530 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.435+0000 7fe4adac8700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 0x7fe49403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.435+0000 7fe4adac8700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 0x7fe49403b5f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe498006fd0 tx=0x7fe498006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.440+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4a404cf50 con 0x7fe4a8108190 2024-03-22T02:03:48.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.728+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe4a807ca10 con 0x7fe4a8108190 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.729+0000 7fe49f7fe700 1 -- 172.21.15.28:0/254317483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe4a404f5b0 con 0x7fe4a8108190 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 msgr2=0x7fe49403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 0x7fe49403b5f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe498006fd0 tx=0x7fe498006e40 comp rx=0 tx=0).stop 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 msgr2=0x7fe4a807db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4011670 tx=0x7fe4a4011b60 comp rx=0 tx=0).stop 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 shutdown_connections 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fe494039130 0x7fe49403b5f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 --2- 172.21.15.28:0/254317483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4a8108190 0x7fe4a807db20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 >> 172.21.15.28:0/254317483 conn(0x7fe4a8103750 msgr2=0x7fe4a8104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 shutdown_connections 2024-03-22T02:03:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:42.731+0000 7fe4b052d700 1 -- 172.21.15.28:0/254317483 wait complete. 2024-03-22T02:03:48.639 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-03-22T02:03:51.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2182324703' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T02:03:51.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:51.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-03-22T02:03:51.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:03:51.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-22T02:03:51.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.165+0000 7f7373676700 1 Processor -- start 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.165+0000 7f7373676700 1 -- start start 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.165+0000 7f7373676700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.165+0000 7f7373676700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1079f0 con 0x7f736c107000 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.165+0000 7f7371412700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7371412700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59720/0 (socket says 172.21.15.28:59720) 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7371412700 1 -- 172.21.15.28:0/3833569291 learned_addr learned my addr 172.21.15.28:0/3833569291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7371412700 1 -- 172.21.15.28:0/3833569291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c108200 con 0x7f736c107000 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7371412700 1 --2- 172.21.15.28:0/3833569291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73680098a0 tx=0x7f7368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873055f7cff8d641 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7363fff700 1 -- 172.21.15.28:0/3833569291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7368004160 con 0x7f736c107000 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7363fff700 1 -- 172.21.15.28:0/3833569291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7368004300 con 0x7f736c107000 2024-03-22T02:03:51.963 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.166+0000 7f7363fff700 1 -- 172.21.15.28:0/3833569291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7368011b50 con 0x7f736c107000 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 -- 172.21.15.28:0/3833569291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 msgr2=0x7f736c107420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 --2- 172.21.15.28:0/3833569291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f73680098a0 tx=0x7f7368009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 -- 172.21.15.28:0/3833569291 shutdown_connections 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 --2- 172.21.15.28:0/3833569291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c107420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 -- 172.21.15.28:0/3833569291 >> 172.21.15.28:0/3833569291 conn(0x7f736c102580 msgr2=0x7f736c1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 -- 172.21.15.28:0/3833569291 shutdown_connections 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.167+0000 7f7373676700 1 -- 172.21.15.28:0/3833569291 wait complete. 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7373676700 1 Processor -- start 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7373676700 1 -- start start 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7373676700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7373676700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f736c10b5b0 con 0x7f736c107000 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7371412700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:51.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7371412700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59724/0 (socket says 172.21.15.28:59724) 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7371412700 1 -- 172.21.15.28:0/2182324703 learned_addr learned my addr 172.21.15.28:0/2182324703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7371412700 1 -- 172.21.15.28:0/2182324703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c1926a0 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.168+0000 7f7371412700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7368000c00 tx=0x7f7368011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7368003b20 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7368003cc0 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736c1908e0 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7368024910 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736c190dc0 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.169+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7f7368022020 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.170+0000 7f73627fc700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 0x7f735803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.170+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7368011e00 con 0x7f736c107000 2024-03-22T02:03:51.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.170+0000 7f7370c11700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 0x7f735803b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.170+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f736c063cb0 con 0x7f736c107000 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.170+0000 7f7370c11700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 0x7f735803b640 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f735c006fd0 tx=0x7f735c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.175+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f736800b3e0 con 0x7f736c107000 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.381+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f736c195db0 con 0x7f736c107000 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.382+0000 7f73627fc700 1 -- 172.21.15.28:0/2182324703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f7368032020 con 0x7f736c107000 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 msgr2=0x7f735803b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 0x7f735803b640 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f735c006fd0 tx=0x7f735c006e40 comp rx=0 tx=0).stop 2024-03-22T02:03:51.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 msgr2=0x7f736c192160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7368000c00 tx=0x7f7368011c10 comp rx=0 tx=0).stop 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 shutdown_connections 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7358039180 0x7f735803b640 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 --2- 172.21.15.28:0/2182324703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c107000 0x7f736c192160 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.384+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 >> 172.21.15.28:0/2182324703 conn(0x7f736c102580 msgr2=0x7f736c103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:51.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.385+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 shutdown_connections 2024-03-22T02:03:51.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:51.385+0000 7f7373676700 1 -- 172.21.15.28:0/2182324703 wait complete. 2024-03-22T02:03:51.968 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-03-22T02:03:52.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3873242736' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-22T02:03:52.999 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:52 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: ignoring --setuser ceph since I am not root 2024-03-22T02:03:52.999 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:52 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: ignoring --setgroup ceph since I am not root 2024-03-22T02:03:53.274 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.350+0000 7f41cff2d700 1 Processor -- start 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.350+0000 7f41cff2d700 1 -- start start 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.350+0000 7f41cff2d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.350+0000 7f41cff2d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41c807bdc0 con 0x7f41c8106a30 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cdcc9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cdcc9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55550/0 (socket says 172.21.15.28:55550) 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cdcc9700 1 -- 172.21.15.28:0/2846110149 learned_addr learned my addr 172.21.15.28:0/2846110149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cdcc9700 1 -- 172.21.15.28:0/2846110149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c8107390 con 0x7f41c8106a30 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cdcc9700 1 --2- 172.21.15.28:0/2846110149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009e80 tx=0x7f41c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ced23b235d3159c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:53.275 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cccc7700 1 -- 172.21.15.28:0/2846110149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c0017070 con 0x7f41c8106a30 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cccc7700 1 -- 172.21.15.28:0/2846110149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f41c000b8c0 con 0x7f41c8106a30 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.351+0000 7f41cccc7700 1 -- 172.21.15.28:0/2846110149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c00045c0 con 0x7f41c8106a30 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 -- 172.21.15.28:0/2846110149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 msgr2=0x7f41c8106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 --2- 172.21.15.28:0/2846110149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f41c0009e80 tx=0x7f41c000b330 comp rx=0 tx=0).stop 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 -- 172.21.15.28:0/2846110149 shutdown_connections 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 --2- 172.21.15.28:0/2846110149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c8106e50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 -- 172.21.15.28:0/2846110149 >> 172.21.15.28:0/2846110149 conn(0x7f41c81025d0 msgr2=0x7f41c8104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 -- 172.21.15.28:0/2846110149 shutdown_connections 2024-03-22T02:03:53.276 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 -- 172.21.15.28:0/2846110149 wait complete. 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.352+0000 7f41cff2d700 1 Processor -- start 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cff2d700 1 -- start start 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cff2d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cff2d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41c8108420 con 0x7f41c8106a30 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cdcc9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cdcc9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55566/0 (socket says 172.21.15.28:55566) 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cdcc9700 1 -- 172.21.15.28:0/3873242736 learned_addr learned my addr 172.21.15.28:0/3873242736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cdcc9700 1 -- 172.21.15.28:0/3873242736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c819e7b0 con 0x7f41c8106a30 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cdcc9700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41c00072a0 tx=0x7f41c000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c0020020 con 0x7f41c8106a30 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c819ea40 con 0x7f41c8106a30 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f41c0003b50 con 0x7f41c8106a30 2024-03-22T02:03:53.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f41c0017040 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.353+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c81a1730 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.354+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46283+0+0 (secure 0 0 0) 0x7f41c0023020 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.354+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c8063cb0 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.354+0000 7f41c6ffd700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 0x7f41b003b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.354+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f41c000f050 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.355+0000 7f41cd4c8700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 0x7f41b003b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.355+0000 7f41cd4c8700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 0x7f41b003b5a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.359+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41c002a020 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.624+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f41c802e0e0 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.675+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f41c001b730 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.677+0000 7f41c6ffd700 1 -- 172.21.15.28:0/3873242736 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7f41c00129f0 con 0x7f41c8106a30 2024-03-22T02:03:53.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 msgr2=0x7f41b003b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 0x7f41b003b5a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f41bc006fd0 tx=0x7f41bc006e40 comp rx=0 tx=0).stop 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 msgr2=0x7f41c819e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41c00072a0 tx=0x7f41c000bd50 comp rx=0 tx=0).stop 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 shutdown_connections 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f41b00390e0 0x7f41b003b5a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 --2- 172.21.15.28:0/3873242736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41c8106a30 0x7f41c819e270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 >> 172.21.15.28:0/3873242736 conn(0x7f41c81025d0 msgr2=0x7f41c8194e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 shutdown_connections 2024-03-22T02:03:53.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:52.679+0000 7f41cff2d700 1 -- 172.21.15.28:0/3873242736 wait complete. 2024-03-22T02:03:53.510 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:53 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:53.252+0000 7f5421847200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T02:03:53.510 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:53 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:53.358+0000 7f5421847200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T02:03:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3873242736' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-22T02:03:53.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:53 smithi028 ceph-mon[31916]: mgrmap e5: x(active, since 15s) 2024-03-22T02:03:54.270 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.030+0000 7f5421847200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T02:03:54.270 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.125+0000 7f5421847200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T02:03:54.270 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.269+0000 7f5421847200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T02:03:54.571 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.367+0000 7f5421847200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T02:03:54.571 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.453+0000 7f5421847200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T02:03:54.664 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b0b0da700 1 Processor -- start 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b0b0da700 1 -- start start 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b0b0da700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b0b0da700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0407be00 con 0x7f7b041069d0 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b08e76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b08e76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55594/0 (socket says 172.21.15.28:55594) 2024-03-22T02:03:54.665 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.702+0000 7f7b08e76700 1 -- 172.21.15.28:0/516635903 learned_addr learned my addr 172.21.15.28:0/516635903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b08e76700 1 -- 172.21.15.28:0/516635903 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b04107330 con 0x7f7b041069d0 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b08e76700 1 --2- 172.21.15.28:0/516635903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7af80098a0 tx=0x7f7af8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e874b84eec11746a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b037fe700 1 -- 172.21.15.28:0/516635903 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af8004160 con 0x7f7b041069d0 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b037fe700 1 -- 172.21.15.28:0/516635903 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7af8004300 con 0x7f7b041069d0 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b037fe700 1 -- 172.21.15.28:0/516635903 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af8011b50 con 0x7f7b041069d0 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b0b0da700 1 -- 172.21.15.28:0/516635903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 msgr2=0x7f7b04106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.703+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/516635903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7af80098a0 tx=0x7f7af8009bb0 comp rx=0 tx=0).stop 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 -- 172.21.15.28:0/516635903 shutdown_connections 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/516635903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b04106df0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 -- 172.21.15.28:0/516635903 >> 172.21.15.28:0/516635903 conn(0x7f7b041025d0 msgr2=0x7f7b041049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:54.666 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 -- 172.21.15.28:0/516635903 shutdown_connections 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 -- 172.21.15.28:0/516635903 wait complete. 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 Processor -- start 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.704+0000 7f7b0b0da700 1 -- start start 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b0b0da700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b0b0da700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b041083c0 con 0x7f7b041069d0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b08e76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b08e76700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55604/0 (socket says 172.21.15.28:55604) 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b08e76700 1 -- 172.21.15.28:0/3505076411 learned_addr learned my addr 172.21.15.28:0/3505076411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b08e76700 1 -- 172.21.15.28:0/3505076411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0419e7c0 con 0x7f7b041069d0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b08e76700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7af8011670 tx=0x7f7af8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af80039c0 con 0x7f7b041069d0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7af8011e50 con 0x7f7b041069d0 2024-03-22T02:03:54.667 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7af801a5c0 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.705+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0419ea50 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.706+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b041a1740 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b04063cb0 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7f7af8024430 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b01ffb700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 0x7f7af403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7af804d580 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b03fff700 1 -- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 msgr2=0x7f7af403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.707+0000 7f7b03fff700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 0x7f7af403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.712+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7af800f2e0 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.908+0000 7f7b03fff700 1 -- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 msgr2=0x7f7af403b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.908+0000 7f7b03fff700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 0x7f7af403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.977+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7b040008d0 con 0x7f7b041069d0 2024-03-22T02:03:54.668 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.977+0000 7f7b01ffb700 1 -- 172.21.15.28:0/3505076411 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f7af804cfa0 con 0x7f7b041069d0 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 msgr2=0x7f7af403b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 0x7f7af403b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 msgr2=0x7f7b0419e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7af8011670 tx=0x7f7af8011b60 comp rx=0 tx=0).stop 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 shutdown_connections 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7f7af4039180 0x7f7af403b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 --2- 172.21.15.28:0/3505076411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b041069d0 0x7f7b0419e280 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 >> 172.21.15.28:0/3505076411 conn(0x7f7b041025d0 msgr2=0x7f7b04194f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 shutdown_connections 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:53.979+0000 7f7b0b0da700 1 -- 172.21.15.28:0/3505076411 wait complete. 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-03-22T02:03:54.669 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-03-22T02:03:54.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3505076411' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T02:03:54.944 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.739+0000 7f5421847200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T02:03:54.944 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.853+0000 7f5421847200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T02:03:54.945 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:54.943+0000 7f5421847200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T02:03:55.224 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:55.043+0000 7f5421847200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T02:03:55.571 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:55.223+0000 7f5421847200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T02:03:56.467 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:56.216+0000 7f5421847200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T02:03:56.585 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:56.350+0000 7f5421847200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T02:03:56.781 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:56.689+0000 7f5421847200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T02:03:56.782 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:56.781+0000 7f5421847200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T02:03:57.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:56.981+0000 7f5421847200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:03:57.462 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:57 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:57.171+0000 7f5421847200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T02:03:57.723 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:57 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:57.462+0000 7f5421847200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:03:57.723 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:57 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:57.723+0000 7f5421847200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T02:03:58.068 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:57 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:57.966+0000 7f5421847200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T02:03:59.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:58 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:58.793+0000 7f5421847200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T02:03:59.696 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:59.374+0000 7f5421847200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T02:03:59.697 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:59.475+0000 7f5421847200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T02:03:59.697 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:03:59.696+0000 7f5421847200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: Active manager daemon x restarted 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: Activating manager daemon x 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: osdmap e2: 0 total, 0 up, 0 in 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: mgrmap e6: x(active, starting, since 0.00336865s) 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T02:04:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: Manager daemon x is now available 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:03:59 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: Found migration_current of "None". Setting to last migration. 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: mgrmap e7: x(active, since 1.0053s) 2024-03-22T02:04:01.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:00 smithi028 ceph-mon[31916]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d3a14700 1 Processor -- start 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d3a14700 1 -- start start 2024-03-22T02:04:01.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d3a14700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d3a14700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8cc107190 con 0x7fd8cc1067a0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d17b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d17b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55606/0 (socket says 172.21.15.28:55606) 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d17b0700 1 -- 172.21.15.28:0/3448297035 learned_addr learned my addr 172.21.15.28:0/3448297035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.058+0000 7fd8d17b0700 1 -- 172.21.15.28:0/3448297035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc1079a0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8d17b0700 1 --2- 172.21.15.28:0/3448297035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80098a0 tx=0x7fd8c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c1c70fa0e32b11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8c3fff700 1 -- 172.21.15.28:0/3448297035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8c8004160 con 0x7fd8cc1067a0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8c3fff700 1 -- 172.21.15.28:0/3448297035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8c8004300 con 0x7fd8cc1067a0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8c3fff700 1 -- 172.21.15.28:0/3448297035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8c8011b50 con 0x7fd8cc1067a0 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8d3a14700 1 -- 172.21.15.28:0/3448297035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 msgr2=0x7fd8cc106bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.059+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/3448297035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80098a0 tx=0x7fd8c8009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- 172.21.15.28:0/3448297035 shutdown_connections 2024-03-22T02:04:01.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/3448297035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc106bc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- 172.21.15.28:0/3448297035 >> 172.21.15.28:0/3448297035 conn(0x7fd8cc101d80 msgr2=0x7fd8cc1041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- 172.21.15.28:0/3448297035 shutdown_connections 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- 172.21.15.28:0/3448297035 wait complete. 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 Processor -- start 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- start start 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.060+0000 7fd8d3a14700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8cc10ad50 con 0x7fd8cc1067a0 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d17b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d17b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55620/0 (socket says 172.21.15.28:55620) 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d17b0700 1 -- 172.21.15.28:0/1763670211 learned_addr learned my addr 172.21.15.28:0/1763670211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d17b0700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8cc19b570 con 0x7fd8cc1067a0 2024-03-22T02:04:01.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d17b0700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8011670 tx=0x7fd8c8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8c80039c0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd8c8011e50 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8cc19b800 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8c801a5c0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.061+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8cc19e4f0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.062+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46294+0+0 (secure 0 0 0) 0x7fd8c801a870 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.063+0000 7fd8c27fc700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.063+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd8b803bd50 con 0x7fd8b8039180 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.063+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd8c804d6b0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.063+0000 7fd8d0faf700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.063+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:04:01.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.264+0000 7fd8d0faf700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.264+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.665+0000 7fd8d0faf700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:55.665+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:56.466+0000 7fd8d0faf700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:56.466+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:58.068+0000 7fd8d0faf700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/857886266 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:58.068+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:59.712+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46061+0+0 (secure 0 0 0) 0x7fd8c801abb0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:59.712+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 msgr2=0x7fd8b803b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:03:59.712+0000 7fd8c27fc700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.713+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46188+0+0 (secure 0 0 0) 0x7fd8c804f1a0 con 0x7fd8cc1067a0 2024-03-22T02:04:01.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.713+0000 7fd8c27fc700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 0x7fd8b803ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.713+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd8b803bd50 con 0x7fd8b803ca70 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.713+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 0x7fd8b803ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.714+0000 7fd8d0faf700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 0x7fd8b803ee60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc003a10 tx=0x7fd8bc0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.715+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd8b803bd50 con 0x7fd8b803ca70 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.721+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd8cc063cb0 con 0x7fd8b803ca70 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.721+0000 7fd8c27fc700 1 -- 172.21.15.28:0/1763670211 <== mgr.14120 v2:172.21.15.28:6800/3429985942 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fd8cc063cb0 con 0x7fd8b803ca70 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 msgr2=0x7fd8b803ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 0x7fd8b803ee60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc003a10 tx=0x7fd8bc0092b0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 msgr2=0x7fd8cc19b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8011670 tx=0x7fd8c8011b60 comp rx=0 tx=0).stop 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 shutdown_connections 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8cc1067a0 0x7fd8cc19b030 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.252 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd8b803ca70 0x7fd8b803ee60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 --2- 172.21.15.28:0/1763670211 >> [v2:172.21.15.28:6800/857886266,v1:172.21.15.28:6801/857886266] conn(0x7fd8b8039180 0x7fd8b803b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 >> 172.21.15.28:0/1763670211 conn(0x7fd8cc101d80 msgr2=0x7fd8cc1041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 shutdown_connections 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:00.722+0000 7fd8d3a14700 1 -- 172.21.15.28:0/1763670211 wait complete. 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-03-22T02:04:01.253 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-03-22T02:04:01.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:01 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:01] ENGINE Bus STARTING 2024-03-22T02:04:01.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:01 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:01] ENGINE Serving on http://172.21.15.28:8765 2024-03-22T02:04:01.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:01 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:01] ENGINE Serving on https://172.21.15.28:7150 2024-03-22T02:04:01.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:01 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:01] ENGINE Bus STARTED 2024-03-22T02:04:01.927 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:01 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:02.497 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7593fcd700 1 Processor -- start 2024-03-22T02:04:02.507 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7593fcd700 1 -- start start 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7593fcd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7593fcd700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f758c07c810 con 0x7f758c07dd10 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7591d69700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7591d69700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55702/0 (socket says 172.21.15.28:55702) 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7591d69700 1 -- 172.21.15.28:0/1192761595 learned_addr learned my addr 172.21.15.28:0/1192761595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.676+0000 7f7591d69700 1 -- 172.21.15.28:0/1192761595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c07c990 con 0x7f758c07dd10 2024-03-22T02:04:02.508 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7591d69700 1 --2- 172.21.15.28:0/1192761595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75880098a0 tx=0x7f7588009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37067990b81512a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7590d67700 1 -- 172.21.15.28:0/1192761595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7588004160 con 0x7f758c07dd10 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7590d67700 1 -- 172.21.15.28:0/1192761595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7588004300 con 0x7f758c07dd10 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7590d67700 1 -- 172.21.15.28:0/1192761595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7588011b50 con 0x7f758c07dd10 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7593fcd700 1 -- 172.21.15.28:0/1192761595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 msgr2=0x7f758c07c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7593fcd700 1 --2- 172.21.15.28:0/1192761595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f75880098a0 tx=0x7f7588009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7593fcd700 1 -- 172.21.15.28:0/1192761595 shutdown_connections 2024-03-22T02:04:02.509 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7593fcd700 1 --2- 172.21.15.28:0/1192761595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c07c2d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.677+0000 7f7593fcd700 1 -- 172.21.15.28:0/1192761595 >> 172.21.15.28:0/1192761595 conn(0x7f758c103860 msgr2=0x7f758c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 -- 172.21.15.28:0/1192761595 shutdown_connections 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 -- 172.21.15.28:0/1192761595 wait complete. 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 Processor -- start 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 -- start start 2024-03-22T02:04:02.510 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7593fcd700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f758c10e650 con 0x7f758c07dd10 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7591d69700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7591d69700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56096/0 (socket says 172.21.15.28:56096) 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.678+0000 7f7591d69700 1 -- 172.21.15.28:0/3250882498 learned_addr learned my addr 172.21.15.28:0/3250882498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7591d69700 1 -- 172.21.15.28:0/3250882498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f758c19e7a0 con 0x7f758c07dd10 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7591d69700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7588011b60 tx=0x7f7588011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:02.511 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7588003b20 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7588003cc0 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7588024840 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f758c19ea30 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.679+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f758c1a1720 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.680+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 7) ==== 46188+0+0 (secure 0 0 0) 0x7f7588022020 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.680+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f758c063cb0 con 0x7f758c07dd10 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.680+0000 7f7582ffd700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 0x7f757803b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:02.512 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.680+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7588011e00 con 0x7f758c07dd10 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.680+0000 7f7591568700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 0x7f757803b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.681+0000 7f7591568700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 0x7f757803b570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f757c006fd0 tx=0x7f757c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.686+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f758801a7b0 con 0x7f758c07dd10 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.764+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f7588050030 con 0x7f758c07dd10 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.908+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f758c104a30 con 0x7f75780390b0 2024-03-22T02:04:02.513 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.913+0000 7f7582ffd700 1 -- 172.21.15.28:0/3250882498 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f758c104a30 con 0x7f75780390b0 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.915+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 msgr2=0x7f757803b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.915+0000 7f7593fcd700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 0x7f757803b570 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f757c006fd0 tx=0x7f757c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.915+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 msgr2=0x7f758c19e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.915+0000 7f7593fcd700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7588011b60 tx=0x7f7588011c40 comp rx=0 tx=0).stop 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 shutdown_connections 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f75780390b0 0x7f757803b570 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:02.514 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 --2- 172.21.15.28:0/3250882498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f758c07dd10 0x7f758c19e260 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:02.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 >> 172.21.15.28:0/3250882498 conn(0x7f758c103860 msgr2=0x7f758c104320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:02.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 shutdown_connections 2024-03-22T02:04:02.515 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:01.916+0000 7f7593fcd700 1 -- 172.21.15.28:0/3250882498 wait complete. 2024-03-22T02:04:02.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:02 smithi028 ceph-mon[31916]: mgrmap e8: x(active, since 2s) 2024-03-22T02:04:02.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:02 smithi028 ceph-mon[31916]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:02.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:02 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:02.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:02 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:03.619 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-22T02:04:03.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.913+0000 7efe03c6f700 1 Processor -- start 2024-03-22T02:04:03.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.913+0000 7efe03c6f700 1 -- start start 2024-03-22T02:04:03.620 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.913+0000 7efe03c6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.913+0000 7efe03c6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc07be60 con 0x7efdfc108b70 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.913+0000 7efe01a0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe01a0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56102/0 (socket says 172.21.15.28:56102) 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe01a0b700 1 -- 172.21.15.28:0/3156657043 learned_addr learned my addr 172.21.15.28:0/3156657043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe01a0b700 1 -- 172.21.15.28:0/3156657043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdfc10b4a0 con 0x7efdfc108b70 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe01a0b700 1 --2- 172.21.15.28:0/3156657043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdf40098a0 tx=0x7efdf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803dff554b969ca8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe00a09700 1 -- 172.21.15.28:0/3156657043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdf4004160 con 0x7efdfc108b70 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe00a09700 1 -- 172.21.15.28:0/3156657043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efdf4004300 con 0x7efdfc108b70 2024-03-22T02:04:03.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.914+0000 7efe00a09700 1 -- 172.21.15.28:0/3156657043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdf4011b50 con 0x7efdfc108b70 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- 172.21.15.28:0/3156657043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 msgr2=0x7efdfc10af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 --2- 172.21.15.28:0/3156657043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdf40098a0 tx=0x7efdf4009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- 172.21.15.28:0/3156657043 shutdown_connections 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 --2- 172.21.15.28:0/3156657043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc10af60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- 172.21.15.28:0/3156657043 >> 172.21.15.28:0/3156657043 conn(0x7efdfc102580 msgr2=0x7efdfc1049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- 172.21.15.28:0/3156657043 shutdown_connections 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- 172.21.15.28:0/3156657043 wait complete. 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 Processor -- start 2024-03-22T02:04:03.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.915+0000 7efe03c6f700 1 -- start start 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe03c6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe03c6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc10c4d0 con 0x7efdfc108b70 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe01a0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe01a0b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56104/0 (socket says 172.21.15.28:56104) 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe01a0b700 1 -- 172.21.15.28:0/1553870451 learned_addr learned my addr 172.21.15.28:0/1553870451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe01a0b700 1 -- 172.21.15.28:0/1553870451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdfc108930 con 0x7efdfc108b70 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe01a0b700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdf4011670 tx=0x7efdf4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdf40039c0 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efdf4011e50 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdfc106b20 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efdf401a5c0 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.916+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdfc107060 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.917+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7efdf401a870 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.917+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdfc063cb0 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.917+0000 7efdfaffd700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 0x7efde403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.917+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7efdf404d5d0 con 0x7efdfc108b70 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.918+0000 7efe0120a700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 0x7efde403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:03.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.918+0000 7efe0120a700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 0x7efde403b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efdf0006fd0 tx=0x7efdf0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:02.923+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efdf402cc80 con 0x7efdfc108b70 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.127+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7efdfc1037f0 con 0x7efde4039180 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.128+0000 7efdfaffd700 1 -- 172.21.15.28:0/1553870451 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7efdfc1037f0 con 0x7efde4039180 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 msgr2=0x7efde403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 0x7efde403b640 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7efdf0006fd0 tx=0x7efdf0006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 msgr2=0x7efdfc1083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efdf4011670 tx=0x7efdf4011b60 comp rx=0 tx=0).stop 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 shutdown_connections 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7efde4039180 0x7efde403b640 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 --2- 172.21.15.28:0/1553870451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc108b70 0x7efdfc1083f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:03.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 >> 172.21.15.28:0/1553870451 conn(0x7efdfc102580 msgr2=0x7efdfc1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:03.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 shutdown_connections 2024-03-22T02:04:03.632 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:03.130+0000 7efe03c6f700 1 -- 172.21.15.28:0/1553870451 wait complete. 2024-03-22T02:04:03.632 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-03-22T02:04:03.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:03 smithi028 ceph-mon[31916]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:05.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-mon[31916]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:05.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-mon[31916]: Generating ssh key... 2024-03-22T02:04:05.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:05.030 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:05.030 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: Generating public/private rsa key pair. 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: Your identification has been saved in /tmp/tmp2blsr5gq/key. 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: Your public key has been saved in /tmp/tmp2blsr5gq/key.pub. 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: The key fingerprint is: 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: SHA256:uQl4+nz77h29wkIgTTmD0/rqOBFPw6THpMKgc4rH5Xg ceph-426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: The key's randomart image is: 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: +---[RSA 3072]----+ 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: | o . | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: |. = * | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: |.o B = o | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: |o + =.O o. | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: |.= =.=o+S. | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: |o + Eo...o. . | 2024-03-22T02:04:05.031 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: | . ... .o. .. . | 2024-03-22T02:04:05.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: | .+. . ..o. . | 2024-03-22T02:04:05.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: | .o+..=+.... | 2024-03-22T02:04:05.032 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: +----[SHA256]-----+ 2024-03-22T02:04:05.297 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d8165c700 1 Processor -- start 2024-03-22T02:04:05.297 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d8165c700 1 -- start start 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d8165c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d8165c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c108b70 con 0x7f9d7c108180 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56120/0 (socket says 172.21.15.28:56120) 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d7affd700 1 -- 172.21.15.28:0/2144910341 learned_addr learned my addr 172.21.15.28:0/2144910341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d7affd700 1 -- 172.21.15.28:0/2144910341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c1092e0 con 0x7f9d7c108180 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d7affd700 1 --2- 172.21.15.28:0/2144910341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098a0 tx=0x7f9d70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5542b3b5473f3105 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.008+0000 7f9d79ffb700 1 -- 172.21.15.28:0/2144910341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d70004160 con 0x7f9d7c108180 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d79ffb700 1 -- 172.21.15.28:0/2144910341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9d70004300 con 0x7f9d7c108180 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d79ffb700 1 -- 172.21.15.28:0/2144910341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d70011b50 con 0x7f9d7c108180 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 -- 172.21.15.28:0/2144910341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 msgr2=0x7f9d7c1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:05.298 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2144910341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9d700098a0 tx=0x7f9d70009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 -- 172.21.15.28:0/2144910341 shutdown_connections 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2144910341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c1085a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 -- 172.21.15.28:0/2144910341 >> 172.21.15.28:0/2144910341 conn(0x7f9d7c103760 msgr2=0x7f9d7c105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 -- 172.21.15.28:0/2144910341 shutdown_connections 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.009+0000 7f9d8165c700 1 -- 172.21.15.28:0/2144910341 wait complete. 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d8165c700 1 Processor -- start 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d8165c700 1 -- start start 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d8165c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d8165c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7c10c690 con 0x7f9d7c108180 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d7affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56132/0 (socket says 172.21.15.28:56132) 2024-03-22T02:04:05.299 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.010+0000 7f9d7affd700 1 -- 172.21.15.28:0/2054462721 learned_addr learned my addr 172.21.15.28:0/2054462721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d7affd700 1 -- 172.21.15.28:0/2054462721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7c07e060 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d7affd700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009fb0 tx=0x7f9d70011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d700038b0 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9d70011e50 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7c07c250 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d70019550 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.011+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7c07c790 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f9d70019850 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d7c063cb0 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d63fff700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 0x7f9d6403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9d7004d410 con 0x7f9d7c108180 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d7a7fc700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 0x7f9d6403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.012+0000 7f9d7a7fc700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 0x7f9d6403b5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006fd0 tx=0x7f9d6c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:05.300 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.017+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d70016dc0 con 0x7f9d7c108180 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.221+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f9d7c1049d0 con 0x7f9d64039130 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.738+0000 7f9d63fff700 1 -- 172.21.15.28:0/2054462721 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9d7c1049d0 con 0x7f9d64039130 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 msgr2=0x7f9d6403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 0x7f9d6403b5f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006fd0 tx=0x7f9d6c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 msgr2=0x7f9d7c07db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009fb0 tx=0x7f9d70011bf0 comp rx=0 tx=0).stop 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 shutdown_connections 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9d64039130 0x7f9d6403b5f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 --2- 172.21.15.28:0/2054462721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d7c108180 0x7f9d7c07db20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 >> 172.21.15.28:0/2054462721 conn(0x7f9d7c103760 msgr2=0x7f9d7c1042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 shutdown_connections 2024-03-22T02:04:05.301 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:04.740+0000 7f9d8165c700 1 -- 172.21.15.28:0/2054462721 wait complete. 2024-03-22T02:04:06.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:04:06.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.676+0000 7f5fbbe95700 1 Processor -- start 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.676+0000 7f5fbbe95700 1 -- start start 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fbbe95700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fbbe95700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb407c7f0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb9c31700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb9c31700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56136/0 (socket says 172.21.15.28:56136) 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb9c31700 1 -- 172.21.15.28:0/3115928749 learned_addr learned my addr 172.21.15.28:0/3115928749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb9c31700 1 -- 172.21.15.28:0/3115928749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb407c970 con 0x7f5fb407ddb0 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb9c31700 1 --2- 172.21.15.28:0/3115928749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00098a0 tx=0x7f5fb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf2b085d87f0ce3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.677+0000 7f5fb8c2f700 1 -- 172.21.15.28:0/3115928749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0004160 con 0x7f5fb407ddb0 2024-03-22T02:04:06.404 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fb8c2f700 1 -- 172.21.15.28:0/3115928749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fb0004300 con 0x7f5fb407ddb0 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fb8c2f700 1 -- 172.21.15.28:0/3115928749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0011b50 con 0x7f5fb407ddb0 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 -- 172.21.15.28:0/3115928749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 msgr2=0x7f5fb407c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/3115928749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5fb00098a0 tx=0x7f5fb0009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 -- 172.21.15.28:0/3115928749 shutdown_connections 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/3115928749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb407c2b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 -- 172.21.15.28:0/3115928749 >> 172.21.15.28:0/3115928749 conn(0x7f5fb4103760 msgr2=0x7f5fb4105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:06.405 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 -- 172.21.15.28:0/3115928749 shutdown_connections 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.678+0000 7f5fbbe95700 1 -- 172.21.15.28:0/3115928749 wait complete. 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fbbe95700 1 Processor -- start 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fbbe95700 1 -- start start 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fbbe95700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fbbe95700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb410e750 con 0x7f5fb407ddb0 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fb9c31700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fb9c31700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56152/0 (socket says 172.21.15.28:56152) 2024-03-22T02:04:06.406 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.679+0000 7f5fb9c31700 1 -- 172.21.15.28:0/124759904 learned_addr learned my addr 172.21.15.28:0/124759904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5fb9c31700 1 -- 172.21.15.28:0/124759904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb419e860 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5fb9c31700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009fb0 tx=0x7f5fb0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb00038b0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5fb0011e50 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb419eaf0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5fb0019420 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.680+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb41a17e0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.681+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f5fb0019740 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.681+0000 7f5faaffd700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 0x7f5fa003b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.681+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5fb004d5f0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.681+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb4063cb0 con 0x7f5fb407ddb0 2024-03-22T02:04:06.407 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.681+0000 7f5fb9430700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 0x7f5fa003b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.682+0000 7f5fb9430700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 0x7f5fa003b5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.686+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5fb000bc70 con 0x7f5fb407ddb0 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.894+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f5fb4104a00 con 0x7f5fa0039130 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.895+0000 7f5faaffd700 1 -- 172.21.15.28:0/124759904 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f5fb4104a00 con 0x7f5fa0039130 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 msgr2=0x7f5fa003b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 0x7f5fa003b5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006fd0 tx=0x7f5fa4006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 msgr2=0x7f5fb419e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0009fb0 tx=0x7f5fb0011bf0 comp rx=0 tx=0).stop 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 shutdown_connections 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f5fa0039130 0x7f5fa003b5f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 --2- 172.21.15.28:0/124759904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb407ddb0 0x7f5fb419e320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 >> 172.21.15.28:0/124759904 conn(0x7f5fb4103760 msgr2=0x7f5fb41042f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:06.408 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 shutdown_connections 2024-03-22T02:04:06.409 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:05.897+0000 7f5fbbe95700 1 -- 172.21.15.28:0/124759904 wait complete. 2024-03-22T02:04:06.409 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-22T02:04:06.409 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-03-22T02:04:06.409 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-03-22T02:04:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:06 smithi028 ceph-mon[31916]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:07.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:07 smithi028 ceph-mon[31916]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:07.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:07 smithi028 ceph-mon[31916]: Deploying cephadm binary to smithi028 2024-03-22T02:04:11.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-03-22T02:04:11.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.818+0000 7fe10eed2700 1 Processor -- start 2024-03-22T02:04:11.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10eed2700 1 -- start start 2024-03-22T02:04:11.953 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10eed2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10eed2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe10807c7f0 con 0x7fe10807ddb0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10cc6e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10cc6e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56156/0 (socket says 172.21.15.28:56156) 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10cc6e700 1 -- 172.21.15.28:0/3648786759 learned_addr learned my addr 172.21.15.28:0/3648786759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10cc6e700 1 -- 172.21.15.28:0/3648786759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10807c970 con 0x7fe10807ddb0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.819+0000 7fe10cc6e700 1 --2- 172.21.15.28:0/3648786759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0098a0 tx=0x7fe0fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27e207d831850a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe1077fe700 1 -- 172.21.15.28:0/3648786759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0fc004160 con 0x7fe10807ddb0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe1077fe700 1 -- 172.21.15.28:0/3648786759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0fc004300 con 0x7fe10807ddb0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe1077fe700 1 -- 172.21.15.28:0/3648786759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0fc011b50 con 0x7fe10807ddb0 2024-03-22T02:04:11.954 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 -- 172.21.15.28:0/3648786759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 msgr2=0x7fe10807c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 --2- 172.21.15.28:0/3648786759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0098a0 tx=0x7fe0fc009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 -- 172.21.15.28:0/3648786759 shutdown_connections 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 --2- 172.21.15.28:0/3648786759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10807c2b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 -- 172.21.15.28:0/3648786759 >> 172.21.15.28:0/3648786759 conn(0x7fe108103790 msgr2=0x7fe108105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 -- 172.21.15.28:0/3648786759 shutdown_connections 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.820+0000 7fe10eed2700 1 -- 172.21.15.28:0/3648786759 wait complete. 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10eed2700 1 Processor -- start 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10eed2700 1 -- start start 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10eed2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:11.955 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10eed2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe10810e760 con 0x7fe10807ddb0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10cc6e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10cc6e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56164/0 (socket says 172.21.15.28:56164) 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10cc6e700 1 -- 172.21.15.28:0/2998620259 learned_addr learned my addr 172.21.15.28:0/2998620259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.821+0000 7fe10cc6e700 1 -- 172.21.15.28:0/2998620259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10819e7d0 con 0x7fe10807ddb0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe10cc6e700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc011b60 tx=0x7fe0fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0fc003b20 con 0x7fe10807ddb0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe0fc003cc0 con 0x7fe10807ddb0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10819ea60 con 0x7fe10807ddb0 2024-03-22T02:04:11.956 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe0fc024840 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.822+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1081a1750 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.823+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fe0fc022020 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.823+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe108063cb0 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.823+0000 7fe105ffb700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 0x7fe0f803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.823+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe0fc011e00 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.823+0000 7fe107fff700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 0x7fe0f803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.824+0000 7fe107fff700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 0x7fe0f803b690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4006fd0 tx=0x7fe0f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:06.828+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe0fc00ba30 con 0x7fe10807ddb0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:07.039+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7fe108104990 con 0x7fe0f80391d0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.369+0000 7fe105ffb700 1 -- 172.21.15.28:0/2998620259 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fe108104990 con 0x7fe0f80391d0 2024-03-22T02:04:11.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 msgr2=0x7fe0f803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 0x7fe0f803b690 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4006fd0 tx=0x7fe0f4006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 msgr2=0x7fe10819e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc011b60 tx=0x7fe0fc011c40 comp rx=0 tx=0).stop 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 shutdown_connections 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fe0f80391d0 0x7fe0f803b690 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 --2- 172.21.15.28:0/2998620259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe10807ddb0 0x7fe10819e290 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 >> 172.21.15.28:0/2998620259 conn(0x7fe108103790 msgr2=0x7fe108104280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 shutdown_connections 2024-03-22T02:04:11.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:11.372+0000 7fe10eed2700 1 -- 172.21.15.28:0/2998620259 wait complete. 2024-03-22T02:04:11.959 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-03-22T02:04:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:12 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:12 smithi028 ceph-mon[31916]: Added host smithi028 2024-03-22T02:04:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:12 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:13.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-22T02:04:13.277 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.350+0000 7f9c8071d700 1 Processor -- start 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.350+0000 7f9c8071d700 1 -- start start 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.350+0000 7f9c8071d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.350+0000 7f9c8071d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c78108aa0 con 0x7f9c78108140 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.350+0000 7f9c7e4b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7e4b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43456/0 (socket says 172.21.15.28:43456) 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7e4b9700 1 -- 172.21.15.28:0/2143374024 learned_addr learned my addr 172.21.15.28:0/2143374024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7e4b9700 1 -- 172.21.15.28:0/2143374024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c78108c20 con 0x7f9c78108140 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7e4b9700 1 --2- 172.21.15.28:0/2143374024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9c740098a0 tx=0x7f9c74009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6563e98b6f659512 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7d4b7700 1 -- 172.21.15.28:0/2143374024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74004160 con 0x7f9c78108140 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7d4b7700 1 -- 172.21.15.28:0/2143374024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9c74004300 con 0x7f9c78108140 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.351+0000 7f9c7d4b7700 1 -- 172.21.15.28:0/2143374024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74011b50 con 0x7f9c78108140 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- 172.21.15.28:0/2143374024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 msgr2=0x7f9c78108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:13.278 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 --2- 172.21.15.28:0/2143374024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9c740098a0 tx=0x7f9c74009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- 172.21.15.28:0/2143374024 shutdown_connections 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 --2- 172.21.15.28:0/2143374024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c78108560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- 172.21.15.28:0/2143374024 >> 172.21.15.28:0/2143374024 conn(0x7f9c78103780 msgr2=0x7f9c78105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- 172.21.15.28:0/2143374024 shutdown_connections 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- 172.21.15.28:0/2143374024 wait complete. 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 Processor -- start 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.352+0000 7f9c8071d700 1 -- start start 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c8071d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c8071d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c7810c5b0 con 0x7f9c78108140 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c7e4b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c7e4b9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43470/0 (socket says 172.21.15.28:43470) 2024-03-22T02:04:13.279 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c7e4b9700 1 -- 172.21.15.28:0/1026012362 learned_addr learned my addr 172.21.15.28:0/1026012362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c7e4b9700 1 -- 172.21.15.28:0/1026012362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c7819c5b0 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c7e4b9700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c74011b60 tx=0x7f9c74011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74003b20 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9c74003cc0 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9c74024840 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c7819c840 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.353+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c7819f530 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.354+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f9c74022020 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.354+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c78063cb0 con 0x7f9c78108140 2024-03-22T02:04:13.280 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.354+0000 7f9c6f7fe700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 0x7f9c6403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.354+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9c74011e00 con 0x7f9c78108140 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.355+0000 7f9c7dcb8700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 0x7f9c6403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.355+0000 7f9c7dcb8700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 0x7f9c6403b640 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006fd0 tx=0x7f9c68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.360+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c7401a430 con 0x7f9c78108140 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.602+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9c781049d0 con 0x7f9c64039180 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.607+0000 7f9c6f7fe700 1 -- 172.21.15.28:0/1026012362 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9c781049d0 con 0x7f9c64039180 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 msgr2=0x7f9c6403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 0x7f9c6403b640 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9c68006fd0 tx=0x7f9c68006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 msgr2=0x7f9c7819c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9c74011b60 tx=0x7f9c74011c40 comp rx=0 tx=0).stop 2024-03-22T02:04:13.281 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 shutdown_connections 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f9c64039180 0x7f9c6403b640 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 --2- 172.21.15.28:0/1026012362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c78108140 0x7f9c7819c070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 >> 172.21.15.28:0/1026012362 conn(0x7f9c78103780 msgr2=0x7f9c781042c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 shutdown_connections 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:12.609+0000 7f9c8071d700 1 -- 172.21.15.28:0/1026012362 wait complete. 2024-03-22T02:04:13.282 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-03-22T02:04:13.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:13 smithi028 ceph-mon[31916]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:13.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:13 smithi028 ceph-mon[31916]: Saving service mon spec with placement count:5 2024-03-22T02:04:13.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:13 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb9b1ad700 1 Processor -- start 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb9b1ad700 1 -- start start 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb9b1ad700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb9b1ad700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb94109070 con 0x7fdb94108680 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb98f49700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb98f49700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43480/0 (socket says 172.21.15.28:43480) 2024-03-22T02:04:14.544 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb98f49700 1 -- 172.21.15.28:0/2242042242 learned_addr learned my addr 172.21.15.28:0/2242042242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb98f49700 1 -- 172.21.15.28:0/2242042242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb941072e0 con 0x7fdb94108680 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.698+0000 7fdb98f49700 1 --2- 172.21.15.28:0/2242042242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800bb40 tx=0x7fdb8800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf09833716f1a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb937fe700 1 -- 172.21.15.28:0/2242042242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb8800dbe0 con 0x7fdb94108680 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb937fe700 1 -- 172.21.15.28:0/2242042242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb880044f0 con 0x7fdb94108680 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb937fe700 1 -- 172.21.15.28:0/2242042242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb88010470 con 0x7fdb94108680 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2242042242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 msgr2=0x7fdb94108aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2242042242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800bb40 tx=0x7fdb8800be50 comp rx=0 tx=0).stop 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2242042242 shutdown_connections 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2242042242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb94108aa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2242042242 >> 172.21.15.28:0/2242042242 conn(0x7fdb94101df0 msgr2=0x7fdb94104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2242042242 shutdown_connections 2024-03-22T02:04:14.545 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.699+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2242042242 wait complete. 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb9b1ad700 1 Processor -- start 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb9b1ad700 1 -- start start 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb9b1ad700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb9b1ad700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb941061d0 con 0x7fdb94108680 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb98f49700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb98f49700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43484/0 (socket says 172.21.15.28:43484) 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb98f49700 1 -- 172.21.15.28:0/2933874304 learned_addr learned my addr 172.21.15.28:0/2933874304 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.700+0000 7fdb98f49700 1 -- 172.21.15.28:0/2933874304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9419e840 con 0x7fdb94108680 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb98f49700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800b340 tx=0x7fdb880040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb88010720 con 0x7fdb94108680 2024-03-22T02:04:14.546 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb9419ead0 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdb8801d070 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb88021c20 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.701+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb941a17c0 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.702+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fdb8801f030 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.702+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb94063cb0 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.702+0000 7fdb91ffb700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 0x7fdb8403b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.702+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdb8804d650 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.702+0000 7fdb93fff700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 0x7fdb8403b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.703+0000 7fdb93fff700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 0x7fdb8403b5f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb80006fd0 tx=0x7fdb80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.707+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb88015070 con 0x7fdb94108680 2024-03-22T02:04:14.547 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.937+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fdb94062ac0 con 0x7fdb84039130 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.941+0000 7fdb91ffb700 1 -- 172.21.15.28:0/2933874304 <== mgr.14120 v2:172.21.15.28:6800/3429985942 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fdb94062ac0 con 0x7fdb84039130 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.943+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 msgr2=0x7fdb8403b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.943+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 0x7fdb8403b5f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdb80006fd0 tx=0x7fdb80006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.943+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 msgr2=0x7fdb9419e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.943+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800b340 tx=0x7fdb880040b0 comp rx=0 tx=0).stop 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 shutdown_connections 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fdb84039130 0x7fdb8403b5f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 --2- 172.21.15.28:0/2933874304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb94108680 0x7fdb9419e300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 >> 172.21.15.28:0/2933874304 conn(0x7fdb94101df0 msgr2=0x7fdb94190510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 shutdown_connections 2024-03-22T02:04:14.548 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:13.944+0000 7fdb9b1ad700 1 -- 172.21.15.28:0/2933874304 wait complete. 2024-03-22T02:04:15.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:14 smithi028 ceph-mon[31916]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:15.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:14 smithi028 ceph-mon[31916]: Saving service mgr spec with placement count:2 2024-03-22T02:04:15.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:14 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:16.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:16 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:16.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3684563095' entity='client.admin' 2024-03-22T02:04:16.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:16 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:16.637 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.824+0000 7fd7c77e8700 1 Processor -- start 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.824+0000 7fd7c77e8700 1 -- start start 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.824+0000 7fd7c77e8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.824+0000 7fd7c77e8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c007c7d0 con 0x7fd7c007dd90 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7c5584700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7c5584700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43496/0 (socket says 172.21.15.28:43496) 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7c5584700 1 -- 172.21.15.28:0/1896648156 learned_addr learned my addr 172.21.15.28:0/1896648156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7c5584700 1 -- 172.21.15.28:0/1896648156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c007c950 con 0x7fd7c007dd90 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7c5584700 1 --2- 172.21.15.28:0/1896648156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00098a0 tx=0x7fd7b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864017011a4e7e8a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:16.638 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7b7fff700 1 -- 172.21.15.28:0/1896648156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b0004160 con 0x7fd7c007dd90 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7b7fff700 1 -- 172.21.15.28:0/1896648156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b0004300 con 0x7fd7c007dd90 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.825+0000 7fd7b7fff700 1 -- 172.21.15.28:0/1896648156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b0011b20 con 0x7fd7c007dd90 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 -- 172.21.15.28:0/1896648156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 msgr2=0x7fd7c007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/1896648156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00098a0 tx=0x7fd7b0009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 -- 172.21.15.28:0/1896648156 shutdown_connections 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/1896648156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c007c290 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 -- 172.21.15.28:0/1896648156 >> 172.21.15.28:0/1896648156 conn(0x7fd7c0103720 msgr2=0x7fd7c0105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 -- 172.21.15.28:0/1896648156 shutdown_connections 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.826+0000 7fd7c77e8700 1 -- 172.21.15.28:0/1896648156 wait complete. 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c77e8700 1 Processor -- start 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c77e8700 1 -- start start 2024-03-22T02:04:16.639 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c77e8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c77e8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c010e6e0 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c5584700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c5584700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43510/0 (socket says 172.21.15.28:43510) 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c5584700 1 -- 172.21.15.28:0/3684563095 learned_addr learned my addr 172.21.15.28:0/3684563095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c5584700 1 -- 172.21.15.28:0/3684563095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c019e720 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.827+0000 7fd7c5584700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00117e0 tx=0x7fd7b0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.828+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b0003ba0 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.828+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b001a910 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.828+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c019e9b0 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.828+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7b00248f0 con 0x7fd7c007dd90 2024-03-22T02:04:16.640 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.828+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c019ee10 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c0063cb0 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7fd7b0022020 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7b67fc700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 0x7fd7ac03b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd7b004dad0 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7c4d83700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 0x7fd7ac03b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.829+0000 7fd7c4d83700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 0x7fd7ac03b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006fd0 tx=0x7fd7bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:15.838+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7b002d910 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.042+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd7c01a16a0 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.045+0000 7fd7b67fc700 1 -- 172.21.15.28:0/3684563095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fd7b002ce40 con 0x7fd7c007dd90 2024-03-22T02:04:16.641 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 msgr2=0x7fd7ac03b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 0x7fd7ac03b640 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc006fd0 tx=0x7fd7bc006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 msgr2=0x7fd7c019e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd7b00117e0 tx=0x7fd7b0011ba0 comp rx=0 tx=0).stop 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 shutdown_connections 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7fd7ac039180 0x7fd7ac03b640 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 --2- 172.21.15.28:0/3684563095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7c007dd90 0x7fd7c019e1e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 >> 172.21.15.28:0/3684563095 conn(0x7fd7c0103720 msgr2=0x7fd7c0104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 shutdown_connections 2024-03-22T02:04:16.642 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:16.048+0000 7fd7c77e8700 1 -- 172.21.15.28:0/3684563095 wait complete. 2024-03-22T02:04:18.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1172925449' entity='client.admin' 2024-03-22T02:04:18.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:18 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:18.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.168+0000 7f6f61447700 1 Processor -- start 2024-03-22T02:04:18.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.168+0000 7f6f61447700 1 -- start start 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.168+0000 7f6f61447700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.168+0000 7f6f61447700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c07be20 con 0x7f6f5c108be0 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f5affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f5affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43526/0 (socket says 172.21.15.28:43526) 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f5affd700 1 -- 172.21.15.28:0/2700709495 learned_addr learned my addr 172.21.15.28:0/2700709495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:18.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f5affd700 1 -- 172.21.15.28:0/2700709495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c10b510 con 0x7f6f5c108be0 2024-03-22T02:04:18.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f5affd700 1 --2- 172.21.15.28:0/2700709495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098a0 tx=0x7f6f54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2ad4f2ad5aab56 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:18.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f59ffb700 1 -- 172.21.15.28:0/2700709495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f54004160 con 0x7f6f5c108be0 2024-03-22T02:04:18.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f59ffb700 1 -- 172.21.15.28:0/2700709495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f54004300 con 0x7f6f5c108be0 2024-03-22T02:04:18.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.169+0000 7f6f59ffb700 1 -- 172.21.15.28:0/2700709495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f54011b20 con 0x7f6f5c108be0 2024-03-22T02:04:18.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 -- 172.21.15.28:0/2700709495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 msgr2=0x7f6f5c10afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 --2- 172.21.15.28:0/2700709495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6f540098a0 tx=0x7f6f54009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 -- 172.21.15.28:0/2700709495 shutdown_connections 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 --2- 172.21.15.28:0/2700709495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c10afd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 -- 172.21.15.28:0/2700709495 >> 172.21.15.28:0/2700709495 conn(0x7f6f5c1025d0 msgr2=0x7f6f5c104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 -- 172.21.15.28:0/2700709495 shutdown_connections 2024-03-22T02:04:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.170+0000 7f6f61447700 1 -- 172.21.15.28:0/2700709495 wait complete. 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f61447700 1 Processor -- start 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f61447700 1 -- start start 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f61447700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f61447700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5c107380 con 0x7f6f5c108be0 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f5affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:18.597 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f5affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43534/0 (socket says 172.21.15.28:43534) 2024-03-22T02:04:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f5affd700 1 -- 172.21.15.28:0/1172925449 learned_addr learned my addr 172.21.15.28:0/1172925449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.171+0000 7f6f5affd700 1 -- 172.21.15.28:0/1172925449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5c19a380 con 0x7f6f5c108be0 2024-03-22T02:04:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f5affd700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6f540117e0 tx=0x7f6f54011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f54003bf0 con 0x7f6f5c108be0 2024-03-22T02:04:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f5401a910 con 0x7f6f5c108be0 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f5c19a610 con 0x7f6f5c108be0 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f54024970 con 0x7f6f5c108be0 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.172+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5c19aa70 con 0x7f6f5c108be0 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f6f54022020 con 0x7f6f5c108be0 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f53fff700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 0x7f6f4803fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:18.599 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f5c063cb0 con 0x7f6f5c108be0 2024-03-22T02:04:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6f5404d4d0 con 0x7f6f5c108be0 2024-03-22T02:04:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f5a7fc700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 0x7f6f4803fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.173+0000 7f6f5a7fc700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 0x7f6f4803fa00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6f44006fd0 tx=0x7f6f44006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.178+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f54011e20 con 0x7f6f5c108be0 2024-03-22T02:04:18.600 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.395+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f6f5c063290 con 0x7f6f5c108be0 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.397+0000 7f6f53fff700 1 -- 172.21.15.28:0/1172925449 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f6f5400b3e0 con 0x7f6f5c108be0 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.399+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 msgr2=0x7f6f4803fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.399+0000 7f6f61447700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 0x7f6f4803fa00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6f44006fd0 tx=0x7f6f44006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.399+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 msgr2=0x7f6f5c199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.399+0000 7f6f61447700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6f540117e0 tx=0x7f6f54011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:18.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 shutdown_connections 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f6f4803d540 0x7f6f4803fa00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 --2- 172.21.15.28:0/1172925449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f5c108be0 0x7f6f5c199e40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 >> 172.21.15.28:0/1172925449 conn(0x7f6f5c1025d0 msgr2=0x7f6f5c1908d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 shutdown_connections 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:17.400+0000 7f6f61447700 1 -- 172.21.15.28:0/1172925449 wait complete. 2024-03-22T02:04:18.602 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-03-22T02:04:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:19 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3601769264' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-22T02:04:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:19 smithi028 ceph-mon[31916]: from='mgr.14120 172.21.15.28:0/2428654770' entity='mgr.x' 2024-03-22T02:04:19.572 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: ignoring --setuser ceph since I am not root 2024-03-22T02:04:19.572 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: ignoring --setgroup ceph since I am not root 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.014+0000 7f43176a8700 1 Processor -- start 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.014+0000 7f43176a8700 1 -- start start 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.014+0000 7f43176a8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.014+0000 7f43176a8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4310107a20 con 0x7f4310107030 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4315444700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4315444700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43542/0 (socket says 172.21.15.28:43542) 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4315444700 1 -- 172.21.15.28:0/3549481502 learned_addr learned my addr 172.21.15.28:0/3549481502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4315444700 1 -- 172.21.15.28:0/3549481502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4310108230 con 0x7f4310107030 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4315444700 1 --2- 172.21.15.28:0/3549481502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43000098a0 tx=0x7f4300009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846d639f419b7c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:19.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4307fff700 1 -- 172.21.15.28:0/3549481502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300004160 con 0x7f4310107030 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4307fff700 1 -- 172.21.15.28:0/3549481502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4300004300 con 0x7f4310107030 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.015+0000 7f4307fff700 1 -- 172.21.15.28:0/3549481502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300011b20 con 0x7f4310107030 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 -- 172.21.15.28:0/3549481502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 msgr2=0x7f4310107450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 --2- 172.21.15.28:0/3549481502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f43000098a0 tx=0x7f4300009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 -- 172.21.15.28:0/3549481502 shutdown_connections 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 --2- 172.21.15.28:0/3549481502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f4310107450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 -- 172.21.15.28:0/3549481502 >> 172.21.15.28:0/3549481502 conn(0x7f43101025d0 msgr2=0x7f4310104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 -- 172.21.15.28:0/3549481502 shutdown_connections 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.016+0000 7f43176a8700 1 -- 172.21.15.28:0/3549481502 wait complete. 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f43176a8700 1 Processor -- start 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f43176a8700 1 -- start start 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f43176a8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:19.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f43176a8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f431010b5e0 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f4315444700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f4315444700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43552/0 (socket says 172.21.15.28:43552) 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f4315444700 1 -- 172.21.15.28:0/3601769264 learned_addr learned my addr 172.21.15.28:0/3601769264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.017+0000 7f4315444700 1 -- 172.21.15.28:0/3601769264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431019edf0 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f4315444700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43000117e0 tx=0x7f4300011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300003bf0 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f430001a910 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431019efc0 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4300024970 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.018+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431019f360 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46294+0+0 (secure 0 0 0) 0x7f4300022020 con 0x7f4310107030 2024-03-22T02:04:19.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f43067fc700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 0x7f42fc03b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f430004d5a0 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4310063cb0 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f4314c43700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 0x7f42fc03b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.019+0000 7f4314c43700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 0x7f42fc03b690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f430c006fd0 tx=0x7f430c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.024+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43000172b0 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.292+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f431019f590 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.399+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f431019f590 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.401+0000 7f43067fc700 1 -- 172.21.15.28:0/3601769264 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7f430002dc90 con 0x7f4310107030 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.402+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 msgr2=0x7f42fc03b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.402+0000 7f43176a8700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 0x7f42fc03b690 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f430c006fd0 tx=0x7f430c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.402+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 msgr2=0x7f431007ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:19.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.402+0000 7f43176a8700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43000117e0 tx=0x7f4300011ba0 comp rx=0 tx=0).stop 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 shutdown_connections 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f42fc0391d0 0x7f42fc03b690 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 --2- 172.21.15.28:0/3601769264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4310107030 0x7f431007ba00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 >> 172.21.15.28:0/3601769264 conn(0x7f43101025d0 msgr2=0x7f4310103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 shutdown_connections 2024-03-22T02:04:20.000 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:19.403+0000 7f43176a8700 1 -- 172.21.15.28:0/3601769264 wait complete. 2024-03-22T02:04:20.241 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:19.988+0000 7f0d839d4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T02:04:20.241 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:20.095+0000 7f0d839d4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T02:04:20.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:20 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3601769264' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-22T02:04:20.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:20 smithi028 ceph-mon[31916]: mgrmap e9: x(active, since 19s) 2024-03-22T02:04:20.821 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:20.773+0000 7f0d839d4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T02:04:21.197 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:20.870+0000 7f0d839d4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T02:04:21.197 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.014+0000 7f0d839d4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T02:04:21.197 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.111+0000 7f0d839d4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T02:04:21.197 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.197+0000 7f0d839d4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T02:04:21.468 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2664214732' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T02:04:21.767 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.468+0000 7f0d839d4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T02:04:21.768 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.581+0000 7f0d839d4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T02:04:21.768 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.670+0000 7f0d839d4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T02:04:21.768 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.767+0000 7f0d839d4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T02:04:22.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:21.949+0000 7f0d839d4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T02:04:22.175 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.449+0000 7f1ec2b05700 1 Processor -- start 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec2b05700 1 -- start start 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec2b05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec2b05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ebc07be20 con 0x7f1ebc108c00 2024-03-22T02:04:22.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec08a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec08a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43570/0 (socket says 172.21.15.28:43570) 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.450+0000 7f1ec08a1700 1 -- 172.21.15.28:0/1890561355 learned_addr learned my addr 172.21.15.28:0/1890561355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.451+0000 7f1ec08a1700 1 -- 172.21.15.28:0/1890561355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc10b530 con 0x7f1ebc108c00 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.451+0000 7f1ec08a1700 1 --2- 172.21.15.28:0/1890561355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098a0 tx=0x7f1eb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b814bc65d273b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.451+0000 7f1ebb7fe700 1 -- 172.21.15.28:0/1890561355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0004160 con 0x7f1ebc108c00 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.451+0000 7f1ebb7fe700 1 -- 172.21.15.28:0/1890561355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb0004300 con 0x7f1ebc108c00 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.451+0000 7f1ebb7fe700 1 -- 172.21.15.28:0/1890561355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0011b20 con 0x7f1ebc108c00 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 -- 172.21.15.28:0/1890561355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 msgr2=0x7f1ebc10aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/1890561355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098a0 tx=0x7f1eb0009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 -- 172.21.15.28:0/1890561355 shutdown_connections 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/1890561355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc10aff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 -- 172.21.15.28:0/1890561355 >> 172.21.15.28:0/1890561355 conn(0x7f1ebc1025d0 msgr2=0x7f1ebc104a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:22.177 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 -- 172.21.15.28:0/1890561355 shutdown_connections 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.452+0000 7f1ec2b05700 1 -- 172.21.15.28:0/1890561355 wait complete. 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec2b05700 1 Processor -- start 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec2b05700 1 -- start start 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec2b05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec2b05700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ebc10c590 con 0x7f1ebc108c00 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec08a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec08a1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43580/0 (socket says 172.21.15.28:43580) 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.453+0000 7f1ec08a1700 1 -- 172.21.15.28:0/2664214732 learned_addr learned my addr 172.21.15.28:0/2664214732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1ec08a1700 1 -- 172.21.15.28:0/2664214732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ebc19c590 con 0x7f1ebc108c00 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1ec08a1700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00117e0 tx=0x7f1eb0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:22.178 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0003bf0 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ebc19c820 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eb001a910 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0024970 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.454+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ebc19cc80 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.455+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7f1eb0022020 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.456+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ebc063cb0 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.456+0000 7f1eb9ffb700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 0x7f1eac03b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.456+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1eb004d5a0 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.456+0000 7f1ebbfff700 1 -- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 msgr2=0x7f1eac03b6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.456+0000 7f1ebbfff700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 0x7f1eac03b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.461+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1eb0011e20 con 0x7f1ebc108c00 2024-03-22T02:04:22.179 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.657+0000 7f1ebbfff700 1 -- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 msgr2=0x7f1eac03b6a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.657+0000 7f1ebbfff700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 0x7f1eac03b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.732+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f1ebc19f630 con 0x7f1ebc108c00 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.733+0000 7f1eb9ffb700 1 -- 172.21.15.28:0/2664214732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f1eb000b3e0 con 0x7f1ebc108c00 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 msgr2=0x7f1eac03b6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 0x7f1eac03b6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 msgr2=0x7f1ebc19c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00117e0 tx=0x7f1eb0011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 shutdown_connections 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f1eac0391e0 0x7f1eac03b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 --2- 172.21.15.28:0/2664214732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ebc108c00 0x7f1ebc19c050 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 >> 172.21.15.28:0/2664214732 conn(0x7f1ebc1025d0 msgr2=0x7f1ebc103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 shutdown_connections 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:20.735+0000 7f1ec2b05700 1 -- 172.21.15.28:0/2664214732 wait complete. 2024-03-22T02:04:22.180 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-03-22T02:04:22.181 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-03-22T02:04:23.071 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:22 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:22.948+0000 7f0d839d4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T02:04:23.422 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:23.082+0000 7f0d839d4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T02:04:23.422 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:23.421+0000 7f0d839d4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T02:04:23.716 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:23.513+0000 7f0d839d4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T02:04:23.716 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:23.715+0000 7f0d839d4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:04:23.998 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:23.906+0000 7f0d839d4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T02:04:24.321 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:24.200+0000 7f0d839d4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:04:24.757 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:24.461+0000 7f0d839d4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T02:04:24.758 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:24.706+0000 7f0d839d4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T02:04:25.599 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:25.546+0000 7f0d839d4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T02:04:26.454 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:26.129+0000 7f0d839d4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T02:04:26.454 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:26.230+0000 7f0d839d4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T02:04:26.454 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:04:26.452+0000 7f0d839d4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T02:04:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: Active manager daemon x restarted 2024-03-22T02:04:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: Activating manager daemon x 2024-03-22T02:04:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: osdmap e3: 0 total, 0 up, 0 in 2024-03-22T02:04:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: mgrmap e10: x(active, starting, since 0.00325168s) 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: Manager daemon x is now available 2024-03-22T02:04:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T02:04:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T02:04:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:27 smithi028 ceph-mon[31916]: mgrmap e11: x(active, since 1.00607s) 2024-03-22T02:04:28.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-03-22T02:04:28.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-22T02:04:28.094 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.589+0000 7f0d9577c700 1 Processor -- start 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.589+0000 7f0d9577c700 1 -- start start 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.589+0000 7f0d9577c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.589+0000 7f0d9577c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d90107a00 con 0x7f0d90107010 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.589+0000 7f0d8effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36744/0 (socket says 172.21.15.28:36744) 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8effd700 1 -- 172.21.15.28:0/2620008780 learned_addr learned my addr 172.21.15.28:0/2620008780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8effd700 1 -- 172.21.15.28:0/2620008780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d90108210 con 0x7f0d90107010 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8effd700 1 --2- 172.21.15.28:0/2620008780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d780098a0 tx=0x7f0d78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedea10c58c76836 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8dffb700 1 -- 172.21.15.28:0/2620008780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d78004160 con 0x7f0d90107010 2024-03-22T02:04:28.095 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8dffb700 1 -- 172.21.15.28:0/2620008780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d78004300 con 0x7f0d90107010 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.590+0000 7f0d8dffb700 1 -- 172.21.15.28:0/2620008780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d78011b20 con 0x7f0d90107010 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 -- 172.21.15.28:0/2620008780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 msgr2=0x7f0d90107430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2620008780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d780098a0 tx=0x7f0d78009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 -- 172.21.15.28:0/2620008780 shutdown_connections 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2620008780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d90107430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 -- 172.21.15.28:0/2620008780 >> 172.21.15.28:0/2620008780 conn(0x7f0d901025d0 msgr2=0x7f0d901049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 -- 172.21.15.28:0/2620008780 shutdown_connections 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 -- 172.21.15.28:0/2620008780 wait complete. 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.591+0000 7f0d9577c700 1 Processor -- start 2024-03-22T02:04:28.096 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d9577c700 1 -- start start 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d9577c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d9577c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9007af80 con 0x7f0d90107010 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d8effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d8effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36748/0 (socket says 172.21.15.28:36748) 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d8effd700 1 -- 172.21.15.28:0/2215421654 learned_addr learned my addr 172.21.15.28:0/2215421654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d8effd700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9019e7a0 con 0x7f0d90107010 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d8effd700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d780117e0 tx=0x7f0d78011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.592+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d78003bf0 con 0x7f0d90107010 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.593+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9019ea30 con 0x7f0d90107010 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.593+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d7801a910 con 0x7f0d90107010 2024-03-22T02:04:28.097 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.593+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0d780249a0 con 0x7f0d90107010 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.593+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9019ee90 con 0x7f0d90107010 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.593+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46307+0+0 (secure 0 0 0) 0x7f0d78022020 con 0x7f0d90107010 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.594+0000 7f0d87fff700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.594+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0d7c03bd50 con 0x7f0d7c039180 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.594+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0d7804d520 con 0x7f0d90107010 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.594+0000 7f0d8e7fc700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:28.098 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.594+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.795+0000 7f0d8e7fc700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:22.795+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:23.196+0000 7f0d8e7fc700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:23.196+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:23.997+0000 7f0d8e7fc700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:23.997+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:25.598+0000 7f0d8e7fc700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3429985942 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:25.598+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:26.468+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46074+0+0 (secure 0 0 0) 0x7f0d7802dad0 con 0x7f0d90107010 2024-03-22T02:04:28.099 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:26.468+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 msgr2=0x7f0d7c03b640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:26.468+0000 7f0d87fff700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.469+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46201+0+0 (secure 0 0 0) 0x7f0d7804f1d0 con 0x7f0d90107010 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.470+0000 7f0d87fff700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 0x7f0d7c03eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.470+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0d7800be90 con 0x7f0d7c03cc00 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.470+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 0x7f0d7c03eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.471+0000 7f0d8e7fc700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 0x7f0d7c03eff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d80003a10 tx=0x7f0d800092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.471+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f0d7800be90 con 0x7f0d7c03cc00 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.484+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f0d90063cb0 con 0x7f0d7c03cc00 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d87fff700 1 -- 172.21.15.28:0/2215421654 <== mgr.14152 v2:172.21.15.28:6800/4205492780 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f0d90063cb0 con 0x7f0d7c03cc00 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 msgr2=0x7f0d7c03eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:28.100 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 0x7f0d7c03eff0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d80003a10 tx=0x7f0d800092b0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 msgr2=0x7f0d9019e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0d780117e0 tx=0x7f0d78011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 shutdown_connections 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d7c03cc00 0x7f0d7c03eff0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d90107010 0x7f0d9019e260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 --2- 172.21.15.28:0/2215421654 >> [v2:172.21.15.28:6800/3429985942,v1:172.21.15.28:6801/3429985942] conn(0x7f0d7c039180 0x7f0d7c03b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 >> 172.21.15.28:0/2215421654 conn(0x7f0d901025d0 msgr2=0x7f0d901948a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 shutdown_connections 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:27.485+0000 7f0d9577c700 1 -- 172.21.15.28:0/2215421654 wait complete. 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-03-22T02:04:28.101 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-03-22T02:04:28.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:28 smithi028 ceph-mon[31916]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T02:04:28.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:28 smithi028 ceph-mon[31916]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T02:04:29.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: mgrmap e12: x(active, since 2s) 2024-03-22T02:04:29.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:29.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:29.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:28] ENGINE Bus STARTING 2024-03-22T02:04:29.524 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:29] ENGINE Serving on https://172.21.15.28:7150 2024-03-22T02:04:29.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:29] ENGINE Serving on http://172.21.15.28:8765 2024-03-22T02:04:29.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: [22/Mar/2024:02:04:29] ENGINE Bus STARTED 2024-03-22T02:04:29.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:29.531 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.500+0000 7fcf19c5d700 1 Processor -- start 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.500+0000 7fcf19c5d700 1 -- start start 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf19c5d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf19c5d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1407be20 con 0x7fcf14108bc0 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf137fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf137fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36836/0 (socket says 172.21.15.28:36836) 2024-03-22T02:04:29.532 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf137fe700 1 -- 172.21.15.28:0/3781834495 learned_addr learned my addr 172.21.15.28:0/3781834495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf137fe700 1 -- 172.21.15.28:0/3781834495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1410b4f0 con 0x7fcf14108bc0 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.501+0000 7fcf137fe700 1 --2- 172.21.15.28:0/3781834495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009e80 tx=0x7fcf0400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5acc7cf86310ad11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf127fc700 1 -- 172.21.15.28:0/3781834495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf04017070 con 0x7fcf14108bc0 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf127fc700 1 -- 172.21.15.28:0/3781834495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf0400b8c0 con 0x7fcf14108bc0 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf127fc700 1 -- 172.21.15.28:0/3781834495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf040045c0 con 0x7fcf14108bc0 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 -- 172.21.15.28:0/3781834495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 msgr2=0x7fcf1410afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/3781834495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcf04009e80 tx=0x7fcf0400b330 comp rx=0 tx=0).stop 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 -- 172.21.15.28:0/3781834495 shutdown_connections 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/3781834495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1410afb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 -- 172.21.15.28:0/3781834495 >> 172.21.15.28:0/3781834495 conn(0x7fcf141025d0 msgr2=0x7fcf141049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 -- 172.21.15.28:0/3781834495 shutdown_connections 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.502+0000 7fcf19c5d700 1 -- 172.21.15.28:0/3781834495 wait complete. 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.503+0000 7fcf19c5d700 1 Processor -- start 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.503+0000 7fcf19c5d700 1 -- start start 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf19c5d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf19c5d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf1410c550 con 0x7fcf14108bc0 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf137fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf137fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36846/0 (socket says 172.21.15.28:36846) 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf137fe700 1 -- 172.21.15.28:0/1227405313 learned_addr learned my addr 172.21.15.28:0/1227405313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf137fe700 1 -- 172.21.15.28:0/1227405313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1419c590 con 0x7fcf14108bc0 2024-03-22T02:04:29.534 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf137fe700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400b7b0 tx=0x7fcf0400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf04020020 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf04003d90 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1419c820 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcf04017040 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.504+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf1419f510 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.505+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 11) ==== 46201+0+0 (secure 0 0 0) 0x7fcf04023020 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.505+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf14063cb0 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.505+0000 7fcf10ff9700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 0x7fcf0003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.506+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcf0400f050 con 0x7fcf14108bc0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.506+0000 7fcf12ffd700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 0x7fcf0003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:29.535 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.506+0000 7fcf12ffd700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 0x7fcf0003b530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcefc006fd0 tx=0x7fcefc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.511+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcf0402a020 con 0x7fcf14108bc0 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.520+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7fcf04011c90 con 0x7fcf14108bc0 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.725+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fcf14103750 con 0x7fcf00039070 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.782+0000 7fcf10ff9700 1 -- 172.21.15.28:0/1227405313 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fcf14103750 con 0x7fcf00039070 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 msgr2=0x7fcf0003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 0x7fcf0003b530 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcefc006fd0 tx=0x7fcefc006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 msgr2=0x7fcf1419c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400b7b0 tx=0x7fcf0400bd50 comp rx=0 tx=0).stop 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 shutdown_connections 2024-03-22T02:04:29.536 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf00039070 0x7fcf0003b530 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 --2- 172.21.15.28:0/1227405313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf14108bc0 0x7fcf1419c050 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 >> 172.21.15.28:0/1227405313 conn(0x7fcf141025d0 msgr2=0x7fcf14103040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 shutdown_connections 2024-03-22T02:04:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:28.784+0000 7fcf19c5d700 1 -- 172.21.15.28:0/1227405313 wait complete. 2024-03-22T02:04:29.537 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-03-22T02:04:31.245 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$2Mx.XZmBC6lGRyPqCATzfO/SqeeL.oJ2Cef7hSozuE/.4kEsgqG2K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711073070, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0342d38700 1 Processor -- start 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0342d38700 1 -- start start 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0342d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0342d38700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f033c106e70 con 0x7f033c106480 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0340ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0340ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36860/0 (socket says 172.21.15.28:36860) 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0340ad4700 1 -- 172.21.15.28:0/3746464924 learned_addr learned my addr 172.21.15.28:0/3746464924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.951+0000 7f0340ad4700 1 -- 172.21.15.28:0/3746464924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033c109420 con 0x7f033c106480 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.952+0000 7f0340ad4700 1 --2- 172.21.15.28:0/3746464924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098a0 tx=0x7f032c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b03b6bbd5cdded server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.952+0000 7f033b7fe700 1 -- 172.21.15.28:0/3746464924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f032c004160 con 0x7f033c106480 2024-03-22T02:04:31.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.952+0000 7f033b7fe700 1 -- 172.21.15.28:0/3746464924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032c004300 con 0x7f033c106480 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.952+0000 7f033b7fe700 1 -- 172.21.15.28:0/3746464924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f032c011b20 con 0x7f033c106480 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 -- 172.21.15.28:0/3746464924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 msgr2=0x7f033c1068a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 --2- 172.21.15.28:0/3746464924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f032c0098a0 tx=0x7f032c009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 -- 172.21.15.28:0/3746464924 shutdown_connections 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 --2- 172.21.15.28:0/3746464924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c1068a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 -- 172.21.15.28:0/3746464924 >> 172.21.15.28:0/3746464924 conn(0x7f033c101d80 msgr2=0x7f033c1041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 -- 172.21.15.28:0/3746464924 shutdown_connections 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 -- 172.21.15.28:0/3746464924 wait complete. 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.953+0000 7f0342d38700 1 Processor -- start 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0342d38700 1 -- start start 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0342d38700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0342d38700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f033c10c7d0 con 0x7f033c106480 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0340ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:31.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0340ad4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36862/0 (socket says 172.21.15.28:36862) 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0340ad4700 1 -- 172.21.15.28:0/3655389594 learned_addr learned my addr 172.21.15.28:0/3655389594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0340ad4700 1 -- 172.21.15.28:0/3655389594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033c19c5a0 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0340ad4700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f032c0117e0 tx=0x7f032c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f032c003bf0 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f032c01a910 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f032c024970 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.954+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033c19c830 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.955+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f033c19cc90 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f032c022020 con 0x7f033c106480 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f0339ffb700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 0x7f032803b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:31.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f033c063cb0 con 0x7f033c106480 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f032c04d5c0 con 0x7f033c106480 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f033bfff700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 0x7f032803b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.956+0000 7f033bfff700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 0x7f032803b6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0324006fd0 tx=0x7f0324006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:29.962+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f032c011e20 con 0x7f033c106480 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.186+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f033c19f6c0 con 0x7f03280391e0 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.431+0000 7f0339ffb700 1 -- 172.21.15.28:0/3655389594 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f033c19f6c0 con 0x7f03280391e0 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.433+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 msgr2=0x7f032803b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.433+0000 7f0342d38700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 0x7f032803b6a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0324006fd0 tx=0x7f0324006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 msgr2=0x7f033c19c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f032c0117e0 tx=0x7f032c011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 shutdown_connections 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03280391e0 0x7f032803b6a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 --2- 172.21.15.28:0/3655389594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f033c106480 0x7f033c19c060 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 >> 172.21.15.28:0/3655389594 conn(0x7f033c101d80 msgr2=0x7f033c102820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 shutdown_connections 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:30.434+0000 7f0342d38700 1 -- 172.21.15.28:0/3655389594 wait complete. 2024-03-22T02:04:31.250 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-03-22T02:04:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:31 smithi028 ceph-mon[31916]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:31.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:32 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1635096295' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-22T02:04:32.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-03-22T02:04:32.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.640+0000 7f8de0840700 1 Processor -- start 2024-03-22T02:04:32.957 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.640+0000 7f8de0840700 1 -- start start 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8de0840700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8de0840700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc106dd0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8dda59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8dda59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36866/0 (socket says 172.21.15.28:36866) 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8dda59c700 1 -- 172.21.15.28:0/3800135267 learned_addr learned my addr 172.21.15.28:0/3800135267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8dda59c700 1 -- 172.21.15.28:0/3800135267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc1093c0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.641+0000 7f8dda59c700 1 --2- 172.21.15.28:0/3800135267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0098a0 tx=0x7f8dcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0f2dcd56d632f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.642+0000 7f8dd959a700 1 -- 172.21.15.28:0/3800135267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dcc004160 con 0x7f8ddc1063e0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.642+0000 7f8dd959a700 1 -- 172.21.15.28:0/3800135267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dcc004300 con 0x7f8ddc1063e0 2024-03-22T02:04:32.958 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.642+0000 7f8dd959a700 1 -- 172.21.15.28:0/3800135267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dcc011b20 con 0x7f8ddc1063e0 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.642+0000 7f8de0840700 1 -- 172.21.15.28:0/3800135267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 msgr2=0x7f8ddc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.642+0000 7f8de0840700 1 --2- 172.21.15.28:0/3800135267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0098a0 tx=0x7f8dcc009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 -- 172.21.15.28:0/3800135267 shutdown_connections 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 --2- 172.21.15.28:0/3800135267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc106800 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 -- 172.21.15.28:0/3800135267 >> 172.21.15.28:0/3800135267 conn(0x7f8ddc101dc0 msgr2=0x7f8ddc1041e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 -- 172.21.15.28:0/3800135267 shutdown_connections 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 -- 172.21.15.28:0/3800135267 wait complete. 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.643+0000 7f8de0840700 1 Processor -- start 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8de0840700 1 -- start start 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8de0840700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:32.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8de0840700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ddc10c770 con 0x7f8ddc1063e0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dda59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dda59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36874/0 (socket says 172.21.15.28:36874) 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dda59c700 1 -- 172.21.15.28:0/1635096295 learned_addr learned my addr 172.21.15.28:0/1635096295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dda59c700 1 -- 172.21.15.28:0/1635096295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc109180 con 0x7f8ddc1063e0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dda59c700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0117e0 tx=0x7f8dcc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.644+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dcc003bf0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.645+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ddc107370 con 0x7f8ddc1063e0 2024-03-22T02:04:32.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.645+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8dcc01a910 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.645+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8dcc0249a0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.645+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ddc107830 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.646+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ddc063cb0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.646+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46307+0+0 (secure 0 0 0) 0x7f8dcc022020 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.646+0000 7f8dcb7fe700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 0x7f8dc403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.646+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8dcc04d7e0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.646+0000 7f8dd9d9b700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 0x7f8dc403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.647+0000 7f8dd9d9b700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 0x7f8dc403b640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0006fd0 tx=0x7f8dd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.651+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8dcc011e20 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.857+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f8ddc107ab0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.961 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.857+0000 7f8dcb7fe700 1 -- 172.21.15.28:0/1635096295 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f8dcc00b9d0 con 0x7f8ddc1063e0 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 msgr2=0x7f8dc403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 0x7f8dc403b640 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8dd0006fd0 tx=0x7f8dd0006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 msgr2=0x7f8ddc108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc0117e0 tx=0x7f8dcc011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 shutdown_connections 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8dc4039180 0x7f8dc403b640 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 --2- 172.21.15.28:0/1635096295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ddc1063e0 0x7f8ddc108c40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 >> 172.21.15.28:0/1635096295 conn(0x7f8ddc101dc0 msgr2=0x7f8ddc1028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 shutdown_connections 2024-03-22T02:04:32.962 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:31.860+0000 7f8de0840700 1 -- 172.21.15.28:0/1635096295 wait complete. 2024-03-22T02:04:32.964 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-22T02:04:32.964 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-03-22T02:04:32.972 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-22T02:04:32.972 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-03-22T02:04:32.972 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-03-22T02:04:32.972 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-22T02:04:32.973 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-03-22T02:04:32.973 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:32.973 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-03-22T02:04:32.974 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-03-22T02:04:32.974 INFO:teuthology.orchestra.run.smithi028.stdout: Password: 64h1dck1i6 2024-03-22T02:04:32.974 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:32.974 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config directory 2024-03-22T02:04:32.974 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-03-22T02:04:34.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:34 smithi028 ceph-mon[31916]: mgrmap e13: x(active, since 6s) 2024-03-22T02:04:34.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.755+0000 7f5ebdb2e700 1 Processor -- start 2024-03-22T02:04:34.672 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5ebdb2e700 1 -- start start 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5ebdb2e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5ebdb2e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb810a160 con 0x7f5eb8109770 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36432/0 (socket says 172.21.15.28:36432) 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5eb77fe700 1 -- 172.21.15.28:0/3367052894 learned_addr learned my addr 172.21.15.28:0/3367052894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5eb77fe700 1 -- 172.21.15.28:0/3367052894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb810a2e0 con 0x7f5eb8109770 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.756+0000 7f5eb77fe700 1 --2- 172.21.15.28:0/3367052894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098a0 tx=0x7f5ea8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bed7290f444a565 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5eb67fc700 1 -- 172.21.15.28:0/3367052894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8004160 con 0x7f5eb8109770 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5eb67fc700 1 -- 172.21.15.28:0/3367052894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea8004300 con 0x7f5eb8109770 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5eb67fc700 1 -- 172.21.15.28:0/3367052894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8011b20 con 0x7f5eb8109770 2024-03-22T02:04:34.673 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/3367052894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 msgr2=0x7f5eb8109b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/3367052894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80098a0 tx=0x7f5ea8009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/3367052894 shutdown_connections 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/3367052894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb8109b90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/3367052894 >> 172.21.15.28:0/3367052894 conn(0x7f5eb8101df0 msgr2=0x7f5eb8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/3367052894 shutdown_connections 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.757+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/3367052894 wait complete. 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5ebdb2e700 1 Processor -- start 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5ebdb2e700 1 -- start start 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5ebdb2e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5ebdb2e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb8107080 con 0x7f5eb8109770 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5eb77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36444/0 (socket says 172.21.15.28:36444) 2024-03-22T02:04:34.674 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.758+0000 7f5eb77fe700 1 -- 172.21.15.28:0/1613458578 learned_addr learned my addr 172.21.15.28:0/1613458578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5eb77fe700 1 -- 172.21.15.28:0/1613458578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb819c550 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5eb77fe700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80117e0 tx=0x7f5ea8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea8003bf0 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea801a910 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb819c840 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ea80249a0 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.759+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb819f4d0 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.760+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f5ea8022020 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.760+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb8063cb0 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.760+0000 7f5eb4ff9700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 0x7f5ea003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.760+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5ea804d890 con 0x7f5eb8109770 2024-03-22T02:04:34.675 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.760+0000 7f5eb6ffd700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 0x7f5ea003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.761+0000 7f5eb6ffd700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 0x7f5ea003b6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5eac006fd0 tx=0x7f5eac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.765+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ea8011e20 con 0x7f5eb8109770 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.971+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f5eb819f7f0 con 0x7f5eb8109770 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.971+0000 7f5eb4ff9700 1 -- 172.21.15.28:0/1613458578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f5ea8019c70 con 0x7f5eb8109770 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 msgr2=0x7f5ea003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 0x7f5ea003b6e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5eac006fd0 tx=0x7f5eac006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 msgr2=0x7f5eb819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80117e0 tx=0x7f5ea8011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 shutdown_connections 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5ea0039220 0x7f5ea003b6e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 --2- 172.21.15.28:0/1613458578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb8109770 0x7f5eb819c010 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 >> 172.21.15.28:0/1613458578 conn(0x7f5eb8101df0 msgr2=0x7f5eb8102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:34.676 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 shutdown_connections 2024-03-22T02:04:34.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:33.974+0000 7f5ebdb2e700 1 -- 172.21.15.28:0/1613458578 wait complete. 2024-03-22T02:04:35.980 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.076+0000 7fb32377e700 1 Processor -- start 2024-03-22T02:04:35.980 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32377e700 1 -- start start 2024-03-22T02:04:35.980 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32377e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:35.980 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32377e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c107a10 con 0x7fb31c107020 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32151a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32151a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36452/0 (socket says 172.21.15.28:36452) 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32151a700 1 -- 172.21.15.28:0/3061821017 learned_addr learned my addr 172.21.15.28:0/3061821017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.077+0000 7fb32151a700 1 -- 172.21.15.28:0/3061821017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31c108220 con 0x7fb31c107020 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32151a700 1 --2- 172.21.15.28:0/3061821017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3180098a0 tx=0x7fb318009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8c20ce494beaa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb30ffff700 1 -- 172.21.15.28:0/3061821017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb318004160 con 0x7fb31c107020 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb30ffff700 1 -- 172.21.15.28:0/3061821017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb318004300 con 0x7fb31c107020 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb30ffff700 1 -- 172.21.15.28:0/3061821017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb318011b20 con 0x7fb31c107020 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32377e700 1 -- 172.21.15.28:0/3061821017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 msgr2=0x7fb31c107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32377e700 1 --2- 172.21.15.28:0/3061821017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb3180098a0 tx=0x7fb318009bb0 comp rx=0 tx=0).stop 2024-03-22T02:04:35.981 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32377e700 1 -- 172.21.15.28:0/3061821017 shutdown_connections 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32377e700 1 --2- 172.21.15.28:0/3061821017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c107440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.078+0000 7fb32377e700 1 -- 172.21.15.28:0/3061821017 >> 172.21.15.28:0/3061821017 conn(0x7fb31c1025e0 msgr2=0x7fb31c104a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 -- 172.21.15.28:0/3061821017 shutdown_connections 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 -- 172.21.15.28:0/3061821017 wait complete. 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 Processor -- start 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 -- start start 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32377e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb31c10b5d0 con 0x7fb31c107020 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32151a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32151a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36454/0 (socket says 172.21.15.28:36454) 2024-03-22T02:04:35.982 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.079+0000 7fb32151a700 1 -- 172.21.15.28:0/19255909 learned_addr learned my addr 172.21.15.28:0/19255909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb32151a700 1 -- 172.21.15.28:0/19255909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb31c192710 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb32151a700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3180117e0 tx=0x7fb318011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb318003bf0 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb31801a910 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb31c190900 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3180249a0 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.080+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb31c190dc0 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.081+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fb318022020 con 0x7fb31c107020 2024-03-22T02:04:35.983 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.081+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31c063cb0 con 0x7fb31c107020 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.081+0000 7fb30e7fc700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 0x7fb30803b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.081+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb31804d680 con 0x7fb31c107020 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.081+0000 7fb320d19700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 0x7fb30803b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.082+0000 7fb320d19700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 0x7fb30803b6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb310006fd0 tx=0x7fb310006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.087+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb318011e20 con 0x7fb31c107020 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.408+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fb31c191510 con 0x7fb31c107020 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.410+0000 7fb30e7fc700 1 -- 172.21.15.28:0/19255909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fb31800b3e0 con 0x7fb31c107020 2024-03-22T02:04:35.984 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 msgr2=0x7fb30803b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 0x7fb30803b6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb310006fd0 tx=0x7fb310006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 msgr2=0x7fb31c1921d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb3180117e0 tx=0x7fb318011c10 comp rx=0 tx=0).stop 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 shutdown_connections 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb308039220 0x7fb30803b6e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 --2- 172.21.15.28:0/19255909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb31c107020 0x7fb31c1921d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.412+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 >> 172.21.15.28:0/19255909 conn(0x7fb31c1025e0 msgr2=0x7fb31c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.413+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 shutdown_connections 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-03-22T02:04:35.413+0000 7fb32377e700 1 -- 172.21.15.28:0/19255909 wait complete. 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-22T02:04:35.985 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-22T02:04:35.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:35.987 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-03-22T02:04:36.022 INFO:tasks.cephadm:Fetching config... 2024-03-22T02:04:36.022 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:04:36.022 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-22T02:04:36.039 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-22T02:04:36.039 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:04:36.039 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-22T02:04:36.093 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-22T02:04:36.094 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:04:36.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-22T02:04:36.161 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-22T02:04:36.161 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:04:36.161 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-22T02:04:36.216 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-22T02:04:36.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T02:04:36.297 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:04:36.311 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T02:04:36.357 INFO:teuthology.orchestra.run.smithi083.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:04:36.372 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T02:04:36.417 INFO:teuthology.orchestra.run.smithi120.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp7wffH7kItKJ5Cn7jjW4qnsxOPAaQ20Mhy+tNreIdZIIOW1cXgYqdQaUiq7dKmKVAS0lG4PKYCjWeuB6GQ9gUImKmjCcwXuIIwjRYJlq7kJ4otwPtUWx02UZV9KpA/itX4VtJ+0VIs0WQPugTIBYyG4D8OwvAzKhNIwU1khL34tBQCPn3+AnHWa8D7j5yiVuQ/+49g0DMNXQfO0zJsLpe9HcNmxP6+AmRJVkSdRsReTNrYTexZAXSO/giK0ETdTzUx9w2rgEQbMJP1Qai0piY5D4xPZYpg+5pJSeh8d1PS8XV2SsTjLc2sL4m75VUbzuwcZR3JvWRNP7T6jcwV/x2AZW6ND2QWTQhiBdjsuXzKBn/y3UAuobcAfh6ZCxP2eU0rioBakr0whPFZ35EVDQZF2Gf8RsnpyIVB//Z2dmOghdr4LloOar7DIK66H2sI+lPvavJAec2br+m/gMTSAwzukdYCjIwz/dKNrZ9ct33T3AmirTy3a5vBg5ASPFffzU= ceph-426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:04:36.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-22T02:04:36.556 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:36 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/19255909' entity='client.admin' 2024-03-22T02:04:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:04:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 -- 172.21.15.28:0/2935973912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 msgr2=0x7fd8ec1024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 --2- 172.21.15.28:0/2935973912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec1024f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00b0a0 tx=0x7fd8dc00b3b0 comp rx=0 tx=0).stop 2024-03-22T02:04:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 -- 172.21.15.28:0/2935973912 shutdown_connections 2024-03-22T02:04:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 --2- 172.21.15.28:0/2935973912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec1024f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:39.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 -- 172.21.15.28:0/2935973912 >> 172.21.15.28:0/2935973912 conn(0x7fd8ec0fd670 msgr2=0x7fd8ec0ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 -- 172.21.15.28:0/2935973912 shutdown_connections 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.148+0000 7fd8f222a700 1 -- 172.21.15.28:0/2935973912 wait complete. 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8f222a700 1 Processor -- start 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8f222a700 1 -- start start 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8f222a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8f222a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8ec108160 con 0x7fd8ec1020d0 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8eb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36464/0 (socket says 172.21.15.28:36464) 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.149+0000 7fd8eb7fe700 1 -- 172.21.15.28:0/3004098344 learned_addr learned my addr 172.21.15.28:0/3004098344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8eb7fe700 1 -- 172.21.15.28:0/3004098344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec076880 con 0x7fd8ec1020d0 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8eb7fe700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc007890 tx=0x7fd8dc00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8dc017040 con 0x7fd8ec1020d0 2024-03-22T02:04:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8dc022070 con 0x7fd8ec1020d0 2024-03-22T02:04:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd8dc01d5b0 con 0x7fd8ec1020d0 2024-03-22T02:04:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8ec074a70 con 0x7fd8ec1020d0 2024-03-22T02:04:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.150+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8ec074fb0 con 0x7fd8ec1020d0 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fd8dc01d860 con 0x7fd8ec1020d0 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8e8ff9700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 0x7fd8d803b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd8dc00f050 con 0x7fd8ec1020d0 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8eaffd700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 0x7fd8d803b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8ec1088e0 con 0x7fd8ec1020d0 2024-03-22T02:04:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.151+0000 7fd8eaffd700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 0x7fd8d803b720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.158+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd8dc007e20 con 0x7fd8ec1020d0 2024-03-22T02:04:39.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.363+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fd8ec190a90 con 0x7fd8ec1020d0 2024-03-22T02:04:39.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.366+0000 7fd8e8ff9700 1 -- 172.21.15.28:0/3004098344 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fd8dc010db0 con 0x7fd8ec1020d0 2024-03-22T02:04:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 msgr2=0x7fd8d803b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 0x7fd8d803b720 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd8d4006fd0 tx=0x7fd8d4006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 msgr2=0x7fd8ec076340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc007890 tx=0x7fd8dc00bd40 comp rx=0 tx=0).stop 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 shutdown_connections 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd8d8039260 0x7fd8d803b720 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 --2- 172.21.15.28:0/3004098344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8ec1020d0 0x7fd8ec076340 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 >> 172.21.15.28:0/3004098344 conn(0x7fd8ec0fd670 msgr2=0x7fd8ec0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 shutdown_connections 2024-03-22T02:04:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:39.368+0000 7fd8f222a700 1 -- 172.21.15.28:0/3004098344 wait complete. 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3004098344' entity='client.admin' 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:40.625 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:40.626 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:40.626 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-22T02:04:40.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-22T02:04:40.867 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.251+0000 7feb3ddab700 1 -- 172.21.15.28:0/3100188361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 msgr2=0x7feb38102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.251+0000 7feb3ddab700 1 --2- 172.21.15.28:0/3100188361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38102980 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7feb28009910 tx=0x7feb28009c20 comp rx=0 tx=0).stop 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 -- 172.21.15.28:0/3100188361 shutdown_connections 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 --2- 172.21.15.28:0/3100188361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38102980 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 -- 172.21.15.28:0/3100188361 >> 172.21.15.28:0/3100188361 conn(0x7feb380fbd10 msgr2=0x7feb380fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 -- 172.21.15.28:0/3100188361 shutdown_connections 2024-03-22T02:04:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 -- 172.21.15.28:0/3100188361 wait complete. 2024-03-22T02:04:42.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 Processor -- start 2024-03-22T02:04:42.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.252+0000 7feb3ddab700 1 -- start start 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb3ddab700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb3ddab700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb38107060 con 0x7feb38102560 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb377fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb377fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45006/0 (socket says 172.21.15.28:45006) 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb377fe700 1 -- 172.21.15.28:0/95814613 learned_addr learned my addr 172.21.15.28:0/95814613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb377fe700 1 -- 172.21.15.28:0/95814613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb38197d60 con 0x7feb38102560 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb377fe700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb28006010 tx=0x7feb28005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb28027070 con 0x7feb38102560 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb38197ff0 con 0x7feb38102560 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb2800be40 con 0x7feb38102560 2024-03-22T02:04:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb2800f460 con 0x7feb38102560 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.253+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb3819ace0 con 0x7feb38102560 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.254+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7feb2802a070 con 0x7feb38102560 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.254+0000 7feb34ff9700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 0x7feb2003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.254+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7feb2801e030 con 0x7feb38102560 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.255+0000 7feb36ffd700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 0x7feb2003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.255+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb381077e0 con 0x7feb38102560 2024-03-22T02:04:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.255+0000 7feb36ffd700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 0x7feb2003b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006fd0 tx=0x7feb2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.260+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb28013070 con 0x7feb38102560 2024-03-22T02:04:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.466+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7feb3802de10 con 0x7feb20039220 2024-03-22T02:04:42.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.469+0000 7feb34ff9700 1 -- 172.21.15.28:0/95814613 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7feb3802de10 con 0x7feb20039220 2024-03-22T02:04:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 msgr2=0x7feb2003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 0x7feb2003b6e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feb2c006fd0 tx=0x7feb2c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 msgr2=0x7feb38197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7feb28006010 tx=0x7feb28005050 comp rx=0 tx=0).stop 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 shutdown_connections 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb20039220 0x7feb2003b6e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 --2- 172.21.15.28:0/95814613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb38102560 0x7feb38197820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.471+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 >> 172.21.15.28:0/95814613 conn(0x7feb380fbd10 msgr2=0x7feb381005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.472+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 shutdown_connections 2024-03-22T02:04:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:42.472+0000 7feb3ddab700 1 -- 172.21.15.28:0/95814613 wait complete. 2024-03-22T02:04:43.192 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi083 2024-03-22T02:04:43.192 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:04:43.192 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/etc/ceph/ceph.conf 2024-03-22T02:04:43.211 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:04:43.211 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:04:43.269 INFO:tasks.cephadm:Adding host smithi083 to orchestrator... 2024-03-22T02:04:43.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch host add smithi083 2024-03-22T02:04:43.515 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-22T02:04:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: Updating smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:04:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:04:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: Updating smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:04:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 -- 172.21.15.28:0/1515354903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 msgr2=0x7fdb24103640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 --2- 172.21.15.28:0/1515354903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24103640 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdb14009910 tx=0x7fdb14009c20 comp rx=0 tx=0).stop 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 -- 172.21.15.28:0/1515354903 shutdown_connections 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 --2- 172.21.15.28:0/1515354903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24103640 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 -- 172.21.15.28:0/1515354903 >> 172.21.15.28:0/1515354903 conn(0x7fdb240fca30 msgr2=0x7fdb240fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 -- 172.21.15.28:0/1515354903 shutdown_connections 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.006+0000 7fdb288e3700 1 -- 172.21.15.28:0/1515354903 wait complete. 2024-03-22T02:04:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb288e3700 1 Processor -- start 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb288e3700 1 -- start start 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb288e3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb288e3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2410c6e0 con 0x7fdb24103260 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb2259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb2259c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45014/0 (socket says 172.21.15.28:45014) 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb2259c700 1 -- 172.21.15.28:0/2550150568 learned_addr learned my addr 172.21.15.28:0/2550150568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.007+0000 7fdb2259c700 1 -- 172.21.15.28:0/2550150568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb24102db0 con 0x7fdb24103260 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb2259c700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb14006010 tx=0x7fdb14005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb14026070 con 0x7fdb24103260 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb1400bc30 con 0x7fdb24103260 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdb14003ab0 con 0x7fdb24103260 2024-03-22T02:04:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb24101400 con 0x7fdb24103260 2024-03-22T02:04:45.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.008+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb241018c0 con 0x7fdb24103260 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.009+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fdb14029070 con 0x7fdb24103260 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.009+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb2406ab80 con 0x7fdb24103260 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.009+0000 7fdb137fe700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 0x7fdb0c03b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.009+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fdb1401e070 con 0x7fdb24103260 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.009+0000 7fdb21d9b700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 0x7fdb0c03b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.010+0000 7fdb21d9b700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 0x7fdb0c03b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdb18006fd0 tx=0x7fdb18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.014+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb14013070 con 0x7fdb24103260 2024-03-22T02:04:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:45.221+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi083", "target": ["mon-mgr", ""]}) -- 0x7fdb241023f0 con 0x7fdb0c039220 2024-03-22T02:04:45.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:45 smithi028 ceph-mon[31916]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi083", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:46 smithi028 ceph-mon[31916]: Deploying cephadm binary to smithi083 2024-03-22T02:04:47.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:47 smithi028 ceph-mon[31916]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:49.580 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi083' with addr '172.21.15.83' 2024-03-22T02:04:49.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.580+0000 7fdb137fe700 1 -- 172.21.15.28:0/2550150568 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fdb241023f0 con 0x7fdb0c039220 2024-03-22T02:04:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 msgr2=0x7fdb0c03b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 0x7fdb0c03b6e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdb18006fd0 tx=0x7fdb18006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 msgr2=0x7fdb24100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdb14006010 tx=0x7fdb14005050 comp rx=0 tx=0).stop 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 shutdown_connections 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb0c039220 0x7fdb0c03b6e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 --2- 172.21.15.28:0/2550150568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb24103260 0x7fdb24100ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.582+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 >> 172.21.15.28:0/2550150568 conn(0x7fdb240fca30 msgr2=0x7fdb24104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.583+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 shutdown_connections 2024-03-22T02:04:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:49.583+0000 7fdb288e3700 1 -- 172.21.15.28:0/2550150568 wait complete. 2024-03-22T02:04:49.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:49 smithi028 ceph-mon[31916]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:50.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T02:04:50.401 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:04:50.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:50.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:50 smithi028 ceph-mon[31916]: Added host smithi083 2024-03-22T02:04:50.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:50.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:51.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:51 smithi028 ceph-mon[31916]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:51.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:51 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 -- 172.21.15.28:0/1868528744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 msgr2=0x7f2764101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 --2- 172.21.15.28:0/1868528744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764101580 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2754009910 tx=0x7f2754009c20 comp rx=0 tx=0).stop 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 -- 172.21.15.28:0/1868528744 shutdown_connections 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 --2- 172.21.15.28:0/1868528744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764101580 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 -- 172.21.15.28:0/1868528744 >> 172.21.15.28:0/1868528744 conn(0x7f27640fca30 msgr2=0x7f27640fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 -- 172.21.15.28:0/1868528744 shutdown_connections 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.830+0000 7f276a2b6700 1 -- 172.21.15.28:0/1868528744 wait complete. 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f276a2b6700 1 Processor -- start 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f276a2b6700 1 -- start start 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f276a2b6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f276a2b6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f276410b570 con 0x7f27641011a0 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f2763fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f2763fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41218/0 (socket says 172.21.15.28:41218) 2024-03-22T02:04:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.831+0000 7f2763fff700 1 -- 172.21.15.28:0/2329477608 learned_addr learned my addr 172.21.15.28:0/2329477608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f2763fff700 1 -- 172.21.15.28:0/2329477608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2764104b30 con 0x7f27641011a0 2024-03-22T02:04:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f2763fff700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2754006010 tx=0x7f2754005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2754026070 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f275400bc30 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27641031e0 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27540039d0 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.832+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27641036a0 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f2754029070 con 0x7f27641011a0 2024-03-22T02:04:51.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2764102680 con 0x7f27641011a0 2024-03-22T02:04:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f27617fa700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 0x7f275003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f275401e070 con 0x7f27641011a0 2024-03-22T02:04:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f27637fe700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 0x7f275003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:51.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.833+0000 7f27637fe700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 0x7f275003b6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:51.838+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2754013070 con 0x7f27641011a0 2024-03-22T02:04:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.047+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2764103930 con 0x7f2750039220 2024-03-22T02:04:52.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.049+0000 7f27617fa700 1 -- 172.21.15.28:0/2329477608 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f2764103930 con 0x7f2750039220 2024-03-22T02:04:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:04:52.049 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}] 2024-03-22T02:04:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 msgr2=0x7f275003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 0x7f275003b6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f274c006fd0 tx=0x7f274c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 msgr2=0x7f2764102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2754006010 tx=0x7f2754005030 comp rx=0 tx=0).stop 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 shutdown_connections 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2750039220 0x7f275003b6e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 --2- 172.21.15.28:0/2329477608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27641011a0 0x7f2764102c40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.051+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 >> 172.21.15.28:0/2329477608 conn(0x7f27640fca30 msgr2=0x7f276406a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.052+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 shutdown_connections 2024-03-22T02:04:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:52.052+0000 7f276a2b6700 1 -- 172.21.15.28:0/2329477608 wait complete. 2024-03-22T02:04:52.676 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi120 2024-03-22T02:04:52.676 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:04:52.676 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.conf 2024-03-22T02:04:52.695 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:04:52.696 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:04:52.753 INFO:tasks.cephadm:Adding host smithi120 to orchestrator... 2024-03-22T02:04:52.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch host add smithi120 2024-03-22T02:04:52.991 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:04:53.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:53 smithi028 ceph-mon[31916]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:04:53.418 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:54.457 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:54 smithi028 ceph-mon[31916]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.488+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1095325329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 msgr2=0x7fd2681026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.488+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1095325329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd2681026a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd258009910 tx=0x7fd258009c20 comp rx=0 tx=0).stop 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1095325329 shutdown_connections 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1095325329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd2681026a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1095325329 >> 172.21.15.28:0/1095325329 conn(0x7fd2680fdb90 msgr2=0x7fd2680fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1095325329 shutdown_connections 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1095325329 wait complete. 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 Processor -- start 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.489+0000 7fd26e9c9700 1 -- start start 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd26e9c9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd26e9c9700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd26810c690 con 0x7fd2681022c0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd267fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd267fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41242/0 (socket says 172.21.15.28:41242) 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd267fff700 1 -- 172.21.15.28:0/1178863862 learned_addr learned my addr 172.21.15.28:0/1178863862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd267fff700 1 -- 172.21.15.28:0/1178863862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd26819a2e0 con 0x7fd2681022c0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd267fff700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd258005850 tx=0x7fd258005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258026070 con 0x7fd2681022c0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd25800bbd0 con 0x7fd2681022c0 2024-03-22T02:04:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd26819a5d0 con 0x7fd2681022c0 2024-03-22T02:04:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd258003970 con 0x7fd2681022c0 2024-03-22T02:04:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.490+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd26819e670 con 0x7fd2681022c0 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.491+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fd258029070 con 0x7fd2681022c0 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.491+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2681037a0 con 0x7fd2681022c0 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.491+0000 7fd2657fa700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 0x7fd25003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.491+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd25801e070 con 0x7fd2681022c0 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.491+0000 7fd2677fe700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 0x7fd25003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:04:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.492+0000 7fd2677fe700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 0x7fd25003b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd25c006fd0 tx=0x7fd25c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:04:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.496+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd258013070 con 0x7fd2681022c0 2024-03-22T02:04:54.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:54.706+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}) -- 0x7fd26819b100 con 0x7fd250039220 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: Deploying cephadm binary to smithi120 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:04:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:04:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:04:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:56 smithi028 ceph-mon[31916]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:04:57.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:57 smithi028 ceph-mon[31916]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:04:57.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:57 smithi028 ceph-mon[31916]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:04:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:58 smithi028 ceph-mon[31916]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:04:58.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.958+0000 7fd2657fa700 1 -- 172.21.15.28:0/1178863862 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fd26819b100 con 0x7fd250039220 2024-03-22T02:04:58.960 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi120' with addr '172.21.15.120' 2024-03-22T02:04:58.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.960+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 msgr2=0x7fd25003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:58.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.960+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 0x7fd25003b6e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd25c006fd0 tx=0x7fd25c006e40 comp rx=0 tx=0).stop 2024-03-22T02:04:58.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.960+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 msgr2=0x7fd268199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:04:58.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.960+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd258005850 tx=0x7fd258005030 comp rx=0 tx=0).stop 2024-03-22T02:04:58.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 shutdown_connections 2024-03-22T02:04:58.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd250039220 0x7fd25003b6e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:58.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 --2- 172.21.15.28:0/1178863862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2681022c0 0x7fd268199da0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:04:58.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 >> 172.21.15.28:0/1178863862 conn(0x7fd2680fdb90 msgr2=0x7fd2680748b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:04:58.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 shutdown_connections 2024-03-22T02:04:58.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:04:58.961+0000 7fd26e9c9700 1 -- 172.21.15.28:0/1178863862 wait complete. 2024-03-22T02:04:59.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:04:59.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:59.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:59.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:59.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:04:59.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:04:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:04:59.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T02:04:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:05:00.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:00 smithi028 ceph-mon[31916]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:00.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:00 smithi028 ceph-mon[31916]: Added host smithi120 2024-03-22T02:05:00.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:00.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 -- 172.21.15.28:0/3234884265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 msgr2=0x7f7d20107470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 --2- 172.21.15.28:0/3234884265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d20107470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000b0a0 tx=0x7f7d1000b3b0 comp rx=0 tx=0).stop 2024-03-22T02:05:01.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 -- 172.21.15.28:0/3234884265 shutdown_connections 2024-03-22T02:05:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 --2- 172.21.15.28:0/3234884265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d20107470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 -- 172.21.15.28:0/3234884265 >> 172.21.15.28:0/3234884265 conn(0x7f7d200fc1d0 msgr2=0x7f7d200fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 -- 172.21.15.28:0/3234884265 shutdown_connections 2024-03-22T02:05:01.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.306+0000 7f7d2527d700 1 -- 172.21.15.28:0/3234884265 wait complete. 2024-03-22T02:05:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d2527d700 1 Processor -- start 2024-03-22T02:05:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d2527d700 1 -- start start 2024-03-22T02:05:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d2527d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:01.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d2527d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2010c7f0 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d1effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d1effd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41268/0 (socket says 172.21.15.28:41268) 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d1effd700 1 -- 172.21.15.28:0/2259479833 learned_addr learned my addr 172.21.15.28:0/2259479833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.307+0000 7f7d1effd700 1 -- 172.21.15.28:0/2259479833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2019dd90 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d1effd700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7d100062a0 tx=0x7f7d10003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d10017040 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d10022070 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2019e020 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d1001d440 con 0x7f7d20107090 2024-03-22T02:05:01.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.308+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d20197a10 con 0x7f7d20107090 2024-03-22T02:05:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f7d1000bb90 con 0x7f7d20107090 2024-03-22T02:05:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d201037d0 con 0x7f7d20107090 2024-03-22T02:05:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d17fff700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 0x7f7d0c03b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:01.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7d1000f050 con 0x7f7d20107090 2024-03-22T02:05:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d1e7fc700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 0x7f7d0c03b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:01.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.309+0000 7f7d1e7fc700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 0x7f7d0c03b6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006fd0 tx=0x7f7d08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:01.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.314+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d10007e20 con 0x7f7d20107090 2024-03-22T02:05:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.527+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d20063cb0 con 0x7f7d0c039220 2024-03-22T02:05:01.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.528+0000 7f7d17fff700 1 -- 172.21.15.28:0/2259479833 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7f7d20063cb0 con 0x7f7d0c039220 2024-03-22T02:05:01.529 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:05:01.529 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.83", "hostname": "smithi083", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}] 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 msgr2=0x7f7d0c03b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 0x7f7d0c03b6e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006fd0 tx=0x7f7d08006e40 comp rx=0 tx=0).stop 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 msgr2=0x7f7d2019d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7d100062a0 tx=0x7f7d10003ce0 comp rx=0 tx=0).stop 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 shutdown_connections 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d0c039220 0x7f7d0c03b6e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:01.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 --2- 172.21.15.28:0/2259479833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d20107090 0x7f7d2019d850 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 >> 172.21.15.28:0/2259479833 conn(0x7f7d200fc1d0 msgr2=0x7f7d20109b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 shutdown_connections 2024-03-22T02:05:01.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:01.530+0000 7f7d2527d700 1 -- 172.21.15.28:0/2259479833 wait complete. 2024-03-22T02:05:02.128 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-22T02:05:02.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-22T02:05:02.370 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:05:02.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:02 smithi028 ceph-mon[31916]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:02.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:02 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:03.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:03 smithi028 ceph-mon[31916]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:05:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 -- 172.21.15.28:0/2259177388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 msgr2=0x7fd6a4074e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/2259177388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4074e90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00b0a0 tx=0x7fd69c00b3b0 comp rx=0 tx=0).stop 2024-03-22T02:05:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 -- 172.21.15.28:0/2259177388 shutdown_connections 2024-03-22T02:05:03.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/2259177388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4074e90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 -- 172.21.15.28:0/2259177388 >> 172.21.15.28:0/2259177388 conn(0x7fd6a40fdb40 msgr2=0x7fd6a40fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 -- 172.21.15.28:0/2259177388 shutdown_connections 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 -- 172.21.15.28:0/2259177388 wait complete. 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.778+0000 7fd6a9a10700 1 Processor -- start 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a9a10700 1 -- start start 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a9a10700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a9a10700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a410e900 con 0x7fd6a4074ab0 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a2ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34382/0 (socket says 172.21.15.28:34382) 2024-03-22T02:05:03.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a2ffd700 1 -- 172.21.15.28:0/1335394797 learned_addr learned my addr 172.21.15.28:0/1335394797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:05:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a2ffd700 1 -- 172.21.15.28:0/1335394797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a419a300 con 0x7fd6a4074ab0 2024-03-22T02:05:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.779+0000 7fd6a2ffd700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd69c0062a0 tx=0x7fd69c00bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd69c017050 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd69c023070 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd69c004000 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a419a590 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a419a9f0 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.780+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fd69c0042c0 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.781+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a4076160 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.781+0000 7fd6a8a0e700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 0x7fd69003b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.781+0000 7fd6a27fc700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 0x7fd69003b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.781+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd69c00f050 con 0x7fd6a4074ab0 2024-03-22T02:05:03.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.781+0000 7fd6a27fc700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 0x7fd69003b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd68c006fd0 tx=0x7fd68c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:03.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.786+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd69c026bd0 con 0x7fd6a4074ab0 2024-03-22T02:05:03.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:03.997+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fd6a419b120 con 0x7fd6a4074ab0 2024-03-22T02:05:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.457+0000 7fd6a8a0e700 1 -- 172.21.15.28:0/1335394797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fd69c02d200 con 0x7fd6a4074ab0 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.459+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 msgr2=0x7fd69003b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 0x7fd69003b6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd68c006fd0 tx=0x7fd68c006e40 comp rx=0 tx=0).stop 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 msgr2=0x7fd6a4199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd69c0062a0 tx=0x7fd69c00bdd0 comp rx=0 tx=0).stop 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 shutdown_connections 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd690039220 0x7fd69003b6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:04.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 --2- 172.21.15.28:0/1335394797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6a4074ab0 0x7fd6a4199dc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 >> 172.21.15.28:0/1335394797 conn(0x7fd6a40fdb40 msgr2=0x7fd6a41062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 shutdown_connections 2024-03-22T02:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:04.460+0000 7fd6a9a10700 1 -- 172.21.15.28:0/1335394797 wait complete. 2024-03-22T02:05:04.462 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-03-22T02:05:04.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:04 smithi028 ceph-mon[31916]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:04.817 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:04 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T02:05:05.162 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-03-22T02:05:05.162 INFO:tasks.cephadm:Adding mon.b on smithi083 2024-03-22T02:05:05.162 INFO:tasks.cephadm:Adding mon.c on smithi120 2024-03-22T02:05:05.163 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c' 2024-03-22T02:05:05.405 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T02:05:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 -- 172.21.15.120:0/1574435589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 msgr2=0x7f6cdc1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:06.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/1574435589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc1015d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009910 tx=0x7f6ccc009c20 comp rx=0 tx=0).stop 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 -- 172.21.15.120:0/1574435589 shutdown_connections 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/1574435589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc1015d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 -- 172.21.15.120:0/1574435589 >> 172.21.15.120:0/1574435589 conn(0x7f6cdc0fcac0 msgr2=0x7f6cdc0feee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 -- 172.21.15.120:0/1574435589 shutdown_connections 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 -- 172.21.15.120:0/1574435589 wait complete. 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.794+0000 7f6ce1a63700 1 Processor -- start 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6ce1a63700 1 -- start start 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6ce1a63700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6ce1a63700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6cdc10b5c0 con 0x7f6cdc1011f0 2024-03-22T02:05:06.796 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6cdb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:06.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6cdb7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.120:54750/0 (socket says 172.21.15.120:54750) 2024-03-22T02:05:06.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.795+0000 7f6cdb7fe700 1 -- 172.21.15.120:0/2030700688 learned_addr learned my addr 172.21.15.120:0/2030700688 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:05:06.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6cdb7fe700 1 -- 172.21.15.120:0/2030700688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cdc198f50 con 0x7f6cdc1011f0 2024-03-22T02:05:06.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6cdb7fe700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc005850 tx=0x7f6ccc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:06.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ccc026070 con 0x7f6cdc1011f0 2024-03-22T02:05:06.798 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cdc199240 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ccc00bbd0 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6ccc003970 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.796+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cdc199700 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.797+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cdc1026d0 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.797+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f6ccc029070 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.797+0000 7f6cd8ff9700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 0x7f6cc403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.797+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ccc01e070 con 0x7f6cdc1011f0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.797+0000 7f6cdaffd700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 0x7f6cc403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:06.799 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.798+0000 7f6cdaffd700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 0x7f6cc403b6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0006fd0 tx=0x7f6cd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:06.803 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:06.802+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ccc013070 con 0x7f6cdc1011f0 2024-03-22T02:05:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:06 smithi028 ceph-mon[31916]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:06 smithi028 ceph-mon[31916]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:06 smithi028 ceph-mon[31916]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:06 smithi028 ceph-mon[31916]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:06 smithi028 ceph-mon[31916]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:05:07.008 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.008+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}) -- 0x7f6cdc06aad0 con 0x7f6cc4039220 2024-03-22T02:05:07.015 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.015+0000 7f6cd8ff9700 1 -- 172.21.15.120:0/2030700688 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6cdc06aad0 con 0x7f6cc4039220 2024-03-22T02:05:07.015 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mon update... 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 msgr2=0x7f6cc403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 0x7f6cc403b6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0006fd0 tx=0x7f6cd0006e40 comp rx=0 tx=0).stop 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 msgr2=0x7f6cdc198a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc005850 tx=0x7f6ccc005030 comp rx=0 tx=0).stop 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 shutdown_connections 2024-03-22T02:05:07.018 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6cc4039220 0x7f6cc403b6e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:07.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 --2- 172.21.15.120:0/2030700688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cdc1011f0 0x7f6cdc198a10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:07.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 >> 172.21.15.120:0/2030700688 conn(0x7f6cdc0fcac0 msgr2=0x7f6cdc102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:07.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 shutdown_connections 2024-03-22T02:05:07.019 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:07.017+0000 7f6ce1a63700 1 -- 172.21.15.120:0/2030700688 wait complete. 2024-03-22T02:05:07.585 DEBUG:teuthology.orchestra.run.smithi083:mon.b> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.b.service 2024-03-22T02:05:07.588 DEBUG:teuthology.orchestra.run.smithi120:mon.c> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.c.service 2024-03-22T02:05:07.590 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T02:05:07.591 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:05:08.304 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c;count:3 2024-03-22T02:05:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:08 smithi028 ceph-mon[31916]: Deploying daemon mon.c on smithi120 2024-03-22T02:05:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:10 smithi028 ceph-mon[31916]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- 172.21.15.120:0/2343606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 msgr2=0x7f36e4101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 --2- 172.21.15.120:0/2343606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e4101580 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009910 tx=0x7f36d8009c20 comp rx=0 tx=0).stop 2024-03-22T02:05:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- 172.21.15.120:0/2343606557 shutdown_connections 2024-03-22T02:05:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 --2- 172.21.15.120:0/2343606557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e4101580 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:11.497 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- 172.21.15.120:0/2343606557 >> 172.21.15.120:0/2343606557 conn(0x7f36e40fca30 msgr2=0x7f36e40fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- 172.21.15.120:0/2343606557 shutdown_connections 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- 172.21.15.120:0/2343606557 wait complete. 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 Processor -- start 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.495+0000 7f36eab63700 1 -- start start 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36eab63700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36eab63700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36e410b570 con 0x7f36e41011a0 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e88ff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e88ff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.120:54774/0 (socket says 172.21.15.120:54774) 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e88ff700 1 -- 172.21.15.120:0/2348264538 learned_addr learned my addr 172.21.15.120:0/2348264538 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e88ff700 1 -- 172.21.15.120:0/2348264538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e406b100 con 0x7f36e41011a0 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e88ff700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f36d8005850 tx=0x7f36d8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:11.498 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d8026070 con 0x7f36e41011a0 2024-03-22T02:05:11.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e4196d10 con 0x7f36e41011a0 2024-03-22T02:05:11.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.496+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d800bbd0 con 0x7f36e41011a0 2024-03-22T02:05:11.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.497+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f36d8003970 con 0x7f36e41011a0 2024-03-22T02:05:11.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.497+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e41971d0 con 0x7f36e41011a0 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.498+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f36d8029070 con 0x7f36e41011a0 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.498+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36e4102680 con 0x7f36e41011a0 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.498+0000 7f36e1ffb700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 0x7f36d403b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.498+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f36d801e070 con 0x7f36e41011a0 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.498+0000 7f36e3fff700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 0x7f36d403b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:11.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.499+0000 7f36e3fff700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 0x7f36d403b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f36d0006fd0 tx=0x7f36d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:11.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.504+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f36d8013070 con 0x7f36e41011a0 2024-03-22T02:05:11.745 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:11 smithi120 systemd[1]: Starting Ceph mon.c for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:05:11.819 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.818+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f36e406a8f0 con 0x7f36e41011a0 2024-03-22T02:05:11.819 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.819+0000 7f36e1ffb700 1 -- 172.21.15.120:0/2348264538 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f36d8030090 con 0x7f36e41011a0 2024-03-22T02:05:11.819 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:05:11.820 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":1,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","modified":"2024-03-22T02:03:20.403582Z","created":"2024-03-22T02:03:20.403582Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T02:05:11.821 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 msgr2=0x7f36d403b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 0x7f36d403b6e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f36d0006fd0 tx=0x7f36d0006e40 comp rx=0 tx=0).stop 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 msgr2=0x7f36e41967d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f36d8005850 tx=0x7f36d8005030 comp rx=0 tx=0).stop 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 shutdown_connections 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36d4039220 0x7f36d403b6e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 --2- 172.21.15.120:0/2348264538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36e41011a0 0x7f36e41967d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 >> 172.21.15.120:0/2348264538 conn(0x7f36e40fca30 msgr2=0x7f36e4102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 shutdown_connections 2024-03-22T02:05:11.822 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:11.821+0000 7f36eab63700 1 -- 172.21.15.120:0/2348264538 wait complete. 2024-03-22T02:05:11.826 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 1 2024-03-22T02:05:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:11 smithi120 podman[36768]: 2024-03-22 02:05:11.650478198 +0000 UTC m=+0.019434592 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:05:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:11 smithi120 podman[36768]: 2024-03-22 02:05:11.761496781 +0000 UTC m=+0.130453159 container create 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-22T02:05:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:12 smithi028 ceph-mon[31916]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:12 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/2348264538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 podman[36768]: 2024-03-22 02:05:12.475195572 +0000 UTC m=+0.844151956 container init 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:05:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 podman[36768]: 2024-03-22 02:05:12.478340091 +0000 UTC m=+0.847296471 container start 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-22T02:05:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:05:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pidfile_write: ignore empty --pid-file 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: load: jerasure load: lrc 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: RocksDB version: 7.9.2 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Git sha 0 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: DB SUMMARY 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: DB Session ID: P9S3VQSSCPX6D2R3BH6U 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-22T02:05:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.env: 0x55f175557c60 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.info_log: 0x55f177785ae0 2024-03-22T02:05:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.statistics: (nil) 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.use_fsync: 0 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:05:12.752 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.db_log_dir: 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.wal_dir: 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:05:12.753 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.write_buffer_manager: 0x55f1777561e0 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:05:12.754 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.unordered_write: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.row_cache: None 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.wal_filter: None 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:05:12.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.wal_compression: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:05:12.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_open_files: -1 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:05:12.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Compression algorithms supported: 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kZSTD supported: 0 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-22T02:05:12.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.merge_operator: 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_filter: None 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f177785be0) 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:05:12.759 INFO:journalctl@ceph.mon.c.smithi120.stdout: index_type: 0 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: data_block_index_type: 0 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: index_shortening: 1 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: checksum: 4 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: no_block_cache: 0 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_cache: 0x55f177789090 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_cache_options: 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: capacity : 536870912 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: num_shard_bits : 4 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: strict_capacity_limit : 0 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:05:12.760 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_cache_compressed: (nil) 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: persistent_cache: (nil) 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_size: 4096 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_size_deviation: 10 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_restart_interval: 16 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: index_block_restart_interval: 1 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: metadata_block_size: 4096 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: partition_filters: 0 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: use_delta_encoding: 1 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: filter_policy: bloomfilter 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: whole_key_filtering: 1 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: verify_compression: 0 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: format_version: 5 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: enable_index_compression: 1 2024-03-22T02:05:12.761 INFO:journalctl@ceph.mon.c.smithi120.stdout: block_align: 0 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout: max_auto_readahead_size: 262144 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout: prepopulate_block_cache: 0 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout: initial_auto_readahead_size: 8192 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression: NoCompression 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.num_levels: 7 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:05:12.762 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:05:12.763 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:05:12.764 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:05:12.765 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:05:12.766 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.ttl: 2592000 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:05:12.767 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: dee38c1f-6846-4714-b567-b110a9571d2c 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073112509439, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-22T02:05:12.768 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073112509908, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711073112, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "dee38c1f-6846-4714-b567-b110a9571d2c", "db_session_id": "P9S3VQSSCPX6D2R3BH6U", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073112509998, "job": 1, "event": "recovery_finished"} 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f177878000 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: DB pointer 0x55f177862000 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: ** DB Stats ** 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:05:12.769 INFO:journalctl@ceph.mon.c.smithi120.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: ** Compaction Stats [default] ** 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: ** Compaction Stats [default] ** 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:05:12.770 INFO:journalctl@ceph.mon.c.smithi120.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:05:12.771 INFO:journalctl@ceph.mon.c.smithi120.stdout: Block cache BinnedLRUCache@0x55f177789090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: using public_addr v2:172.21.15.120:0/0 -> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: starting mon.c rank -1 at public addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] at bind addrs [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(???) e0 preinit fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).mds e1 new map 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: e1 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: legacy client fscid: -1 2024-03-22T02:05:12.772 INFO:journalctl@ceph.mon.c.smithi120.stdout: 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout: No filesystems configured 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.773 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Deploying cephadm binary to smithi120 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Added host smithi120 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.775 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.776 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c;count:3 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:12.777 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: Deploying daemon mon.c on smithi120 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/2348264538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 ceph-mon[36803]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 bash[36768]: 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 2024-03-22T02:05:12.778 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:12 smithi120 systemd[1]: Started Ceph mon.c for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:05:13.448 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T02:05:13.448 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:05:13.690 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:05:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: mon.a calling monitor election 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: mon.c calling monitor election 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: monmap epoch 2 2024-03-22T02:05:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: last_changed 2024-03-22T02:05:12.534027+0000 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: min_mon_release 19 (squid) 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: election_strategy: 1 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: fsmap 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: mgrmap e13: x(active, since 51s) 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: overall HEALTH_OK 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:18.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:18.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 -- 172.21.15.120:0/573766720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 msgr2=0x7f1adc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:18.033 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/573766720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1adc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1af0100fb0 tx=0x7f1ae000feb0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 -- 172.21.15.120:0/573766720 shutdown_connections 2024-03-22T02:05:18.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/573766720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1adc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 -- 172.21.15.120:0/573766720 >> 172.21.15.120:0/573766720 conn(0x7f1af00fcb90 msgr2=0x7f1af00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:18.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 -- 172.21.15.120:0/573766720 shutdown_connections 2024-03-22T02:05:18.034 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 -- 172.21.15.120:0/573766720 wait complete. 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.033+0000 7f1af5f9b700 1 Processor -- start 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1af5f9b700 1 -- start start 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1af5f9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1af0197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1af5f9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1af5f9b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0109110 con 0x7f1adc0035c0 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1af5f9b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1af0108f90 con 0x7f1af0198370 2024-03-22T02:05:18.037 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:55184/0 (socket says 172.21.15.120:55184) 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 -- 172.21.15.120:0/3906652369 learned_addr learned my addr 172.21.15.120:0/3906652369 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 -- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 msgr2=0x7f1af019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 -- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 msgr2=0x7f1af019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1aef7fe700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1af0197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.034+0000 7f1ae7fff700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aef7fe700 1 -- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 msgr2=0x7f1af019c660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aef7fe700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aef7fe700 1 -- 172.21.15.120:0/3906652369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af01989b0 con 0x7f1adc0035c0 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aef7fe700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1af0197e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00059d0 tx=0x7f1ae00057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:18.038 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1ae0004b20 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ae0013e40 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af019cba0 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1ae0011ca0 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.035+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af019d100 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.036+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f1ae0023070 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.036+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af01027c0 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.036+0000 7f1aed7fa700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 0x7f1ad003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.036+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1ae0020050 con 0x7f1adc0035c0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.037+0000 7f1ae7fff700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 0x7f1ad003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:18.039 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.037+0000 7f1ae7fff700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 0x7f1ad003b480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009180 tx=0x7f1ad8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:18.042 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.042+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ae0031020 con 0x7f1adc0035c0 2024-03-22T02:05:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: mon.a calling monitor election 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: mon.c calling monitor election 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: monmap epoch 2 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: last_changed 2024-03-22T02:05:12.534027+0000 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: min_mon_release 19 (squid) 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: election_strategy: 1 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: fsmap 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: mgrmap e13: x(active, since 51s) 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: overall HEALTH_OK 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:18.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:18.318 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.318+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1af019d3b0 con 0x7f1adc0035c0 2024-03-22T02:05:18.319 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.319+0000 7f1aed7fa700 1 -- 172.21.15.120:0/3906652369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f1ae0028030 con 0x7f1adc0035c0 2024-03-22T02:05:18.319 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:05:18.319 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":2,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","modified":"2024-03-22T02:05:12.534027Z","created":"2024-03-22T02:03:20.403582Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T02:05:18.321 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 msgr2=0x7f1ad003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:18.321 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 0x7f1ad003b480 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009180 tx=0x7f1ad8008040 comp rx=0 tx=0).stop 2024-03-22T02:05:18.321 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 msgr2=0x7f1af0197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1af0197e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00059d0 tx=0x7f1ae00057d0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 shutdown_connections 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ad0038fc0 0x7f1ad003b480 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1adc0035c0 0x7f1af0197e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 --2- 172.21.15.120:0/3906652369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af0198370 0x7f1af019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 >> 172.21.15.120:0/3906652369 conn(0x7f1af00fcb90 msgr2=0x7f1af006b0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 shutdown_connections 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:18.321+0000 7f1af5f9b700 1 -- 172.21.15.120:0/3906652369 wait complete. 2024-03-22T02:05:18.322 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 2 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: Deploying daemon mon.b on smithi083 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/3906652369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:18.861 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: Deploying daemon mon.b on smithi083 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/3906652369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:19.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:19.886 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T02:05:19.887 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T02:05:19.915 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:19 smithi120 ceph-mon[36803]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:19.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:19 smithi028 ceph-mon[31916]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:20.124 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:05:21.189 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 podman[36321]: 2024-03-22 02:05:20.931561533 +0000 UTC m=+0.017499364 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:05:21.189 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 podman[36321]: 2024-03-22 02:05:21.189904895 +0000 UTC m=+0.275842717 container create a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-22T02:05:21.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 podman[36321]: 2024-03-22 02:05:21.409329724 +0000 UTC m=+0.495267553 container init a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 podman[36321]: 2024-03-22 02:05:21.412209884 +0000 UTC m=+0.498147710 container start a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev), process ceph-mon, pid 2 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pidfile_write: ignore empty --pid-file 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: load: jerasure load: lrc 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: RocksDB version: 7.9.2 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Git sha 0 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Compile date 2024-03-21 19:33:57 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: DB SUMMARY 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: DB Session ID: 8878392Z8R2UH2B5AVBY 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: CURRENT file: CURRENT 2024-03-22T02:05:21.441 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.error_if_exists: 0 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.create_if_missing: 0 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.paranoid_checks: 1 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T02:05:21.442 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.env: 0x55724816bc60 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.info_log: 0x55724a861ae0 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.statistics: (nil) 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.use_fsync: 0 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_log_file_size: 0 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_fallocate: 1 2024-03-22T02:05:21.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T02:05:21.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.use_direct_reads: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.db_log_dir: 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.wal_dir: 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T02:05:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.write_buffer_manager: 0x55724a8321e0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.unordered_write: 0 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T02:05:21.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.row_cache: None 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.wal_filter: None 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.two_write_queues: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.wal_compression: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.atomic_flush: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T02:05:21.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.log_readahead_size: 0 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T02:05:21.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_background_jobs: 2 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_background_compactions: -1 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_subcompactions: 1 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T02:05:21.770 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_open_files: -1 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_background_flushes: -1 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Compression algorithms supported: 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kZSTD supported: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kXpressCompression supported: 0 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T02:05:21.771 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kZlibCompression supported: 1 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kSnappyCompression supported: 1 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kLZ4Compression supported: 1 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: kBZip2Compression supported: 0 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.merge_operator: 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_filter: None 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T02:05:21.772 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55724a861be0) 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: cache_index_and_filter_blocks: 1 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: pin_top_level_index_and_filter: 1 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: index_type: 0 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: data_block_index_type: 0 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: index_shortening: 1 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: checksum: 4 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: no_block_cache: 0 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_cache: 0x55724a865090 2024-03-22T02:05:21.773 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_cache_name: BinnedLRUCache 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_cache_options: 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: capacity : 536870912 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: num_shard_bits : 4 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: strict_capacity_limit : 0 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: high_pri_pool_ratio: 0.000 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_cache_compressed: (nil) 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: persistent_cache: (nil) 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_size: 4096 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_size_deviation: 10 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_restart_interval: 16 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: index_block_restart_interval: 1 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: metadata_block_size: 4096 2024-03-22T02:05:21.774 INFO:journalctl@ceph.mon.b.smithi083.stdout: partition_filters: 0 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: use_delta_encoding: 1 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: filter_policy: bloomfilter 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: whole_key_filtering: 1 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: verify_compression: 0 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: read_amp_bytes_per_bit: 0 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: format_version: 5 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: enable_index_compression: 1 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: block_align: 0 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: max_auto_readahead_size: 262144 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: prepopulate_block_cache: 0 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: initial_auto_readahead_size: 8192 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T02:05:21.775 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression: NoCompression 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.num_levels: 7 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T02:05:21.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T02:05:21.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T02:05:21.778 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T02:05:21.779 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.table_properties_collectors: 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.inplace_update_support: 0 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T02:05:21.780 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.bloom_locality: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.max_successive_merges: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.ttl: 2592000 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enable_blob_files: false 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.min_blob_size: 0 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T02:05:21.781 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3d6a4418-8156-4062-9d2f-f54c032c142a 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073121441858, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-22T02:05:21.782 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073121442308, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711073121, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3d6a4418-8156-4062-9d2f-f54c032c142a", "db_session_id": "8878392Z8R2UH2B5AVBY", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711073121442388, "job": 1, "event": "recovery_finished"} 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55724a954000 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: DB pointer 0x55724a93e000 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: ** DB Stats ** 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T02:05:21.783 INFO:journalctl@ceph.mon.b.smithi083.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: ** Compaction Stats [default] ** 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T02:05:21.784 INFO:journalctl@ceph.mon.b.smithi083.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T02:05:21.785 INFO:journalctl@ceph.mon.b.smithi083.stdout: Block cache BinnedLRUCache@0x55724a865090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: using public_addr v2:172.21.15.83:0/0 -> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: starting mon.b rank -1 at public addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] at bind addrs [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(???) e0 preinit fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).mds e1 new map 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: e1 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:05:21.786 INFO:journalctl@ceph.mon.b.smithi083.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout: legacy client fscid: -1 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout: 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout: No filesystems configured 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T02:05:21.787 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Deploying cephadm binary to smithi120 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi083", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:21.788 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi083:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Added host smithi120 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.789 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1335394797' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:21.790 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c;count:3 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.791 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Deploying daemon mon.c on smithi120 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/2348264538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.a calling monitor election 2024-03-22T02:05:21.792 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.c calling monitor election 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: monmap epoch 2 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: last_changed 2024-03-22T02:05:12.534027+0000 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:21.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: min_mon_release 19 (squid) 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: election_strategy: 1 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: fsmap 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mgrmap e13: x(active, since 51s) 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: overall HEALTH_OK 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:21.794 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: Deploying daemon mon.b on smithi083 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/3906652369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 ceph-mon[36340]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 bash[36321]: a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 2024-03-22T02:05:21.795 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:21 smithi083 systemd[1]: Started Ceph mon.b for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: mon.a calling monitor election 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: mon.c calling monitor election 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: mon.b calling monitor election 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: monmap epoch 3 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: last_changed 2024-03-22T02:05:21.464970+0000 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: min_mon_release 19 (squid) 2024-03-22T02:05:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: election_strategy: 1 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: 2: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: fsmap 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: mgrmap e13: x(active, since 60s) 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: overall HEALTH_OK 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:26.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:26.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: mon.a calling monitor election 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: mon.c calling monitor election 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: mon.b calling monitor election 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: monmap epoch 3 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: last_changed 2024-03-22T02:05:21.464970+0000 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: min_mon_release 19 (squid) 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: election_strategy: 1 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: 2: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: fsmap 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: mgrmap e13: x(active, since 60s) 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: overall HEALTH_OK 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: mon.a calling monitor election 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: mon.c calling monitor election 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: mon.b calling monitor election 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: monmap epoch 3 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: last_changed 2024-03-22T02:05:21.464970+0000 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: created 2024-03-22T02:03:20.403582+0000 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: min_mon_release 19 (squid) 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: election_strategy: 1 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: 1: [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] mon.c 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: 2: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.b 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: fsmap 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T02:05:27.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: mgrmap e13: x(active, since 60s) 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: overall HEALTH_OK 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:27.659 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3233357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 msgr2=0x7f9cb8008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3233357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9cb8008710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc069880 tx=0x7f9cbc00fe90 comp rx=0 tx=0).stop 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3233357109 shutdown_connections 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3233357109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9cb8008710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3233357109 >> 172.21.15.120:0/3233357109 conn(0x7f9ccc0fc790 msgr2=0x7f9ccc0feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3233357109 shutdown_connections 2024-03-22T02:05:27.660 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.659+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3233357109 wait complete. 2024-03-22T02:05:27.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 Processor -- start 2024-03-22T02:05:27.662 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 -- start start 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9ccc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc19ccc0 0x7f9ccc19d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc10a220 con 0x7f9cb80062a0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc10a0a0 con 0x7f9ccc19ccc0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd40b5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9ccc10a3a0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.660+0000 7f9cd1650700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:38086/0 (socket says 172.21.15.120:38086) 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 -- 172.21.15.120:0/3878114154 learned_addr learned my addr 172.21.15.120:0/3878114154 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1e51700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9ccc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd2652700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc19ccc0 0x7f9ccc19d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:27.663 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 -- 172.21.15.120:0/3878114154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc19ccc0 msgr2=0x7f9ccc19d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc19ccc0 0x7f9ccc19d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 -- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 msgr2=0x7f9ccc197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9ccc197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 -- 172.21.15.120:0/3878114154 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ccc1a4a30 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd1650700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800b780 tx=0x7f9cc800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8016070 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ccc1a4c60 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cc80036a0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8011af0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.661+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ccc1a51a0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.662+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f9cc8019070 con 0x7f9ccc1983f0 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.662+0000 7f9cc2ffd700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 0x7f9cb003bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:27.664 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.662+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9cc804b070 con 0x7f9ccc1983f0 2024-03-22T02:05:27.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.662+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ccc063cb0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.663+0000 7f9cd1e51700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 0x7f9cb003bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:27.665 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.663+0000 7f9cd1e51700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 0x7f9cb003bb70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc01f040 tx=0x7f9cbc029040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:27.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.668+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9cc8015170 con 0x7f9ccc1983f0 2024-03-22T02:05:27.747 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:27 smithi120 ceph-mon[36803]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:27 smithi083 ceph-mon[36340]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:27.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:27 smithi028 ceph-mon[31916]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:27.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:05:27.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.968+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f9ccc02dd70 con 0x7f9ccc1983f0 2024-03-22T02:05:27.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.968+0000 7f9cc2ffd700 1 -- 172.21.15.120:0/3878114154 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f9cc8019bb0 con 0x7f9ccc1983f0 2024-03-22T02:05:27.969 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-22T02:05:27.969 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":3,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","modified":"2024-03-22T02:05:21.464970Z","created":"2024-03-22T02:03:20.403582Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:3300","nonce":0},{"type":"v1","addr":"172.21.15.83:6789","nonce":0}]},"addr":"172.21.15.83:6789/0","public_addr":"172.21.15.83:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 msgr2=0x7f9cb003bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 0x7f9cb003bb70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc01f040 tx=0x7f9cbc029040 comp rx=0 tx=0).stop 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 msgr2=0x7f9ccc19c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cc800b780 tx=0x7f9cc800ba90 comp rx=0 tx=0).stop 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 shutdown_connections 2024-03-22T02:05:27.971 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9cb00396b0 0x7f9cb003bb70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cb80062a0 0x7f9ccc197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ccc19ccc0 0x7f9ccc19d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 --2- 172.21.15.120:0/3878114154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ccc1983f0 0x7f9ccc19c780 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 >> 172.21.15.120:0/3878114154 conn(0x7f9ccc0fc790 msgr2=0x7f9ccc0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.970+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 shutdown_connections 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:27.971+0000 7f9cd40b5700 1 -- 172.21.15.120:0/3878114154 wait complete. 2024-03-22T02:05:27.972 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 3 2024-03-22T02:05:28.683 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-22T02:05:28.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-22T02:05:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:05:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/3878114154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:29.003 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:29.003 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:29.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:29.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:29.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/3878114154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.047 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:29.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:29.049 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:29.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.953 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.953 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.953 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.953 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi028:/etc/ceph/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi083:/etc/ceph/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/3878114154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: Updating smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.954 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:29.955 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:29 smithi120 ceph-mon[36803]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:29 smithi120 ceph-mon[36803]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T02:05:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:29 smithi120 ceph-mon[36803]: Reconfiguring daemon mon.a on smithi028 2024-03-22T02:05:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:29 smithi083 ceph-mon[36340]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:29 smithi083 ceph-mon[36340]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T02:05:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:29 smithi083 ceph-mon[36340]: Reconfiguring daemon mon.a on smithi028 2024-03-22T02:05:30.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:29 smithi028 ceph-mon[31916]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:30.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:29 smithi028 ceph-mon[31916]: Reconfiguring mon.a (unknown last config time)... 2024-03-22T02:05:30.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:29 smithi028 ceph-mon[31916]: Reconfiguring daemon mon.a on smithi028 2024-03-22T02:05:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 -- 172.21.15.28:0/3228116837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08107a50 msgr2=0x7fba0810f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 --2- 172.21.15.28:0/3228116837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08107a50 0x7fba0810f660 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fba0000b0a0 tx=0x7fba0000b3b0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 -- 172.21.15.28:0/3228116837 shutdown_connections 2024-03-22T02:05:31.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 --2- 172.21.15.28:0/3228116837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08101280 0x7fba08101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 --2- 172.21.15.28:0/3228116837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08107a50 0x7fba0810f660 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 --2- 172.21.15.28:0/3228116837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 0x7fba08107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 -- 172.21.15.28:0/3228116837 >> 172.21.15.28:0/3228116837 conn(0x7fba08076900 msgr2=0x7fba08076d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 -- 172.21.15.28:0/3228116837 shutdown_connections 2024-03-22T02:05:31.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.419+0000 7fba0f49d700 1 -- 172.21.15.28:0/3228116837 wait complete. 2024-03-22T02:05:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 Processor -- start 2024-03-22T02:05:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 -- start start 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 0x7fba08102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08107a50 0x7fba08102c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba08112a90 con 0x7fba08107a50 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fba08112910 con 0x7fba08101280 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0f49d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fba08112c10 con 0x7fba08101c30 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:39328/0 (socket says 172.21.15.28:39328) 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 -- 172.21.15.28:0/891977324 learned_addr learned my addr 172.21.15.28:0/891977324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0da3a700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08107a50 0x7fba08102c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0ca38700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 0x7fba08102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 -- 172.21.15.28:0/891977324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 msgr2=0x7fba08102740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:31.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 0x7fba08102740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 -- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08107a50 msgr2=0x7fba08102c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08107a50 0x7fba08102c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 -- 172.21.15.28:0/891977324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba081a7400 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.420+0000 7fba0d239700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fba0400e9a0 tx=0x7fba0400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.421+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0400ca10 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.421+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba081a76f0 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.421+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba0400cbb0 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.421+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba04018680 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.421+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba081a7c30 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba08069340 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7fba04010430 con 0x7fba08101280 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fb9fe7fc700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 0x7fb9f403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fba0404c770 con 0x7fba08101280 2024-03-22T02:05:31.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fba0ca38700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 0x7fb9f403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:31.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.422+0000 7fba0ca38700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 0x7fb9f403b920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba08100a20 tx=0x7fb9f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:31.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.427+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba04014230 con 0x7fba08101280 2024-03-22T02:05:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.632+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fba0802dd70 con 0x7fba08101280 2024-03-22T02:05:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.633+0000 7fb9fe7fc700 1 -- 172.21.15.28:0/891977324 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fba04018870 con 0x7fba08101280 2024-03-22T02:05:31.634 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:31.634 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-03-22T02:05:31.634 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:05:31.634 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-03-22T02:05:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 msgr2=0x7fb9f403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 0x7fb9f403b920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fba08100a20 tx=0x7fb9f8009260 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 msgr2=0x7fba08102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fba0400e9a0 tx=0x7fba0400ecb0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 shutdown_connections 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9f4039460 0x7fb9f403b920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba08107a50 0x7fba08102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fba08101280 0x7fba08102200 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 --2- 172.21.15.28:0/891977324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fba08101c30 0x7fba08102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 >> 172.21.15.28:0/891977324 conn(0x7fba08076900 msgr2=0x7fba080fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 shutdown_connections 2024-03-22T02:05:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:31.635+0000 7fba0f49d700 1 -- 172.21.15.28:0/891977324 wait complete. 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: Reconfiguring mon.b (monmap changed)... 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: Reconfiguring daemon mon.b on smithi083 2024-03-22T02:05:32.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:31 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/891977324' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: Reconfiguring mon.b (monmap changed)... 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: Reconfiguring daemon mon.b on smithi083 2024-03-22T02:05:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:31 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/891977324' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: Reconfiguring mon.b (monmap changed)... 2024-03-22T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: Reconfiguring daemon mon.b on smithi083 2024-03-22T02:05:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:31 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/891977324' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:32.312 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-22T02:05:32.312 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:05:32.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T02:05:32.341 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:05:32.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:32.408 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:05:32.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T02:05:32.440 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:05:32.440 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:32.509 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:05:32.509 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T02:05:32.544 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:05:32.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T02:05:32.613 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-03-22T02:05:32.613 INFO:tasks.cephadm:Adding mgr.y on smithi083 2024-03-22T02:05:32.613 INFO:tasks.cephadm:Adding mgr.z on smithi120 2024-03-22T02:05:32.614 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi028=x;smithi083=y;smithi120=z' 2024-03-22T02:05:32.896 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:05:33.655 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:33.655 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:33.655 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:33.655 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: Reconfiguring mon.c (monmap changed)... 2024-03-22T02:05:33.656 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:33.656 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:33.656 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:33.656 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:33 smithi120 ceph-mon[36803]: Reconfiguring daemon mon.c on smithi120 2024-03-22T02:05:33.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:33.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: Reconfiguring mon.c (monmap changed)... 2024-03-22T02:05:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:33.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:33 smithi028 ceph-mon[31916]: Reconfiguring daemon mon.c on smithi120 2024-03-22T02:05:34.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: Reconfiguring mon.c (monmap changed)... 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T02:05:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:34.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:33 smithi083 ceph-mon[36340]: Reconfiguring daemon mon.c on smithi120 2024-03-22T02:05:35.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- 172.21.15.120:0/1407018907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc103130 msgr2=0x7f01dc105520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:35.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 --2- 172.21.15.120:0/1407018907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc103130 0x7f01dc105520 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f01cc009990 tx=0x7f01cc009ca0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- 172.21.15.120:0/1407018907 shutdown_connections 2024-03-22T02:05:35.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 --2- 172.21.15.120:0/1407018907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 0x7f01dc10a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.279 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 --2- 172.21.15.120:0/1407018907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc103130 0x7f01dc105520 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 --2- 172.21.15.120:0/1407018907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc105a60 0x7f01dc107e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- 172.21.15.120:0/1407018907 >> 172.21.15.120:0/1407018907 conn(0x7f01dc0fcb40 msgr2=0x7f01dc0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- 172.21.15.120:0/1407018907 shutdown_connections 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- 172.21.15.120:0/1407018907 wait complete. 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 Processor -- start 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.276+0000 7f01e21da700 1 -- start start 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 0x7f01dc1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 0x7f01dc19feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc10e2d0 con 0x7f01dc103130 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc10e150 con 0x7f01dc108390 2024-03-22T02:05:35.280 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01e21da700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc10e450 con 0x7f01dc105a60 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:38112/0 (socket says 172.21.15.120:38112) 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 -- 172.21.15.120:0/4251510790 learned_addr learned my addr 172.21.15.120:0/4251510790 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01db7fe700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 0x7f01dc1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01dbfff700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 0x7f01dc19feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 -- 172.21.15.120:0/4251510790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 msgr2=0x7f01dc19feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 0x7f01dc19feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 -- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 msgr2=0x7f01dc1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 0x7f01dc1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 -- 172.21.15.120:0/4251510790 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc1a0550 con 0x7f01dc105a60 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01db7fe700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 0x7f01dc1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.277+0000 7f01d2ffd700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f01c400d9b0 tx=0x7f01c400dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:35.281 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4016070 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc1a0840 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01c4004500 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4004d00 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc1a0d80 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.278+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f01c4005020 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.279+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc069340 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.279+0000 7f01d97fa700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 0x7f01c803b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.279+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f01c404c2f0 con 0x7f01dc105a60 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.279+0000 7f01db7fe700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 0x7f01c803b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:35.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.279+0000 7f01db7fe700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 0x7f01c803b920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005cd0 tx=0x7f01cc005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:35.284 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.284+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01c401f020 con 0x7f01dc105a60 2024-03-22T02:05:35.495 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.494+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi083=y;smithi120=z", "target": ["mon-mgr", ""]}) -- 0x7f01dc062b50 con 0x7f01c8039460 2024-03-22T02:05:35.501 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.501+0000 7f01d97fa700 1 -- 172.21.15.120:0/4251510790 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f01dc062b50 con 0x7f01c8039460 2024-03-22T02:05:35.501 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mgr update... 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 msgr2=0x7f01c803b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 0x7f01c803b920 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f01cc005cd0 tx=0x7f01cc005c60 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 msgr2=0x7f01dc198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f01c400d9b0 tx=0x7f01c400dd70 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 shutdown_connections 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f01c8039460 0x7f01c803b920 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01dc103130 0x7f01dc1988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f01dc108390 0x7f01dc19feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 --2- 172.21.15.120:0/4251510790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01dc105a60 0x7f01dc198e30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 >> 172.21.15.120:0/4251510790 conn(0x7f01dc0fcb40 msgr2=0x7f01dc103a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:35.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 shutdown_connections 2024-03-22T02:05:35.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:05:35.503+0000 7f01e21da700 1 -- 172.21.15.120:0/4251510790 wait complete. 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:35.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:35.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:35.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:35.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.142 DEBUG:teuthology.orchestra.run.smithi083:mgr.y> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.y.service 2024-03-22T02:05:36.145 DEBUG:teuthology.orchestra.run.smithi120:mgr.z> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.z.service 2024-03-22T02:05:36.147 INFO:tasks.cephadm:Deploying OSDs... 2024-03-22T02:05:36.148 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:05:36.148 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T02:05:36.166 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T02:05:36.167 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-03-22T02:05:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T02:05:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:36.225 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-22T02:05:36.225 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:36.225 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:36.226 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-22 02:04:39.348506829 +0000 2024-03-22T02:05:36.226 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-22 01:59:09.675240190 +0000 2024-03-22T02:05:36.226 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-22 01:59:09.675240190 +0000 2024-03-22T02:05:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-22 01:59:09.675240190 +0000 2024-03-22T02:05:36.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T02:05:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-22T02:05:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-22T02:05:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000121097 s, 4.2 MB/s 2024-03-22T02:05:36.297 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T02:05:36.354 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-03-22T02:05:36.411 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T02:05:36.411 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-22 02:04:39.383505816 +0000 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-22 01:59:10.000231033 +0000 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-22 01:59:10.000231033 +0000 2024-03-22T02:05:36.412 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-22 01:59:10.000231033 +0000 2024-03-22T02:05:36.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T02:05:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-22T02:05:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-22T02:05:36.480 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000117241 s, 4.4 MB/s 2024-03-22T02:05:36.481 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T02:05:36.540 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-03-22T02:05:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.587 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-22 02:04:39.413504949 +0000 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-22 01:59:10.325221875 +0000 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-22 01:59:10.325221875 +0000 2024-03-22T02:05:36.588 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-22 01:59:10.325221875 +0000 2024-03-22T02:05:36.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T02:05:36.637 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:36.637 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.637 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:36.637 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.637 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:36.638 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.638 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:36.638 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:36.638 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:36.638 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-22T02:05:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-22T02:05:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000107629 s, 4.8 MB/s 2024-03-22T02:05:36.656 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T02:05:36.713 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-03-22T02:05:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:36.768 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T02:05:36.768 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:36.768 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-22T02:05:36.768 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:36.769 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:36.769 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-22 02:04:39.441504139 +0000 2024-03-22T02:05:36.769 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-22 01:59:10.657212520 +0000 2024-03-22T02:05:36.769 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-22 01:59:10.657212520 +0000 2024-03-22T02:05:36.769 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-22 01:59:10.657212520 +0000 2024-03-22T02:05:36.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T02:05:36.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-03-22T02:05:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-03-22T02:05:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 9.4149e-05 s, 5.4 MB/s 2024-03-22T02:05:36.836 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T02:05:36.892 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-03-22T02:05:36.893 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T02:05:36.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T02:05:36.912 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-03-22T02:05:36.969 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-22 02:05:20.740459466 +0000 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-22 01:59:43.835218870 +0000 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-22 01:59:43.835218870 +0000 2024-03-22T02:05:36.970 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-22 01:59:43.835218870 +0000 2024-03-22T02:05:36.970 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T02:05:37.038 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-22T02:05:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-22T02:05:37.039 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000270503 s, 1.9 MB/s 2024-03-22T02:05:37.040 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T02:05:37.098 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-22 02:05:20.740459466 +0000 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-22 01:59:44.174210554 +0000 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-22 01:59:44.174210554 +0000 2024-03-22T02:05:37.155 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-22 01:59:44.174210554 +0000 2024-03-22T02:05:37.156 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T02:05:37.222 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-22T02:05:37.222 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-22T02:05:37.223 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000196244 s, 2.6 MB/s 2024-03-22T02:05:37.224 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T02:05:37.280 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-03-22T02:05:37.336 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T02:05:37.336 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:37.336 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-22T02:05:37.336 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:37.337 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:37.337 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-22 02:05:20.740459466 +0000 2024-03-22T02:05:37.337 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-22 01:59:44.529201845 +0000 2024-03-22T02:05:37.337 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-22 01:59:44.529201845 +0000 2024-03-22T02:05:37.337 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-22 01:59:44.529201845 +0000 2024-03-22T02:05:37.337 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T02:05:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-22T02:05:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-22T02:05:37.403 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000186399 s, 2.7 MB/s 2024-03-22T02:05:37.404 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T02:05:37.461 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-03-22 02:05:20.741459441 +0000 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-03-22 01:59:44.858193775 +0000 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-03-22 01:59:44.858193775 +0000 2024-03-22T02:05:37.516 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-03-22 01:59:44.858193775 +0000 2024-03-22T02:05:37.517 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T02:05:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-03-22T02:05:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-03-22T02:05:37.582 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000240208 s, 2.1 MB/s 2024-03-22T02:05:37.583 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T02:05:37.641 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-22T02:05:37.641 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T02:05:37.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T02:05:37.660 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-03-22T02:05:37.680 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:37 smithi120 ceph-mon[36803]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi083=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:37.680 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:37 smithi120 ceph-mon[36803]: Saving service mgr spec with placement smithi028=x;smithi083=y;smithi120=z;count:3 2024-03-22T02:05:37.680 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:37 smithi120 ceph-mon[36803]: Deploying daemon mgr.z on smithi120 2024-03-22T02:05:37.681 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:37 smithi120 systemd[1]: Starting Ceph mgr.z for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-22 02:05:37.544844132 +0000 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-22 01:59:32.231656745 +0000 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-22 01:59:32.231656745 +0000 2024-03-22T02:05:37.695 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-22 01:59:32.231656745 +0000 2024-03-22T02:05:37.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T02:05:37.763 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-22T02:05:37.763 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-22T02:05:37.763 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000257095 s, 2.0 MB/s 2024-03-22T02:05:37.764 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T02:05:37.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:37 smithi083 ceph-mon[36340]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi083=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:37.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:37 smithi083 ceph-mon[36340]: Saving service mgr spec with placement smithi028=x;smithi083=y;smithi120=z;count:3 2024-03-22T02:05:37.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:37 smithi083 ceph-mon[36340]: Deploying daemon mgr.z on smithi120 2024-03-22T02:05:37.819 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-03-22T02:05:37.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:37 smithi028 ceph-mon[31916]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi083=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:37.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:37 smithi028 ceph-mon[31916]: Saving service mgr spec with placement smithi028=x;smithi083=y;smithi120=z;count:3 2024-03-22T02:05:37.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:37 smithi028 ceph-mon[31916]: Deploying daemon mgr.z on smithi120 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-22 02:05:37.544844132 +0000 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-22 01:59:32.573649306 +0000 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-22 01:59:32.573649306 +0000 2024-03-22T02:05:37.875 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-22 01:59:32.573649306 +0000 2024-03-22T02:05:37.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T02:05:37.939 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:37 smithi120 podman[38474]: 2024-03-22 02:05:37.733428849 +0000 UTC m=+0.017658788 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:05:37.939 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:37 smithi120 podman[38474]: 2024-03-22 02:05:37.851235058 +0000 UTC m=+0.135464976 container create 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD) 2024-03-22T02:05:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-22T02:05:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-22T02:05:37.941 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000204245 s, 2.5 MB/s 2024-03-22T02:05:37.942 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T02:05:37.997 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-03-22T02:05:38.052 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T02:05:38.052 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:38.052 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-22T02:05:38.052 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:38.053 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:38.053 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-22 02:05:37.544844132 +0000 2024-03-22T02:05:38.053 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-22 01:59:32.917641823 +0000 2024-03-22T02:05:38.053 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-22 01:59:32.917641823 +0000 2024-03-22T02:05:38.053 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-22 01:59:32.917641823 +0000 2024-03-22T02:05:38.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T02:05:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-22T02:05:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-22T02:05:38.120 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000180663 s, 2.8 MB/s 2024-03-22T02:05:38.121 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T02:05:38.177 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-22 02:05:37.545844110 +0000 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-22 01:59:33.247634644 +0000 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-22 01:59:33.247634644 +0000 2024-03-22T02:05:38.232 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-22 01:59:33.247634644 +0000 2024-03-22T02:05:38.232 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T02:05:38.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 podman[38474]: 2024-03-22 02:05:38.086820931 +0000 UTC m=+0.371050859 container init 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-22T02:05:38.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 podman[38474]: 2024-03-22 02:05:38.09006672 +0000 UTC m=+0.374296647 container start 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-22T02:05:38.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mgr[38609]: -- 172.21.15.120:0/233942492 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd751365a0 con 0x55bd7510b000 2024-03-22T02:05:38.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 bash[38474]: 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 2024-03-22T02:05:38.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 systemd[1]: Started Ceph mgr.z for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:05:38.272 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-22T02:05:38.273 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-22T02:05:38.273 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000188992 s, 2.7 MB/s 2024-03-22T02:05:38.274 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T02:05:38.329 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-03-22T02:05:38.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T02:05:38.574 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:05:38.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:38.650 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:38.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T02:05:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:38.998 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:38.649+0000 7fa739e92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T02:05:38.998 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:38.742+0000 7fa739e92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T02:05:39.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:39 smithi083 ceph-mon[36340]: Deploying daemon mgr.y on smithi083 2024-03-22T02:05:39.606 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-mon[36803]: Deploying daemon mgr.y on smithi083 2024-03-22T02:05:39.607 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:39.375+0000 7fa739e92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T02:05:39.607 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:39.466+0000 7fa739e92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T02:05:39.607 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:39.606+0000 7fa739e92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T02:05:39.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:39 smithi028 ceph-mon[31916]: Deploying daemon mgr.y on smithi083 2024-03-22T02:05:39.998 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:39.700+0000 7fa739e92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T02:05:39.998 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:39.781+0000 7fa739e92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T02:05:40.333 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:40.049+0000 7fa739e92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T02:05:40.334 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:40.156+0000 7fa739e92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T02:05:40.334 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:40.240+0000 7fa739e92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T02:05:40.334 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:40.333+0000 7fa739e92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00830259 s, 1.3 GB/s 2024-03-22T02:05:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-22T02:05:40.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:40 smithi028 ceph-mon[31916]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-mon[36803]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:40.748 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:40.506+0000 7fa739e92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T02:05:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:40 smithi083 ceph-mon[36340]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:40.765 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:40 smithi083 systemd[1]: Started Ceph mgr.y for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:05:41.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-03-22T02:05:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:05:41.515 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:41.133+0000 7f977c1b0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T02:05:41.516 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:41.227+0000 7f977c1b0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T02:05:41.903 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:41.903 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:41.545+0000 7fa739e92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T02:05:41.904 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:41.673+0000 7fa739e92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T02:05:42.002 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:41.911+0000 7f977c1b0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.001+0000 7f977c1b0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:42.003 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:42.247 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:42.011+0000 7fa739e92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T02:05:42.248 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:42.100+0000 7fa739e92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T02:05:42.264 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.156+0000 7f977c1b0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T02:05:42.264 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.249+0000 7f977c1b0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T02:05:42.594 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.330+0000 7f977c1b0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T02:05:42.594 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.593+0000 7f977c1b0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T02:05:42.748 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:42.315+0000 7fa739e92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:05:42.748 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:42.499+0000 7fa739e92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 -- 172.21.15.28:0/2940707289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 msgr2=0x7f9c30102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 --2- 172.21.15.28:0/2940707289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30102af0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c0099c0 tx=0x7f9c2c009cd0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 -- 172.21.15.28:0/2940707289 shutdown_connections 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 --2- 172.21.15.28:0/2940707289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30102af0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 --2- 172.21.15.28:0/2940707289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c30104270 0x7f9c3010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 --2- 172.21.15.28:0/2940707289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c301038d0 0x7f9c30103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.842+0000 7f9c381fb700 1 -- 172.21.15.28:0/2940707289 >> 172.21.15.28:0/2940707289 conn(0x7f9c300fdc90 msgr2=0x7f9c301000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.843+0000 7f9c381fb700 1 -- 172.21.15.28:0/2940707289 shutdown_connections 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.843+0000 7f9c381fb700 1 -- 172.21.15.28:0/2940707289 wait complete. 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.843+0000 7f9c381fb700 1 Processor -- start 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.843+0000 7f9c381fb700 1 -- start start 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c301038d0 0x7f9c30074a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 0x7f9c30074fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3010e300 con 0x7f9c301026d0 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3010e180 con 0x7f9c30104270 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c381fb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3010e480 con 0x7f9c301038d0 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c36798700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 0x7f9c30074fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49662/0 (socket says 172.21.15.28:49662) 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 -- 172.21.15.28:0/3021189444 learned_addr learned my addr 172.21.15.28:0/3021189444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35796700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c301038d0 0x7f9c30074a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 -- 172.21.15.28:0/3021189444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c301038d0 msgr2=0x7f9c30074a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c301038d0 0x7f9c30074a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 -- 172.21.15.28:0/3021189444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 msgr2=0x7f9c30074fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:05:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 0x7f9c30074fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 -- 172.21.15.28:0/3021189444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c30075860 con 0x7f9c301026d0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c36798700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 0x7f9c30074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.844+0000 7f9c35f97700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c010 tx=0x7f9c2c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.845+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c026070 con 0x7f9c301026d0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.845+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c301a2a60 con 0x7f9c301026d0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.845+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c2c004150 con 0x7f9c301026d0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.845+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c00fde0 con 0x7f9c301026d0 2024-03-22T02:05:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.845+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c301a2eb0 con 0x7f9c301026d0 2024-03-22T02:05:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.846+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46350+0+0 (secure 0 0 0) 0x7f9c2c029070 con 0x7f9c301026d0 2024-03-22T02:05:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.846+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c30069340 con 0x7f9c301026d0 2024-03-22T02:05:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.846+0000 7f9c26ffd700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 0x7f9c1c03b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:05:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.846+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9c2c04d280 con 0x7f9c301026d0 2024-03-22T02:05:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.846+0000 7f9c35796700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 0x7f9c1c03b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:05:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.847+0000 7f9c35796700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 0x7f9c1c03b920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9c20007c60 tx=0x7f9c200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:05:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:42.855+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c2c013070 con 0x7f9c301026d0 2024-03-22T02:05:42.876 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.700+0000 7f977c1b0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T02:05:42.876 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.783+0000 7f977c1b0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T02:05:42.876 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:42.875+0000 7f977c1b0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T02:05:43.025 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:42.772+0000 7fa739e92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:05:43.026 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:43.025+0000 7fa739e92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T02:05:43.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:43.097+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f9c30062ac0 con 0x7f9c1c039460 2024-03-22T02:05:43.129 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:43.047+0000 7f977c1b0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T02:05:43.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:05:43.149 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:43.255+0000 7fa739e92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:05:43.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:05:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:44.264 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.002+0000 7f977c1b0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T02:05:44.264 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.134+0000 7f977c1b0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T02:05:44.342 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:44.037+0000 7fa739e92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T02:05:44.598 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: Reconfiguring daemon mgr.x on smithi028 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:44.599 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T02:05:44.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: Reconfiguring daemon mgr.x on smithi028 2024-03-22T02:05:44.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:44.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:44.729 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:44.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:44.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:44.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.742 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.463+0000 7f977c1b0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T02:05:44.742 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.551+0000 7f977c1b0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T02:05:44.742 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.741+0000 7f977c1b0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: Reconfiguring mgr.x (unknown last config time)... 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: Reconfiguring daemon mgr.x on smithi028 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:05:44.743 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:44.901 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:44.598+0000 7fa739e92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T02:05:44.901 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:44.692+0000 7fa739e92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T02:05:44.901 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:05:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z[38568]: 2024-03-22T02:05:44.900+0000 7fa739e92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T02:05:45.014 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:44.922+0000 7f977c1b0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T02:05:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:45.349+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 14) ==== 92074+0+0 (secure 0 0 0) 0x7f9c2c02bce0 con 0x7f9c301026d0 2024-03-22T02:05:45.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:45 smithi028 ceph-mon[31916]: Standby manager daemon z started 2024-03-22T02:05:45.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:45 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T02:05:45.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:45 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:45.391 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:45 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T02:05:45.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:45 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:45.458 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:45.201+0000 7f977c1b0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:45.457+0000 7f977c1b0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-mon[36340]: Standby manager daemon z started 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T02:05:45.459 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:45 smithi120 ceph-mon[36803]: Standby manager daemon z started 2024-03-22T02:05:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:45 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-22T02:05:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:45 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:45 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-22T02:05:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:45 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.120:0/616679267' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:45.764 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:45.690+0000 7f977c1b0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]: dispatch 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]': finished 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T02:05:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:46.480+0000 7f977c1b0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]: dispatch 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]': finished 2024-03-22T02:05:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T02:05:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]: dispatch 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4052005146' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edd58ef4-5561-4043-85e2-d205cb7c9396"}]': finished 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T02:05:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:05:47.343 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:47.037+0000 7f977c1b0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T02:05:47.343 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:47.132+0000 7f977c1b0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T02:05:47.343 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:05:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y[37762]: 2024-03-22T02:05:47.342+0000 7f977c1b0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T02:05:47.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:47 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1375066699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:05:47.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:47 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1375066699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:05:47.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:47 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1375066699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:05:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:05:48.366+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7f9c2c00f460 con 0x7f9c301026d0 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: Standby manager daemon y started 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T02:05:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:48 smithi120 ceph-mon[36803]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: Standby manager daemon y started 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T02:05:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:48 smithi083 ceph-mon[36340]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: Standby manager daemon y started 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T02:05:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:48 smithi028 ceph-mon[31916]: from='mgr.? 172.21.15.83:0/2931265791' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T02:05:49.675 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:49 smithi083 ceph-mon[36340]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-03-22T02:05:49.675 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T02:05:49.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:49 smithi120 ceph-mon[36803]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-03-22T02:05:49.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T02:05:49.748 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:49 smithi028 ceph-mon[31916]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-03-22T02:05:49.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T02:05:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:50 smithi120 ceph-mon[36803]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:50 smithi083 ceph-mon[36340]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:50 smithi028 ceph-mon[31916]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:52 smithi028 ceph-mon[31916]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:52 smithi120 ceph-mon[36803]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:52 smithi083 ceph-mon[36340]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:05:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:53.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:05:53.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:53.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T02:05:53.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:05:54.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:54 smithi028 ceph-mon[31916]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:54.381 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:54 smithi028 ceph-mon[31916]: Deploying daemon osd.0 on smithi028 2024-03-22T02:05:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:54 smithi120 ceph-mon[36803]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:54 smithi120 ceph-mon[36803]: Deploying daemon osd.0 on smithi028 2024-03-22T02:05:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:54 smithi083 ceph-mon[36340]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:54 smithi083 ceph-mon[36340]: Deploying daemon osd.0 on smithi028 2024-03-22T02:05:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:56 smithi028 ceph-mon[31916]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:56 smithi120 ceph-mon[36803]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:56 smithi083 ceph-mon[36340]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:58.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:58 smithi028 ceph-mon[31916]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:58 smithi120 ceph-mon[36803]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:58 smithi083 ceph-mon[36340]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:59 smithi120 ceph-mon[36803]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:05:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:59 smithi083 ceph-mon[36340]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:05:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:05:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:59 smithi028 ceph-mon[31916]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:05:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:05:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:05:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:02.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:02.082 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:02.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:02.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.126 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-03-22T02:06:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.126+0000 7f9c26ffd700 1 -- 172.21.15.28:0/3021189444 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9c30062ac0 con 0x7f9c1c039460 2024-03-22T02:06:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 msgr2=0x7f9c1c03b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 0x7f9c1c03b920 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9c20007c60 tx=0x7f9c200073d0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 msgr2=0x7f9c30076420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c010 tx=0x7f9c2c0057c0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 shutdown_connections 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9c1c039460 0x7f9c1c03b920 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c301026d0 0x7f9c30076420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c30104270 0x7f9c30074fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 --2- 172.21.15.28:0/3021189444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c301038d0 0x7f9c30074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 >> 172.21.15.28:0/3021189444 conn(0x7f9c300fdc90 msgr2=0x7f9c30106b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 shutdown_connections 2024-03-22T02:06:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:02.128+0000 7f9c381fb700 1 -- 172.21.15.28:0/3021189444 wait complete. 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:02.861 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.0.service 2024-03-22T02:06:02.864 INFO:tasks.cephadm:Deploying osd.1 on smithi083 with /dev/vg_nvme/lv_4... 2024-03-22T02:06:02.864 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T02:06:03.105 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:06:03.229 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.229 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.229 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:03.229 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:03 smithi083 ceph-mon[36340]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:06:03.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.430 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:03.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:03 smithi028 ceph-mon[31916]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:06:03.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:03.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:03.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:03 smithi120 ceph-mon[36803]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T02:06:04.212 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:04 smithi083 ceph-mon[36340]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:04.212 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:04 smithi083 ceph-mon[36340]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:06:04.213 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:04 smithi083 ceph-mon[36340]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T02:06:04.213 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:04 smithi083 ceph-mon[36340]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:06:04.213 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:04.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-mon[31916]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:04.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-mon[31916]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:06:04.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-mon[31916]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T02:06:04.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-mon[31916]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:06:04.398 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:04 smithi120 ceph-mon[36803]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:04 smithi120 ceph-mon[36803]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T02:06:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:04 smithi120 ceph-mon[36803]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T02:06:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:04 smithi120 ceph-mon[36803]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:06:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T02:06:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T02:06:05.012 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-22T02:06:05.012 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-22T02:06:05.012 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121879 s, 860 MB/s 2024-03-22T02:06:05.013 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-22T02:06:05.072 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0[46820]: 2024-03-22T02:06:04.676+0000 7f894d18c700 -1 osd.0 0 waiting for initial osdmap 2024-03-22T02:06:05.072 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0[46820]: 2024-03-22T02:06:04.679+0000 7f8944776700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:06:05.304 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:05 smithi083 ceph-mon[36340]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:06:05.305 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:05 smithi083 ceph-mon[36340]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T02:06:05.305 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.305 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.305 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:05 smithi083 ceph-mon[36340]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' 2024-03-22T02:06:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:05 smithi120 ceph-mon[36803]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:06:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:05 smithi120 ceph-mon[36803]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T02:06:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:05 smithi120 ceph-mon[36803]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' 2024-03-22T02:06:05.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:05 smithi028 ceph-mon[31916]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:06:05.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:05 smithi028 ceph-mon[31916]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T02:06:05.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:05.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:05 smithi028 ceph-mon[31916]: from='osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761]' entity='osd.0' 2024-03-22T02:06:05.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_4 2024-03-22T02:06:05.936 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] boot 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:06.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.330 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:06:06.330 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:06:06.330 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:06.330 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.330 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] boot 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:06.331 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:06:06.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:06:06.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T02:06:06.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: osd.0 [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] boot 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:06.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:07.413 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:07 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi028 2024-03-22T02:06:07.413 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:07 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 17093M 2024-03-22T02:06:07.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:07 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi028 2024-03-22T02:06:07.461 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:07 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 17093M 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 -- 172.21.15.83:0/2921709292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c1038a0 msgr2=0x7ff27c105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 --2- 172.21.15.83:0/2921709292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c1038a0 0x7ff27c105d30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009990 tx=0x7ff26c009ca0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 -- 172.21.15.83:0/2921709292 shutdown_connections 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 --2- 172.21.15.83:0/2921709292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 --2- 172.21.15.83:0/2921709292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c100f30 0x7ff27c103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 --2- 172.21.15.83:0/2921709292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c1038a0 0x7ff27c105d30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.480 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.479+0000 7ff283331700 1 -- 172.21.15.83:0/2921709292 >> 172.21.15.83:0/2921709292 conn(0x7ff27c0fcb50 msgr2=0x7ff27c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:07.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- 172.21.15.83:0/2921709292 shutdown_connections 2024-03-22T02:06:07.481 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- 172.21.15.83:0/2921709292 wait complete. 2024-03-22T02:06:07.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 Processor -- start 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- start start 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c1038a0 0x7ff27c19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1a6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c1147f0 con 0x7ff27c106270 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c114670 con 0x7ff27c100f30 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.480+0000 7ff283331700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c114970 con 0x7ff27c1038a0 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:42300/0 (socket says 172.21.15.83:42300) 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 -- 172.21.15.83:0/1830079688 learned_addr learned my addr 172.21.15.83:0/1830079688 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2808cc700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c1038a0 0x7ff27c19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:07.484 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2818ce700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1a6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 -- 172.21.15.83:0/1830079688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c1038a0 msgr2=0x7ff27c19f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c1038a0 0x7ff27c19f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 -- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 msgr2=0x7ff27c1a6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1a6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 -- 172.21.15.83:0/1830079688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c1a6ad0 con 0x7ff27c100f30 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2818ce700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1a6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2810cd700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff27800e9a0 tx=0x7ff27800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27800cbd0 con 0x7ff27c100f30 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff27800cd70 con 0x7ff27c100f30 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27c1a6dc0 con 0x7ff27c100f30 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2780187a0 con 0x7ff27c100f30 2024-03-22T02:06:07.485 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.481+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27c1a7220 con 0x7ff27c100f30 2024-03-22T02:06:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.482+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27c02dd70 con 0x7ff27c100f30 2024-03-22T02:06:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.483+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7ff2780109a0 con 0x7ff27c100f30 2024-03-22T02:06:07.486 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.484+0000 7ff2727fc700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 0x7ff2680a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.484+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7ff2780cdaa0 con 0x7ff27c100f30 2024-03-22T02:06:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.484+0000 7ff2808cc700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 0x7ff2680a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:07.487 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.485+0000 7ff2808cc700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 0x7ff2680a5460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00b7b0 tx=0x7ff26c00b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:07.488 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.489+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff27809c340 con 0x7ff27c100f30 2024-03-22T02:06:07.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:07 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi028 2024-03-22T02:06:07.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:07 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 17093M 2024-03-22T02:06:07.690 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:07.690+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7ff27c0634c0 con 0x7ff2680a2fa0 2024-03-22T02:06:08.471 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:08 smithi083 ceph-mon[36340]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:08.472 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:08 smithi083 ceph-mon[36340]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T02:06:08.472 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:08.472 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:08.472 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:08 smithi120 ceph-mon[36803]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:08 smithi120 ceph-mon[36803]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T02:06:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:08 smithi028 ceph-mon[31916]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:08 smithi028 ceph-mon[31916]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T02:06:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:09.478 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:09 smithi083 ceph-mon[36340]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:09 smithi120 ceph-mon[36803]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:09.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:09 smithi028 ceph-mon[31916]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:10 smithi120 ceph-mon[36803]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:10.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:10 smithi083 ceph-mon[36340]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:10 smithi028 ceph-mon[31916]: pgmap v50: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/732799690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]': finished 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/226827837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/732799690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]': finished 2024-03-22T02:06:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T02:06:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:11.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/226827837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/732799690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]: dispatch 2024-03-22T02:06:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd32d180-5e18-4813-8e6d-74ed124ee68f"}]': finished 2024-03-22T02:06:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T02:06:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:11.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/226827837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:12 smithi028 ceph-mon[31916]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:12 smithi120 ceph-mon[36803]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:12 smithi083 ceph-mon[36340]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:14.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:14 smithi083 ceph-mon[36340]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:14.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:14 smithi028 ceph-mon[31916]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:14 smithi120 ceph-mon[36803]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:16.417 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:16 smithi083 ceph-mon[36340]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:16 smithi028 ceph-mon[31916]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:16 smithi120 ceph-mon[36803]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:17.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:06:17.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:17.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:06:17.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:17.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T02:06:17.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:18 smithi120 ceph-mon[36803]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:18 smithi120 ceph-mon[36803]: Deploying daemon osd.1 on smithi083 2024-03-22T02:06:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:18 smithi028 ceph-mon[31916]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:18 smithi028 ceph-mon[31916]: Deploying daemon osd.1 on smithi083 2024-03-22T02:06:18.684 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:18 smithi083 ceph-mon[36340]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:18.685 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:18 smithi083 ceph-mon[36340]: Deploying daemon osd.1 on smithi083 2024-03-22T02:06:20.427 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:20 smithi083 ceph-mon[36340]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:20.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:20 smithi028 ceph-mon[31916]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:20 smithi120 ceph-mon[36803]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:22.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:22 smithi083 ceph-mon[36340]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:22.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:22 smithi028 ceph-mon[31916]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:22 smithi120 ceph-mon[36803]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:23.684 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.684 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.684 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:23.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:23.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:24 smithi083 ceph-mon[36340]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:24 smithi120 ceph-mon[36803]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:24.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:24 smithi028 ceph-mon[31916]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:25.907 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 1 on host 'smithi083' 2024-03-22T02:06:25.908 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.906+0000 7ff2727fc700 1 -- 172.21.15.83:0/1830079688 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff27c0634c0 con 0x7ff2680a2fa0 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 msgr2=0x7ff2680a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 0x7ff2680a5460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00b7b0 tx=0x7ff26c00b880 comp rx=0 tx=0).stop 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 msgr2=0x7ff27c19ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff27800e9a0 tx=0x7ff27800ecb0 comp rx=0 tx=0).stop 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 shutdown_connections 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2680a2fa0 0x7ff2680a5460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:25.909 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27c106270 0x7ff27c1a6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:25.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff27c100f30 0x7ff27c19ee70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:25.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 --2- 172.21.15.83:0/1830079688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff27c1038a0 0x7ff27c19f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:25.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 >> 172.21.15.83:0/1830079688 conn(0x7ff27c0fcb50 msgr2=0x7ff27c101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:25.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 shutdown_connections 2024-03-22T02:06:25.910 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:06:25.908+0000 7ff283331700 1 -- 172.21.15.83:0/1830079688 wait complete. 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.230 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.231 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.231 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:26.578 DEBUG:teuthology.orchestra.run.smithi083:osd.1> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.1.service 2024-03-22T02:06:26.580 INFO:tasks.cephadm:Deploying osd.2 on smithi120 with /dev/vg_nvme/lv_4... 2024-03-22T02:06:26.580 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T02:06:26.825 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:06:27.239 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:27 smithi120 ceph-mon[36803]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:27.239 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:27 smithi120 ceph-mon[36803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:27.419 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:27 smithi083 ceph-mon[36340]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:27.419 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:27 smithi083 ceph-mon[36340]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:27.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:27 smithi028 ceph-mon[31916]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:27.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:27 smithi028 ceph-mon[31916]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T02:06:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:06:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T02:06:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:28 smithi120 ceph-mon[36803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:06:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T02:06:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-mon[36340]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.515 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1[40978]: 2024-03-22T02:06:28.137+0000 7f3cb89e5700 -1 osd.1 0 waiting for initial osdmap 2024-03-22T02:06:28.515 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1[40978]: 2024-03-22T02:06:28.141+0000 7f3cb47d8700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:06:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: pgmap v60: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T02:06:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: from='osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T02:06:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:28.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:28 smithi028 ceph-mon[31916]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112509 s, 932 MB/s 2024-03-22T02:06:28.709 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-22T02:06:29.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:29 smithi120 ceph-mon[36803]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:06:29.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:29 smithi120 ceph-mon[36803]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T02:06:29.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:29.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:29.275 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_4 2024-03-22T02:06:29.436 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:29 smithi083 ceph-mon[36340]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:06:29.436 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:29 smithi083 ceph-mon[36340]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T02:06:29.437 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:29.437 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:29.514 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:06:29.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:29 smithi028 ceph-mon[31916]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:06:29.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:29 smithi028 ceph-mon[31916]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T02:06:29.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:29.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:30.480 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:06:30.480 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:06:30.480 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:30.480 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] boot 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi083 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:30.481 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:30.482 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] boot 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi083 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:30.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:30.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:06:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:06:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: pgmap v63: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-22T02:06:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: osd.1 [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] boot 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi083 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:31.068 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 -- 172.21.15.120:0/3338359690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc48103150 msgr2=0x7fcc48109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 --2- 172.21.15.120:0/3338359690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc48103150 0x7fcc48109a00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000b0a0 tx=0x7fcc4000b3b0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 -- 172.21.15.120:0/3338359690 shutdown_connections 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 --2- 172.21.15.120:0/3338359690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc48103150 0x7fcc48109a00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 --2- 172.21.15.120:0/3338359690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481015b0 0x7fcc481019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 --2- 172.21.15.120:0/3338359690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc481027b0 0x7fcc48102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 -- 172.21.15.120:0/3338359690 >> 172.21.15.120:0/3338359690 conn(0x7fcc480fcb50 msgr2=0x7fcc480fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 -- 172.21.15.120:0/3338359690 shutdown_connections 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 -- 172.21.15.120:0/3338359690 wait complete. 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.066+0000 7fcc4d182700 1 Processor -- start 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 -- start start 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 0x7fcc48103b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:31.069 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 0x7fcc481040d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:31.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4810d220 con 0x7fcc48103150 2024-03-22T02:06:31.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4810d0a0 con 0x7fcc481027b0 2024-03-22T02:06:31.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4d182700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4810d3a0 con 0x7fcc481015b0 2024-03-22T02:06:31.070 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:57490/0 (socket says 172.21.15.120:57490) 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 -- 172.21.15.120:0/1404777725 learned_addr learned my addr 172.21.15.120:0/1404777725 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4659c700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 0x7fcc48103b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4759e700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 0x7fcc481040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 -- 172.21.15.120:0/1404777725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 msgr2=0x7fcc48103b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 0x7fcc48103b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 -- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 msgr2=0x7fcc481040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 0x7fcc481040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:31.071 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 -- 172.21.15.120:0/1404777725 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc48104980 con 0x7fcc481015b0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4659c700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 0x7fcc48103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc4759e700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 0x7fcc481040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.067+0000 7fcc46d9d700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800b500 tx=0x7fcc3800b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.068+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38015020 con 0x7fcc481015b0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.068+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc38011d10 con 0x7fcc481015b0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.068+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc481a2a30 con 0x7fcc481015b0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.068+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc3800f2e0 con 0x7fcc481015b0 2024-03-22T02:06:31.072 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.068+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc481a2ff0 con 0x7fcc481015b0 2024-03-22T02:06:31.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.069+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc48069370 con 0x7fcc481015b0 2024-03-22T02:06:31.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.070+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7fcc38018020 con 0x7fcc481015b0 2024-03-22T02:06:31.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.070+0000 7fcc2ffff700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 0x7fcc300a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:31.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.071+0000 7fcc4659c700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 0x7fcc300a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:31.073 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.071+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fcc380cd930 con 0x7fcc481015b0 2024-03-22T02:06:31.074 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.071+0000 7fcc4659c700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 0x7fcc300a5460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c005fd0 tx=0x7fcc3c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:31.075 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.074+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc48069370 con 0x7fcc481015b0 2024-03-22T02:06:31.282 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:31.281+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fcc480fec50 con 0x7fcc300a2fa0 2024-03-22T02:06:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:31 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 17080M 2024-03-22T02:06:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:31 smithi120 ceph-mon[36803]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:31 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 17080M 2024-03-22T02:06:31.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:31 smithi083 ceph-mon[36340]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:31 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 17080M 2024-03-22T02:06:31.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:31 smithi028 ceph-mon[31916]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T02:06:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:32 smithi120 ceph-mon[36803]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:32 smithi120 ceph-mon[36803]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:32.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:32 smithi083 ceph-mon[36340]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:32.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:32 smithi083 ceph-mon[36340]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:32.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:32.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:32.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:32 smithi028 ceph-mon[31916]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:32 smithi028 ceph-mon[31916]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/404761238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]': finished 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T02:06:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:34.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:34.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/404761238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]': finished 2024-03-22T02:06:34.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T02:06:34.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: pgmap v67: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/404761238' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]: dispatch 2024-03-22T02:06:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9244c70-b1b3-42f4-a4e9-285e6933cc9b"}]': finished 2024-03-22T02:06:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T02:06:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:35.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:35 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/3966093696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:35.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:35 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/3966093696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:35.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:35 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/3966093696' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:36.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:36 smithi120 ceph-mon[36803]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:36.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:36 smithi083 ceph-mon[36340]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:36 smithi028 ceph-mon[31916]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:38 smithi120 ceph-mon[36803]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:38 smithi083 ceph-mon[36340]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:38 smithi028 ceph-mon[31916]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:40.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:40 smithi120 ceph-mon[36803]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:40.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:40 smithi083 ceph-mon[36340]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:40.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:40 smithi028 ceph-mon[31916]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:41.461 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:06:41.461 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:41.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:06:41.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:41.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T02:06:41.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:42.396 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:42 smithi120 ceph-mon[36803]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:42.396 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:42 smithi120 ceph-mon[36803]: Deploying daemon osd.2 on smithi120 2024-03-22T02:06:42.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:42 smithi083 ceph-mon[36340]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:42.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:42 smithi083 ceph-mon[36340]: Deploying daemon osd.2 on smithi120 2024-03-22T02:06:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:42 smithi028 ceph-mon[31916]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:42 smithi028 ceph-mon[31916]: Deploying daemon osd.2 on smithi120 2024-03-22T02:06:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:44 smithi120 ceph-mon[36803]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:44 smithi083 ceph-mon[36340]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:44 smithi028 ceph-mon[31916]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:46.438 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:46 smithi120 ceph-mon[36803]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:46.439 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.439 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.439 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:46 smithi083 ceph-mon[36340]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:46 smithi028 ceph-mon[31916]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:48.411 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:48 smithi120 ceph-mon[36803]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:48.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:48 smithi083 ceph-mon[36340]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:48 smithi028 ceph-mon[31916]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:49.496 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.496+0000 7fcc2ffff700 1 -- 172.21.15.120:0/1404777725 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc480fec50 con 0x7fcc300a2fa0 2024-03-22T02:06:49.497 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 2 on host 'smithi120' 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 msgr2=0x7fcc300a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 0x7fcc300a5460 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c005fd0 tx=0x7fcc3c005e30 comp rx=0 tx=0).stop 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 msgr2=0x7fcc48105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc3800b500 tx=0x7fcc3800b8c0 comp rx=0 tx=0).stop 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 shutdown_connections 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc300a2fa0 0x7fcc300a5460 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc48103150 0x7fcc481040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:49.499 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc481015b0 0x7fcc48105540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:49.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 --2- 172.21.15.120:0/1404777725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc481027b0 0x7fcc48103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:49.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 >> 172.21.15.120:0/1404777725 conn(0x7fcc480fcb50 msgr2=0x7fcc480fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:49.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 shutdown_connections 2024-03-22T02:06:49.500 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:06:49.499+0000 7fcc4d182700 1 -- 172.21.15.120:0/1404777725 wait complete. 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:49.718 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:50.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: pgmap v76: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:50.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:06:50.135 DEBUG:teuthology.orchestra.run.smithi120:osd.2> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.2.service 2024-03-22T02:06:50.138 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-03-22T02:06:50.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T02:06:50.389 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:06:50.747 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:50 smithi120 ceph-mon[36803]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:50 smithi120 ceph-mon[36803]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:50.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:50 smithi028 ceph-mon[31916]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:50.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:50 smithi028 ceph-mon[31916]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:51.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:50 smithi083 ceph-mon[36340]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:51.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:50 smithi083 ceph-mon[36340]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T02:06:51.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:51.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:06:51.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:51.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T02:06:51.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:51.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:51 smithi028 ceph-mon[31916]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:51.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:51.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:06:51.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:51.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T02:06:51.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:51.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-mon[36803]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:51.999 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2[41989]: 2024-03-22T02:06:51.588+0000 7fe92e91e700 -1 osd.2 0 waiting for initial osdmap 2024-03-22T02:06:51.999 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2[41989]: 2024-03-22T02:06:51.592+0000 7fe92a711700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:06:52.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: pgmap v77: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:52.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T02:06:52.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: from='osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:52.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T02:06:52.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:52.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:51 smithi083 ceph-mon[36340]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00858914 s, 1.2 GB/s 2024-03-22T02:06:52.460 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-22T02:06:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:52 smithi028 ceph-mon[31916]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:06:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:52 smithi028 ceph-mon[31916]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T02:06:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:52.853 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:52.960 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:52 smithi120 ceph-mon[36803]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:06:52.960 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:52 smithi120 ceph-mon[36803]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T02:06:52.960 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:52.960 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:52 smithi083 ceph-mon[36340]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:06:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:52 smithi083 ceph-mon[36340]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T02:06:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:53.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-03-22T02:06:53.366 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:06:53.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] boot 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi120 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:53.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:53.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:53 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:06:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:06:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] boot 2024-03-22T02:06:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi120 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:53.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:53 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:54.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:06:54.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:06:54.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: pgmap v80: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: osd.2 [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] boot 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi120 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:06:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:53 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:54 smithi028 ceph-mon[31916]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T02:06:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:54 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 -- 172.21.15.28:0/766082414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0101650 msgr2=0x7f06b0101a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 --2- 172.21.15.28:0/766082414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0101650 0x7f06b0101a70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06a00099c0 tx=0x7f06a0009cd0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 -- 172.21.15.28:0/766082414 shutdown_connections 2024-03-22T02:06:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 --2- 172.21.15.28:0/766082414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0101650 0x7f06b0101a70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 --2- 172.21.15.28:0/766082414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b01031f0 0x7f06b0109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 --2- 172.21.15.28:0/766082414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06b0102850 0x7f06b0102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 -- 172.21.15.28:0/766082414 >> 172.21.15.28:0/766082414 conn(0x7f06b00fcc10 msgr2=0x7f06b00ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 -- 172.21.15.28:0/766082414 shutdown_connections 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.936+0000 7f06b5950700 1 -- 172.21.15.28:0/766082414 wait complete. 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 Processor -- start 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 -- start start 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b0101650 0x7f06b01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06b0102850 0x7f06b0198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:54.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010d1c0 con 0x7f06b01031f0 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010d040 con 0x7f06b0101650 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06b5950700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010d340 con 0x7f06b0102850 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06af7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06af7fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35736/0 (socket says 172.21.15.28:35736) 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.937+0000 7f06af7fe700 1 -- 172.21.15.28:0/1985002895 learned_addr learned my addr 172.21.15.28:0/1985002895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06aeffd700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b0101650 0x7f06b01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:54.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 -- 172.21.15.28:0/1985002895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06b0102850 msgr2=0x7f06b0198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06b0102850 0x7f06b0198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 -- 172.21.15.28:0/1985002895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b0101650 msgr2=0x7f06b01988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b0101650 0x7f06b01988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 -- 172.21.15.28:0/1985002895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b01a0540 con 0x7f06b01031f0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.938+0000 7f06af7fe700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06a800add0 tx=0x7f06a800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.939+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8017020 con 0x7f06b01031f0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.939+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06a800ce20 con 0x7f06b01031f0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.939+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b01a0830 con 0x7f06b01031f0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.939+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a8005520 con 0x7f06b01031f0 2024-03-22T02:06:54.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.939+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01a0d70 con 0x7f06b01031f0 2024-03-22T02:06:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.940+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b0069340 con 0x7f06b01031f0 2024-03-22T02:06:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.941+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 137798+0+0 (secure 0 0 0) 0x7f06a8004ad0 con 0x7f06b01031f0 2024-03-22T02:06:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.941+0000 7f06b494e700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 0x7f06980a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:06:54.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.941+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f06a80cd640 con 0x7f06b01031f0 2024-03-22T02:06:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.941+0000 7f06aeffd700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 0x7f06980a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:06:54.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.942+0000 7f06aeffd700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 0x7f06980a5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06a000b580 tx=0x7f06a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:06:54.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:54.946+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f06a8021020 con 0x7f06b01031f0 2024-03-22T02:06:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:54 smithi120 ceph-mon[36803]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T02:06:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:54 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:55.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:54 smithi083 ceph-mon[36340]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T02:06:55.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:54 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:55.162+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f06b00fedc0 con 0x7f06980a2fa0 2024-03-22T02:06:55.891 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51368]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:55.892 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51368]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.892 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51368]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.892 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51368]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.892 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51371]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:55.892 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51371]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.893 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51371]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.893 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51371]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:55.893 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:55.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:55.928 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:55.928 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:55.928 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T02:06:55.929 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:55.929 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:55.929 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:55.929 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:55.929 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:55.930 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45761]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:55.930 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45761]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.930 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45761]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.930 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45761]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.930 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45764]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:55.931 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45764]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.931 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45764]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.931 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45764]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44855]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44855]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44855]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:55.964 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:55.965 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44858]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:55.965 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44858]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: pgmap v83: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:06:55.965 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51374]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51377]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51377]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:55 smithi028 sudo[51377]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45767]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45767]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45767]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45767]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45770]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45770]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45770]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:55 smithi120 sudo[45770]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44861]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T02:06:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44861]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44861]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44861]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T02:06:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-22T02:06:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-22T02:06:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:55 smithi083 sudo[44864]: pam_unix(sudo:session): session closed for user root 2024-03-22T02:06:56.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:56.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:56.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:56.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:56 smithi028 ceph-mon[31916]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:57.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:57.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:57.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:57.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:57.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:56 smithi120 ceph-mon[36803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:57.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T02:06:57.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:56 smithi083 ceph-mon[36340]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T02:06:57.891 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:57 smithi028 ceph-mon[31916]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:57.892 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:57 smithi028 ceph-mon[31916]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T02:06:57.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:06:57.890+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f06a8097930 con 0x7f06b01031f0 2024-03-22T02:06:57.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:57 smithi120 ceph-mon[36803]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:57.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:57 smithi120 ceph-mon[36803]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T02:06:58.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:57 smithi083 ceph-mon[36340]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:06:58.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:57 smithi083 ceph-mon[36340]: osdmap e22: 3 total, 3 up, 3 in 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]: dispatch 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]': finished 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T02:06:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1850139033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]: dispatch 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]': finished 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T02:06:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1850139033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:06:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]: dispatch 2024-03-22T02:06:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/160886980' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "242938b2-bb3e-4a38-8c8e-7491101ee0bd"}]': finished 2024-03-22T02:06:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T02:06:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:06:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-22T02:06:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1850139033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:06:59 smithi120 ceph-mon[36803]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:00.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:06:59 smithi083 ceph-mon[36340]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:06:59 smithi028 ceph-mon[31916]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:02.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:01 smithi028 ceph-mon[31916]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:01 smithi120 ceph-mon[36803]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:02.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:01 smithi083 ceph-mon[36340]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:04.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:03 smithi028 ceph-mon[31916]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:03 smithi120 ceph-mon[36803]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:04.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:03 smithi083 ceph-mon[36340]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:06.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:05 smithi028 ceph-mon[31916]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:06.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:07:06.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:06.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:05 smithi028 ceph-mon[31916]: Deploying daemon osd.3 on smithi028 2024-03-22T02:07:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:05 smithi120 ceph-mon[36803]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:07:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:05 smithi120 ceph-mon[36803]: Deploying daemon osd.3 on smithi028 2024-03-22T02:07:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:05 smithi083 ceph-mon[36340]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T02:07:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:05 smithi083 ceph-mon[36340]: Deploying daemon osd.3 on smithi028 2024-03-22T02:07:08.057 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:07 smithi028 ceph-mon[31916]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:07 smithi120 ceph-mon[36803]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:08.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:07 smithi083 ceph-mon[36340]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:10.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:09 smithi028 ceph-mon[31916]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:09 smithi120 ceph-mon[36803]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:10.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:09 smithi083 ceph-mon[36340]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:11 smithi028 ceph-mon[31916]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:11 smithi120 ceph-mon[36803]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:11 smithi083 ceph-mon[36340]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:13 smithi120 ceph-mon[36803]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:14.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:13 smithi083 ceph-mon[36340]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:14.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:13 smithi028 ceph-mon[31916]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:14.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.118+0000 7f06b494e700 1 -- 172.21.15.28:0/1985002895 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f06b00fedc0 con 0x7f06980a2fa0 2024-03-22T02:07:14.119 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.120+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 msgr2=0x7f06980a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.120+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 0x7f06980a5460 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f06a000b580 tx=0x7f06a0005fb0 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.120+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 msgr2=0x7f06b019fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.120+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f06a800add0 tx=0x7f06a800c330 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 shutdown_connections 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06980a2fa0 0x7f06980a5460 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b01031f0 0x7f06b019fea0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f06b0101650 0x7f06b01988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 --2- 172.21.15.28:0/1985002895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06b0102850 0x7f06b0198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 >> 172.21.15.28:0/1985002895 conn(0x7f06b00fcc10 msgr2=0x7f06b00fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 shutdown_connections 2024-03-22T02:07:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:07:14.121+0000 7f06b5950700 1 -- 172.21.15.28:0/1985002895 wait complete. 2024-03-22T02:07:14.821 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.3.service 2024-03-22T02:07:14.823 INFO:tasks.cephadm:Deploying osd.4 on smithi083 with /dev/vg_nvme/lv_3... 2024-03-22T02:07:14.823 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T02:07:15.070 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:07:15.191 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.191 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.191 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:15 smithi083 ceph-mon[36340]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:07:15.192 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.192 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:15 smithi028 ceph-mon[31916]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:07:15.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:15 smithi120 ceph-mon[36803]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T02:07:15.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:15.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:16.348 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:16 smithi083 ceph-mon[36340]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:16.349 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:16 smithi083 ceph-mon[36340]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:07:16.349 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:16 smithi083 ceph-mon[36340]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T02:07:16.349 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:16 smithi083 ceph-mon[36340]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:07:16.349 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:16 smithi120 ceph-mon[36803]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:16 smithi120 ceph-mon[36803]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:07:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:16 smithi120 ceph-mon[36803]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T02:07:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:16 smithi120 ceph-mon[36803]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:07:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:16.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-mon[31916]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:16.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-mon[31916]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T02:07:16.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-mon[31916]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T02:07:16.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-mon[31916]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:07:16.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:16.773 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:07:16.535+0000 7f65caa00700 -1 osd.3 0 waiting for initial osdmap 2024-03-22T02:07:16.773 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:07:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:07:16.539+0000 7f65c27eb700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:07:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T02:07:17.260 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T02:07:17.261 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-22T02:07:17.261 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-22T02:07:17.261 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120867 s, 868 MB/s 2024-03-22T02:07:17.261 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-22T02:07:17.284 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:17 smithi083 ceph-mon[36340]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:07:17.285 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:17 smithi083 ceph-mon[36340]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T02:07:17.285 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.285 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.285 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:17 smithi083 ceph-mon[36340]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' 2024-03-22T02:07:17.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:17 smithi028 ceph-mon[31916]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:07:17.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:17 smithi028 ceph-mon[31916]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T02:07:17.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:17 smithi028 ceph-mon[31916]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' 2024-03-22T02:07:17.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:17 smithi120 ceph-mon[36803]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:07:17.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:17 smithi120 ceph-mon[36803]: osdmap e25: 4 total, 3 up, 4 in 2024-03-22T02:07:17.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:17.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:17 smithi120 ceph-mon[36803]: from='osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849]' entity='osd.3' 2024-03-22T02:07:18.000 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_3 2024-03-22T02:07:18.244 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:07:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:07:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:07:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] boot 2024-03-22T02:07:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T02:07:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:18.304 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] boot 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T02:07:18.305 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: osd.3 [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] boot 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T02:07:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi028 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 8546M 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.372 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.373 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi028 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 8546M 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: osdmap e27: 4 total, 4 up, 4 in 2024-03-22T02:07:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi028 2024-03-22T02:07:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 8546M 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 -- 172.21.15.83:0/158758434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381026b0 msgr2=0x7f1138102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 --2- 172.21.15.83:0/158758434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381026b0 0x7f1138102ad0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11280099c0 tx=0x7f1128009cd0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 -- 172.21.15.83:0/158758434 shutdown_connections 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 --2- 172.21.15.83:0/158758434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113810ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 --2- 172.21.15.83:0/158758434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381026b0 0x7f1138102ad0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 --2- 172.21.15.83:0/158758434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381038b0 0x7f1138103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 -- 172.21.15.83:0/158758434 >> 172.21.15.83:0/158758434 conn(0x7f11380fdc90 msgr2=0x7f11381000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:07:19.962 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 -- 172.21.15.83:0/158758434 shutdown_connections 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.959+0000 7f113ea46700 1 -- 172.21.15.83:0/158758434 wait complete. 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 Processor -- start 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 -- start start 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 0x7f1138198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113819ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f113810e300 con 0x7f11381026b0 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f113810e180 con 0x7f11381038b0 2024-03-22T02:07:19.963 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113ea46700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f113810e480 con 0x7f1138104250 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f11377fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f11377fe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:60198/0 (socket says 172.21.15.83:60198) 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f11377fe700 1 -- 172.21.15.83:0/349048285 learned_addr learned my addr 172.21.15.83:0/349048285 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f1137fff700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 0x7f1138198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.960+0000 7f113cfe3700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113819ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 -- 172.21.15.83:0/349048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 msgr2=0x7f113819ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113819ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 -- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 msgr2=0x7f1138198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 0x7f1138198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 -- 172.21.15.83:0/349048285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11381a05a0 con 0x7f11381038b0 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f113cfe3700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113819ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:07:19.964 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f1137fff700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 0x7f1138198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11377fe700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d980 tx=0x7f112c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c016070 con 0x7f11381038b0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11381a0890 con 0x7f11381038b0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f112c004d10 con 0x7f11381038b0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c005680 con 0x7f11381038b0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.961+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11381a0dd0 con 0x7f11381038b0 2024-03-22T02:07:19.965 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.962+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1138069340 con 0x7f11381038b0 2024-03-22T02:07:19.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.963+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f112c004340 con 0x7f11381038b0 2024-03-22T02:07:19.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.964+0000 7f11357fa700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 0x7f11200a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:19.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.964+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7f112c0ce560 con 0x7f11381038b0 2024-03-22T02:07:19.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.964+0000 7f1137fff700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 0x7f11200a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:19.967 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.964+0000 7f1137fff700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 0x7f11200a5470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f112800b580 tx=0x7f1128005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:07:19.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:19.968+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f112c01f020 con 0x7f11381038b0 2024-03-22T02:07:20.174 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:20.174+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1138062b80 con 0x7f11200a2fb0 2024-03-22T02:07:20.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:20 smithi028 ceph-mon[31916]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:20.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:20 smithi028 ceph-mon[31916]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T02:07:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:20 smithi120 ceph-mon[36803]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:20 smithi120 ceph-mon[36803]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T02:07:20.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:20 smithi083 ceph-mon[36340]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:20.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:20 smithi083 ceph-mon[36340]: osdmap e28: 4 total, 4 up, 4 in 2024-03-22T02:07:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:21 smithi028 ceph-mon[31916]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T02:07:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:21 smithi028 ceph-mon[31916]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:21.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:21 smithi120 ceph-mon[36803]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T02:07:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:21 smithi120 ceph-mon[36803]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:21 smithi083 ceph-mon[36340]: osdmap e29: 4 total, 4 up, 4 in 2024-03-22T02:07:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:21 smithi083 ceph-mon[36340]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:21.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:21.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:21.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:22 smithi120 ceph-mon[36803]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:07:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:22 smithi120 ceph-mon[36803]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T02:07:22.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:22 smithi083 ceph-mon[36340]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:07:22.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:22 smithi083 ceph-mon[36340]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T02:07:22.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:22 smithi028 ceph-mon[31916]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:07:22.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:22 smithi028 ceph-mon[31916]: osdmap e30: 4 total, 4 up, 4 in 2024-03-22T02:07:23.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:23 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/763290880' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.502 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:23 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.502 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:23 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]': finished 2024-03-22T02:07:23.502 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:23 smithi120 ceph-mon[36803]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T02:07:23.502 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:23.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:23 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/763290880' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:23 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:23 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]': finished 2024-03-22T02:07:23.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:23 smithi083 ceph-mon[36340]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T02:07:23.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:23 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/763290880' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:23 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]: dispatch 2024-03-22T02:07:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:23 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "39ccc6c8-4269-4d9e-b30f-40311d5558b3"}]': finished 2024-03-22T02:07:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:23 smithi028 ceph-mon[31916]: osdmap e31: 5 total, 4 up, 5 in 2024-03-22T02:07:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:24 smithi120 ceph-mon[36803]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-22T02:07:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:24 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/1191116777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:24 smithi083 ceph-mon[36340]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-22T02:07:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:24 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/1191116777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:24 smithi028 ceph-mon[31916]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 131 KiB/s, 0 objects/s recovering 2024-03-22T02:07:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:24 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/1191116777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:26 smithi120 ceph-mon[36803]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-22T02:07:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:26 smithi083 ceph-mon[36340]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-22T02:07:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:26 smithi028 ceph-mon[31916]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-22T02:07:28.446 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:28 smithi083 ceph-mon[36340]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-22T02:07:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:28 smithi120 ceph-mon[36803]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-22T02:07:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:28 smithi028 ceph-mon[31916]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-22T02:07:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:30 smithi083 ceph-mon[36340]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:30 smithi120 ceph-mon[36803]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:30 smithi028 ceph-mon[31916]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:31.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:07:31.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:31.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:31 smithi083 ceph-mon[36340]: Deploying daemon osd.4 on smithi083 2024-03-22T02:07:31.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:07:31.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:31.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:31 smithi028 ceph-mon[31916]: Deploying daemon osd.4 on smithi083 2024-03-22T02:07:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T02:07:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:31 smithi120 ceph-mon[36803]: Deploying daemon osd.4 on smithi083 2024-03-22T02:07:32.483 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:32 smithi083 ceph-mon[36340]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:32 smithi120 ceph-mon[36803]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:32 smithi028 ceph-mon[31916]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:34.456 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:34 smithi083 ceph-mon[36340]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:34 smithi120 ceph-mon[36803]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:34 smithi028 ceph-mon[31916]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:36.369 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:36 smithi083 ceph-mon[36340]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:36.369 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.369 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.370 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:36.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:36 smithi120 ceph-mon[36803]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:36.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:36 smithi028 ceph-mon[31916]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:38.219 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:38 smithi083 ceph-mon[36340]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:38 smithi120 ceph-mon[36803]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:38 smithi028 ceph-mon[31916]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:39.040 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 4 on host 'smithi083' 2024-03-22T02:07:39.040 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.038+0000 7f11357fa700 1 -- 172.21.15.83:0/349048285 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1138062b80 con 0x7f11200a2fb0 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.040+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 msgr2=0x7f11200a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.040+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 0x7f11200a5470 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f112800b580 tx=0x7f1128005fb0 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 msgr2=0x7f1138198e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d980 tx=0x7f112c00dc90 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 shutdown_connections 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11200a2fb0 0x7f11200a5470 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11381026b0 0x7f1138198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1138104250 0x7f113819ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 --2- 172.21.15.83:0/349048285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11381038b0 0x7f1138198e80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 >> 172.21.15.83:0/349048285 conn(0x7f11380fdc90 msgr2=0x7f1138106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:07:39.042 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 shutdown_connections 2024-03-22T02:07:39.043 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:07:39.041+0000 7f113ea46700 1 -- 172.21.15.83:0/349048285 wait complete. 2024-03-22T02:07:39.686 DEBUG:teuthology.orchestra.run.smithi083:osd.4> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.4.service 2024-03-22T02:07:39.688 INFO:tasks.cephadm:Deploying osd.5 on smithi120 with /dev/vg_nvme/lv_3... 2024-03-22T02:07:39.688 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T02:07:39.931 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:07:40.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:40 smithi083 ceph-mon[36340]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:40.292 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:40.293 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:40 smithi120 ceph-mon[36803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:40.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:40.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:40 smithi028 ceph-mon[31916]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T02:07:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:41 smithi120 ceph-mon[36803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:07:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:41 smithi120 ceph-mon[36803]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T02:07:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:41 smithi120 ceph-mon[36803]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:41 smithi120 ceph-mon[36803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-mon[36340]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:07:41.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-mon[36340]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T02:07:41.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:41.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-mon[36340]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.440 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-mon[36340]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:41 smithi028 ceph-mon[31916]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T02:07:41.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:41 smithi028 ceph-mon[31916]: osdmap e32: 5 total, 4 up, 5 in 2024-03-22T02:07:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:41 smithi028 ceph-mon[31916]: from='osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:41 smithi028 ceph-mon[31916]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:07:41.765 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4[47694]: 2024-03-22T02:07:41.655+0000 7f90fd603700 -1 osd.4 0 waiting for initial osdmap 2024-03-22T02:07:41.765 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:07:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4[47694]: 2024-03-22T02:07:41.662+0000 7f90f53ee700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125716 s, 834 MB/s 2024-03-22T02:07:41.853 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-22T02:07:42.422 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:42.422 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:07:42.422 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T02:07:42.422 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.422 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.423 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:42 smithi120 ceph-mon[36803]: from='osd.4 ' entity='osd.4' 2024-03-22T02:07:42.424 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_3 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.460 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:42 smithi083 ceph-mon[36340]: from='osd.4 ' entity='osd.4' 2024-03-22T02:07:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:07:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: osdmap e33: 5 total, 4 up, 5 in 2024-03-22T02:07:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:42 smithi028 ceph-mon[31916]: from='osd.4 ' entity='osd.4' 2024-03-22T02:07:42.668 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] boot 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T02:07:43.469 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:07:43.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] boot 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:43.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:07:43.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:07:43.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: osd.4 [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] boot 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:43.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:07:43.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:43.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:07:43.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:07:44.303 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 -- 172.21.15.120:0/3837173351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124100970 msgr2=0x7fe124100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 --2- 172.21.15.120:0/3837173351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124100970 0x7fe124100d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe118009990 tx=0x7fe118009ca0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 -- 172.21.15.120:0/3837173351 shutdown_connections 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 --2- 172.21.15.120:0/3837173351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1241042f0 0x7fe12410aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 --2- 172.21.15.120:0/3837173351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124100970 0x7fe124100d90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 --2- 172.21.15.120:0/3837173351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe124103950 0x7fe124103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 -- 172.21.15.120:0/3837173351 >> 172.21.15.120:0/3837173351 conn(0x7fe1240fc2f0 msgr2=0x7fe1240fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 -- 172.21.15.120:0/3837173351 shutdown_connections 2024-03-22T02:07:44.304 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.300+0000 7fe12aca7700 1 -- 172.21.15.120:0/3837173351 wait complete. 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 Processor -- start 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 -- start start 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe124100970 0x7fe1241032b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 0x7fe124101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:44.305 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe12410e2f0 con 0x7fe1241042f0 2024-03-22T02:07:44.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe12410e170 con 0x7fe124100970 2024-03-22T02:07:44.306 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe12aca7700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe12410e470 con 0x7fe124103950 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe123fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe123fff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:60838/0 (socket says 172.21.15.120:60838) 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe123fff700 1 -- 172.21.15.120:0/2481863489 learned_addr learned my addr 172.21.15.120:0/2481863489 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe128a43700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe124100970 0x7fe1241032b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.301+0000 7fe129244700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 0x7fe124101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 -- 172.21.15.120:0/2481863489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe124100970 msgr2=0x7fe1241032b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe124100970 0x7fe1241032b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 -- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 msgr2=0x7fe124101e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 0x7fe124101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 -- 172.21.15.120:0/2481863489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1241026f0 con 0x7fe124103950 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe129244700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 0x7fe124101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:07:44.307 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe123fff700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe11000d950 tx=0x7fe11000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110016070 con 0x7fe124103950 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe110009e20 con 0x7fe124103950 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1241a2af0 con 0x7fe124103950 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11000f620 con 0x7fe124103950 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.302+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1241a3000 con 0x7fe124103950 2024-03-22T02:07:44.308 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.303+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe124069340 con 0x7fe124103950 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.304+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe110019070 con 0x7fe124103950 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.305+0000 7fe121ffb700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 0x7fe1140a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.305+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7fe1100cdf10 con 0x7fe124103950 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.305+0000 7fe128a43700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 0x7fe1140a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.305+0000 7fe128a43700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 0x7fe1140a5470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe118003890 tx=0x7fe118005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:07:44.309 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.309+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe11009bdb0 con 0x7fe124103950 2024-03-22T02:07:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:44 smithi120 ceph-mon[36803]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:44 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi083 2024-03-22T02:07:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:44 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 8540M 2024-03-22T02:07:44.512 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:07:44.511+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe124062ac0 con 0x7fe1140a2fb0 2024-03-22T02:07:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:44 smithi083 ceph-mon[36340]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:44 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi083 2024-03-22T02:07:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:44 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 8540M 2024-03-22T02:07:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:44 smithi028 ceph-mon[31916]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-22T02:07:44.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:44 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi083 2024-03-22T02:07:44.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:44 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 8540M 2024-03-22T02:07:45.444 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:45 smithi120 ceph-mon[36803]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T02:07:45.444 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:45.445 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:45.445 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:45.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:45 smithi083 ceph-mon[36340]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T02:07:45.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:45.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:45.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:45.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:45 smithi028 ceph-mon[31916]: osdmap e35: 5 total, 5 up, 5 in 2024-03-22T02:07:45.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:07:45.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:07:45.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:46.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:46 smithi120 ceph-mon[36803]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:46.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:46 smithi120 ceph-mon[36803]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:46 smithi083 ceph-mon[36340]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:46 smithi083 ceph-mon[36340]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:46 smithi028 ceph-mon[31916]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:46 smithi028 ceph-mon[31916]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/2711082885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]': finished 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:07:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/238337127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:48.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/2711082885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]': finished 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:07:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/238337127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/2711082885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]: dispatch 2024-03-22T02:07:48.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "17fac3fc-aa99-48a3-8ce4-565522ae800d"}]': finished 2024-03-22T02:07:48.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T02:07:48.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:07:48.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/238337127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:07:50.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:50 smithi120 ceph-mon[36803]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:50.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:50 smithi083 ceph-mon[36340]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:50.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:50 smithi028 ceph-mon[31916]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:52 smithi120 ceph-mon[36803]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:52.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:52 smithi083 ceph-mon[36340]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:52 smithi028 ceph-mon[31916]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:54.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:54 smithi120 ceph-mon[36803]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:54.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:54 smithi083 ceph-mon[36340]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:54 smithi028 ceph-mon[31916]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:55.475 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:07:55.478 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:55.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:07:55.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:55.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T02:07:55.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:07:56.445 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:56 smithi120 ceph-mon[36803]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:56.445 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:56 smithi120 ceph-mon[36803]: Deploying daemon osd.5 on smithi120 2024-03-22T02:07:56.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:56 smithi083 ceph-mon[36340]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:56.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:56 smithi083 ceph-mon[36340]: Deploying daemon osd.5 on smithi120 2024-03-22T02:07:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:56 smithi028 ceph-mon[31916]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:56 smithi028 ceph-mon[31916]: Deploying daemon osd.5 on smithi120 2024-03-22T02:07:58.470 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:07:58 smithi120 ceph-mon[36803]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:58.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:07:58 smithi083 ceph-mon[36340]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:07:58.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:07:58 smithi028 ceph-mon[31916]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:00 smithi120 ceph-mon[36803]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:00.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:00 smithi083 ceph-mon[36340]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:00 smithi028 ceph-mon[31916]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:01.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:01.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:01.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:02.336 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:02 smithi120 ceph-mon[36803]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:02.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:02 smithi083 ceph-mon[36340]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:02 smithi028 ceph-mon[31916]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:03.932 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.932+0000 7fe121ffb700 1 -- 172.21.15.120:0/2481863489 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe124062ac0 con 0x7fe1140a2fb0 2024-03-22T02:08:03.932 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 5 on host 'smithi120' 2024-03-22T02:08:03.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 msgr2=0x7fe1140a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:03.934 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 0x7fe1140a5470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe118003890 tx=0x7fe118005fd0 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 msgr2=0x7fe124101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe11000d950 tx=0x7fe11000dc60 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 shutdown_connections 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1140a2fb0 0x7fe1140a5470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1241042f0 0x7fe124101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe124100970 0x7fe1241032b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 --2- 172.21.15.120:0/2481863489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe124103950 0x7fe124101900 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 >> 172.21.15.120:0/2481863489 conn(0x7fe1240fc2f0 msgr2=0x7fe124106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 shutdown_connections 2024-03-22T02:08:03.935 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:08:03.934+0000 7fe12aca7700 1 -- 172.21.15.120:0/2481863489 wait complete. 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:04.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:04.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:04.592 DEBUG:teuthology.orchestra.run.smithi120:osd.5> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.5.service 2024-03-22T02:08:04.594 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-03-22T02:08:04.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T02:08:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:08:05.145 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:04 smithi120 ceph-mon[36803]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:05.145 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:04 smithi120 ceph-mon[36803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:05.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:04 smithi028 ceph-mon[31916]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:05.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:04 smithi028 ceph-mon[31916]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:05.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:04 smithi083 ceph-mon[36340]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:05.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:04 smithi083 ceph-mon[36340]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T02:08:06.078 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:08:05.986+0000 7ffb29075700 -1 osd.5 0 waiting for initial osdmap 2024-03-22T02:08:06.079 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:08:05.994+0000 7ffb24e68700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.079 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:05 smithi120 ceph-mon[36803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:05 smithi083 ceph-mon[36340]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:06.265 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: from='osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.266 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:05 smithi028 ceph-mon[31916]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:08:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T02:08:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T02:08:06.963 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-22T02:08:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-22T02:08:06.963 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00813631 s, 1.3 GB/s 2024-03-22T02:08:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-22T02:08:07.198 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:06 smithi120 ceph-mon[36803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:08:07.199 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:06 smithi120 ceph-mon[36803]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T02:08:07.199 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.199 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:06 smithi083 ceph-mon[36340]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:08:07.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:06 smithi083 ceph-mon[36340]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T02:08:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:06 smithi028 ceph-mon[31916]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:08:07.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:06 smithi028 ceph-mon[31916]: osdmap e38: 6 total, 5 up, 6 in 2024-03-22T02:08:07.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.309 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:07.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-03-22T02:08:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:08:08.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:08:08.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:08:08.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:08.238 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] boot 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:08.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:08.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:08:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:08:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] boot 2024-03-22T02:08:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T02:08:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:08.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: osd.5 [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] boot 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T02:08:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:08.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:08.267 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:09.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:08 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi120 2024-03-22T02:08:09.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:08 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-22T02:08:09.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:08 smithi083 ceph-mon[36340]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T02:08:09.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:08 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi120 2024-03-22T02:08:09.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:08 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-22T02:08:09.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:08 smithi028 ceph-mon[31916]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T02:08:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:08 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi120 2024-03-22T02:08:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:08 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-22T02:08:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:08 smithi120 ceph-mon[36803]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T02:08:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 -- 172.21.15.28:0/440205174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 msgr2=0x7f41a410ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 --2- 172.21.15.28:0/440205174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a410ad00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f419c00b0a0 tx=0x7f419c00b3b0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 -- 172.21.15.28:0/440205174 shutdown_connections 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 --2- 172.21.15.28:0/440205174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a410ad00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 --2- 172.21.15.28:0/440205174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 0x7f41a4104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 --2- 172.21.15.28:0/440205174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 -- 172.21.15.28:0/440205174 >> 172.21.15.28:0/440205174 conn(0x7f41a40fc310 msgr2=0x7f41a40fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 -- 172.21.15.28:0/440205174 shutdown_connections 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.634+0000 7f41aadf8700 1 -- 172.21.15.28:0/440205174 wait complete. 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 Processor -- start 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 -- start start 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4103410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 0x7f41a4101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410e2d0 con 0x7f41a4101130 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410e150 con 0x7f41a4100790 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41aadf8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410e450 con 0x7f41a4103d10 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44358/0 (socket says 172.21.15.28:44358) 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a8b94700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4103410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 -- 172.21.15.28:0/1618593114 learned_addr learned my addr 172.21.15.28:0/1618593114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a9395700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 0x7f41a4101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 -- 172.21.15.28:0/1618593114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 msgr2=0x7f41a4101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 0x7f41a4101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 -- 172.21.15.28:0/1618593114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 msgr2=0x7f41a4103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4103410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.635+0000 7f41a3fff700 1 -- 172.21.15.28:0/1618593114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a4102850 con 0x7f41a4101130 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41a8b94700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41a3fff700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f419000d980 tx=0x7f419000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190016070 con 0x7f41a4101130 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a41a29f0 con 0x7f41a4101130 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4190004d10 con 0x7f41a4101130 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190005670 con 0x7f41a4101130 2024-03-22T02:08:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.636+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a41a2f30 con 0x7f41a4101130 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.637+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a402dd70 con 0x7f41a4101130 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.638+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4190004750 con 0x7f41a4101130 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.639+0000 7f41a1ffb700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 0x7f41940a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.639+0000 7f41a8b94700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 0x7f41940a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.639+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7f41900ce340 con 0x7f41a4101130 2024-03-22T02:08:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.639+0000 7f41a8b94700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 0x7f41940a5470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f41a406aaa0 tx=0x7f419800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:08:09.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.643+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f419001f020 con 0x7f41a4101130 2024-03-22T02:08:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:09.862+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f41a4062ac0 con 0x7f41940a2fb0 2024-03-22T02:08:10.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:10 smithi083 ceph-mon[36340]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:10.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:10 smithi083 ceph-mon[36340]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T02:08:10.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:10.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:10.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:10 smithi028 ceph-mon[31916]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:10 smithi028 ceph-mon[31916]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T02:08:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:10 smithi120 ceph-mon[36803]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:10 smithi120 ceph-mon[36803]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T02:08:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:11.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:11 smithi028 ceph-mon[31916]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:11 smithi028 ceph-mon[31916]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T02:08:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:11 smithi120 ceph-mon[36803]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:11 smithi120 ceph-mon[36803]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T02:08:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:11 smithi083 ceph-mon[36340]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:11 smithi083 ceph-mon[36340]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T02:08:12.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:12 smithi083 ceph-mon[36340]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:08:12.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:12 smithi083 ceph-mon[36340]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T02:08:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:12 smithi028 ceph-mon[31916]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:08:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:12 smithi028 ceph-mon[31916]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T02:08:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:12 smithi120 ceph-mon[36803]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:08:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:12 smithi120 ceph-mon[36803]: osdmap e43: 6 total, 6 up, 6 in 2024-03-22T02:08:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:13 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]: dispatch 2024-03-22T02:08:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:13 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]': finished 2024-03-22T02:08:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:13 smithi028 ceph-mon[31916]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T02:08:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:13 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]: dispatch 2024-03-22T02:08:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:13 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]': finished 2024-03-22T02:08:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:13 smithi120 ceph-mon[36803]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T02:08:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:13 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]: dispatch 2024-03-22T02:08:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:13 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/394243564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b4222ad-6096-499f-ba76-609404830d4d"}]': finished 2024-03-22T02:08:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:13 smithi083 ceph-mon[36340]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T02:08:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:14.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:14 smithi028 ceph-mon[31916]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T02:08:14.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:14 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/461801339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:14.621 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:14 smithi083 ceph-mon[36340]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T02:08:14.621 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:14 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/461801339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:14.621 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:14 smithi120 ceph-mon[36803]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-22T02:08:14.621 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:14 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/461801339' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:16.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:16 smithi028 ceph-mon[31916]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-22T02:08:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:16 smithi120 ceph-mon[36803]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-22T02:08:16.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:16 smithi083 ceph-mon[36340]: pgmap v148: 1 pgs: 1 peering; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-22T02:08:18.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:18 smithi028 ceph-mon[31916]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T02:08:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:18 smithi120 ceph-mon[36803]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T02:08:18.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:18 smithi083 ceph-mon[36340]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T02:08:20.236 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:20 smithi028 ceph-mon[31916]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:20.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:20 smithi083 ceph-mon[36340]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:20 smithi120 ceph-mon[36803]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:21.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T02:08:21.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:21.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:21 smithi028 ceph-mon[31916]: Deploying daemon osd.6 on smithi028 2024-03-22T02:08:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T02:08:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:21 smithi120 ceph-mon[36803]: Deploying daemon osd.6 on smithi028 2024-03-22T02:08:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T02:08:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:21 smithi083 ceph-mon[36340]: Deploying daemon osd.6 on smithi028 2024-03-22T02:08:22.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:22 smithi028 ceph-mon[31916]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:22 smithi120 ceph-mon[36803]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:22.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:22 smithi083 ceph-mon[36340]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:24.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:24 smithi028 ceph-mon[31916]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:24 smithi120 ceph-mon[36803]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:24 smithi083 ceph-mon[36340]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:26 smithi028 ceph-mon[31916]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:26 smithi120 ceph-mon[36803]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:26 smithi083 ceph-mon[36340]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:28 smithi120 ceph-mon[36803]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:28 smithi083 ceph-mon[36340]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:28 smithi028 ceph-mon[31916]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:29.922 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-03-22T02:08:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.920+0000 7f41a1ffb700 1 -- 172.21.15.28:0/1618593114 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f41a4062ac0 con 0x7f41940a2fb0 2024-03-22T02:08:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 msgr2=0x7f41940a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 0x7f41940a5470 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f41a406aaa0 tx=0x7f419800b410 comp rx=0 tx=0).stop 2024-03-22T02:08:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 msgr2=0x7f41a4101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f419000d980 tx=0x7f419000dc90 comp rx=0 tx=0).stop 2024-03-22T02:08:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 shutdown_connections 2024-03-22T02:08:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41a4103d10 0x7f41a4101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41a4100790 0x7f41a4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41940a2fb0 0x7f41940a5470 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 --2- 172.21.15.28:0/1618593114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41a4101130 0x7f41a4101a60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 >> 172.21.15.28:0/1618593114 conn(0x7f41a40fc310 msgr2=0x7f41a4106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:08:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 shutdown_connections 2024-03-22T02:08:29.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:08:29.922+0000 7f41aadf8700 1 -- 172.21.15.28:0/1618593114 wait complete. 2024-03-22T02:08:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:30.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T02:08:30.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:30.630 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.6.service 2024-03-22T02:08:30.632 INFO:tasks.cephadm:Deploying osd.7 on smithi083 with /dev/vg_nvme/lv_2... 2024-03-22T02:08:30.632 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T02:08:30.877 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:08:31.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-mon[31916]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T02:08:31.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-mon[31916]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T02:08:31.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-mon[31916]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:08:31.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:31.166 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:08:31.086+0000 7f9feabd0700 -1 osd.6 0 waiting for initial osdmap 2024-03-22T02:08:31.166 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:08:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:08:31.094+0000 7f9fe69c3700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:08:31.262 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:31 smithi083 ceph-mon[36340]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T02:08:31.262 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:31 smithi083 ceph-mon[36340]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T02:08:31.262 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:31 smithi083 ceph-mon[36340]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:08:31.262 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:31 smithi120 ceph-mon[36803]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T02:08:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:31 smithi120 ceph-mon[36803]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T02:08:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:31 smithi120 ceph-mon[36803]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:08:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.419 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:32 smithi083 ceph-mon[36340]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:32.420 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:32 smithi083 ceph-mon[36340]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:08:32.420 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:32 smithi083 ceph-mon[36340]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T02:08:32.420 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.420 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:32 smithi120 ceph-mon[36803]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:32 smithi120 ceph-mon[36803]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:08:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:32 smithi120 ceph-mon[36803]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T02:08:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:32 smithi028 ceph-mon[31916]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 177 MiB used, 536 GiB / 536 GiB avail 2024-03-22T02:08:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:32 smithi028 ceph-mon[31916]: from='osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:08:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:32 smithi028 ceph-mon[31916]: osdmap e46: 7 total, 6 up, 7 in 2024-03-22T02:08:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:33.003 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T02:08:33.003 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T02:08:33.003 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-22T02:08:33.004 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-22T02:08:33.004 INFO:teuthology.orchestra.run.smithi083.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120817 s, 868 MB/s 2024-03-22T02:08:33.004 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-22T02:08:33.306 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:33 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:08:33.307 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:33 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:08:33.307 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:33 smithi083 ceph-mon[36340]: osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] boot 2024-03-22T02:08:33.307 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:33 smithi083 ceph-mon[36340]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T02:08:33.307 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:33.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:33 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:08:33.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:33 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:08:33.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:33 smithi028 ceph-mon[31916]: osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] boot 2024-03-22T02:08:33.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:33 smithi028 ceph-mon[31916]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T02:08:33.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:33.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:33 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:08:33.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:33 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:08:33.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:33 smithi120 ceph-mon[36803]: osd.6 [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] boot 2024-03-22T02:08:33.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:33 smithi120 ceph-mon[36803]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T02:08:33.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T02:08:33.653 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_2 2024-03-22T02:08:33.898 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:08:34.280 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:34.280 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T02:08:34.280 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:34.281 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:34.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:34.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:08:34.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:35.328 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:35 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi028 2024-03-22T02:08:35.328 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:35 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 5697M 2024-03-22T02:08:35.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:35 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi028 2024-03-22T02:08:35.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:35 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 5697M 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 -- 172.21.15.83:0/1057758685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01038b0 msgr2=0x7f9ae0103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/1057758685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01038b0 0x7f9ae0103d10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0099c0 tx=0x7f9adc009cd0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 -- 172.21.15.83:0/1057758685 shutdown_connections 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/1057758685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae0104250 0x7f9ae010ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/1057758685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae01026b0 0x7f9ae0102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/1057758685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01038b0 0x7f9ae0103d10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 -- 172.21.15.83:0/1057758685 >> 172.21.15.83:0/1057758685 conn(0x7f9ae00fdc90 msgr2=0x7f9ae01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:08:35.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 -- 172.21.15.83:0/1057758685 shutdown_connections 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.555+0000 7f9ae84b4700 1 -- 172.21.15.83:0/1057758685 wait complete. 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 Processor -- start 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 -- start start 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae01038b0 0x7f9ae0198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 0x7f9ae019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010e2a0 con 0x7f9ae01038b0 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010e120 con 0x7f9ae01026b0 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae84b4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9ae010e420 con 0x7f9ae0104250 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:35.562 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:44972/0 (socket says 172.21.15.83:44972) 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 -- 172.21.15.83:0/2637999705 learned_addr learned my addr 172.21.15.83:0/2637999705 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae5a4f700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae01038b0 0x7f9ae0198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6a51700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 0x7f9ae019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 -- 172.21.15.83:0/2637999705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 msgr2=0x7f9ae019fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 0x7f9ae019fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 -- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae01038b0 msgr2=0x7f9ae0198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae01038b0 0x7f9ae0198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.556+0000 7f9ae6250700 1 -- 172.21.15.83:0/2637999705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ae01a0540 con 0x7f9ae01026b0 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ae6a51700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 0x7f9ae019fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ae6250700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c8f0 tx=0x7f9ad000cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0004220 con 0x7f9ae01026b0 2024-03-22T02:08:35.563 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ae01a0830 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ad00043c0 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0003d40 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.557+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ae01a0d20 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.558+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ae0069340 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.559+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9ad0007740 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.559+0000 7f9ad77fe700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 0x7f9acc0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.560+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7f9ad00ce340 con 0x7f9ae01026b0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.560+0000 7f9ae5a4f700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 0x7f9acc0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.560+0000 7f9ae5a4f700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 0x7f9acc0a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9adc000c00 tx=0x7f9adc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:08:35.564 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.564+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9ad0014030 con 0x7f9ae01026b0 2024-03-22T02:08:35.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:35 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi028 2024-03-22T02:08:35.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:35 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 5697M 2024-03-22T02:08:35.769 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:35.768+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9ae0062ac0 con 0x7f9acc0a2fb0 2024-03-22T02:08:36.362 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:36 smithi083 ceph-mon[36340]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:36.363 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:36.363 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:36.363 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:36.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:36 smithi120 ceph-mon[36803]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:36.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:36.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:36.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:36 smithi028 ceph-mon[31916]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:08:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:08:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:37.366 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:37 smithi083 ceph-mon[36340]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:37.420 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:37 smithi120 ceph-mon[36803]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:37.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:37 smithi028 ceph-mon[31916]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:08:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:38 smithi120 ceph-mon[36803]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:38.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:38 smithi083 ceph-mon[36340]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:38 smithi028 ceph-mon[31916]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:39.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/2163346928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]': finished 2024-03-22T02:08:39.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T02:08:39.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:39.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:39 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/3002155096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:39.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/2163346928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]': finished 2024-03-22T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:39.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:39 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/3002155096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:39.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/2163346928' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]: dispatch 2024-03-22T02:08:39.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d99f5cf-caaa-4c34-96da-37dd87bc1491"}]': finished 2024-03-22T02:08:39.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: osdmap e49: 8 total, 7 up, 8 in 2024-03-22T02:08:39.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:39.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:39 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/3002155096' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:08:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:40 smithi120 ceph-mon[36803]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:40 smithi083 ceph-mon[36340]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:40 smithi028 ceph-mon[31916]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:42 smithi120 ceph-mon[36803]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:42.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:42 smithi083 ceph-mon[36340]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:42 smithi028 ceph-mon[31916]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:44.698 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:44 smithi083 ceph-mon[36340]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:44 smithi120 ceph-mon[36803]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:44 smithi028 ceph-mon[31916]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:46.730 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:46 smithi083 ceph-mon[36340]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:46.730 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T02:08:46.731 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:46 smithi120 ceph-mon[36803]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T02:08:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:46 smithi028 ceph-mon[31916]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T02:08:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:08:47.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:47 smithi120 ceph-mon[36803]: Deploying daemon osd.7 on smithi083 2024-03-22T02:08:47.756 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:47 smithi083 ceph-mon[36340]: Deploying daemon osd.7 on smithi083 2024-03-22T02:08:47.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:47 smithi028 ceph-mon[31916]: Deploying daemon osd.7 on smithi083 2024-03-22T02:08:48.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:48 smithi028 ceph-mon[31916]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:48 smithi120 ceph-mon[36803]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:48 smithi083 ceph-mon[36340]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:50.710 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:50 smithi083 ceph-mon[36340]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:50 smithi120 ceph-mon[36803]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:50.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:50 smithi028 ceph-mon[31916]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:52 smithi028 ceph-mon[31916]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:52 smithi120 ceph-mon[36803]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:52 smithi083 ceph-mon[36340]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:08:54.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:54 smithi083 ceph-mon[36340]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:54 smithi028 ceph-mon[31916]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:54 smithi120 ceph-mon[36803]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:54.984 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.983+0000 7f9ad77fe700 1 -- 172.21.15.83:0/2637999705 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9ae0062ac0 con 0x7f9acc0a2fb0 2024-03-22T02:08:54.984 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 7 on host 'smithi083' 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 msgr2=0x7f9acc0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 0x7f9acc0a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9adc000c00 tx=0x7f9adc005fb0 comp rx=0 tx=0).stop 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 msgr2=0x7f9ae01988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000c8f0 tx=0x7f9ad000cc00 comp rx=0 tx=0).stop 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 shutdown_connections 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9ae0104250 0x7f9ae019fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9ae01026b0 0x7f9ae01988e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9acc0a2fb0 0x7f9acc0a5470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:54.986 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 --2- 172.21.15.83:0/2637999705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ae01038b0 0x7f9ae0198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:08:54.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 >> 172.21.15.83:0/2637999705 conn(0x7f9ae00fdc90 msgr2=0x7f9ae0106c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:08:54.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 shutdown_connections 2024-03-22T02:08:54.987 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:08:54.986+0000 7f9ae84b4700 1 -- 172.21.15.83:0/2637999705 wait complete. 2024-03-22T02:08:55.693 DEBUG:teuthology.orchestra.run.smithi083:osd.7> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.7.service 2024-03-22T02:08:55.695 INFO:tasks.cephadm:Deploying osd.8 on smithi120 with /dev/vg_nvme/lv_2... 2024-03-22T02:08:55.695 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T02:08:55.947 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T02:08:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:08:57.205 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:56 smithi120 ceph-mon[36803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T02:08:57.205 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:56 smithi120 ceph-mon[36803]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T02:08:57.205 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:57.205 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:56 smithi120 ceph-mon[36803]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:57.206 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:56 smithi120 ceph-mon[36803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-mon[36340]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T02:08:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-mon[36340]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T02:08:57.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:57.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-mon[36340]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:57.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-mon[36340]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:57.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:08:56.947+0000 7fb523647700 -1 osd.7 0 waiting for initial osdmap 2024-03-22T02:08:57.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:08:56 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:08:56.959+0000 7fb51f43a700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:08:57.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:56 smithi028 ceph-mon[31916]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T02:08:57.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:56 smithi028 ceph-mon[31916]: osdmap e50: 8 total, 7 up, 8 in 2024-03-22T02:08:57.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:57.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:56 smithi028 ceph-mon[31916]: from='osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:57.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:56 smithi028 ceph-mon[31916]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:08:58.119 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T02:08:58.119 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T02:08:58.120 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-22T02:08:58.120 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-22T02:08:58.120 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115918 s, 905 MB/s 2024-03-22T02:08:58.120 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-22T02:08:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:57 smithi120 ceph-mon[36803]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:57 smithi120 ceph-mon[36803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:08:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:57 smithi120 ceph-mon[36803]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T02:08:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:57 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:57 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:57 smithi083 ceph-mon[36340]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:57 smithi083 ceph-mon[36340]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:08:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:57 smithi083 ceph-mon[36340]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T02:08:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:57 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:57 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:57 smithi028 ceph-mon[31916]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:08:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:57 smithi028 ceph-mon[31916]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:08:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:57 smithi028 ceph-mon[31916]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T02:08:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:57 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:57 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:58.866 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_2 2024-03-22T02:08:59.105 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] boot 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.188 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:08:58 smithi083 ceph-mon[36340]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] boot 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T02:08:59.227 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.228 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:08:58 smithi120 ceph-mon[36803]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T02:08:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:08:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:08:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: osd.7 [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] boot 2024-03-22T02:08:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: osdmap e52: 8 total, 8 up, 8 in 2024-03-22T02:08:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T02:08:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:08:58 smithi028 ceph-mon[31916]: osdmap e53: 8 total, 8 up, 8 in 2024-03-22T02:09:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:09:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi083 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 5693M 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:00 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi083 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 5693M 2024-03-22T02:09:00.415 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.416 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:00 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: pgmap v179: 1 pgs: 1 remapped; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi083 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 5693M 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:00.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:00 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 -- 172.21.15.120:0/2858585540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 msgr2=0x7fb940109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 --2- 172.21.15.120:0/2858585540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb940109a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb93800b0a0 tx=0x7fb93800b3b0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 -- 172.21.15.120:0/2858585540 shutdown_connections 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 --2- 172.21.15.120:0/2858585540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb940109a00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 --2- 172.21.15.120:0/2858585540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401015b0 0x7fb9401019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 --2- 172.21.15.120:0/2858585540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401027b0 0x7fb940102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 -- 172.21.15.120:0/2858585540 >> 172.21.15.120:0/2858585540 conn(0x7fb9400fcb50 msgr2=0x7fb9400fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:09:00.779 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 -- 172.21.15.120:0/2858585540 shutdown_connections 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.761+0000 7fb944962700 1 -- 172.21.15.120:0/2858585540 wait complete. 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 Processor -- start 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 -- start start 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 0x7fb940197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401027b0 0x7fb940197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb94010d1f0 con 0x7fb9401015b0 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb94010d070 con 0x7fb9401027b0 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb944962700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb94010d370 con 0x7fb940103150 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb93ed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb93ed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:32788/0 (socket says 172.21.15.120:32788) 2024-03-22T02:09:00.780 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb93ed9d700 1 -- 172.21.15.120:0/1518157289 learned_addr learned my addr 172.21.15.120:0/1518157289 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb93dd9b700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401027b0 0x7fb940197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.762+0000 7fb93e59c700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 0x7fb940197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 -- 172.21.15.120:0/1518157289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401027b0 msgr2=0x7fb940197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401027b0 0x7fb940197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 -- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 msgr2=0x7fb940197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 0x7fb940197750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 -- 172.21.15.120:0/1518157289 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb94019f3b0 con 0x7fb940103150 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93e59c700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 0x7fb940197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb93ed9d700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb938007d40 tx=0x7fb938007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938021020 con 0x7fb940103150 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb94006a330 con 0x7fb940103150 2024-03-22T02:09:00.781 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93801d950 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9380054f0 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.763+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb94006a8f0 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.764+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb940069340 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.765+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fb938024020 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.766+0000 7fb92f7fe700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 0x7fb9280a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.766+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5397+0+0 (secure 0 0 0) 0x7fb93800f020 con 0x7fb940103150 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.766+0000 7fb93e59c700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 0x7fb9280a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.766+0000 7fb93e59c700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 0x7fb9280a5470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb940102610 tx=0x7fb93400a680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:00.782 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.769+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb93809d9a0 con 0x7fb940103150 2024-03-22T02:09:00.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:00.971+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb94006ab50 con 0x7fb9280a2fb0 2024-03-22T02:09:01.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:01 smithi028 ceph-mon[31916]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T02:09:01.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:01.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:01.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:01.581 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:01 smithi120 ceph-mon[36803]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T02:09:01.581 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:01.581 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:01.581 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:01.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:01 smithi083 ceph-mon[36340]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T02:09:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:02 smithi028 ceph-mon[31916]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:02 smithi028 ceph-mon[31916]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:02 smithi028 ceph-mon[31916]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T02:09:02.678 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:02 smithi120 ceph-mon[36803]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:02.679 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:02 smithi120 ceph-mon[36803]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:02.679 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:02 smithi120 ceph-mon[36803]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T02:09:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:02 smithi083 ceph-mon[36340]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:02 smithi083 ceph-mon[36340]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:02 smithi083 ceph-mon[36340]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: pgmap v184: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/608887627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]': finished 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T02:09:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: pgmap v184: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/608887627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]': finished 2024-03-22T02:09:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T02:09:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: pgmap v184: 1 pgs: 1 remapped; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/608887627' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]: dispatch 2024-03-22T02:09:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "574e783a-b2d6-439e-8bde-68e2d4b577b7"}]': finished 2024-03-22T02:09:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: osdmap e57: 9 total, 8 up, 9 in 2024-03-22T02:09:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:05.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:05 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/2142707259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:05 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/2142707259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:05 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/2142707259' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:06.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:06 smithi028 ceph-mon[31916]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:06.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:06 smithi120 ceph-mon[36803]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:06.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:06 smithi083 ceph-mon[36340]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:08 smithi120 ceph-mon[36803]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:08.566 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:08 smithi083 ceph-mon[36340]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:08 smithi028 ceph-mon[31916]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:10.479 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:10 smithi120 ceph-mon[36803]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:10 smithi028 ceph-mon[31916]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:10 smithi083 ceph-mon[36340]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T02:09:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T02:09:11.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:11.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-22T02:09:11.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:12 smithi120 ceph-mon[36803]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:12 smithi120 ceph-mon[36803]: Deploying daemon osd.8 on smithi120 2024-03-22T02:09:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:12 smithi028 ceph-mon[31916]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:12 smithi028 ceph-mon[31916]: Deploying daemon osd.8 on smithi120 2024-03-22T02:09:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:12 smithi083 ceph-mon[36340]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:12 smithi083 ceph-mon[36340]: Deploying daemon osd.8 on smithi120 2024-03-22T02:09:14.467 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:14 smithi120 ceph-mon[36803]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:14.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:14 smithi028 ceph-mon[31916]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:14 smithi083 ceph-mon[36340]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:16 smithi120 ceph-mon[36803]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:16 smithi028 ceph-mon[31916]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:16 smithi083 ceph-mon[36340]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:17 smithi120 ceph-mon[36803]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:17 smithi028 ceph-mon[31916]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:17 smithi083 ceph-mon[36340]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:19.912 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:19 smithi120 ceph-mon[36803]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:19.987 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.988+0000 7fb92f7fe700 1 -- 172.21.15.120:0/1518157289 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb94006ab50 con 0x7fb9280a2fb0 2024-03-22T02:09:19.988 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 8 on host 'smithi120' 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 msgr2=0x7fb9280a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 0x7fb9280a5470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb940102610 tx=0x7fb93400a680 comp rx=0 tx=0).stop 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 msgr2=0x7fb94019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb938007d40 tx=0x7fb938007d70 comp rx=0 tx=0).stop 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 shutdown_connections 2024-03-22T02:09:19.990 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9280a2fb0 0x7fb9280a5470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9401015b0 0x7fb940197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb940103150 0x7fb94019ed10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 --2- 172.21.15.120:0/1518157289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9401027b0 0x7fb940197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 >> 172.21.15.120:0/1518157289 conn(0x7fb9400fcb50 msgr2=0x7fb940105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 shutdown_connections 2024-03-22T02:09:19.991 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:09:19.990+0000 7fb944962700 1 -- 172.21.15.120:0/1518157289 wait complete. 2024-03-22T02:09:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:19 smithi028 ceph-mon[31916]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:20.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:19 smithi083 ceph-mon[36340]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:21.020 DEBUG:teuthology.orchestra.run.smithi120:osd.8> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.8.service 2024-03-22T02:09:21.022 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-03-22T02:09:21.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T02:09:21.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:20 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:20 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:20 smithi120 ceph-mon[36803]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:21.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:20 smithi120 ceph-mon[36803]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:21.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:20 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:20 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:20 smithi083 ceph-mon[36340]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:21.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:20 smithi083 ceph-mon[36340]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:21.278 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:09:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:20 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:20 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:20 smithi028 ceph-mon[31916]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:21.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:20 smithi028 ceph-mon[31916]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:22.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:22.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:09:21.974+0000 7f88dd3a3700 -1 osd.8 0 waiting for initial osdmap 2024-03-22T02:09:22.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:09:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:09:21.983+0000 7f88d9196700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:09:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T02:09:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T02:09:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:22.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: osdmap e58: 9 total, 8 up, 9 in 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:22 smithi120 ceph-mon[36803]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:09:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:22 smithi120 ceph-mon[36803]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T02:09:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:22 smithi083 ceph-mon[36340]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:09:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:22 smithi083 ceph-mon[36340]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T02:09:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:22 smithi028 ceph-mon[31916]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:09:23.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:22 smithi028 ceph-mon[31916]: osdmap e59: 9 total, 8 up, 9 in 2024-03-22T02:09:23.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:23.310 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T02:09:23.310 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T02:09:23.310 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-03-22T02:09:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-03-22T02:09:23.311 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121152 s, 866 MB/s 2024-03-22T02:09:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-03-22T02:09:23.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-03-22T02:09:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:09:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:09:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:09:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] boot 2024-03-22T02:09:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T02:09:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:23 smithi120 ceph-mon[36803]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T02:09:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:09:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:09:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] boot 2024-03-22T02:09:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T02:09:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:23 smithi083 ceph-mon[36340]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: osd.8 [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] boot 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: osdmap e60: 9 total, 9 up, 9 in 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-22T02:09:24.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:23 smithi028 ceph-mon[31916]: osdmap e61: 9 total, 9 up, 9 in 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.466 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: osdmap e62: 9 total, 9 up, 9 in 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:25.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:25.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:25.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:25.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 -- 172.21.15.28:0/4131160556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d501009b0 msgr2=0x7f0d50100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 --2- 172.21.15.28:0/4131160556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d501009b0 0x7f0d50100dd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009990 tx=0x7f0d40009ca0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 -- 172.21.15.28:0/4131160556 shutdown_connections 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 --2- 172.21.15.28:0/4131160556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d501009b0 0x7f0d50100dd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 --2- 172.21.15.28:0/4131160556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 0x7f0d5010abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 --2- 172.21.15.28:0/4131160556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d50103990 0x7f0d50103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.149+0000 7f0d54f86700 1 -- 172.21.15.28:0/4131160556 >> 172.21.15.28:0/4131160556 conn(0x7f0d500fc310 msgr2=0x7f0d500fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:09:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 -- 172.21.15.28:0/4131160556 shutdown_connections 2024-03-22T02:09:26.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 -- 172.21.15.28:0/4131160556 wait complete. 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 Processor -- start 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 -- start start 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d501009b0 0x7f0d50198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.150+0000 7f0d54f86700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 0x7f0d5019fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d54f86700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5010e320 con 0x7f0d50103990 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d54f86700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5010e1a0 con 0x7f0d50104330 2024-03-22T02:09:26.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d54f86700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5010e4a0 con 0x7f0d501009b0 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53932/0 (socket says 172.21.15.28:53932) 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 -- 172.21.15.28:0/3728695781 learned_addr learned my addr 172.21.15.28:0/3728695781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4e59c700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d501009b0 0x7f0d50198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4ed9d700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 0x7f0d5019fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 -- 172.21.15.28:0/3728695781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d501009b0 msgr2=0x7f0d50198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d501009b0 0x7f0d50198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 -- 172.21.15.28:0/3728695781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 msgr2=0x7f0d5019fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 0x7f0d5019fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 -- 172.21.15.28:0/3728695781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d501a04d0 con 0x7f0d50103990 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d4dd9b700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c9e0 tx=0x7f0d4400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:26.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44007ac0 con 0x7f0d50103990 2024-03-22T02:09:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d501a07c0 con 0x7f0d50103990 2024-03-22T02:09:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d44007c60 con 0x7f0d50103990 2024-03-22T02:09:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d44018730 con 0x7f0d50103990 2024-03-22T02:09:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.151+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d501a0dd0 con 0x7f0d50103990 2024-03-22T02:09:26.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.152+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d50069340 con 0x7f0d50103990 2024-03-22T02:09:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.154+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f0d440188d0 con 0x7f0d50103990 2024-03-22T02:09:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.154+0000 7f0d3f7fe700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 0x7f0d380a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.155+0000 7f0d4e59c700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 0x7f0d380a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.155+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f0d440cec70 con 0x7f0d50103990 2024-03-22T02:09:26.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.155+0000 7f0d4e59c700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 0x7f0d380a5470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003890 tx=0x7f0d40003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:26.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.158+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d44014030 con 0x7f0d50103990 2024-03-22T02:09:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:26 smithi028 ceph-mon[31916]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:26 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi120 2024-03-22T02:09:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:26 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-22T02:09:26.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:26 smithi028 ceph-mon[31916]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T02:09:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:26.374+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f0d50062ac0 con 0x7f0d380a2fb0 2024-03-22T02:09:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:26 smithi120 ceph-mon[36803]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:26 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi120 2024-03-22T02:09:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:26 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-22T02:09:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:26 smithi120 ceph-mon[36803]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T02:09:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:26 smithi083 ceph-mon[36340]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:26 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi120 2024-03-22T02:09:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:26 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-22T02:09:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:26 smithi083 ceph-mon[36340]: osdmap e63: 9 total, 9 up, 9 in 2024-03-22T02:09:27.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:27 smithi028 ceph-mon[31916]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T02:09:27.246 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:27 smithi028 ceph-mon[31916]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:27.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:27 smithi120 ceph-mon[36803]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T02:09:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:27 smithi120 ceph-mon[36803]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:27.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:27 smithi083 ceph-mon[36340]: osdmap e64: 9 total, 9 up, 9 in 2024-03-22T02:09:27.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:27 smithi083 ceph-mon[36340]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:28 smithi028 ceph-mon[31916]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:28 smithi120 ceph-mon[36803]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:28 smithi083 ceph-mon[36340]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]: dispatch 2024-03-22T02:09:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]': finished 2024-03-22T02:09:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T02:09:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:30.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/956546587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]: dispatch 2024-03-22T02:09:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]': finished 2024-03-22T02:09:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T02:09:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/956546587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: pgmap v205: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]: dispatch 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/999092169' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f26642a2-c7eb-495a-979a-6002cf9f9438"}]': finished 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: osdmap e65: 10 total, 9 up, 10 in 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/956546587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:32 smithi120 ceph-mon[36803]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T02:09:32.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:32 smithi083 ceph-mon[36340]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T02:09:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:32 smithi028 ceph-mon[31916]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-22T02:09:34.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:34 smithi028 ceph-mon[31916]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-22T02:09:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:34 smithi120 ceph-mon[36803]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-22T02:09:34.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:34 smithi083 ceph-mon[36340]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-22T02:09:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:36 smithi028 ceph-mon[31916]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-22T02:09:36.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:36 smithi120 ceph-mon[36803]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-22T02:09:36.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:36 smithi083 ceph-mon[36340]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-22T02:09:37.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T02:09:37.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:37.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T02:09:37.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:37.518 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-22T02:09:37.518 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:38 smithi120 ceph-mon[36803]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:38 smithi120 ceph-mon[36803]: Deploying daemon osd.9 on smithi028 2024-03-22T02:09:38.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:38 smithi028 ceph-mon[31916]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:38.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:38 smithi028 ceph-mon[31916]: Deploying daemon osd.9 on smithi028 2024-03-22T02:09:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:38 smithi083 ceph-mon[36340]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:38 smithi083 ceph-mon[36340]: Deploying daemon osd.9 on smithi028 2024-03-22T02:09:40.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:40 smithi120 ceph-mon[36803]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:40.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:40 smithi083 ceph-mon[36340]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:40.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:40 smithi028 ceph-mon[31916]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-22T02:09:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:42 smithi028 ceph-mon[31916]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:42.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:42 smithi120 ceph-mon[36803]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:42.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:42 smithi083 ceph-mon[36340]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:43 smithi120 ceph-mon[36803]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:43 smithi083 ceph-mon[36340]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:44.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:44.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:43 smithi028 ceph-mon[31916]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:44.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:44.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:44.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:45.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:45 smithi028 ceph-mon[31916]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:45.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:45 smithi120 ceph-mon[36803]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:46.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:45 smithi083 ceph-mon[36340]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:46.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.687+0000 7f0d3f7fe700 1 -- 172.21.15.28:0/3728695781 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0d50062ac0 con 0x7f0d380a2fb0 2024-03-22T02:09:46.688 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.689+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 msgr2=0x7f0d380a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.689+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 0x7f0d380a5470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d40003890 tx=0x7f0d40003960 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.689+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 msgr2=0x7f0d50198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.689+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400c9e0 tx=0x7f0d4400ccf0 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.689+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 shutdown_connections 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0d50104330 0x7f0d5019fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0d501009b0 0x7f0d50198870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0d380a2fb0 0x7f0d380a5470 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 --2- 172.21.15.28:0/3728695781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d50103990 0x7f0d50198db0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:46.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 >> 172.21.15.28:0/3728695781 conn(0x7f0d500fc310 msgr2=0x7f0d50106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:09:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 shutdown_connections 2024-03-22T02:09:46.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:09:46.690+0000 7f0d54f86700 1 -- 172.21.15.28:0/3728695781 wait complete. 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:46.991 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T02:09:46.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:46.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:46.992 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:46 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T02:09:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:46 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:47.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:46 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:09:47.327 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.9.service 2024-03-22T02:09:47.329 INFO:tasks.cephadm:Deploying osd.10 on smithi083 with /dev/vg_nvme/lv_1... 2024-03-22T02:09:47.330 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T02:09:47.576 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:09:47.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-mon[31916]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:47.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-mon[31916]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T02:09:47.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-mon[31916]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T02:09:47.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-mon[31916]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:09:47.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:47.911 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:09:47.897+0000 7f1e9a030700 -1 osd.9 0 waiting for initial osdmap 2024-03-22T02:09:47.911 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:09:47 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:09:47.910+0000 7f1e95e23700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:09:48.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:47 smithi083 ceph-mon[36340]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:48.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:47 smithi083 ceph-mon[36340]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T02:09:48.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:47 smithi083 ceph-mon[36340]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T02:09:48.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:47 smithi083 ceph-mon[36340]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:09:48.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:47 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:47 smithi120 ceph-mon[36803]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:47 smithi120 ceph-mon[36803]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-22T02:09:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:47 smithi120 ceph-mon[36803]: osdmap e66: 10 total, 9 up, 10 in 2024-03-22T02:09:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:47 smithi120 ceph-mon[36803]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-03-22T02:09:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:47 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:48 smithi083 ceph-mon[36340]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:09:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:48 smithi083 ceph-mon[36340]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T02:09:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:48 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:48 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:48 smithi028 ceph-mon[31916]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:09:49.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:48 smithi028 ceph-mon[31916]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T02:09:49.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:48 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:48 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:48 smithi120 ceph-mon[36803]: from='osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-03-22T02:09:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:48 smithi120 ceph-mon[36803]: osdmap e67: 10 total, 9 up, 10 in 2024-03-22T02:09:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:48 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:48 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:49.623 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T02:09:49.623 INFO:teuthology.orchestra.run.smithi083.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T02:09:49.624 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10+0 records in 2024-03-22T02:09:49.624 INFO:teuthology.orchestra.run.smithi083.stderr:10+0 records out 2024-03-22T02:09:49.624 INFO:teuthology.orchestra.run.smithi083.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013313 s, 788 MB/s 2024-03-22T02:09:49.624 INFO:teuthology.orchestra.run.smithi083.stderr:--> Zapping successful for: 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] boot 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T02:09:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:49 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:09:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:09:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] boot 2024-03-22T02:09:50.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T02:09:50.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:49 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: osd.9 [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] boot 2024-03-22T02:09:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: osdmap e68: 10 total, 10 up, 10 in 2024-03-22T02:09:50.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:49 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-22T02:09:50.355 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi083:vg_nvme/lv_1 2024-03-22T02:09:50.597 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:09:50.973 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T02:09:50.973 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:50.973 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:50.973 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:50.973 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:50.974 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:50 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:51.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:51.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:50 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: osdmap e69: 10 total, 10 up, 10 in 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:09:51.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:50 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:09:52.089 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:51 smithi083 ceph-mon[36340]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:52.089 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:51 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi028 2024-03-22T02:09:52.089 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:51 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 4273M 2024-03-22T02:09:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:51 smithi120 ceph-mon[36803]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:51 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi028 2024-03-22T02:09:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:51 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 4273M 2024-03-22T02:09:52.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 -- 172.21.15.83:0/834643882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf8103150 msgr2=0x7facf8109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:52.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 --2- 172.21.15.83:0/834643882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf8103150 0x7facf8109a00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7facf000b0d0 tx=0x7facf000b3e0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 -- 172.21.15.83:0/834643882 shutdown_connections 2024-03-22T02:09:52.298 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 --2- 172.21.15.83:0/834643882 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf8103150 0x7facf8109a00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 --2- 172.21.15.83:0/834643882 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf81015b0 0x7facf81019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 --2- 172.21.15.83:0/834643882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 0x7facf8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 -- 172.21.15.83:0/834643882 >> 172.21.15.83:0/834643882 conn(0x7facf80fcb50 msgr2=0x7facf80fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 -- 172.21.15.83:0/834643882 shutdown_connections 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.296+0000 7facfeb64700 1 -- 172.21.15.83:0/834643882 wait complete. 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 Processor -- start 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 -- start start 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 0x7facf8103b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 0x7facf81040d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7facf810d1a0 con 0x7facf81027b0 2024-03-22T02:09:52.299 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7facf810d020 con 0x7facf81015b0 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfeb64700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7facf810d320 con 0x7facf8103150 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.83:33614/0 (socket says 172.21.15.83:33614) 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 -- 172.21.15.83:0/2383507849 learned_addr learned my addr 172.21.15.83:0/2383507849 (peer_addr_for_me v2:172.21.15.83:0/0) 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facf7fff700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 0x7facf8103b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfd101700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 0x7facf81040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 -- 172.21.15.83:0/2383507849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 msgr2=0x7facf81040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 0x7facf81040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 -- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 msgr2=0x7facf8103b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 0x7facf8103b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfc900700 1 -- 172.21.15.83:0/2383507849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facf8104980 con 0x7facf81015b0 2024-03-22T02:09:52.300 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.297+0000 7facfd101700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 0x7facf81040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facfc900700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facec00d980 tx=0x7facec00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec017070 con 0x7facf81015b0 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facf81a2a70 con 0x7facf81015b0 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facec009e10 con 0x7facf81015b0 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec00f620 con 0x7facf81015b0 2024-03-22T02:09:52.301 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.298+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facf81a2f80 con 0x7facf81015b0 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.299+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facf81a31b0 con 0x7facf81015b0 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.300+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7facec009930 con 0x7facf81015b0 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.300+0000 7facf5ffb700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 0x7face80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.301+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7facec0ce7a0 con 0x7facf81015b0 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.301+0000 7facf7fff700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 0x7face80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:09:52.302 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.301+0000 7facf7fff700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 0x7face80a5470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7face4005950 tx=0x7face400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:09:52.305 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.305+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7facec09bdd0 con 0x7facf81015b0 2024-03-22T02:09:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:51 smithi028 ceph-mon[31916]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:51 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi028 2024-03-22T02:09:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:51 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 4273M 2024-03-22T02:09:52.508 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:09:52.508+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7facf80fecf0 con 0x7face80a2fb0 2024-03-22T02:09:53.233 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:53.234 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:53.234 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:52 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:53.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:53.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:53.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:52 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:53.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:09:53.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:09:53.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:52 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:09:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:53 smithi120 ceph-mon[36803]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:53 smithi120 ceph-mon[36803]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:54.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:53 smithi083 ceph-mon[36340]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:54.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:53 smithi083 ceph-mon[36340]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:53 smithi028 ceph-mon[31916]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi083:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:09:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:53 smithi028 ceph-mon[31916]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/1777143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]': finished 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T02:09:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:55 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/1777143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]': finished 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T02:09:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:55 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:09:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/1777143182' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]: dispatch 2024-03-22T02:09:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9eafc799-33ff-4ef4-b4ec-235fcd7b9366"}]': finished 2024-03-22T02:09:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: osdmap e70: 11 total, 10 up, 11 in 2024-03-22T02:09:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:55 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:09:57.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:56 smithi120 ceph-mon[36803]: from='client.? 172.21.15.83:0/1249673240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:56 smithi083 ceph-mon[36340]: from='client.? 172.21.15.83:0/1249673240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:57.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:56 smithi028 ceph-mon[31916]: from='client.? 172.21.15.83:0/1249673240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:09:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:57 smithi120 ceph-mon[36803]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:57 smithi083 ceph-mon[36340]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:09:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:57 smithi028 ceph-mon[31916]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:09:59 smithi028 ceph-mon[31916]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:00.215 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:09:59 smithi083 ceph-mon[36340]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:09:59 smithi120 ceph-mon[36803]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:01.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:00 smithi120 ceph-mon[36803]: overall HEALTH_OK 2024-03-22T02:10:01.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:00 smithi083 ceph-mon[36340]: overall HEALTH_OK 2024-03-22T02:10:01.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:00 smithi028 ceph-mon[31916]: overall HEALTH_OK 2024-03-22T02:10:02.212 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:01 smithi083 ceph-mon[36340]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:01 smithi120 ceph-mon[36803]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:01 smithi028 ceph-mon[31916]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:04.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:03 smithi083 ceph-mon[36340]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:04.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T02:10:04.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:04.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:03 smithi083 ceph-mon[36340]: Deploying daemon osd.10 on smithi083 2024-03-22T02:10:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:03 smithi120 ceph-mon[36803]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T02:10:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:03 smithi120 ceph-mon[36803]: Deploying daemon osd.10 on smithi083 2024-03-22T02:10:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:03 smithi028 ceph-mon[31916]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-22T02:10:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:03 smithi028 ceph-mon[31916]: Deploying daemon osd.10 on smithi083 2024-03-22T02:10:06.075 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:05 smithi083 ceph-mon[36340]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:05 smithi120 ceph-mon[36803]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:06.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:05 smithi028 ceph-mon[31916]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:08.246 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:07 smithi120 ceph-mon[36803]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:08.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:07 smithi083 ceph-mon[36340]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:07 smithi028 ceph-mon[31916]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:10.132 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:09 smithi083 ceph-mon[36340]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:10.133 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.133 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.133 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:09 smithi120 ceph-mon[36803]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:09 smithi028 ceph-mon[31916]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:11.926 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:11 smithi083 ceph-mon[36340]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:12.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:11 smithi120 ceph-mon[36803]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:11 smithi028 ceph-mon[31916]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:12.827 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.827+0000 7facf5ffb700 1 -- 172.21.15.83:0/2383507849 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7facf80fecf0 con 0x7face80a2fb0 2024-03-22T02:10:12.828 INFO:teuthology.orchestra.run.smithi083.stdout:Created osd(s) 10 on host 'smithi083' 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 msgr2=0x7face80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 0x7face80a5470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7face4005950 tx=0x7face400b410 comp rx=0 tx=0).stop 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 msgr2=0x7facf8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7facec00d980 tx=0x7facec00dc90 comp rx=0 tx=0).stop 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 shutdown_connections 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7facf8103150 0x7facf81040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7facf81015b0 0x7facf8105540 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:12.830 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7face80a2fb0 0x7face80a5470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 --2- 172.21.15.83:0/2383507849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facf81027b0 0x7facf8103b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 >> 172.21.15.83:0/2383507849 conn(0x7facf80fcb50 msgr2=0x7facf80fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 shutdown_connections 2024-03-22T02:10:12.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-03-22T02:10:12.830+0000 7facfeb64700 1 -- 172.21.15.83:0/2383507849 wait complete. 2024-03-22T02:10:13.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:12 smithi083 ceph-mon[36340]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:13.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:12 smithi120 ceph-mon[36803]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:12 smithi028 ceph-mon[31916]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-22T02:10:13.535 DEBUG:teuthology.orchestra.run.smithi083:osd.10> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.10.service 2024-03-22T02:10:13.538 INFO:tasks.cephadm:Deploying osd.11 on smithi120 with /dev/vg_nvme/lv_1... 2024-03-22T02:10:13.538 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T02:10:13.784 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:14.166 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:13 smithi120 ceph-mon[36803]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:14.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:14.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T02:10:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T02:10:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:13 smithi083 ceph-mon[36340]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:14.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:14.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-22T02:10:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: osdmap e71: 11 total, 10 up, 11 in 2024-03-22T02:10:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: from='osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:13 smithi028 ceph-mon[31916]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]: dispatch 2024-03-22T02:10:15.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:10:15.231+0000 7f8fe0017700 -1 osd.10 0 waiting for initial osdmap 2024-03-22T02:10:15.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:10:15.246+0000 7f8fd7e02700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110624 s, 948 MB/s 2024-03-22T02:10:15.971 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-22T02:10:16.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:10:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T02:10:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:15 smithi028 ceph-mon[31916]: from='osd.10 ' entity='osd.10' 2024-03-22T02:10:16.209 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:16.210 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:10:16.210 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T02:10:16.210 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.210 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.210 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:15 smithi083 ceph-mon[36340]: from='osd.10 ' entity='osd.10' 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi083", "root=default"]}]': finished 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: osdmap e72: 11 total, 10 up, 11 in 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:15 smithi120 ceph-mon[36803]: from='osd.10 ' entity='osd.10' 2024-03-22T02:10:16.712 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_1 2024-03-22T02:10:16.952 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:10:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] boot 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: osdmap e73: 11 total, 11 up, 11 in 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:17.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:17.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:10:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] boot 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: osdmap e73: 11 total, 11 up, 11 in 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:17.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:10:17.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:10:17.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: osd.10 [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] boot 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: osdmap e73: 11 total, 11 up, 11 in 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:17 smithi028 ceph-mon[31916]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:17 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi083 2024-03-22T02:10:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:17 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 4270M 2024-03-22T02:10:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:17 smithi028 ceph-mon[31916]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T02:10:18.114 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:17 smithi120 ceph-mon[36803]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:18.114 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:17 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi083 2024-03-22T02:10:18.114 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:17 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 4270M 2024-03-22T02:10:18.114 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:17 smithi120 ceph-mon[36803]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T02:10:18.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:17 smithi083 ceph-mon[36340]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:17 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi083 2024-03-22T02:10:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:17 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 4270M 2024-03-22T02:10:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:17 smithi083 ceph-mon[36340]: osdmap e74: 11 total, 11 up, 11 in 2024-03-22T02:10:18.668 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 -- 172.21.15.120:0/413110187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94101015b0 msgr2=0x7f94101019d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 --2- 172.21.15.120:0/413110187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94101015b0 0x7f94101019d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f940c009990 tx=0x7f940c009ca0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 -- 172.21.15.120:0/413110187 shutdown_connections 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 --2- 172.21.15.120:0/413110187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9410103150 0x7f9410109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 --2- 172.21.15.120:0/413110187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94101015b0 0x7f94101019d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 --2- 172.21.15.120:0/413110187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101027b0 0x7f9410102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 -- 172.21.15.120:0/413110187 >> 172.21.15.120:0/413110187 conn(0x7f94100fcb50 msgr2=0x7f94100fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 -- 172.21.15.120:0/413110187 shutdown_connections 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.661+0000 7f94181e3700 1 -- 172.21.15.120:0/413110187 wait complete. 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 Processor -- start 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 -- start start 2024-03-22T02:10:18.669 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101015b0 0x7f941006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94101027b0 0x7f941019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f941010d1a0 con 0x7f94101027b0 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f941010d020 con 0x7f94101015b0 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f94181e3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f941010d320 con 0x7f9410103150 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40140/0 (socket says 172.21.15.120:40140) 2024-03-22T02:10:18.670 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 -- 172.21.15.120:0/321558892 learned_addr learned my addr 172.21.15.120:0/321558892 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f941577e700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94101027b0 0x7f941019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9415f7f700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101015b0 0x7f941006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 -- 172.21.15.120:0/321558892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101015b0 msgr2=0x7f941006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101015b0 0x7f941006aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 -- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94101027b0 msgr2=0x7f941019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94101027b0 0x7f941019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.662+0000 7f9416780700 1 -- 172.21.15.120:0/321558892 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94101a05a0 con 0x7f9410103150 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f9416780700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9408007ee0 tx=0x7f940800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940800dbd0 con 0x7f9410103150 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94101a0890 con 0x7f9410103150 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f940800dd70 con 0x7f9410103150 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94101a0dd0 con 0x7f9410103150 2024-03-22T02:10:18.671 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.663+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9408005a60 con 0x7f9410103150 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.664+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9410069340 con 0x7f9410103150 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.665+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f940801a020 con 0x7f9410103150 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.665+0000 7f9406ffd700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 0x7f93fc0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.665+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f94080ceb80 con 0x7f9410103150 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.666+0000 7f9415f7f700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 0x7f93fc0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.666+0000 7f9415f7f700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 0x7f93fc0a5470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f940c00b5c0 tx=0x7f940c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:18.672 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.669+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9408010070 con 0x7f9410103150 2024-03-22T02:10:18.873 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:18.873+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9410062ac0 con 0x7f93fc0a2fb0 2024-03-22T02:10:19.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:18 smithi028 ceph-mon[31916]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T02:10:19.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:18 smithi028 ceph-mon[31916]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T02:10:19.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:18 smithi120 ceph-mon[36803]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T02:10:19.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:18 smithi120 ceph-mon[36803]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T02:10:19.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:18 smithi083 ceph-mon[36340]: osdmap e75: 11 total, 11 up, 11 in 2024-03-22T02:10:19.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:18 smithi083 ceph-mon[36340]: osdmap e76: 11 total, 11 up, 11 in 2024-03-22T02:10:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:10:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:10:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:19.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:19 smithi120 ceph-mon[36803]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T02:10:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:20.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:10:20.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:10:20.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:20.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:19 smithi028 ceph-mon[31916]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T02:10:20.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:19 smithi083 ceph-mon[36340]: osdmap e77: 11 total, 11 up, 11 in 2024-03-22T02:10:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:10:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/542773166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]': finished 2024-03-22T02:10:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: osdmap e78: 12 total, 11 up, 12 in 2024-03-22T02:10:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:22.152 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:10:22.152 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/542773166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.152 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.152 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]': finished 2024-03-22T02:10:22.153 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: osdmap e78: 12 total, 11 up, 12 in 2024-03-22T02:10:22.153 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-22T02:10:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/542773166' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]: dispatch 2024-03-22T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e"}]': finished 2024-03-22T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: osdmap e78: 12 total, 11 up, 12 in 2024-03-22T02:10:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:23.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:22 smithi028 ceph-mon[31916]: from='client.? 172.21.15.120:0/928749612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:10:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:22 smithi120 ceph-mon[36803]: from='client.? 172.21.15.120:0/928749612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:10:23.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:22 smithi083 ceph-mon[36340]: from='client.? 172.21.15.120:0/928749612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T02:10:24.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:23 smithi028 ceph-mon[31916]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-22T02:10:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:23 smithi120 ceph-mon[36803]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-22T02:10:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:23 smithi083 ceph-mon[36340]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-22T02:10:26.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:25 smithi120 ceph-mon[36803]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-22T02:10:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:25 smithi083 ceph-mon[36340]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-22T02:10:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:25 smithi028 ceph-mon[31916]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-22T02:10:28.192 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:27 smithi120 ceph-mon[36803]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-22T02:10:28.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:27 smithi083 ceph-mon[36340]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-22T02:10:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:27 smithi028 ceph-mon[31916]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-22T02:10:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:29 smithi028 ceph-mon[31916]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T02:10:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:29 smithi028 ceph-mon[31916]: Deploying daemon osd.11 on smithi120 2024-03-22T02:10:30.104 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:29 smithi120 ceph-mon[36803]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:30.104 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T02:10:30.104 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:30.104 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:29 smithi120 ceph-mon[36803]: Deploying daemon osd.11 on smithi120 2024-03-22T02:10:30.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:29 smithi083 ceph-mon[36340]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-22T02:10:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:29 smithi083 ceph-mon[36340]: Deploying daemon osd.11 on smithi120 2024-03-22T02:10:31.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:31 smithi120 ceph-mon[36803]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:31 smithi083 ceph-mon[36340]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:32.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:31 smithi028 ceph-mon[31916]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:34.159 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:33 smithi120 ceph-mon[36803]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:34.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:33 smithi083 ceph-mon[36340]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:33 smithi028 ceph-mon[31916]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:36 smithi120 ceph-mon[36803]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:36.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:36 smithi083 ceph-mon[36340]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:36.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:36 smithi028 ceph-mon[31916]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:38.360 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:38 smithi120 ceph-mon[36803]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:38 smithi083 ceph-mon[36340]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:38 smithi028 ceph-mon[31916]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:38.741 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 11 on host 'smithi120' 2024-03-22T02:10:38.742 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.740+0000 7f9406ffd700 1 -- 172.21.15.120:0/321558892 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9410062ac0 con 0x7f93fc0a2fb0 2024-03-22T02:10:38.744 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 msgr2=0x7f93fc0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 0x7f93fc0a5470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f940c00b5c0 tx=0x7f940c005fb0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 msgr2=0x7f941019fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9408007ee0 tx=0x7f940800d3b0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 shutdown_connections 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9410103150 0x7f941019fcd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94101015b0 0x7f941006aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93fc0a2fb0 0x7f93fc0a5470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 --2- 172.21.15.120:0/321558892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94101027b0 0x7f941019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 >> 172.21.15.120:0/321558892 conn(0x7f94100fcb50 msgr2=0x7f9410105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:38.745 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 shutdown_connections 2024-03-22T02:10:38.746 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:38.744+0000 7f94181e3700 1 -- 172.21.15.120:0/321558892 wait complete. 2024-03-22T02:10:39.461 DEBUG:teuthology.orchestra.run.smithi120:osd.11> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.11.service 2024-03-22T02:10:39.464 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-22T02:10:39.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T02:10:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:10:39.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:39.776 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.777 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-22T02:10:39.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:39.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:40.902 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:40 smithi120 ceph-mon[36803]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T02:10:40.902 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:40 smithi120 ceph-mon[36803]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T02:10:40.903 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:40 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:40.903 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:40 smithi120 ceph-mon[36803]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:40.903 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:40 smithi120 ceph-mon[36803]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:41.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:40 smithi083 ceph-mon[36340]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T02:10:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:40 smithi083 ceph-mon[36340]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T02:10:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:40 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:40 smithi083 ceph-mon[36340]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:40 smithi083 ceph-mon[36340]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:40 smithi028 ceph-mon[31916]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-22T02:10:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:40 smithi028 ceph-mon[31916]: osdmap e79: 12 total, 11 up, 12 in 2024-03-22T02:10:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:40 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:40 smithi028 ceph-mon[31916]: from='osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:40 smithi028 ceph-mon[31916]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-22T02:10:41.164 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:10:41.150+0000 7f8f14b18700 -1 osd.11 0 waiting for initial osdmap 2024-03-22T02:10:41.164 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:10:41.163+0000 7f8f0c903700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 -- 172.21.15.28:0/1303948424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 msgr2=0x7f1978101d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 --2- 172.21.15.28:0/1303948424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f1978101d10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1968009990 tx=0x7f1968009ca0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 -- 172.21.15.28:0/1303948424 shutdown_connections 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 --2- 172.21.15.28:0/1303948424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978102250 0x7f19781026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 --2- 172.21.15.28:0/1303948424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978100f30 0x7f1978101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 --2- 172.21.15.28:0/1303948424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f1978101d10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 -- 172.21.15.28:0/1303948424 >> 172.21.15.28:0/1303948424 conn(0x7f19780fcb70 msgr2=0x7f19780fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 -- 172.21.15.28:0/1303948424 shutdown_connections 2024-03-22T02:10:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.491+0000 7f197ff15700 1 -- 172.21.15.28:0/1303948424 wait complete. 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 Processor -- start 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 -- start start 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978100f30 0x7f197819ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 0x7f19781a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f19781104f0 con 0x7f1978101890 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1978110370 con 0x7f1978102250 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1978110670 con 0x7f1978100f30 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54924/0 (socket says 172.21.15.28:54924) 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 -- 172.21.15.28:0/1568881236 learned_addr learned my addr 172.21.15.28:0/1568881236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197dcb1700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978100f30 0x7f197819ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:41.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197e4b2700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 0x7f19781a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 -- 172.21.15.28:0/1568881236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978100f30 msgr2=0x7f197819ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978100f30 0x7f197819ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 -- 172.21.15.28:0/1568881236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 msgr2=0x7f19781a20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 0x7f19781a20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 -- 172.21.15.28:0/1568881236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19781a2790 con 0x7f1978101890 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197e4b2700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 0x7f19781a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197d4b0700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1968004790 tx=0x7f1968009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968027070 con 0x7f1978101890 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1968004150 con 0x7f1978101890 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.492+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19781a2a20 con 0x7f1978101890 2024-03-22T02:10:41.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.493+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196800f460 con 0x7f1978101890 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.493+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19781a2ef0 con 0x7f1978101890 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.493+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1978069340 con 0x7f1978101890 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.495+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f196802a070 con 0x7f1978101890 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.495+0000 7f196effd700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 0x7f19640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.495+0000 7f197dcb1700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 0x7f19640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.496+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f19680d04f0 con 0x7f1978101890 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.496+0000 7f197dcb1700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 0x7f19640a5470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1974009a10 tx=0x7f1974009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.499+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1968031070 con 0x7f1978101890 2024-03-22T02:10:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.703+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f197802df80 con 0x7f1978101890 2024-03-22T02:10:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.704+0000 7f196effd700 1 -- 172.21.15.28:0/1568881236 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f1968013070 con 0x7f1978101890 2024-03-22T02:10:41.704 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 msgr2=0x7f19640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 0x7f19640a5470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1974009a10 tx=0x7f1974009190 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 msgr2=0x7f197819b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1968004790 tx=0x7f1968009580 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 shutdown_connections 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1978102250 0x7f19781a20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1978100f30 0x7f197819ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f19640a2fb0 0x7f19640a5470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 --2- 172.21.15.28:0/1568881236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1978101890 0x7f197819b070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 >> 172.21.15.28:0/1568881236 conn(0x7f19780fcb70 msgr2=0x7f19780fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 shutdown_connections 2024-03-22T02:10:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:41.706+0000 7f197ff15700 1 -- 172.21.15.28:0/1568881236 wait complete. 2024-03-22T02:10:41.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:41.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:10:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T02:10:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: from='osd.11 ' entity='osd.11' 2024-03-22T02:10:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:41 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1568881236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:42.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: from='osd.11 ' entity='osd.11' 2024-03-22T02:10:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:41 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1568881236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:42.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:42.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-22T02:10:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: osdmap e80: 12 total, 11 up, 12 in 2024-03-22T02:10:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: from='osd.11 ' entity='osd.11' 2024-03-22T02:10:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:41 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1568881236' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:42.581 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1711073415,"num_in_osds":12,"osd_in_since":1711073421,"num_remapped_pgs":0} 2024-03-22T02:10:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: purged_snaps scrub starts 2024-03-22T02:10:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: purged_snaps scrub ok 2024-03-22T02:10:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] boot 2024-03-22T02:10:42.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: osdmap e81: 12 total, 12 up, 12 in 2024-03-22T02:10:42.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:42 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: purged_snaps scrub starts 2024-03-22T02:10:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: purged_snaps scrub ok 2024-03-22T02:10:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] boot 2024-03-22T02:10:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: osdmap e81: 12 total, 12 up, 12 in 2024-03-22T02:10:42.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:42 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:43.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: purged_snaps scrub starts 2024-03-22T02:10:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: purged_snaps scrub ok 2024-03-22T02:10:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: osd.11 [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] boot 2024-03-22T02:10:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: osdmap e81: 12 total, 12 up, 12 in 2024-03-22T02:10:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:42 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-22T02:10:43.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T02:10:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:10:44.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:44.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T02:10:44.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: Detected new or changed devices on smithi120 2024-03-22T02:10:44.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:44.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:44.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.205 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:44 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T02:10:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: Detected new or changed devices on smithi120 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:44 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: osdmap e82: 12 total, 12 up, 12 in 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: Detected new or changed devices on smithi120 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:44.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:44.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:44.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:44 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:45.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:45.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:45.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:45 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:45.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:45.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:45.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:45 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:45.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:45.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:45.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:45 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- 172.21.15.28:0/1290534985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01038d0 msgr2=0x7fd6e0103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1290534985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01038d0 0x7fd6e0103d30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00099c0 tx=0x7fd6d0009cd0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- 172.21.15.28:0/1290534985 shutdown_connections 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1290534985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 0x7fd6e010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1290534985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01026d0 0x7fd6e0102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1290534985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01038d0 0x7fd6e0103d30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- 172.21.15.28:0/1290534985 >> 172.21.15.28:0/1290534985 conn(0x7fd6e00fdc90 msgr2=0x7fd6e01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- 172.21.15.28:0/1290534985 shutdown_connections 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- 172.21.15.28:0/1290534985 wait complete. 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 Processor -- start 2024-03-22T02:10:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.715+0000 7fd6e737a700 1 -- start start 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01038d0 0x7fd6e0198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 0x7fd6e019fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010e2a0 con 0x7fd6e01026d0 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010e120 con 0x7fd6e01038d0 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e737a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6e010e420 con 0x7fd6e0104270 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36120/0 (socket says 172.21.15.28:36120) 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 -- 172.21.15.28:0/1360810414 learned_addr learned my addr 172.21.15.28:0/1360810414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5917700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 0x7fd6e019fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e4915700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01038d0 0x7fd6e0198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 -- 172.21.15.28:0/1360810414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 msgr2=0x7fd6e019fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 0x7fd6e019fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 -- 172.21.15.28:0/1360810414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01038d0 msgr2=0x7fd6e0198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01038d0 0x7fd6e0198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 -- 172.21.15.28:0/1360810414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e01a0540 con 0x7fd6e01026d0 2024-03-22T02:10:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6e5116700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00d9b0 tx=0x7fd6dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc016070 con 0x7fd6e01026d0 2024-03-22T02:10:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.716+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6dc00be30 con 0x7fd6e01026d0 2024-03-22T02:10:45.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.717+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6e01a0830 con 0x7fd6e01026d0 2024-03-22T02:10:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.717+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc00f3c0 con 0x7fd6e01026d0 2024-03-22T02:10:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.717+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6e01a0dc0 con 0x7fd6e01026d0 2024-03-22T02:10:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.717+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6e0069340 con 0x7fd6e01026d0 2024-03-22T02:10:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.719+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fd6dc019070 con 0x7fd6e01026d0 2024-03-22T02:10:45.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.719+0000 7fd6d67fc700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 0x7fd6cc0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.720+0000 7fd6e4915700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 0x7fd6cc0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.720+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd6dc0cf280 con 0x7fd6e01026d0 2024-03-22T02:10:45.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.720+0000 7fd6e4915700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 0x7fd6cc0a54f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000c00 tx=0x7fd6d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:45.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.723+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd6dc020030 con 0x7fd6e01026d0 2024-03-22T02:10:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.931+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd6e002dd70 con 0x7fd6e01026d0 2024-03-22T02:10:45.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.931+0000 7fd6d67fc700 1 -- 172.21.15.28:0/1360810414 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7fd6dc09c550 con 0x7fd6e01026d0 2024-03-22T02:10:45.932 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 msgr2=0x7fd6cc0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 0x7fd6cc0a54f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0000c00 tx=0x7fd6d0005fb0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 msgr2=0x7fd6e01988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc00d9b0 tx=0x7fd6dc00dcc0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 shutdown_connections 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd6cc0a3030 0x7fd6cc0a54f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e01026d0 0x7fd6e01988e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6e0104270 0x7fd6e019fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 --2- 172.21.15.28:0/1360810414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd6e01038d0 0x7fd6e0198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 >> 172.21.15.28:0/1360810414 conn(0x7fd6e00fdc90 msgr2=0x7fd6e0106c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 shutdown_connections 2024-03-22T02:10:45.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:45.934+0000 7fd6e737a700 1 -- 172.21.15.28:0/1360810414 wait complete. 2024-03-22T02:10:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:46 smithi120 ceph-mon[36803]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1360810414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:46 smithi083 ceph-mon[36340]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1360810414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:46 smithi028 ceph-mon[31916]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1360810414' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T02:10:46.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711073442,"num_in_osds":12,"osd_in_since":1711073421,"num_remapped_pgs":0} 2024-03-22T02:10:46.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:10:46.869 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:10:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:48 smithi028 ceph-mon[31916]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:48 smithi120 ceph-mon[36803]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:48.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:48 smithi083 ceph-mon[36340]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.622+0000 7fce86b9e700 1 -- 172.21.15.28:0/1133178357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 msgr2=0x7fce8006b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.622+0000 7fce86b9e700 1 --2- 172.21.15.28:0/1133178357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8006b250 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fce740099c0 tx=0x7fce74009cd0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- 172.21.15.28:0/1133178357 shutdown_connections 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- 172.21.15.28:0/1133178357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 0x7fce8010f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- 172.21.15.28:0/1133178357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 0x7fce801078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- 172.21.15.28:0/1133178357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8006b250 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- 172.21.15.28:0/1133178357 >> 172.21.15.28:0/1133178357 conn(0x7fce80078590 msgr2=0x7fce800789a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- 172.21.15.28:0/1133178357 shutdown_connections 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- 172.21.15.28:0/1133178357 wait complete. 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 Processor -- start 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- start start 2024-03-22T02:10:48.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 0x7fce8019b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 0x7fce8019f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce80112980 con 0x7fce8006add0 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce80112800 con 0x7fce8006b790 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.623+0000 7fce86b9e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce80112b00 con 0x7fce801074f0 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce7ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 0x7fce8019b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36140/0 (socket says 172.21.15.28:36140) 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 -- 172.21.15.28:0/3699156980 learned_addr learned my addr 172.21.15.28:0/3699156980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8513b700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 0x7fce8019f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 -- 172.21.15.28:0/3699156980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 msgr2=0x7fce8019f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 0x7fce8019f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 -- 172.21.15.28:0/3699156980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 msgr2=0x7fce8019b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 0x7fce8019b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 -- 172.21.15.28:0/3699156980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce801a00b0 con 0x7fce8006add0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce8493a700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fce7000d950 tx=0x7fce7000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70016070 con 0x7fce8006add0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce7000fd70 con 0x7fce8006add0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce801a03a0 con 0x7fce8006add0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7000e610 con 0x7fce8006add0 2024-03-22T02:10:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.624+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce801a7c20 con 0x7fce8006add0 2024-03-22T02:10:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.625+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce80069340 con 0x7fce8006add0 2024-03-22T02:10:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.626+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fce70019070 con 0x7fce8006add0 2024-03-22T02:10:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.627+0000 7fce7dffb700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 0x7fce680a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.627+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fce700ce8b0 con 0x7fce8006add0 2024-03-22T02:10:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.627+0000 7fce7ffff700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 0x7fce680a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.627+0000 7fce7ffff700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 0x7fce680a54f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce80078b00 tx=0x7fce74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.632+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce7009bb80 con 0x7fce8006add0 2024-03-22T02:10:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.849+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce8002dd70 con 0x7fce8006add0 2024-03-22T02:10:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.850+0000 7fce7dffb700 1 -- 172.21.15.28:0/3699156980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18405 (secure 0 0 0) 0x7fce7009b480 con 0x7fce8006add0 2024-03-22T02:10:48.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:48.851 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:10:43.155372+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:10:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 msgr2=0x7fce680a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 0x7fce680a54f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce80078b00 tx=0x7fce74005fb0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 msgr2=0x7fce8019b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fce7000d950 tx=0x7fce7000dc60 comp rx=0 tx=0).stop 2024-03-22T02:10:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 shutdown_connections 2024-03-22T02:10:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce680a3030 0x7fce680a54f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8006add0 0x7fce8019b0a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce801074f0 0x7fce8019f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 --2- 172.21.15.28:0/3699156980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce8006b790 0x7fce8019b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 >> 172.21.15.28:0/3699156980 conn(0x7fce80078590 msgr2=0x7fce8010da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 shutdown_connections 2024-03-22T02:10:48.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:48.852+0000 7fce86b9e700 1 -- 172.21.15.28:0/3699156980 wait complete. 2024-03-22T02:10:49.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:49 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3699156980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:49.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:49 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3699156980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:49.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:49 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3699156980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:49.648 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:10:49.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-22T02:10:49.897 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:10:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:50 smithi028 ceph-mon[31916]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:50.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:50 smithi120 ceph-mon[36803]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:50.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:50 smithi083 ceph-mon[36340]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 -- 172.21.15.28:0/2731475562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894107590 msgr2=0x7f7894107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 --2- 172.21.15.28:0/2731475562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894107590 0x7f7894107970 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7884009990 tx=0x7f7884009ca0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 -- 172.21.15.28:0/2731475562 shutdown_connections 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 --2- 172.21.15.28:0/2731475562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894107590 0x7f7894107970 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 --2- 172.21.15.28:0/2731475562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101f40 0x7f789410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 --2- 172.21.15.28:0/2731475562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894101580 0x7f7894101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 -- 172.21.15.28:0/2731475562 >> 172.21.15.28:0/2731475562 conn(0x7f78940784b0 msgr2=0x7f78940788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:51.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 -- 172.21.15.28:0/2731475562 shutdown_connections 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.706+0000 7f789a12d700 1 -- 172.21.15.28:0/2731475562 wait complete. 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 Processor -- start 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 -- start start 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101580 0x7f7894198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894107590 0x7f789419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7894111890 con 0x7f7894101f40 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7894111710 con 0x7f7894101580 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f789a12d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7894111a10 con 0x7f7894107590 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47970/0 (socket says 172.21.15.28:47970) 2024-03-22T02:10:51.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 -- 172.21.15.28:0/1152804093 learned_addr learned my addr 172.21.15.28:0/1152804093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f78937fe700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101580 0x7f7894198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f7893fff700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894107590 0x7f789419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 -- 172.21.15.28:0/1152804093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894107590 msgr2=0x7f789419d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894107590 0x7f789419d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 -- 172.21.15.28:0/1152804093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101580 msgr2=0x7f7894198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101580 0x7f7894198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.707+0000 7f788affd700 1 -- 172.21.15.28:0/1152804093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f789419dd70 con 0x7f7894101f40 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f788affd700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c920 tx=0x7f787c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c007b00 con 0x7f7894101f40 2024-03-22T02:10:51.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f789419e060 con 0x7f7894101f40 2024-03-22T02:10:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f787c004d10 con 0x7f7894101f40 2024-03-22T02:10:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c005630 con 0x7f7894101f40 2024-03-22T02:10:51.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.708+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78941a58e0 con 0x7f7894101f40 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.709+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7894069340 con 0x7f7894101f40 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.710+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f787c004750 con 0x7f7894101f40 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.711+0000 7f78917fa700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 0x7f78800a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.711+0000 7f78937fe700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 0x7f78800a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.711+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f787c0ceb80 con 0x7f7894101f40 2024-03-22T02:10:51.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.711+0000 7f78937fe700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 0x7f78800a5470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f788400b5c0 tx=0x7f7884005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:51.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.715+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f787c014030 con 0x7f7894101f40 2024-03-22T02:10:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.916+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f78941a5c00 con 0x7f7894101f40 2024-03-22T02:10:51.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.916+0000 7f78917fa700 1 -- 172.21.15.28:0/1152804093 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f787c09bdb0 con 0x7f7894101f40 2024-03-22T02:10:51.918 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:10:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 msgr2=0x7f78800a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 0x7f78800a5470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f788400b5c0 tx=0x7f7884005960 comp rx=0 tx=0).stop 2024-03-22T02:10:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 msgr2=0x7f78941992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c920 tx=0x7f787c00cc30 comp rx=0 tx=0).stop 2024-03-22T02:10:51.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 shutdown_connections 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7894107590 0x7f789419d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7894101580 0x7f7894198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f78800a2fb0 0x7f78800a5470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 --2- 172.21.15.28:0/1152804093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7894101f40 0x7f78941992a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 >> 172.21.15.28:0/1152804093 conn(0x7f78940784b0 msgr2=0x7f7894100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 shutdown_connections 2024-03-22T02:10:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:51.919+0000 7f789a12d700 1 -- 172.21.15.28:0/1152804093 wait complete. 2024-03-22T02:10:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:52 smithi120 ceph-mon[36803]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:52 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1152804093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:52.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:52 smithi083 ceph-mon[36340]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:52.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:52 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1152804093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:52 smithi028 ceph-mon[31916]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1152804093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:52.921 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-03-22T02:10:52.921 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-03-22T02:10:52.921 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-03-22T02:10:52.921 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.b on smithi083 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.e on smithi083 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.h on smithi083 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.k on smithi083 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.c on smithi120 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.f on smithi120 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.i on smithi120 2024-03-22T02:10:52.922 INFO:tasks.cephadm:Adding mds.l on smithi120 2024-03-22T02:10:52.922 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l' 2024-03-22T02:10:53.170 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:10:54.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:54 smithi120 ceph-mon[36803]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:54.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:54 smithi083 ceph-mon[36340]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:54 smithi028 ceph-mon[31916]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:54.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 -- 172.21.15.120:0/3243091068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 msgr2=0x7f4df0103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:54.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/3243091068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0103d30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4de4009990 tx=0x7f4de4009ca0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 -- 172.21.15.120:0/3243091068 shutdown_connections 2024-03-22T02:10:54.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/3243091068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df0104270 0x7f4df010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.805 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/3243091068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df01026d0 0x7f4df0102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/3243091068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0103d30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.804+0000 7f4df5fdc700 1 -- 172.21.15.120:0/3243091068 >> 172.21.15.120:0/3243091068 conn(0x7f4df00fdc90 msgr2=0x7f4df01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:54.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.805+0000 7f4df5fdc700 1 -- 172.21.15.120:0/3243091068 shutdown_connections 2024-03-22T02:10:54.806 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.805+0000 7f4df5fdc700 1 -- 172.21.15.120:0/3243091068 wait complete. 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.805+0000 7f4df5fdc700 1 Processor -- start 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.805+0000 7f4df5fdc700 1 -- start start 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df01026d0 0x7f4df0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df0104270 0x7f4df019ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010e330 con 0x7f4df01026d0 2024-03-22T02:10:54.808 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010e1b0 con 0x7f4df0104270 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4df5fdc700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010e4b0 con 0x7f4df01038d0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:35916/0 (socket says 172.21.15.120:35916) 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 -- 172.21.15.120:0/21248451 learned_addr learned my addr 172.21.15.120:0/21248451 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4def7fe700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df01026d0 0x7f4df0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deffff700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df0104270 0x7f4df019ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 -- 172.21.15.120:0/21248451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df0104270 msgr2=0x7f4df019ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df0104270 0x7f4df019ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 -- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df01026d0 msgr2=0x7f4df0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df01026d0 0x7f4df0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 -- 172.21.15.120:0/21248451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df01a05d0 con 0x7f4df01038d0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4deeffd700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4de40047b0 tx=0x7f4de4003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:54.809 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4027070 con 0x7f4df01038d0 2024-03-22T02:10:54.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de4022470 con 0x7f4df01038d0 2024-03-22T02:10:54.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.806+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de400f6c0 con 0x7f4df01038d0 2024-03-22T02:10:54.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.807+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df01a0860 con 0x7f4df01038d0 2024-03-22T02:10:54.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.807+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df01a0d90 con 0x7f4df01038d0 2024-03-22T02:10:54.810 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.807+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df0069340 con 0x7f4df01038d0 2024-03-22T02:10:54.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.808+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4de402a070 con 0x7f4df01038d0 2024-03-22T02:10:54.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.809+0000 7f4decff9700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 0x7f4dd80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:54.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.809+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4de40cfff0 con 0x7f4df01038d0 2024-03-22T02:10:54.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.809+0000 7f4def7fe700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 0x7f4dd80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:54.811 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.810+0000 7f4def7fe700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 0x7f4dd80a54f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006fd0 tx=0x7f4de0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:54.814 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:54.814+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4de405e740 con 0x7f4df01038d0 2024-03-22T02:10:55.017 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.016+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}) -- 0x7f4df0062cc0 con 0x7f4dd80a3030 2024-03-22T02:10:55.025 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.024+0000 7f4decff9700 1 -- 172.21.15.120:0/21248451 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f4df0062cc0 con 0x7f4dd80a3030 2024-03-22T02:10:55.025 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mds.all update... 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 msgr2=0x7f4dd80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 0x7f4dd80a54f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006fd0 tx=0x7f4de0008040 comp rx=0 tx=0).stop 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 msgr2=0x7f4df0198eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4de40047b0 tx=0x7f4de4003680 comp rx=0 tx=0).stop 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 shutdown_connections 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4dd80a3030 0x7f4dd80a54f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df01026d0 0x7f4df0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:55.027 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4df0104270 0x7f4df019ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:55.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.026+0000 7f4df5fdc700 1 --2- 172.21.15.120:0/21248451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4df01038d0 0x7f4df0198eb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:55.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.027+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 >> 172.21.15.120:0/21248451 conn(0x7f4df00fdc90 msgr2=0x7f4df0106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:55.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.027+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 shutdown_connections 2024-03-22T02:10:55.028 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-22T02:10:55.027+0000 7f4df5fdc700 1 -- 172.21.15.120:0/21248451 wait complete. 2024-03-22T02:10:55.737 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.a.service 2024-03-22T02:10:55.740 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.d.service 2024-03-22T02:10:55.783 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.g.service 2024-03-22T02:10:55.827 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.j.service 2024-03-22T02:10:55.870 DEBUG:teuthology.orchestra.run.smithi083:mds.b> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.b.service 2024-03-22T02:10:55.873 DEBUG:teuthology.orchestra.run.smithi083:mds.e> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.e.service 2024-03-22T02:10:55.916 DEBUG:teuthology.orchestra.run.smithi083:mds.h> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.h.service 2024-03-22T02:10:55.959 DEBUG:teuthology.orchestra.run.smithi083:mds.k> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.k.service 2024-03-22T02:10:56.002 DEBUG:teuthology.orchestra.run.smithi120:mds.c> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.c.service 2024-03-22T02:10:56.005 DEBUG:teuthology.orchestra.run.smithi120:mds.f> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.f.service 2024-03-22T02:10:56.047 DEBUG:teuthology.orchestra.run.smithi120:mds.i> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.i.service 2024-03-22T02:10:56.090 DEBUG:teuthology.orchestra.run.smithi120:mds.l> sudo journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.l.service 2024-03-22T02:10:56.133 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-22T02:10:56.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 -- 172.21.15.28:0/4020586320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c100410 msgr2=0x7fed3c102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 --2- 172.21.15.28:0/4020586320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c100410 0x7fed3c102840 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fed34009900 tx=0x7fed3402bc30 comp rx=0 tx=0).stop 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 -- 172.21.15.28:0/4020586320 shutdown_connections 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 --2- 172.21.15.28:0/4020586320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 0x7fed3c105bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 --2- 172.21.15.28:0/4020586320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c102d80 0x7fed3c1051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 --2- 172.21.15.28:0/4020586320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c100410 0x7fed3c102840 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 -- 172.21.15.28:0/4020586320 >> 172.21.15.28:0/4020586320 conn(0x7fed3c0fc030 msgr2=0x7fed3c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 -- 172.21.15.28:0/4020586320 shutdown_connections 2024-03-22T02:10:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.246+0000 7fed43ca3640 1 -- 172.21.15.28:0/4020586320 wait complete. 2024-03-22T02:10:56.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:56.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:56.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:56 smithi120 ceph-mon[36803]: Deploying daemon mds.l on smithi120 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 Processor -- start 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 -- start start 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 0x7fed3c19e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 0x7fed3c1a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113e40 con 0x7fed3c102d80 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113cc0 con 0x7fed3c100410 2024-03-22T02:10:56.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed43ca3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113fc0 con 0x7fed3c105730 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed4123e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed4123e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47988/0 (socket says 172.21.15.28:47988) 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed4123e640 1 -- 172.21.15.28:0/3011766646 learned_addr learned my addr 172.21.15.28:0/3011766646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.247+0000 7fed42240640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 0x7fed3c1a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed41a3f640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 0x7fed3c19e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 -- 172.21.15.28:0/3011766646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 msgr2=0x7fed3c1a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 0x7fed3c1a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 -- 172.21.15.28:0/3011766646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 msgr2=0x7fed3c19e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 0x7fed3c19e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 -- 172.21.15.28:0/3011766646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c1a60a0 con 0x7fed3c102d80 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed41a3f640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 0x7fed3c19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed4123e640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed3000d950 tx=0x7fed3000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed30017030 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c1a6390 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed30004020 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed300047a0 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.248+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c1a68e0 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.249+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c0fdb20 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.250+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fed300041d0 con 0x7fed3c102d80 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.250+0000 7fed3affd640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 0x7fed240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.251+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fed300ce9c0 con 0x7fed3c102d80 2024-03-22T02:10:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.251+0000 7fed41a3f640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 0x7fed240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.251+0000 7fed41a3f640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 0x7fed240a53e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed340096c0 tx=0x7fed34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.255+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fed300d0e20 con 0x7fed3c102d80 2024-03-22T02:10:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:56 smithi083 ceph-mon[36340]: Deploying daemon mds.l on smithi120 2024-03-22T02:10:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:56.325 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:56 smithi028 ceph-mon[31916]: Deploying daemon mds.l on smithi120 2024-03-22T02:10:56.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.439+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed3c02d600 con 0x7fed3c102d80 2024-03-22T02:10:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.440+0000 7fed3affd640 1 -- 172.21.15.28:0/3011766646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18405 (secure 0 0 0) 0x7fed3009bc90 con 0x7fed3c102d80 2024-03-22T02:10:56.441 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:56.441 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:10:43.155372+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 msgr2=0x7fed240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 0x7fed240a53e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fed340096c0 tx=0x7fed34000bc0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 msgr2=0x7fed3c19e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed3000d950 tx=0x7fed3000de10 comp rx=0 tx=0).stop 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 shutdown_connections 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fed240a2f20 0x7fed240a53e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed3c105730 0x7fed3c1a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed3c102d80 0x7fed3c19e9c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 --2- 172.21.15.28:0/3011766646 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fed3c100410 0x7fed3c19e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 >> 172.21.15.28:0/3011766646 conn(0x7fed3c0fc030 msgr2=0x7fed3c103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 shutdown_connections 2024-03-22T02:10:56.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.442+0000 7fed43ca3640 1 -- 172.21.15.28:0/3011766646 wait complete. 2024-03-22T02:10:56.454 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:10:56.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:10:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 -- 172.21.15.28:0/3702354112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 msgr2=0x7f2f5c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 --2- 172.21.15.28:0/3702354112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c102110 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009900 tx=0x7f2f5002bc30 comp rx=0 tx=0).stop 2024-03-22T02:10:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 -- 172.21.15.28:0/3702354112 shutdown_connections 2024-03-22T02:10:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 --2- 172.21.15.28:0/3702354112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 0x7f2f5c108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 --2- 172.21.15.28:0/3702354112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c102110 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 --2- 172.21.15.28:0/3702354112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 0x7f2f5c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 -- 172.21.15.28:0/3702354112 >> 172.21.15.28:0/3702354112 conn(0x7f2f5c0fc030 msgr2=0x7f2f5c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 -- 172.21.15.28:0/3702354112 shutdown_connections 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.558+0000 7f2f64113640 1 -- 172.21.15.28:0/3702354112 wait complete. 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 Processor -- start 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 -- start start 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 0x7f2f5c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 0x7f2f5c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c10d840 con 0x7f2f5c101c90 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c10d6c0 con 0x7f2f5c100aa0 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f64113640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f5c10d9c0 con 0x7f2f5c102650 2024-03-22T02:10:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48006/0 (socket says 172.21.15.28:48006) 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 -- 172.21.15.28:0/786424821 learned_addr learned my addr 172.21.15.28:0/786424821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f61eaf640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 0x7f2f5c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f626b0640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 0x7f2f5c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 -- 172.21.15.28:0/786424821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 msgr2=0x7f2f5c19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 0x7f2f5c19f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 -- 172.21.15.28:0/786424821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 msgr2=0x7f2f5c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 0x7f2f5c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.559+0000 7f2f616ae640 1 -- 172.21.15.28:0/786424821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f5c19faf0 con 0x7f2f5c101c90 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f616ae640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f50006010 tx=0x7f2f50005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f50041070 con 0x7f2f5c101c90 2024-03-22T02:10:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f500054e0 con 0x7f2f5c101c90 2024-03-22T02:10:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f5c19fd80 con 0x7f2f5c101c90 2024-03-22T02:10:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f50034440 con 0x7f2f5c101c90 2024-03-22T02:10:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.560+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f5c1a0210 con 0x7f2f5c101c90 2024-03-22T02:10:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.561+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f5c107d60 con 0x7f2f5c101c90 2024-03-22T02:10:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.562+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2f50044070 con 0x7f2f5c101c90 2024-03-22T02:10:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.562+0000 7f2f5affd640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 0x7f2f440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.563+0000 7f2f61eaf640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 0x7f2f440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.563+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2f500f0d50 con 0x7f2f5c101c90 2024-03-22T02:10:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.563+0000 7f2f61eaf640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 0x7f2f440a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2f540058c0 tx=0x7f2f54005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.568+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f500bdf80 con 0x7f2f5c101c90 2024-03-22T02:10:56.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.750+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f5c062790 con 0x7f2f5c101c90 2024-03-22T02:10:56.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.750+0000 7f2f5affd640 1 -- 172.21.15.28:0/786424821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f2f500bd880 con 0x7f2f5c101c90 2024-03-22T02:10:56.755 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 msgr2=0x7f2f440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 0x7f2f440a53e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2f540058c0 tx=0x7f2f54005850 comp rx=0 tx=0).stop 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 msgr2=0x7f2f5c198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f50006010 tx=0x7f2f50005dd0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 shutdown_connections 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f440a2f20 0x7f2f440a53e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f5c102650 0x7f2f5c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f5c101c90 0x7f2f5c198410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 --2- 172.21.15.28:0/786424821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f5c100aa0 0x7f2f5c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 >> 172.21.15.28:0/786424821 conn(0x7f2f5c0fc030 msgr2=0x7f2f5c104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:56.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 shutdown_connections 2024-03-22T02:10:56.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.752+0000 7f2f64113640 1 -- 172.21.15.28:0/786424821 wait complete. 2024-03-22T02:10:56.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:10:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 -- 172.21.15.28:0/3585923610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4100da0 msgr2=0x7f04c4101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 --2- 172.21.15.28:0/3585923610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4100da0 0x7f04c4101220 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f04b8009900 tx=0x7f04b802bc30 comp rx=0 tx=0).stop 2024-03-22T02:10:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 -- 172.21.15.28:0/3585923610 shutdown_connections 2024-03-22T02:10:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 --2- 172.21.15.28:0/3585923610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4101760 0x7f04c4109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 --2- 172.21.15.28:0/3585923610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4100da0 0x7f04c4101220 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 --2- 172.21.15.28:0/3585923610 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 0x7f04c4102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 -- 172.21.15.28:0/3585923610 >> 172.21.15.28:0/3585923610 conn(0x7f04c40fb7d0 msgr2=0x7f04c40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 -- 172.21.15.28:0/3585923610 shutdown_connections 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.866+0000 7f04c93ac640 1 -- 172.21.15.28:0/3585923610 wait complete. 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 Processor -- start 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 -- start start 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 0x7f04c410b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 0x7f04c4109fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04c410d890 con 0x7f04c4101760 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f04c410d710 con 0x7f04c4102070 2024-03-22T02:10:56.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c93ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f04c410da10 con 0x7f04c4100da0 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48034/0 (socket says 172.21.15.28:48034) 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.867+0000 7f04c27fc640 1 -- 172.21.15.28:0/1884558774 learned_addr learned my addr 172.21.15.28:0/1884558774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c37fe640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 0x7f04c4109fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c2ffd640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 0x7f04c410b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 -- 172.21.15.28:0/1884558774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 msgr2=0x7f04c410b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 0x7f04c410b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 -- 172.21.15.28:0/1884558774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 msgr2=0x7f04c4109fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 0x7f04c4109fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 -- 172.21.15.28:0/1884558774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c410a850 con 0x7f04c4101760 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c2ffd640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 0x7f04c410b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c27fc640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f04b802efd0 tx=0x7f04b802cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8041070 con 0x7f04c4101760 2024-03-22T02:10:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c41a1f50 con 0x7f04c4101760 2024-03-22T02:10:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b80054e0 con 0x7f04c4101760 2024-03-22T02:10:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b80346a0 con 0x7f04c4101760 2024-03-22T02:10:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.868+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c41a23c0 con 0x7f04c4101760 2024-03-22T02:10:56.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.869+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c4107e80 con 0x7f04c4101760 2024-03-22T02:10:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.870+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f04b8044070 con 0x7f04c4101760 2024-03-22T02:10:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.870+0000 7f04abfff640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 0x7f04ac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.871+0000 7f04c2ffd640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 0x7f04ac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.871+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f04b80f0bb0 con 0x7f04c4101760 2024-03-22T02:10:56.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.871+0000 7f04c2ffd640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 0x7f04ac0a53e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f04c4069f60 tx=0x7f04bc005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:56.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:56.875+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f04b80bde80 con 0x7f04c4101760 2024-03-22T02:10:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.097+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04c402d580 con 0x7f04c4101760 2024-03-22T02:10:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.097+0000 7f04abfff640 1 -- 172.21.15.28:0/1884558774 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f04b80bd780 con 0x7f04c4101760 2024-03-22T02:10:57.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:57.098 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-22T02:10:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-03-22T02:10:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 msgr2=0x7f04ac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 0x7f04ac0a53e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f04c4069f60 tx=0x7f04bc005db0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 msgr2=0x7f04c4109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f04b802efd0 tx=0x7f04b802cdc0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 shutdown_connections 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f04ac0a2f20 0x7f04ac0a53e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f04c4102070 0x7f04c4109fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04c4101760 0x7f04c4109a80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 --2- 172.21.15.28:0/1884558774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f04c4100da0 0x7f04c410b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 >> 172.21.15.28:0/1884558774 conn(0x7f04c40fb7d0 msgr2=0x7f04c4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 shutdown_connections 2024-03-22T02:10:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:57.099+0000 7f04c93ac640 1 -- 172.21.15.28:0/1884558774 wait complete. 2024-03-22T02:10:57.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3011766646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:57.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/786424821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:57.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3011766646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:57.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/786424821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:57.498 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 systemd[1]: Starting Ceph mds.l for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:10:57.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3011766646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:10:57.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/786424821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:10:57.898 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 podman[66872]: 2024-03-22 02:10:57.539941888 +0000 UTC m=+0.017668063 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:10:57.899 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 podman[66872]: 2024-03-22 02:10:57.662910537 +0000 UTC m=+0.140636738 container create 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:10:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:58 smithi120 ceph-mon[36803]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:58.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1884558774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:10:58.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:58 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6832/2414441153,v1:172.21.15.120:6833/2414441153] up:boot 2024-03-22T02:10:58.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:58 smithi120 ceph-mon[36803]: fsmap 1 up:standby 2024-03-22T02:10:58.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:58 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T02:10:58.249 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 podman[66872]: 2024-03-22 02:10:57.899150719 +0000 UTC m=+0.376876882 container init 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:10:58.250 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 podman[66872]: 2024-03-22 02:10:57.90164781 +0000 UTC m=+0.379373983 container start 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T02:10:58.250 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 ceph-mds[66892]: -- 172.21.15.120:0/1291629419 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558047773a40 con 0x558047746c00 2024-03-22T02:10:58.250 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l[66888]: starting mds.l at 2024-03-22T02:10:58.250 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:57 smithi120 bash[66872]: 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd 2024-03-22T02:10:58.250 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:10:58 smithi120 systemd[1]: Started Ceph mds.l for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:10:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:58 smithi028 ceph-mon[31916]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1884558774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:10:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:58 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6832/2414441153,v1:172.21.15.120:6833/2414441153] up:boot 2024-03-22T02:10:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:58 smithi028 ceph-mon[31916]: fsmap 1 up:standby 2024-03-22T02:10:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:58 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:58 smithi083 ceph-mon[36340]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1884558774' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:58 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6832/2414441153,v1:172.21.15.120:6833/2414441153] up:boot 2024-03-22T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:58 smithi083 ceph-mon[36340]: fsmap 1 up:standby 2024-03-22T02:10:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:58 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-22T02:10:59.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- 172.21.15.28:0/3395760095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270100ab0 msgr2=0x7fa270100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 --2- 172.21.15.28:0/3395760095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270100ab0 0x7fa270100ed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa260009900 tx=0x7fa26002bc30 comp rx=0 tx=0).stop 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- 172.21.15.28:0/3395760095 shutdown_connections 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 --2- 172.21.15.28:0/3395760095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270102660 0x7fa270108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 --2- 172.21.15.28:0/3395760095 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 --2- 172.21.15.28:0/3395760095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270100ab0 0x7fa270100ed0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- 172.21.15.28:0/3395760095 >> 172.21.15.28:0/3395760095 conn(0x7fa2700fc020 msgr2=0x7fa2700fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:59.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- 172.21.15.28:0/3395760095 shutdown_connections 2024-03-22T02:10:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- 172.21.15.28:0/3395760095 wait complete. 2024-03-22T02:10:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 Processor -- start 2024-03-22T02:10:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.226+0000 7fa276932640 1 -- start start 2024-03-22T02:10:59.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 0x7fa270196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:59.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 0x7fa27019e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c730 con 0x7fa270102660 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c5b0 con 0x7fa270101ca0 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa276932640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa27010c8b0 con 0x7fa270100ab0 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 0x7fa270196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa274ecf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 0x7fa27019e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50658/0 (socket says 172.21.15.28:50658) 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 0x7fa270196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55508/0 (socket says 172.21.15.28:55508) 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 -- 172.21.15.28:0/741809166 learned_addr learned my addr 172.21.15.28:0/741809166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:10:59.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 -- 172.21.15.28:0/741809166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 msgr2=0x7fa270196dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 0x7fa270196dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 -- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 msgr2=0x7fa27019e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 0x7fa27019e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa26f7fe640 1 -- 172.21.15.28:0/741809166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa27019e9d0 con 0x7fa270101ca0 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.227+0000 7fa274ecf640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 0x7fa27019e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa26f7fe640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa26400d950 tx=0x7fa26400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264017030 con 0x7fa270101ca0 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa27019ecc0 con 0x7fa270101ca0 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa264004020 con 0x7fa270101ca0 2024-03-22T02:10:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640047a0 con 0x7fa270101ca0 2024-03-22T02:10:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.228+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa27019f180 con 0x7fa270101ca0 2024-03-22T02:10:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.229+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa270107dd0 con 0x7fa270101ca0 2024-03-22T02:10:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.230+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa2640041d0 con 0x7fa270101ca0 2024-03-22T02:10:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.230+0000 7fa26d7fa640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 0x7fa2580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:10:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.230+0000 7fa26ffff640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 0x7fa2580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:10:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.230+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa2640ce9e0 con 0x7fa270101ca0 2024-03-22T02:10:59.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.231+0000 7fa26ffff640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 0x7fa2580a53e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa260000c00 tx=0x7fa260005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:10:59.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.235+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa26409bc90 con 0x7fa270101ca0 2024-03-22T02:10:59.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:10:59 smithi083 ceph-mon[36340]: Deploying daemon mds.k on smithi083 2024-03-22T02:10:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:10:59 smithi028 ceph-mon[31916]: Deploying daemon mds.k on smithi083 2024-03-22T02:10:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.458+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa27002d3b0 con 0x7fa270101ca0 2024-03-22T02:10:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.458+0000 7fa26d7fa640 1 -- 172.21.15.28:0/741809166 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa26409b590 con 0x7fa270101ca0 2024-03-22T02:10:59.459 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:10:59.459 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T02:10:59.459 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 msgr2=0x7fa2580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 0x7fa2580a53e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa260000c00 tx=0x7fa260005fd0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 msgr2=0x7fa270197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa26400d950 tx=0x7fa26400de10 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 shutdown_connections 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2580a2f20 0x7fa2580a53e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa270102660 0x7fa27019e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa270101ca0 0x7fa270197300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 --2- 172.21.15.28:0/741809166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa270100ab0 0x7fa270196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 >> 172.21.15.28:0/741809166 conn(0x7fa2700fc020 msgr2=0x7fa270104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 shutdown_connections 2024-03-22T02:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:10:59.460+0000 7fa276932640 1 -- 172.21.15.28:0/741809166 wait complete. 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:10:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:10:59.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:10:59 smithi120 ceph-mon[36803]: Deploying daemon mds.k on smithi083 2024-03-22T02:11:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:00 smithi120 ceph-mon[36803]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:00 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/741809166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:00.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:00 smithi083 ceph-mon[36340]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:00 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/741809166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:00.515 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 systemd[1]: Starting Ceph mds.k for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:00 smithi028 ceph-mon[31916]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:00 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/741809166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:00.816 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 podman[65851]: 2024-03-22 02:11:00.452951834 +0000 UTC m=+0.018588270 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:00.816 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 podman[65851]: 2024-03-22 02:11:00.563108325 +0000 UTC m=+0.128744746 container create e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T02:11:00.816 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 podman[65851]: 2024-03-22 02:11:00.815796983 +0000 UTC m=+0.381433423 container init e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-22T02:11:01.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 podman[65851]: 2024-03-22 02:11:00.818465456 +0000 UTC m=+0.384101874 container start e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:11:01.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 ceph-mds[65870]: -- 172.21.15.83:0/2498980677 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557857e9fa40 con 0x557857e72c00 2024-03-22T02:11:01.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k[65866]: starting mds.k at 2024-03-22T02:11:01.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 bash[65851]: e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d 2024-03-22T02:11:01.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:11:00 smithi083 systemd[1]: Started Ceph mds.k for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:01.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.588+0000 7fb6283b5640 1 -- 172.21.15.28:0/1327655349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620101bd0 msgr2=0x7fb620101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.588+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1327655349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620101bd0 0x7fb620101ff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb610009900 tx=0x7fb61002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- 172.21.15.28:0/1327655349 shutdown_connections 2024-03-22T02:11:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1327655349 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620103780 0x7fb62010a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1327655349 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620102dc0 0x7fb620103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1327655349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620101bd0 0x7fb620101ff0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- 172.21.15.28:0/1327655349 >> 172.21.15.28:0/1327655349 conn(0x7fb6200fd140 msgr2=0x7fb6200ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- 172.21.15.28:0/1327655349 shutdown_connections 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- 172.21.15.28:0/1327655349 wait complete. 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 Processor -- start 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- start start 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620101bd0 0x7fb620075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 0x7fb6200744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d820 con 0x7fb620102dc0 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d6a0 con 0x7fb620101bd0 2024-03-22T02:11:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb6283b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb62010d9a0 con 0x7fb620103780 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.589+0000 7fb625950640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48062/0 (socket says 172.21.15.28:48062) 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 -- 172.21.15.28:0/1027116356 learned_addr learned my addr 172.21.15.28:0/1027116356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb626151640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620101bd0 0x7fb620075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb626952640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 0x7fb6200744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 -- 172.21.15.28:0/1027116356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 msgr2=0x7fb6200744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 0x7fb6200744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 -- 172.21.15.28:0/1027116356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620101bd0 msgr2=0x7fb620075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620101bd0 0x7fb620075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 -- 172.21.15.28:0/1027116356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb620074d50 con 0x7fb620102dc0 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb626952640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 0x7fb6200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb625950640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c890 tx=0x7fb61c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004220 con 0x7fb620102dc0 2024-03-22T02:11:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb61c0043c0 con 0x7fb620102dc0 2024-03-22T02:11:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6201a1fb0 con 0x7fb620102dc0 2024-03-22T02:11:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61c004a00 con 0x7fb620102dc0 2024-03-22T02:11:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.590+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6201a2410 con 0x7fb620102dc0 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.591+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb620108f70 con 0x7fb620102dc0 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.593+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fb61c0079b0 con 0x7fb620102dc0 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.593+0000 7fb6177fe640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 0x7fb60c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.593+0000 7fb626151640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 0x7fb60c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.593+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb61c0cef80 con 0x7fb620102dc0 2024-03-22T02:11:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.594+0000 7fb626151640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 0x7fb60c0a53e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb610004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:01.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.597+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb61c014030 con 0x7fb620102dc0 2024-03-22T02:11:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.819+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb62002d3b0 con 0x7fb620102dc0 2024-03-22T02:11:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.819+0000 7fb6177fe640 1 -- 172.21.15.28:0/1027116356 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7fb61c09c250 con 0x7fb620102dc0 2024-03-22T02:11:01.820 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:01.820 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T02:11:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-03-22T02:11:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 msgr2=0x7fb60c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 0x7fb60c0a53e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb610000c00 tx=0x7fb610004f10 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 msgr2=0x7fb620073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb61c00c890 tx=0x7fb61c00cd50 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 shutdown_connections 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb60c0a2f20 0x7fb60c0a53e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb620103780 0x7fb6200744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb620102dc0 0x7fb620073f80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 --2- 172.21.15.28:0/1027116356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb620101bd0 0x7fb620075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 >> 172.21.15.28:0/1027116356 conn(0x7fb6200fd140 msgr2=0x7fb620106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 shutdown_connections 2024-03-22T02:11:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:01.821+0000 7fb6283b5640 1 -- 172.21.15.28:0/1027116356 wait complete. 2024-03-22T02:11:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: Deploying daemon mds.j on smithi028 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6832/675405105,v1:172.21.15.83:6833/675405105] up:boot 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: fsmap 2 up:standby 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T02:11:02.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:01 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1027116356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:02.267 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:02.267 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.267 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.267 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: Deploying daemon mds.j on smithi028 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6832/675405105,v1:172.21.15.83:6833/675405105] up:boot 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: fsmap 2 up:standby 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T02:11:02.268 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:01 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1027116356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: Deploying daemon mds.j on smithi028 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6832/675405105,v1:172.21.15.83:6833/675405105] up:boot 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: fsmap 2 up:standby 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-22T02:11:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:01 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1027116356' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:03.322 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 systemd[1]: Starting Ceph mds.j for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:03.709 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 podman[72969]: 2024-03-22 02:11:03.331982624 +0000 UTC m=+0.018282402 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:03.710 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 podman[72969]: 2024-03-22 02:11:03.465002199 +0000 UTC m=+0.151301958 container create 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T02:11:03.710 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 podman[72969]: 2024-03-22 02:11:03.709451439 +0000 UTC m=+0.395751223 container init 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-22T02:11:03.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 -- 172.21.15.28:0/3686375871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940102680 msgr2=0x7f6940108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 --2- 172.21.15.28:0/3686375871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940102680 0x7f6940108f10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6938009ca0 tx=0x7f693802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 -- 172.21.15.28:0/3686375871 shutdown_connections 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 --2- 172.21.15.28:0/3686375871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940102680 0x7f6940108f10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 --2- 172.21.15.28:0/3686375871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940101cc0 0x7f6940102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 --2- 172.21.15.28:0/3686375871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f6940100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 -- 172.21.15.28:0/3686375871 >> 172.21.15.28:0/3686375871 conn(0x7f69400fc040 msgr2=0x7f69400fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 -- 172.21.15.28:0/3686375871 shutdown_connections 2024-03-22T02:11:03.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 -- 172.21.15.28:0/3686375871 wait complete. 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.949+0000 7f69450d4640 1 Processor -- start 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 -- start start 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f694010b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 0x7f6940109fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f694010d870 con 0x7f6940101cc0 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f694010d6f0 con 0x7f6940100ad0 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f69450d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f694010d9f0 con 0x7f6940102680 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f6937fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f6937fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54738/0 (socket says 172.21.15.28:54738) 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f6937fff640 1 -- 172.21.15.28:0/2540254175 learned_addr learned my addr 172.21.15.28:0/2540254175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:03.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f694010b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693f59e640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 0x7f6940109fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 -- 172.21.15.28:0/2540254175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 msgr2=0x7f6940109fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 0x7f6940109fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 -- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 msgr2=0x7f6940109a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:03.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 -- 172.21.15.28:0/2540254175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694010a860 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693f59e640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 0x7f6940109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f6937fff640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.950+0000 7f693ed9d640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f694010b450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f693000e9c0 tx=0x7f693000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.951+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693000ccc0 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.951+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69300035f0 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.951+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69401a1fb0 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.951+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693000ce60 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.951+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69401a23c0 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.952+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6940107d90 con 0x7f6940100ad0 2024-03-22T02:11:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.952+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f69300037a0 con 0x7f6940100ad0 2024-03-22T02:11:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.953+0000 7f693cd99640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 0x7f692c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.953+0000 7f6937fff640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 0x7f692c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.953+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f69300cf560 con 0x7f6940100ad0 2024-03-22T02:11:03.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.953+0000 7f6937fff640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 0x7f692c0a53e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f694010b1c0 tx=0x7f692800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:03.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:03.957+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f693009c790 con 0x7f6940100ad0 2024-03-22T02:11:04.000 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 podman[72969]: 2024-03-22 02:11:03.712204002 +0000 UTC m=+0.398503761 container start 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T02:11:04.000 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 ceph-mds[72989]: -- 172.21.15.28:0/2591718601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a72774ba40 con 0x55a72771ec00 2024-03-22T02:11:04.000 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j[72985]: starting mds.j at 2024-03-22T02:11:04.000 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 bash[72969]: 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde 2024-03-22T02:11:04.000 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:11:03 smithi028 systemd[1]: Started Ceph mds.j for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.185+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f694002d3b0 con 0x7f6940100ad0 2024-03-22T02:11:04.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.186+0000 7f693cd99640 1 -- 172.21.15.28:0/2540254175 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f694002d3b0 con 0x7f6940100ad0 2024-03-22T02:11:04.186 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:04.186 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T02:11:04.186 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-03-22T02:11:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 msgr2=0x7f692c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 0x7f692c0a53e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f694010b1c0 tx=0x7f692800a380 comp rx=0 tx=0).stop 2024-03-22T02:11:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 msgr2=0x7f694010b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f694010b450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f693000e9c0 tx=0x7f693000ee80 comp rx=0 tx=0).stop 2024-03-22T02:11:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 shutdown_connections 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f692c0a2f20 0x7f692c0a53e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6940102680 0x7f6940109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6940101cc0 0x7f6940109a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 --2- 172.21.15.28:0/2540254175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6940100ad0 0x7f694010b450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 >> 172.21.15.28:0/2540254175 conn(0x7f69400fc040 msgr2=0x7f6940104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 shutdown_connections 2024-03-22T02:11:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:04.188+0000 7f69450d4640 1 -- 172.21.15.28:0/2540254175 wait complete. 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:03 smithi028 ceph-mon[31916]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:04.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:05.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:05 smithi028 ceph-mon[31916]: Deploying daemon mds.d on smithi028 2024-03-22T02:11:05.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:05 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.28:6834/4228376531,v1:172.21.15.28:6835/4228376531] up:boot 2024-03-22T02:11:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:05 smithi028 ceph-mon[31916]: fsmap 3 up:standby 2024-03-22T02:11:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T02:11:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:05 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2540254175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:05.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:05 smithi083 ceph-mon[36340]: Deploying daemon mds.d on smithi028 2024-03-22T02:11:05.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:05 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.28:6834/4228376531,v1:172.21.15.28:6835/4228376531] up:boot 2024-03-22T02:11:05.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:05 smithi083 ceph-mon[36340]: fsmap 3 up:standby 2024-03-22T02:11:05.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T02:11:05.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:05 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2540254175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:05 smithi120 ceph-mon[36803]: Deploying daemon mds.d on smithi028 2024-03-22T02:11:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:05 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.28:6834/4228376531,v1:172.21.15.28:6835/4228376531] up:boot 2024-03-22T02:11:05.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:05 smithi120 ceph-mon[36803]: fsmap 3 up:standby 2024-03-22T02:11:05.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-22T02:11:05.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:05 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2540254175' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:06.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:06.226 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:06 smithi028 ceph-mon[31916]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:06.227 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 systemd[1]: Starting Ceph mds.d for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:06 smithi120 ceph-mon[36803]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:06.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:06 smithi083 ceph-mon[36340]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:06.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- 172.21.15.28:0/2642917943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 msgr2=0x7f30801073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 --2- 172.21.15.28:0/2642917943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f30801073b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3068009900 tx=0x7f306802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- 172.21.15.28:0/2642917943 shutdown_connections 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 --2- 172.21.15.28:0/2642917943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 0x7f3080109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 --2- 172.21.15.28:0/2642917943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f30801073b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 --2- 172.21.15.28:0/2642917943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f3080104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- 172.21.15.28:0/2642917943 >> 172.21.15.28:0/2642917943 conn(0x7f30800fc020 msgr2=0x7f30800fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- 172.21.15.28:0/2642917943 shutdown_connections 2024-03-22T02:11:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- 172.21.15.28:0/2642917943 wait complete. 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 Processor -- start 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.308+0000 7f3086822640 1 -- start start 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f308019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 0x7f30801a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d830 con 0x7f3080104fc0 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d6b0 con 0x7f3080102690 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3086822640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f308010d9b0 con 0x7f30801078f0 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54762/0 (socket says 172.21.15.28:54762) 2024-03-22T02:11:06.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 -- 172.21.15.28:0/3315480508 learned_addr learned my addr 172.21.15.28:0/3315480508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f3084dbf640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 0x7f30801a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307ffff640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f308019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 -- 172.21.15.28:0/3315480508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 msgr2=0x7f30801a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 0x7f30801a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 -- 172.21.15.28:0/3315480508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 msgr2=0x7f308019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f308019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 -- 172.21.15.28:0/3315480508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30801a1d20 con 0x7f3080104fc0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307ffff640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f308019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.309+0000 7f307f7fe640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3068005ab0 tx=0x7f3068005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.310+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068041070 con 0x7f3080104fc0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.310+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30680054e0 con 0x7f3080104fc0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.310+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30801a1fb0 con 0x7f3080104fc0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.310+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068034440 con 0x7f3080104fc0 2024-03-22T02:11:06.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.310+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30801a2410 con 0x7f3080104fc0 2024-03-22T02:11:06.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.311+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30801093e0 con 0x7f3080104fc0 2024-03-22T02:11:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.312+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f3068044070 con 0x7f3080104fc0 2024-03-22T02:11:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.312+0000 7f307d7fa640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 0x7f306c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.312+0000 7f307ffff640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 0x7f306c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.312+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f30680f0b80 con 0x7f3080104fc0 2024-03-22T02:11:06.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.313+0000 7f307ffff640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 0x7f306c0a53e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3070009b40 tx=0x7f3070005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:06.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.317+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30680bde50 con 0x7f3080104fc0 2024-03-22T02:11:06.489 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 podman[73319]: 2024-03-22 02:11:06.15039371 +0000 UTC m=+0.017655136 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:06.489 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 podman[73319]: 2024-03-22 02:11:06.261154973 +0000 UTC m=+0.128416387 container create 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T02:11:06.489 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 podman[73319]: 2024-03-22 02:11:06.488862536 +0000 UTC m=+0.356123951 container init 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T02:11:06.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.542+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f308002d3b0 con 0x7f3080104fc0 2024-03-22T02:11:06.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.543+0000 7f307d7fa640 1 -- 172.21.15.28:0/3315480508 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f30680bd750 con 0x7f3080104fc0 2024-03-22T02:11:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:06.543 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T02:11:06.544 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 msgr2=0x7f306c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 0x7f306c0a53e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3070009b40 tx=0x7f3070005e50 comp rx=0 tx=0).stop 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 msgr2=0x7f308019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3068005ab0 tx=0x7f3068005dc0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 shutdown_connections 2024-03-22T02:11:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f306c0a2f20 0x7f306c0a53e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30801078f0 0x7f30801a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3080104fc0 0x7f308019a640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 --2- 172.21.15.28:0/3315480508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3080102690 0x7f308019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 >> 172.21.15.28:0/3315480508 conn(0x7f30800fc020 msgr2=0x7f3080101e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 shutdown_connections 2024-03-22T02:11:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:06.545+0000 7f3086822640 1 -- 172.21.15.28:0/3315480508 wait complete. 2024-03-22T02:11:06.821 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 podman[73319]: 2024-03-22 02:11:06.49183428 +0000 UTC m=+0.359095693 container start 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T02:11:06.822 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 ceph-mds[73377]: -- 172.21.15.28:0/2276602981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a979e73a40 con 0x55a979e46c00 2024-03-22T02:11:06.822 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d[73373]: starting mds.d at 2024-03-22T02:11:06.822 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 bash[73319]: 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 2024-03-22T02:11:06.822 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:11:06 smithi028 systemd[1]: Started Ceph mds.d for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:07.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3315480508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:07.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:07.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3315480508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:07.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:07.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3315480508' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:07.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:07.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:07.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:07.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:08.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:08 smithi083 ceph-mon[36340]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:08 smithi083 ceph-mon[36340]: Deploying daemon mds.e on smithi083 2024-03-22T02:11:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:08 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.28:6836/1410938784,v1:172.21.15.28:6837/1410938784] up:boot 2024-03-22T02:11:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:08 smithi083 ceph-mon[36340]: fsmap 4 up:standby 2024-03-22T02:11:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T02:11:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:08 smithi028 ceph-mon[31916]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:08 smithi028 ceph-mon[31916]: Deploying daemon mds.e on smithi083 2024-03-22T02:11:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:08 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.28:6836/1410938784,v1:172.21.15.28:6837/1410938784] up:boot 2024-03-22T02:11:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:08 smithi028 ceph-mon[31916]: fsmap 4 up:standby 2024-03-22T02:11:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T02:11:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:08 smithi120 ceph-mon[36803]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:08 smithi120 ceph-mon[36803]: Deploying daemon mds.e on smithi083 2024-03-22T02:11:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:08 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.28:6836/1410938784,v1:172.21.15.28:6837/1410938784] up:boot 2024-03-22T02:11:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:08 smithi120 ceph-mon[36803]: fsmap 4 up:standby 2024-03-22T02:11:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-22T02:11:08.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 -- 172.21.15.28:0/554611741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4102db0 msgr2=0x7f56d4103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 --2- 172.21.15.28:0/554611741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4102db0 0x7f56d4103230 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f56c8009900 tx=0x7f56c802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 -- 172.21.15.28:0/554611741 shutdown_connections 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 --2- 172.21.15.28:0/554611741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4103770 0x7f56d410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 --2- 172.21.15.28:0/554611741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4102db0 0x7f56d4103230 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 --2- 172.21.15.28:0/554611741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 -- 172.21.15.28:0/554611741 >> 172.21.15.28:0/554611741 conn(0x7f56d40fd170 msgr2=0x7f56d40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 -- 172.21.15.28:0/554611741 shutdown_connections 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.673+0000 7f56dc066640 1 -- 172.21.15.28:0/554611741 wait complete. 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 Processor -- start 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 -- start start 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4102db0 0x7f56d41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410d840 con 0x7f56d4103770 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410d6c0 con 0x7f56d4101bc0 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56dc066640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f56d410d9c0 con 0x7f56d4102db0 2024-03-22T02:11:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56da603640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56da603640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54802/0 (socket says 172.21.15.28:54802) 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56da603640 1 -- 172.21.15.28:0/912868076 learned_addr learned my addr 172.21.15.28:0/912868076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56d9e02640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.674+0000 7f56d9601640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4102db0 0x7f56d41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 -- 172.21.15.28:0/912868076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4102db0 msgr2=0x7f56d41983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4102db0 0x7f56d41983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 -- 172.21.15.28:0/912868076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 msgr2=0x7f56d4197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 -- 172.21.15.28:0/912868076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56d419fab0 con 0x7f56d4103770 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56d9e02640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56da603640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e970 tx=0x7f56cc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00ccc0 con 0x7f56d4103770 2024-03-22T02:11:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56d419fda0 con 0x7f56d4103770 2024-03-22T02:11:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc00ce60 con 0x7f56d4103770 2024-03-22T02:11:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc010450 con 0x7f56d4103770 2024-03-22T02:11:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.675+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56d41a0270 con 0x7f56d4103770 2024-03-22T02:11:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.676+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56d4108ee0 con 0x7f56d4103770 2024-03-22T02:11:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.677+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f56cc005020 con 0x7f56d4103770 2024-03-22T02:11:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.678+0000 7f56d2ffd640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 0x7f56bc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.678+0000 7f56d9e02640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 0x7f56bc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.678+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f56cc0cf3e0 con 0x7f56d4103770 2024-03-22T02:11:08.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.678+0000 7f56d9e02640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 0x7f56bc0a53e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d4102c10 tx=0x7f56c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:08.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.682+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f56cc09c6b0 con 0x7f56d4103770 2024-03-22T02:11:08.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.909+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56d402d3b0 con 0x7f56d4103770 2024-03-22T02:11:08.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.909+0000 7f56d2ffd640 1 -- 172.21.15.28:0/912868076 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f56cc09bfb0 con 0x7f56d4103770 2024-03-22T02:11:08.910 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:08.910 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-22T02:11:08.910 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 msgr2=0x7f56bc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 0x7f56bc0a53e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d4102c10 tx=0x7f56c4009500 comp rx=0 tx=0).stop 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 msgr2=0x7f56d419f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e970 tx=0x7f56cc00ed70 comp rx=0 tx=0).stop 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 shutdown_connections 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f56bc0a2f20 0x7f56bc0a53e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56d4103770 0x7f56d419f440 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f56d4102db0 0x7f56d41983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 --2- 172.21.15.28:0/912868076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f56d4101bc0 0x7f56d4197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 >> 172.21.15.28:0/912868076 conn(0x7f56d40fd170 msgr2=0x7f56d41060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 shutdown_connections 2024-03-22T02:11:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:08.911+0000 7f56dc066640 1 -- 172.21.15.28:0/912868076 wait complete. 2024-03-22T02:11:09.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:09 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/912868076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:09.443 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:09 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/912868076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:09.444 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 systemd[1]: Starting Ceph mds.e for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:09 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/912868076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:09.714 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 podman[66172]: 2024-03-22 02:11:09.346106298 +0000 UTC m=+0.017835460 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:09.714 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 podman[66172]: 2024-03-22 02:11:09.457123934 +0000 UTC m=+0.128853116 container create bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T02:11:09.714 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 podman[66172]: 2024-03-22 02:11:09.714163296 +0000 UTC m=+0.385892466 container init bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:11:10.014 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 podman[66172]: 2024-03-22 02:11:09.717132479 +0000 UTC m=+0.388861626 container start bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-22T02:11:10.015 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 ceph-mds[66191]: -- 172.21.15.83:0/468395542 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c3d4fa9a40 con 0x55c3d4f7c800 2024-03-22T02:11:10.015 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: starting mds.e at 2024-03-22T02:11:10.015 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 bash[66172]: bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 2024-03-22T02:11:10.015 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:11:09 smithi083 systemd[1]: Started Ceph mds.e for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:10.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:10.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:10.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:10.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 -- 172.21.15.28:0/1155970589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650101ca0 msgr2=0x7fa650102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 --2- 172.21.15.28:0/1155970589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650101ca0 0x7fa650102120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa644009900 tx=0x7fa64402bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 -- 172.21.15.28:0/1155970589 shutdown_connections 2024-03-22T02:11:11.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 --2- 172.21.15.28:0/1155970589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650102660 0x7fa650108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 --2- 172.21.15.28:0/1155970589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650101ca0 0x7fa650102120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 --2- 172.21.15.28:0/1155970589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650100ab0 0x7fa650100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 -- 172.21.15.28:0/1155970589 >> 172.21.15.28:0/1155970589 conn(0x7fa6500fc040 msgr2=0x7fa6500fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 -- 172.21.15.28:0/1155970589 shutdown_connections 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.039+0000 7fa65611e640 1 -- 172.21.15.28:0/1155970589 wait complete. 2024-03-22T02:11:11.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 Processor -- start 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 -- start start 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650101ca0 0x7fa650103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650102660 0x7fa650103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa65010c790 con 0x7fa650100ab0 2024-03-22T02:11:11.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa65010c610 con 0x7fa650101ca0 2024-03-22T02:11:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa65611e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa65010c910 con 0x7fa650102660 2024-03-22T02:11:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54828/0 (socket says 172.21.15.28:54828) 2024-03-22T02:11:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3326193259 learned_addr learned my addr 172.21.15.28:0/3326193259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:11.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa64effd640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650101ca0 0x7fa650103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.040+0000 7fa64ffff640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650102660 0x7fa650103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3326193259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650102660 msgr2=0x7fa650103690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650102660 0x7fa650103690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3326193259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650101ca0 msgr2=0x7fa650103150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650101ca0 0x7fa650103150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 -- 172.21.15.28:0/3326193259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa650103f20 con 0x7fa650100ab0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64f7fe640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c970 tx=0x7fa64000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa640007c10 con 0x7fa650100ab0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa640007db0 con 0x7fa650100ab0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6501a1fd0 con 0x7fa650100ab0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64000f940 con 0x7fa650100ab0 2024-03-22T02:11:11.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.041+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6501a23d0 con 0x7fa650100ab0 2024-03-22T02:11:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.042+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6501a26c0 con 0x7fa650100ab0 2024-03-22T02:11:11.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.043+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa64000fb00 con 0x7fa650100ab0 2024-03-22T02:11:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.044+0000 7fa64cff9640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 0x7fa6380a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:11.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.044+0000 7fa64effd640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 0x7fa6380a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.044+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa6400cf310 con 0x7fa650100ab0 2024-03-22T02:11:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.044+0000 7fa64effd640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 0x7fa6380a53e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa644005ab0 tx=0x7fa644004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:11.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.048+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa64009c670 con 0x7fa650100ab0 2024-03-22T02:11:11.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:11 smithi120 ceph-mon[36803]: Deploying daemon mds.f on smithi120 2024-03-22T02:11:11.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:11 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6834/3494340980,v1:172.21.15.83:6835/3494340980] up:boot 2024-03-22T02:11:11.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:11 smithi120 ceph-mon[36803]: fsmap 5 up:standby 2024-03-22T02:11:11.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:11:11.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.274+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6501a2890 con 0x7fa650100ab0 2024-03-22T02:11:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.275+0000 7fa64cff9640 1 -- 172.21.15.28:0/3326193259 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fa64009bf70 con 0x7fa650100ab0 2024-03-22T02:11:11.275 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:11.275 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-22T02:11:11.275 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-03-22T02:11:11.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 msgr2=0x7fa6380a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 0x7fa6380a53e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa644005ab0 tx=0x7fa644004f10 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 msgr2=0x7fa650104b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa64000c970 tx=0x7fa64000ce30 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 shutdown_connections 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa6380a2f20 0x7fa6380a53e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa650102660 0x7fa650103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa650101ca0 0x7fa650103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 --2- 172.21.15.28:0/3326193259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa650100ab0 0x7fa650104b10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 >> 172.21.15.28:0/3326193259 conn(0x7fa6500fc040 msgr2=0x7fa65006a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:11.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 shutdown_connections 2024-03-22T02:11:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:11.277+0000 7fa65611e640 1 -- 172.21.15.28:0/3326193259 wait complete. 2024-03-22T02:11:11.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:11 smithi028 ceph-mon[31916]: Deploying daemon mds.f on smithi120 2024-03-22T02:11:11.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:11 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6834/3494340980,v1:172.21.15.83:6835/3494340980] up:boot 2024-03-22T02:11:11.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:11 smithi028 ceph-mon[31916]: fsmap 5 up:standby 2024-03-22T02:11:11.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:11:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:11 smithi083 ceph-mon[36340]: Deploying daemon mds.f on smithi120 2024-03-22T02:11:11.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:11 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6834/3494340980,v1:172.21.15.83:6835/3494340980] up:boot 2024-03-22T02:11:11.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:11 smithi083 ceph-mon[36340]: fsmap 5 up:standby 2024-03-22T02:11:11.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:11:12.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:12 smithi120 ceph-mon[36803]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:12.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3326193259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:12.243 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 systemd[1]: Starting Ceph mds.f for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:12 smithi028 ceph-mon[31916]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:12 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3326193259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:12.497 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 podman[67192]: 2024-03-22 02:11:12.146234778 +0000 UTC m=+0.017669445 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:12.497 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 podman[67192]: 2024-03-22 02:11:12.244967948 +0000 UTC m=+0.116402592 container create 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-22T02:11:12.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:12 smithi083 ceph-mon[36340]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:12 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3326193259' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 podman[67192]: 2024-03-22 02:11:12.498192982 +0000 UTC m=+0.369627650 container init 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 podman[67192]: 2024-03-22 02:11:12.500682604 +0000 UTC m=+0.372117256 container start 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 ceph-mds[67211]: -- 172.21.15.120:0/3716959469 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b2f4203a40 con 0x55b2f41d6800 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: starting mds.f at 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 bash[67192]: 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 2024-03-22T02:11:12.748 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:11:12 smithi120 systemd[1]: Started Ceph mds.f for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:13.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 -- 172.21.15.28:0/2349250189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8100ad0 msgr2=0x7feef8100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 --2- 172.21.15.28:0/2349250189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8100ad0 0x7feef8100ef0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7feef0009ca0 tx=0x7feef002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 -- 172.21.15.28:0/2349250189 shutdown_connections 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 --2- 172.21.15.28:0/2349250189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8102680 0x7feef8108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 --2- 172.21.15.28:0/2349250189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 0x7feef8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 --2- 172.21.15.28:0/2349250189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8100ad0 0x7feef8100ef0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 -- 172.21.15.28:0/2349250189 >> 172.21.15.28:0/2349250189 conn(0x7feef80fc040 msgr2=0x7feef80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 -- 172.21.15.28:0/2349250189 shutdown_connections 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.405+0000 7feefec83640 1 -- 172.21.15.28:0/2349250189 wait complete. 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 Processor -- start 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 -- start start 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 0x7feef8197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 0x7feef81983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feef810d870 con 0x7feef8102680 2024-03-22T02:11:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feef810d6f0 con 0x7feef8100ad0 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefec83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feef810d9f0 con 0x7feef8101cc0 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52356/0 (socket says 172.21.15.28:52356) 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 -- 172.21.15.28:0/853867844 learned_addr learned my addr 172.21.15.28:0/853867844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feef7fff640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 0x7feef81983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefca1f640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 0x7feef8197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 -- 172.21.15.28:0/853867844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 msgr2=0x7feef81983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 0x7feef81983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 -- 172.21.15.28:0/853867844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 msgr2=0x7feef8197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 0x7feef8197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefd220640 1 -- 172.21.15.28:0/853867844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feef819fac0 con 0x7feef8102680 2024-03-22T02:11:13.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.406+0000 7feefca1f640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 0x7feef8197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feefd220640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feee800c970 tx=0x7feee800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8007bb0 con 0x7feef8102680 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feee8007d50 con 0x7feef8102680 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feef819fdb0 con 0x7feef8102680 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8018800 con 0x7feef8102680 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.407+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feef81a0270 con 0x7feef8102680 2024-03-22T02:11:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.408+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef8107df0 con 0x7feef8102680 2024-03-22T02:11:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.409+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7feee800fa50 con 0x7feef8102680 2024-03-22T02:11:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.409+0000 7feef5ffb640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 0x7feee40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.410+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7feee80cefd0 con 0x7feef8102680 2024-03-22T02:11:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.410+0000 7feefca1f640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 0x7feee40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.410+0000 7feefca1f640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 0x7feee40a5460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feef0006010 tx=0x7feef0005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:13.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.414+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feee8014030 con 0x7feef8102680 2024-03-22T02:11:13.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.638+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feef8062830 con 0x7feef8102680 2024-03-22T02:11:13.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.641+0000 7feef5ffb640 1 -- 172.21.15.28:0/853867844 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7feee809c200 con 0x7feef8102680 2024-03-22T02:11:13.641 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:13.641 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-22T02:11:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-03-22T02:11:13.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 msgr2=0x7feee40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:13.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 0x7feee40a5460 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7feef0006010 tx=0x7feef0005c50 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 msgr2=0x7feef819f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feee800c970 tx=0x7feee800ce30 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 shutdown_connections 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feee40a2fa0 0x7feee40a5460 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feef8102680 0x7feef819f450 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feef8101cc0 0x7feef81983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 --2- 172.21.15.28:0/853867844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feef8100ad0 0x7feef8197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 >> 172.21.15.28:0/853867844 conn(0x7feef80fc040 msgr2=0x7feef8104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 shutdown_connections 2024-03-22T02:11:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:13.643+0000 7feefec83640 1 -- 172.21.15.28:0/853867844 wait complete. 2024-03-22T02:11:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: Deploying daemon mds.c on smithi120 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6834/1917291995,v1:172.21.15.120:6835/1917291995] up:boot 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: fsmap 6 up:standby 2024-03-22T02:11:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: Deploying daemon mds.c on smithi120 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6834/1917291995,v1:172.21.15.120:6835/1917291995] up:boot 2024-03-22T02:11:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: fsmap 6 up:standby 2024-03-22T02:11:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:11:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: Deploying daemon mds.c on smithi120 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6834/1917291995,v1:172.21.15.120:6835/1917291995] up:boot 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: fsmap 6 up:standby 2024-03-22T02:11:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:11:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:14 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/853867844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:14.998 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:14 smithi120 systemd[1]: Starting Ceph mds.c for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:15.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:14 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/853867844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:15.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:14 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/853867844' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:15.330 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 podman[67496]: 2024-03-22 02:11:14.936230633 +0000 UTC m=+0.017800937 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:15.331 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 podman[67496]: 2024-03-22 02:11:15.077873066 +0000 UTC m=+0.159443400 container create 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-22T02:11:15.331 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 podman[67496]: 2024-03-22 02:11:15.330615703 +0000 UTC m=+0.412185988 container init 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-22T02:11:15.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:15.748 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 podman[67496]: 2024-03-22 02:11:15.33358065 +0000 UTC m=+0.415150947 container start 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-22T02:11:15.748 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 ceph-mds[67515]: -- 172.21.15.120:0/4022147788 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fdc8cdba40 con 0x55fdc8caf000 2024-03-22T02:11:15.748 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: starting mds.c at 2024-03-22T02:11:15.748 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 bash[67496]: 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 2024-03-22T02:11:15.748 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:11:15 smithi120 systemd[1]: Started Ceph mds.c for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:15.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/217653155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54100ac0 msgr2=0x7f4a54100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:15.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/217653155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54100ac0 0x7f4a54100ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009900 tx=0x7f4a4402bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:15.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/217653155 shutdown_connections 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/217653155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54102670 0x7f4a54108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/217653155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54101cb0 0x7f4a54102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/217653155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54100ac0 0x7f4a54100ee0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/217653155 >> 172.21.15.28:0/217653155 conn(0x7f4a540fc070 msgr2=0x7f4a540fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/217653155 shutdown_connections 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.769+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/217653155 wait complete. 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 Processor -- start 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 -- start start 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 0x7f4a5410b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54101cb0 0x7f4a54109900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5410d820 con 0x7f4a54102670 2024-03-22T02:11:15.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5410d6a0 con 0x7f4a54101cb0 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5c7e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5410d9a0 con 0x7f4a54100ac0 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52384/0 (socket says 172.21.15.28:52384) 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 -- 172.21.15.28:0/1665846645 learned_addr learned my addr 172.21.15.28:0/1665846645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a59d84640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54101cb0 0x7f4a54109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5a585640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 0x7f4a5410b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 -- 172.21.15.28:0/1665846645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 msgr2=0x7f4a5410b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 0x7f4a5410b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 -- 172.21.15.28:0/1665846645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54101cb0 msgr2=0x7f4a54109900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54101cb0 0x7f4a54109900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5ad86640 1 -- 172.21.15.28:0/1665846645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5410a6d0 con 0x7f4a54102670 2024-03-22T02:11:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.770+0000 7f4a5a585640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 0x7f4a5410b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a5ad86640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009fd0 tx=0x7f4a4c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c017070 con 0x7f4a54102670 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a4c003e50 con 0x7f4a54102670 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a541a1fb0 con 0x7f4a54102670 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c0106c0 con 0x7f4a54102670 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.771+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a541a2420 con 0x7f4a54102670 2024-03-22T02:11:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.772+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a54107de0 con 0x7f4a54102670 2024-03-22T02:11:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.773+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4a4c003970 con 0x7f4a54102670 2024-03-22T02:11:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.774+0000 7f4a537fe640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 0x7f4a3c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.774+0000 7f4a5a585640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 0x7f4a3c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.774+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4a4c0cebd0 con 0x7f4a54102670 2024-03-22T02:11:15.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.774+0000 7f4a5a585640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 0x7f4a3c0a53e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402c7b0 tx=0x7f4a44005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:15.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.778+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a4c09bea0 con 0x7f4a54102670 2024-03-22T02:11:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:15.999+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a5402d580 con 0x7f4a54102670 2024-03-22T02:11:16.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.000+0000 7f4a537fe640 1 -- 172.21.15.28:0/1665846645 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f4a4c09b7a0 con 0x7f4a54102670 2024-03-22T02:11:16.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:16.001 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-22T02:11:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-03-22T02:11:16.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 msgr2=0x7f4a3c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:16.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 0x7f4a3c0a53e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4a4402c7b0 tx=0x7f4a44005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:16.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 msgr2=0x7f4a54109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009fd0 tx=0x7f4a4c00b260 comp rx=0 tx=0).stop 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 shutdown_connections 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a3c0a2f20 0x7f4a3c0a53e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a54102670 0x7f4a54109e40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a54101cb0 0x7f4a54109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 --2- 172.21.15.28:0/1665846645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a54100ac0 0x7f4a5410b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 >> 172.21.15.28:0/1665846645 conn(0x7f4a540fc070 msgr2=0x7f4a54104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 shutdown_connections 2024-03-22T02:11:16.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:16.002+0000 7f4a5c7e9640 1 -- 172.21.15.28:0/1665846645 wait complete. 2024-03-22T02:11:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:boot 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: fsmap 7 up:standby 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:11:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:16 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1665846645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:boot 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: fsmap 7 up:standby 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:11:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:16 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1665846645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:boot 2024-03-22T02:11:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: fsmap 7 up:standby 2024-03-22T02:11:16.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:11:16.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1665846645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:17.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:17 smithi120 ceph-mon[36803]: Deploying daemon mds.h on smithi083 2024-03-22T02:11:17.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:17 smithi083 ceph-mon[36340]: Deploying daemon mds.h on smithi083 2024-03-22T02:11:17.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:17 smithi028 ceph-mon[31916]: Deploying daemon mds.h on smithi083 2024-03-22T02:11:18.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/2358443506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 msgr2=0x7f6b88102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/2358443506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88102140 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009900 tx=0x7f6b7002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/2358443506 shutdown_connections 2024-03-22T02:11:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/2358443506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88102680 0x7f6b88108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/2358443506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88102140 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/2358443506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88100ad0 0x7f6b88100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/2358443506 >> 172.21.15.28:0/2358443506 conn(0x7f6b880fc040 msgr2=0x7f6b880fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/2358443506 shutdown_connections 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/2358443506 wait complete. 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.129+0000 7f6b8d4eb640 1 Processor -- start 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 -- start start 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88100ad0 0x7f6b88197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88102680 0x7f6b8819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810d870 con 0x7f6b88101cc0 2024-03-22T02:11:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810d6f0 con 0x7f6b88100ad0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b8d4eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810d9f0 con 0x7f6b88102680 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52404/0 (socket says 172.21.15.28:52404) 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 -- 172.21.15.28:0/628744955 learned_addr learned my addr 172.21.15.28:0/628744955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b86ffd640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88100ad0 0x7f6b88197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b877fe640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88102680 0x7f6b8819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 -- 172.21.15.28:0/628744955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88102680 msgr2=0x7f6b8819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88102680 0x7f6b8819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 -- 172.21.15.28:0/628744955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88100ad0 msgr2=0x7f6b88197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88100ad0 0x7f6b88197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 -- 172.21.15.28:0/628744955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8819fb00 con 0x7f6b88101cc0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.130+0000 7f6b867fc640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005ab0 tx=0x7f6b7002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.131+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70041070 con 0x7f6b88101cc0 2024-03-22T02:11:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.131+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b700054e0 con 0x7f6b88101cc0 2024-03-22T02:11:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.131+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8819fd90 con 0x7f6b88101cc0 2024-03-22T02:11:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.131+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b70034440 con 0x7f6b88101cc0 2024-03-22T02:11:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.131+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b881a0270 con 0x7f6b88101cc0 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.132+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b88107e70 con 0x7f6b88101cc0 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.133+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6b70044070 con 0x7f6b88101cc0 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.133+0000 7f6b7ffff640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 0x7f6b740a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.133+0000 7f6b86ffd640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 0x7f6b740a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.133+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f6b700f0b70 con 0x7f6b88101cc0 2024-03-22T02:11:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.134+0000 7f6b86ffd640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 0x7f6b740a5460 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b80005fd0 tx=0x7f6b8000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.138+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b700bde40 con 0x7f6b88101cc0 2024-03-22T02:11:18.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.362+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b8802d620 con 0x7f6b88101cc0 2024-03-22T02:11:18.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.363+0000 7f6b7ffff640 1 -- 172.21.15.28:0/628744955 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f6b700bd740 con 0x7f6b88101cc0 2024-03-22T02:11:18.363 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:18.363 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-22T02:11:18.364 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-03-22T02:11:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 msgr2=0x7f6b740a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 0x7f6b740a5460 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6b80005fd0 tx=0x7f6b8000b410 comp rx=0 tx=0).stop 2024-03-22T02:11:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 msgr2=0x7f6b88198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:18.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f6b70005ab0 tx=0x7f6b7002c6c0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 shutdown_connections 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a2fa0 0x7f6b740a5460 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88102680 0x7f6b8819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88101cc0 0x7f6b88198420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 --2- 172.21.15.28:0/628744955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b88100ad0 0x7f6b88197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 >> 172.21.15.28:0/628744955 conn(0x7f6b880fc040 msgr2=0x7f6b88105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 shutdown_connections 2024-03-22T02:11:18.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:18.365+0000 7f6b8d4eb640 1 -- 172.21.15.28:0/628744955 wait complete. 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/628744955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:18.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:18.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/628744955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/628744955' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:19 smithi120 ceph-mon[36803]: Deploying daemon mds.i on smithi120 2024-03-22T02:11:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:19 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/3509701876,v1:172.21.15.83:6837/3509701876] up:boot 2024-03-22T02:11:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:19 smithi120 ceph-mon[36803]: fsmap 8 up:standby 2024-03-22T02:11:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:11:19.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:19 smithi083 ceph-mon[36340]: Deploying daemon mds.i on smithi120 2024-03-22T02:11:19.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:19 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/3509701876,v1:172.21.15.83:6837/3509701876] up:boot 2024-03-22T02:11:19.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:19 smithi083 ceph-mon[36340]: fsmap 8 up:standby 2024-03-22T02:11:19.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:11:19.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:19 smithi028 ceph-mon[31916]: Deploying daemon mds.i on smithi120 2024-03-22T02:11:19.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:19 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/3509701876,v1:172.21.15.83:6837/3509701876] up:boot 2024-03-22T02:11:19.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:19 smithi028 ceph-mon[31916]: fsmap 8 up:standby 2024-03-22T02:11:19.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:11:20.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.495+0000 7f2eb17a0640 1 -- 172.21.15.28:0/1245446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac1019d0 msgr2=0x7f2eac101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.495+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/1245446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac1019d0 0x7f2eac101df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c009900 tx=0x7f2e9c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 -- 172.21.15.28:0/1245446669 shutdown_connections 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/1245446669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac103580 0x7f2eac109e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/1245446669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac102bc0 0x7f2eac103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/1245446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac1019d0 0x7f2eac101df0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 -- 172.21.15.28:0/1245446669 >> 172.21.15.28:0/1245446669 conn(0x7f2eac0fcf20 msgr2=0x7f2eac0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 -- 172.21.15.28:0/1245446669 shutdown_connections 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 -- 172.21.15.28:0/1245446669 wait complete. 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 Processor -- start 2024-03-22T02:11:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.496+0000 7f2eb17a0640 1 -- start start 2024-03-22T02:11:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac1019d0 0x7f2eac197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac103580 0x7f2eac19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac10d650 con 0x7f2eac102bc0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac10d4d0 con 0x7f2eac103580 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac10d7d0 con 0x7f2eac1019d0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52410/0 (socket says 172.21.15.28:52410) 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 -- 172.21.15.28:0/2409012512 learned_addr learned my addr 172.21.15.28:0/2409012512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eabfff640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac1019d0 0x7f2eac197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb0f9f640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac103580 0x7f2eac19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 -- 172.21.15.28:0/2409012512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac1019d0 msgr2=0x7f2eac197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac1019d0 0x7f2eac197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 -- 172.21.15.28:0/2409012512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac103580 msgr2=0x7f2eac19f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac103580 0x7f2eac19f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 -- 172.21.15.28:0/2409012512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac19f900 con 0x7f2eac102bc0 2024-03-22T02:11:20.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eab7fe640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9f0 tx=0x7f2e9400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400cd80 con 0x7f2eac102bc0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac19fbf0 con 0x7f2eac102bc0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.497+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e94010910 con 0x7f2eac102bc0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.498+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e940188f0 con 0x7f2eac102bc0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.498+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac1a0160 con 0x7f2eac102bc0 2024-03-22T02:11:20.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.498+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac108cf0 con 0x7f2eac102bc0 2024-03-22T02:11:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.500+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2e94010ac0 con 0x7f2eac102bc0 2024-03-22T02:11:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.500+0000 7f2ea97fa640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 0x7f2e980a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.500+0000 7f2eabfff640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 0x7f2e980a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.500+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f2e940cf810 con 0x7f2eac102bc0 2024-03-22T02:11:20.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.501+0000 7f2eabfff640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 0x7f2e980a53e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c000c00 tx=0x7f2e9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:20.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.504+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e9409cae0 con 0x7f2eac102bc0 2024-03-22T02:11:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.728+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eac02d3b0 con 0x7f2eac102bc0 2024-03-22T02:11:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.729+0000 7f2ea97fa640 1 -- 172.21.15.28:0/2409012512 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f2e9409c3e0 con 0x7f2eac102bc0 2024-03-22T02:11:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:20.729 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-22T02:11:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 msgr2=0x7f2e980a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 0x7f2e980a53e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c000c00 tx=0x7f2e9c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 msgr2=0x7f2eac198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400e9f0 tx=0x7f2e9400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 shutdown_connections 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2e980a2f20 0x7f2e980a53e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2eac103580 0x7f2eac19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eac102bc0 0x7f2eac198230 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 --2- 172.21.15.28:0/2409012512 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2eac1019d0 0x7f2eac197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 >> 172.21.15.28:0/2409012512 conn(0x7f2eac0fcf20 msgr2=0x7f2eac105ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 shutdown_connections 2024-03-22T02:11:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:20.731+0000 7f2eb17a0640 1 -- 172.21.15.28:0/2409012512 wait complete. 2024-03-22T02:11:20.740 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:20 smithi028 ceph-mon[31916]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:20 smithi120 ceph-mon[36803]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:20.748 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:20 smithi120 systemd[1]: Starting Ceph mds.i for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:20.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:20 smithi083 ceph-mon[36340]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:21.164 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:20 smithi120 podman[67803]: 2024-03-22 02:11:20.797864287 +0000 UTC m=+0.018272800 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:21.164 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:20 smithi120 podman[67803]: 2024-03-22 02:11:20.919899423 +0000 UTC m=+0.140307952 container create 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-22T02:11:21.164 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 podman[67803]: 2024-03-22 02:11:21.16447255 +0000 UTC m=+0.384881072 container init 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-22T02:11:21.498 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 podman[67803]: 2024-03-22 02:11:21.167189869 +0000 UTC m=+0.387598382 container start 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-22T02:11:21.498 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mds[67822]: -- 172.21.15.120:0/2671362261 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556e7d353a40 con 0x556e7d326c00 2024-03-22T02:11:21.498 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: starting mds.i at 2024-03-22T02:11:21.498 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 bash[67803]: 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d 2024-03-22T02:11:21.498 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:11:21 smithi120 systemd[1]: Started Ceph mds.i for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:21.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2409012512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:21.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:21.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:21.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2409012512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:22.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2409012512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:22.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:22.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:22 smithi028 ceph-mon[31916]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:22.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:22 smithi028 ceph-mon[31916]: Deploying daemon mds.g on smithi028 2024-03-22T02:11:22.771 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:22 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:boot 2024-03-22T02:11:22.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:22 smithi028 ceph-mon[31916]: fsmap 9 up:standby 2024-03-22T02:11:22.772 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 -- 172.21.15.28:0/3909520587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e01020a0 msgr2=0x7fe2e01024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/3909520587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e01020a0 0x7fe2e01024c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8009ca0 tx=0x7fe2d802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 -- 172.21.15.28:0/3909520587 shutdown_connections 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/3909520587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 0x7fe2e0108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/3909520587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e0100cf0 0x7fe2e0101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/3909520587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e01020a0 0x7fe2e01024c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 -- 172.21.15.28:0/3909520587 >> 172.21.15.28:0/3909520587 conn(0x7fe2e00fb7c0 msgr2=0x7fe2e00fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 -- 172.21.15.28:0/3909520587 shutdown_connections 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.857+0000 7fe2e78fb640 1 -- 172.21.15.28:0/3909520587 wait complete. 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 Processor -- start 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 -- start start 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 0x7fe2e01097d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e01020a0 0x7fe2e0109d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e010d8b0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e010d730 con 0x7fe2e01016b0 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e78fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e010da30 con 0x7fe2e01020a0 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e5697640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e5697640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49310/0 (socket says 172.21.15.28:49310) 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e5697640 1 -- 172.21.15.28:0/2555024338 learned_addr learned my addr 172.21.15.28:0/2555024338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e5e98640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e01020a0 0x7fe2e0109d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.858+0000 7fe2e4e96640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 0x7fe2e01097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 -- 172.21.15.28:0/2555024338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e01020a0 msgr2=0x7fe2e0109d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e01020a0 0x7fe2e0109d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 -- 172.21.15.28:0/2555024338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 msgr2=0x7fe2e01097d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 0x7fe2e01097d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 -- 172.21.15.28:0/2555024338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2e010a5a0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e5697640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80324c0 tx=0x7fe2d8032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8041020 con 0x7fe2e0100cf0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2d8034cd0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2e01a1fb0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8039040 con 0x7fe2e0100cf0 2024-03-22T02:11:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.859+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2e01a23c0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.860+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2e0107e30 con 0x7fe2e0100cf0 2024-03-22T02:11:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.861+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fe2d8044020 con 0x7fe2e0100cf0 2024-03-22T02:11:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.862+0000 7fe2de7fc640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 0x7fe2c80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.862+0000 7fe2e4e96640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 0x7fe2c80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.862+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe2d80f0df0 con 0x7fe2e0100cf0 2024-03-22T02:11:22.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.863+0000 7fe2e4e96640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 0x7fe2c80a53e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe2e010af00 tx=0x7fe2d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:22.867+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe2d804b020 con 0x7fe2e0100cf0 2024-03-22T02:11:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:22 smithi120 ceph-mon[36803]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:22 smithi120 ceph-mon[36803]: Deploying daemon mds.g on smithi028 2024-03-22T02:11:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:22 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:boot 2024-03-22T02:11:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:22 smithi120 ceph-mon[36803]: fsmap 9 up:standby 2024-03-22T02:11:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:11:23.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:22 smithi083 ceph-mon[36340]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:22 smithi083 ceph-mon[36340]: Deploying daemon mds.g on smithi028 2024-03-22T02:11:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:22 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:boot 2024-03-22T02:11:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:22 smithi083 ceph-mon[36340]: fsmap 9 up:standby 2024-03-22T02:11:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:11:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.093+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2e002d3b0 con 0x7fe2e0100cf0 2024-03-22T02:11:23.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.093+0000 7fe2de7fc640 1 -- 172.21.15.28:0/2555024338 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8001 (secure 0 0 0) 0x7fe2d80be0b0 con 0x7fe2e0100cf0 2024-03-22T02:11:23.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:23.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T02:11:23.094 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 msgr2=0x7fe2c80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 0x7fe2c80a53e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe2e010af00 tx=0x7fe2d4005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 msgr2=0x7fe2e010b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe2d80324c0 tx=0x7fe2d8032f40 comp rx=0 tx=0).stop 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 shutdown_connections 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2c80a2f20 0x7fe2c80a53e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2e01020a0 0x7fe2e0109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe2e01016b0 0x7fe2e01097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:23.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 --2- 172.21.15.28:0/2555024338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e0100cf0 0x7fe2e010b190 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.095+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 >> 172.21.15.28:0/2555024338 conn(0x7fe2e00fb7c0 msgr2=0x7fe2e0105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.096+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 shutdown_connections 2024-03-22T02:11:23.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:23.096+0000 7fe2e78fb640 1 -- 172.21.15.28:0/2555024338 wait complete. 2024-03-22T02:11:23.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:23 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2555024338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:23.822 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:23 smithi028 podman[73964]: 2024-03-22 02:11:23.68933949 +0000 UTC m=+0.124473508 container create 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-22T02:11:23.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:23 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2555024338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:24.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:23 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2555024338' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:24.321 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:23 smithi028 podman[73964]: 2024-03-22 02:11:23.950092671 +0000 UTC m=+0.385226703 container init 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:11:24.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:23 smithi028 podman[73964]: 2024-03-22 02:11:23.95267167 +0000 UTC m=+0.387805689 container start 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-22T02:11:24.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:23 smithi028 ceph-mds[73983]: -- 172.21.15.28:0/1441924978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607f969da40 con 0x5607f9671000 2024-03-22T02:11:24.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g[73979]: starting mds.g at 2024-03-22T02:11:24.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:24 smithi028 bash[73964]: 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 2024-03-22T02:11:24.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:11:24 smithi028 systemd[1]: Started Ceph mds.g for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:24.749 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:24.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:24.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:25.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:25.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:25.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:25.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 -- 172.21.15.28:0/2161755100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86381026e0 msgr2=0x7f8638108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 --2- 172.21.15.28:0/2161755100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86381026e0 0x7f8638108f70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8630009ca0 tx=0x7f863002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 -- 172.21.15.28:0/2161755100 shutdown_connections 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 --2- 172.21.15.28:0/2161755100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86381026e0 0x7f8638108f70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 --2- 172.21.15.28:0/2161755100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8638101d20 0x7f86381021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 --2- 172.21.15.28:0/2161755100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638100b30 0x7f8638100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 -- 172.21.15.28:0/2161755100 >> 172.21.15.28:0/2161755100 conn(0x7f86380fc0c0 msgr2=0x7f86380fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 -- 172.21.15.28:0/2161755100 shutdown_connections 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.214+0000 7f863d177640 1 -- 172.21.15.28:0/2161755100 wait complete. 2024-03-22T02:11:25.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 Processor -- start 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 -- start start 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638101d20 0x7f8638198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86381026e0 0x7f863819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f863810d860 con 0x7f8638100b30 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f863810d6e0 con 0x7f86381026e0 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863d177640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f863810d9e0 con 0x7f8638101d20 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f8636d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f8636d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49356/0 (socket says 172.21.15.28:49356) 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f8636d9d640 1 -- 172.21.15.28:0/710915084 learned_addr learned my addr 172.21.15.28:0/710915084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:25.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863759e640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86381026e0 0x7f863819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.215+0000 7f863659c640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638101d20 0x7f8638198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 -- 172.21.15.28:0/710915084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638101d20 msgr2=0x7f8638198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638101d20 0x7f8638198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 -- 172.21.15.28:0/710915084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86381026e0 msgr2=0x7f863819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86381026e0 0x7f863819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 -- 172.21.15.28:0/710915084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863819fa60 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f8636d9d640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f862800d770 tx=0x7f862800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628017070 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8628009c30 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863819fd50 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8628003dd0 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.216+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86381a0210 con 0x7f8638100b30 2024-03-22T02:11:25.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.217+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8638107e50 con 0x7f8638100b30 2024-03-22T02:11:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.218+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f8628003f70 con 0x7f8638100b30 2024-03-22T02:11:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.218+0000 7f861ffff640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 0x7f86200a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.219+0000 7f863659c640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 0x7f86200a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.219+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f86280ce800 con 0x7f8638100b30 2024-03-22T02:11:25.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.219+0000 7f863659c640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 0x7f86200a5460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8638199350 tx=0x7f862c00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:25.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.223+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f862809bad0 con 0x7f8638100b30 2024-03-22T02:11:25.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.447+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f863802d3b0 con 0x7f8638100b30 2024-03-22T02:11:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.447+0000 7f861ffff640 1 -- 172.21.15.28:0/710915084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8824 (secure 0 0 0) 0x7f8628009de0 con 0x7f8638100b30 2024-03-22T02:11:25.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:25.449 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T02:11:25.449 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 msgr2=0x7f86200a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 0x7f86200a5460 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8638199350 tx=0x7f862c00b590 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 msgr2=0x7f8638197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f862800d770 tx=0x7f862800dc30 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 shutdown_connections 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86200a2fa0 0x7f86200a5460 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f86381026e0 0x7f863819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8638101d20 0x7f8638198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 --2- 172.21.15.28:0/710915084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8638100b30 0x7f8638197e40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.449+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 >> 172.21.15.28:0/710915084 conn(0x7f86380fc0c0 msgr2=0x7f8638105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:25.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.450+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 shutdown_connections 2024-03-22T02:11:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:25.450+0000 7f863d177640 1 -- 172.21.15.28:0/710915084 wait complete. 2024-03-22T02:11:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: Deploying daemon mds.a on smithi028 2024-03-22T02:11:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.28:6838/678476485,v1:172.21.15.28:6839/678476485] up:boot 2024-03-22T02:11:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: fsmap 10 up:standby 2024-03-22T02:11:25.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T02:11:25.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:25.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:25 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/710915084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: Deploying daemon mds.a on smithi028 2024-03-22T02:11:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.28:6838/678476485,v1:172.21.15.28:6839/678476485] up:boot 2024-03-22T02:11:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: fsmap 10 up:standby 2024-03-22T02:11:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T02:11:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:25.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:25 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/710915084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:26.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: Deploying daemon mds.a on smithi028 2024-03-22T02:11:26.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.28:6838/678476485,v1:172.21.15.28:6839/678476485] up:boot 2024-03-22T02:11:26.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: fsmap 10 up:standby 2024-03-22T02:11:26.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-22T02:11:26.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:26.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:25 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/710915084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:26.322 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 systemd[1]: Starting Ceph mds.a for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:26.705 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 podman[74315]: 2024-03-22 02:11:26.34430737 +0000 UTC m=+0.017843021 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:26.705 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 podman[74315]: 2024-03-22 02:11:26.443859185 +0000 UTC m=+0.117394875 container create ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-22T02:11:26.706 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 podman[74315]: 2024-03-22 02:11:26.679502489 +0000 UTC m=+0.353038154 container init ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-22T02:11:26.706 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 podman[74315]: 2024-03-22 02:11:26.682158802 +0000 UTC m=+0.355694450 container start ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-22T02:11:26.706 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 ceph-mds[74334]: -- 172.21.15.28:0/2784109566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56405a28ba40 con 0x56405a25f000 2024-03-22T02:11:27.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a[74330]: starting mds.a at 2024-03-22T02:11:27.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 bash[74315]: ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee 2024-03-22T02:11:27.071 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:11:26 smithi028 systemd[1]: Started Ceph mds.a for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:27.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 -- 172.21.15.28:0/2676630356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 msgr2=0x7f121c10a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 --2- 172.21.15.28:0/2676630356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c10a020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1214009ca0 tx=0x7f121402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 -- 172.21.15.28:0/2676630356 shutdown_connections 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 --2- 172.21.15.28:0/2676630356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c10a020 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 --2- 172.21.15.28:0/2676630356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c102dd0 0x7f121c103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 --2- 172.21.15.28:0/2676630356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c101be0 0x7f121c102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 -- 172.21.15.28:0/2676630356 >> 172.21.15.28:0/2676630356 conn(0x7f121c0fd130 msgr2=0x7f121c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:27.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 -- 172.21.15.28:0/2676630356 shutdown_connections 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 -- 172.21.15.28:0/2676630356 wait complete. 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.572+0000 7f122355e640 1 Processor -- start 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 -- start start 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c101be0 0x7f121c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c102dd0 0x7f121c1983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f121c10d860 con 0x7f121c103790 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f121c10d6e0 con 0x7f121c101be0 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f121c10d9e0 con 0x7f121c102dd0 2024-03-22T02:11:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49394/0 (socket says 172.21.15.28:49394) 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 -- 172.21.15.28:0/4265618223 learned_addr learned my addr 172.21.15.28:0/4265618223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f12212fa640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c101be0 0x7f121c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1220af9640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c102dd0 0x7f121c1983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 -- 172.21.15.28:0/4265618223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c102dd0 msgr2=0x7f121c1983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c102dd0 0x7f121c1983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 -- 172.21.15.28:0/4265618223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c101be0 msgr2=0x7f121c197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c101be0 0x7f121c197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 -- 172.21.15.28:0/4265618223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f121c19fa90 con 0x7f121c103790 2024-03-22T02:11:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f1221afb640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1214037040 tx=0x7f1214032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214039040 con 0x7f121c103790 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1214044070 con 0x7f121c103790 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f121c19fd20 con 0x7f121c103790 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.573+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214004330 con 0x7f121c103790 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.574+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f121c1a0200 con 0x7f121c103790 2024-03-22T02:11:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.574+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121c108f00 con 0x7f121c103790 2024-03-22T02:11:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.576+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f1214003a00 con 0x7f121c103790 2024-03-22T02:11:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.576+0000 7f12127fc640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 0x7f12080a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.576+0000 7f12212fa640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 0x7f12080a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.576+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f12140f1ca0 con 0x7f121c103790 2024-03-22T02:11:27.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.577+0000 7f12212fa640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 0x7f12080a5460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12180058c0 tx=0x7f121800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:27.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.580+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12140bef70 con 0x7f121c103790 2024-03-22T02:11:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.805+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f121c02d3b0 con 0x7f121c103790 2024-03-22T02:11:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.806+0000 7f12127fc640 1 -- 172.21.15.28:0/4265618223 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9650 (secure 0 0 0) 0x7f12140be870 con 0x7f121c103790 2024-03-22T02:11:27.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:27.808 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-22T02:11:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 msgr2=0x7f12080a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 0x7f12080a5460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12180058c0 tx=0x7f121800a2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 msgr2=0x7f121c19f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1214037040 tx=0x7f1214032f90 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 shutdown_connections 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12080a2fa0 0x7f12080a5460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121c103790 0x7f121c19f420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f121c102dd0 0x7f121c1983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 --2- 172.21.15.28:0/4265618223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f121c101be0 0x7f121c197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 >> 172.21.15.28:0/4265618223 conn(0x7f121c0fd130 msgr2=0x7f121c1060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 shutdown_connections 2024-03-22T02:11:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:27.808+0000 7f122355e640 1 -- 172.21.15.28:0/4265618223 wait complete. 2024-03-22T02:11:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: Deploying daemon mds.b on smithi083 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.28:6840/2956844816,v1:172.21.15.28:6841/2956844816] up:boot 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: fsmap 11 up:standby 2024-03-22T02:11:28.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T02:11:28.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:27 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4265618223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: Deploying daemon mds.b on smithi083 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.28:6840/2956844816,v1:172.21.15.28:6841/2956844816] up:boot 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: fsmap 11 up:standby 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T02:11:28.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:27 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4265618223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-22T02:11:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: Deploying daemon mds.b on smithi083 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.28:6840/2956844816,v1:172.21.15.28:6841/2956844816] up:boot 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: fsmap 11 up:standby 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-22T02:11:28.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:27 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4265618223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:29.265 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 systemd[1]: Starting Ceph mds.b for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:11:29.651 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 podman[66790]: 2024-03-22 02:11:29.286446988 +0000 UTC m=+0.018385195 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:11:29.651 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 podman[66790]: 2024-03-22 02:11:29.399233049 +0000 UTC m=+0.131171236 container create 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-22T02:11:29.652 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 podman[66790]: 2024-03-22 02:11:29.652053541 +0000 UTC m=+0.383991742 container init 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-22T02:11:29.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.937+0000 7f30c1050640 1 -- 172.21.15.28:0/801346902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc102da0 msgr2=0x7f30bc103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.937+0000 7f30c1050640 1 --2- 172.21.15.28:0/801346902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc102da0 0x7f30bc103220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009900 tx=0x7f30a402bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 -- 172.21.15.28:0/801346902 shutdown_connections 2024-03-22T02:11:29.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 --2- 172.21.15.28:0/801346902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc103760 0x7f30bc109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 --2- 172.21.15.28:0/801346902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc102da0 0x7f30bc103220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 --2- 172.21.15.28:0/801346902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc101bb0 0x7f30bc101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 -- 172.21.15.28:0/801346902 >> 172.21.15.28:0/801346902 conn(0x7f30bc0fd140 msgr2=0x7f30bc0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 -- 172.21.15.28:0/801346902 shutdown_connections 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 -- 172.21.15.28:0/801346902 wait complete. 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 Processor -- start 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.938+0000 7f30c1050640 1 -- start start 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc101bb0 0x7f30bc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc102da0 0x7f30bc1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d810 con 0x7f30bc103760 2024-03-22T02:11:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d690 con 0x7f30bc101bb0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30bc10d990 con 0x7f30bc102da0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49426/0 (socket says 172.21.15.28:49426) 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 -- 172.21.15.28:0/1067787300 learned_addr learned my addr 172.21.15.28:0/1067787300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bad9d640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc101bb0 0x7f30bc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30b3fff640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc102da0 0x7f30bc1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 -- 172.21.15.28:0/1067787300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc102da0 msgr2=0x7f30bc1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc102da0 0x7f30bc1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 -- 172.21.15.28:0/1067787300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc101bb0 msgr2=0x7f30bc197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc101bb0 0x7f30bc197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 -- 172.21.15.28:0/1067787300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30bc19faa0 con 0x7f30bc103760 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30bb59e640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f30b400cad0 tx=0x7f30b400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400ae20 con 0x7f30bc103760 2024-03-22T02:11:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30bc19fd90 con 0x7f30bc103760 2024-03-22T02:11:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30b4005020 con 0x7f30bc103760 2024-03-22T02:11:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400f6f0 con 0x7f30bc103760 2024-03-22T02:11:29.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.939+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30bc1a0200 con 0x7f30bc103760 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.940+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30bc108ed0 con 0x7f30bc103760 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.942+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f30b40051d0 con 0x7f30bc103760 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.942+0000 7f30b8d99640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 0x7f30a80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.942+0000 7f30bad9d640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 0x7f30a80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.942+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f30b40cf360 con 0x7f30bc103760 2024-03-22T02:11:29.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.943+0000 7f30bad9d640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 0x7f30a80a53e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f30bc102c00 tx=0x7f30ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:29.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:29.946+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30b4014030 con 0x7f30bc103760 2024-03-22T02:11:30.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:29 smithi083 ceph-mon[36340]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:30.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 podman[66790]: 2024-03-22 02:11:29.654726043 +0000 UTC m=+0.386664239 container start 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 ceph-mds[66809]: -- 172.21.15.83:0/2058724980 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5643b0965a40 con 0x5643b0938800 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: starting mds.b at 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 bash[66790]: 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef 2024-03-22T02:11:30.015 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:29 smithi083 systemd[1]: Started Ceph mds.b for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:11:30.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.174+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30bc02d3b0 con 0x7f30bc103760 2024-03-22T02:11:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.174+0000 7f30b8d99640 1 -- 172.21.15.28:0/1067787300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10476 (secure 0 0 0) 0x7f30b409c630 con 0x7f30bc103760 2024-03-22T02:11:30.176 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:30.176 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24509,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-22T02:11:30.176 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-03-22T02:11:30.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 msgr2=0x7f30a80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 0x7f30a80a53e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f30bc102c00 tx=0x7f30ac005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 msgr2=0x7f30bc19f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f30b400cad0 tx=0x7f30b400cf90 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 shutdown_connections 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30a80a2f20 0x7f30a80a53e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30bc103760 0x7f30bc19f430 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30bc102da0 0x7f30bc1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 --2- 172.21.15.28:0/1067787300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30bc101bb0 0x7f30bc197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 >> 172.21.15.28:0/1067787300 conn(0x7f30bc0fd140 msgr2=0x7f30bc1060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 shutdown_connections 2024-03-22T02:11:30.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.177+0000 7f30c1050640 1 -- 172.21.15.28:0/1067787300 wait complete. 2024-03-22T02:11:30.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:29 smithi028 ceph-mon[31916]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:30.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:29 smithi120 ceph-mon[36803]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.292+0000 7fad49cdf640 1 -- 172.21.15.28:0/2280589244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 msgr2=0x7fad44109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2280589244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad44109d20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fad3c009ca0 tx=0x7fad3c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 -- 172.21.15.28:0/2280589244 shutdown_connections 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2280589244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad44109d20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2280589244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad44105000 0x7fad441073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2280589244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad441026d0 0x7fad44104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 -- 172.21.15.28:0/2280589244 >> 172.21.15.28:0/2280589244 conn(0x7fad440fc020 msgr2=0x7fad440fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 -- 172.21.15.28:0/2280589244 shutdown_connections 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 -- 172.21.15.28:0/2280589244 wait complete. 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 Processor -- start 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.293+0000 7fad49cdf640 1 -- start start 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad441026d0 0x7fad44197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 0x7fad441983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad4410d850 con 0x7fad44107930 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fad4410d6d0 con 0x7fad44105000 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad4410d9d0 con 0x7fad441026d0 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49444/0 (socket says 172.21.15.28:49444) 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 -- 172.21.15.28:0/2519143094 learned_addr learned my addr 172.21.15.28:0/2519143094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad437fe640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad441026d0 0x7fad44197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad42ffd640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 0x7fad441983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 -- 172.21.15.28:0/2519143094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad441026d0 msgr2=0x7fad44197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad441026d0 0x7fad44197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 -- 172.21.15.28:0/2519143094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 msgr2=0x7fad441983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 0x7fad441983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 -- 172.21.15.28:0/2519143094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4419fa90 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad42ffd640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 0x7fad441983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad43fff640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad3c037040 tx=0x7fad3c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c041020 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad3c034cd0 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad4419fd20 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.294+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad3c039040 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.295+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad441a01c0 con 0x7fad44107930 2024-03-22T02:11:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.295+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad440fdb10 con 0x7fad44107930 2024-03-22T02:11:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.297+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fad3c044020 con 0x7fad44107930 2024-03-22T02:11:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.298+0000 7fad40ff9640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 0x7fad2c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.298+0000 7fad437fe640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 0x7fad2c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.298+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fad3c0f0e00 con 0x7fad44107930 2024-03-22T02:11:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.298+0000 7fad437fe640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 0x7fad2c0a53e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad34009bd0 tx=0x7fad34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.303+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad3c04b020 con 0x7fad44107930 2024-03-22T02:11:30.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.488+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad44062790 con 0x7fad44107930 2024-03-22T02:11:30.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.490+0000 7fad40ff9640 1 -- 172.21.15.28:0/2519143094 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18405 (secure 0 0 0) 0x7fad3c0be0c0 con 0x7fad44107930 2024-03-22T02:11:30.492 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:30.492 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:10:43.155372+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 msgr2=0x7fad2c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 0x7fad2c0a53e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad34009bd0 tx=0x7fad34005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 msgr2=0x7fad4419f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fad3c037040 tx=0x7fad3c032f40 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 shutdown_connections 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad2c0a2f20 0x7fad2c0a53e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44107930 0x7fad4419f420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad44105000 0x7fad441983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 --2- 172.21.15.28:0/2519143094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad441026d0 0x7fad44197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 >> 172.21.15.28:0/2519143094 conn(0x7fad440fc020 msgr2=0x7fad44103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 shutdown_connections 2024-03-22T02:11:30.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.492+0000 7fad49cdf640 1 -- 172.21.15.28:0/2519143094 wait complete. 2024-03-22T02:11:30.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-22T02:11:30.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:11:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 -- 172.21.15.28:0/1101284178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47781037e0 msgr2=0x7f477810a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1101284178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47781037e0 0x7f477810a070 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4770009cd0 tx=0x7f477002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 -- 172.21.15.28:0/1101284178 shutdown_connections 2024-03-22T02:11:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1101284178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47781037e0 0x7f477810a070 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1101284178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778102e20 0x7f47781032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1101284178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f47780ffe50 0x7f4778100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 -- 172.21.15.28:0/1101284178 >> 172.21.15.28:0/1101284178 conn(0x7f47780fb780 msgr2=0x7f47780fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 -- 172.21.15.28:0/1101284178 shutdown_connections 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.627+0000 7f477d7ac640 1 -- 172.21.15.28:0/1101284178 wait complete. 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 Processor -- start 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 -- start start 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778102e20 0x7f4778100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 0x7f4778101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f477810d820 con 0x7f47780ffe50 2024-03-22T02:11:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f477810d6a0 con 0x7f47781037e0 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f477d7ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f477810d9a0 con 0x7f4778102e20 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49462/0 (socket says 172.21.15.28:49462) 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 -- 172.21.15.28:0/1698915666 learned_addr learned my addr 172.21.15.28:0/1698915666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f476ffff640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778102e20 0x7f4778100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f47777fe640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 0x7f4778101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 -- 172.21.15.28:0/1698915666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778102e20 msgr2=0x7f4778100dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778102e20 0x7f4778100dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 -- 172.21.15.28:0/1698915666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 msgr2=0x7f4778101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 0x7f4778101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f4776ffd640 1 -- 172.21.15.28:0/1698915666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4778101ba0 con 0x7f47780ffe50 2024-03-22T02:11:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.628+0000 7f47777fe640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 0x7f4778101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f4776ffd640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f476800e9c0 tx=0x7f476800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476800cd80 con 0x7f47780ffe50 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47781a1fe0 con 0x7f47780ffe50 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47680035f0 con 0x7f47780ffe50 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768010690 con 0x7f47780ffe50 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.629+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781a23e0 con 0x7f47780ffe50 2024-03-22T02:11:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.630+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4778108f50 con 0x7f47780ffe50 2024-03-22T02:11:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.631+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f47680037a0 con 0x7f47780ffe50 2024-03-22T02:11:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.632+0000 7f4774ff9640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 0x7f47640a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.632+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f47680cf3c0 con 0x7f47780ffe50 2024-03-22T02:11:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.632+0000 7f476ffff640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 0x7f47640a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.632+0000 7f476ffff640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 0x7f47640a53e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47600095f0 tx=0x7f4760005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.636+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f476809c690 con 0x7f47780ffe50 2024-03-22T02:11:30.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.821+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4778062890 con 0x7f47780ffe50 2024-03-22T02:11:30.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.821+0000 7f4774ff9640 1 -- 172.21.15.28:0/1698915666 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f476809bf90 con 0x7f47780ffe50 2024-03-22T02:11:30.826 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:11:30.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 msgr2=0x7f47640a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 0x7f47640a53e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f47600095f0 tx=0x7f4760005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 msgr2=0x7f4778102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f476800e9c0 tx=0x7f476800ee80 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 shutdown_connections 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a2f20 0x7f47640a53e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f47781037e0 0x7f4778101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778102e20 0x7f4778100dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 --2- 172.21.15.28:0/1698915666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47780ffe50 0x7f4778102790 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 >> 172.21.15.28:0/1698915666 conn(0x7f47780fb780 msgr2=0x7f4778106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 shutdown_connections 2024-03-22T02:11:30.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.823+0000 7f477d7ac640 1 -- 172.21.15.28:0/1698915666 wait complete. 2024-03-22T02:11:30.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-22T02:11:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- 172.21.15.28:0/4136584994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45481078f0 msgr2=0x7f4548109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- 172.21.15.28:0/4136584994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45481078f0 0x7f4548109ce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4540009ca0 tx=0x7f454002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- 172.21.15.28:0/4136584994 shutdown_connections 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- 172.21.15.28:0/4136584994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45481078f0 0x7f4548109ce0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- 172.21.15.28:0/4136584994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4548104fc0 0x7f45481073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- 172.21.15.28:0/4136584994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548102690 0x7f4548104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- 172.21.15.28:0/4136584994 >> 172.21.15.28:0/4136584994 conn(0x7f45480fc020 msgr2=0x7f45480fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- 172.21.15.28:0/4136584994 shutdown_connections 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- 172.21.15.28:0/4136584994 wait complete. 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 Processor -- start 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- start start 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548104fc0 0x7f4548198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45481078f0 0x7f454819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f454810d850 con 0x7f4548102690 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f454810d6d0 con 0x7f4548104fc0 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.954+0000 7f455035c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f454810d9d0 con 0x7f45481078f0 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49480/0 (socket says 172.21.15.28:49480) 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 -- 172.21.15.28:0/1968806573 learned_addr learned my addr 172.21.15.28:0/1968806573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454d8f7640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548104fc0 0x7f4548198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e8f9640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45481078f0 0x7f454819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 -- 172.21.15.28:0/1968806573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45481078f0 msgr2=0x7f454819f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45481078f0 0x7f454819f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 -- 172.21.15.28:0/1968806573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548104fc0 msgr2=0x7f4548198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548104fc0 0x7f4548198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 -- 172.21.15.28:0/1968806573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454819fae0 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f454e0f8640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f454400e9f0 tx=0x7f454400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454400cd80 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454819fdd0 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4544010910 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45440188f0 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.955+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45481a0230 con 0x7f4548102690 2024-03-22T02:11:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.956+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45480fdb10 con 0x7f4548102690 2024-03-22T02:11:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.957+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4544010ac0 con 0x7f4548102690 2024-03-22T02:11:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.958+0000 7f453f7fe640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 0x7f45340a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.958+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f45440cf2e0 con 0x7f4548102690 2024-03-22T02:11:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.958+0000 7f454d8f7640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 0x7f45340a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.958+0000 7f454d8f7640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 0x7f45340a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45481993d0 tx=0x7f4538005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:30.962+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f454409bf60 con 0x7f4548102690 2024-03-22T02:11:31.097 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:boot 2024-03-22T02:11:31.097 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: fsmap 12 up:standby 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1067787300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2519143094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.098 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1698915666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:boot 2024-03-22T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: fsmap 12 up:standby 2024-03-22T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.109 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.110 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.110 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.110 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1067787300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.110 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2519143094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.110 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1698915666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:31.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:boot 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: fsmap 12 up:standby 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1067787300' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2519143094' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1698915666' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.155+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f454802d600 con 0x7f4548102690 2024-03-22T02:11:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.156+0000 7f453f7fe640 1 -- 172.21.15.28:0/1968806573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f45440a0000 con 0x7f4548102690 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:11:31.158 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-03-22T02:11:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 msgr2=0x7f45340a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 0x7f45340a53e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45481993d0 tx=0x7f4538005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 msgr2=0x7f4548197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f454400e9f0 tx=0x7f454400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 shutdown_connections 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45340a2f20 0x7f45340a53e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45481078f0 0x7f454819f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4548104fc0 0x7f4548198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 --2- 172.21.15.28:0/1968806573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4548102690 0x7f4548197ec0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 >> 172.21.15.28:0/1968806573 conn(0x7f45480fc020 msgr2=0x7f4548102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:31.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 shutdown_connections 2024-03-22T02:11:31.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.158+0000 7f455035c640 1 -- 172.21.15.28:0/1968806573 wait complete. 2024-03-22T02:11:31.170 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-22T02:11:31.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-22T02:11:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 -- 172.21.15.28:0/2139653122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928100a90 msgr2=0x7f5928100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 --2- 172.21.15.28:0/2139653122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928100a90 0x7f5928100eb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5918009900 tx=0x7f591802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 -- 172.21.15.28:0/2139653122 shutdown_connections 2024-03-22T02:11:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 --2- 172.21.15.28:0/2139653122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928102640 0x7f5928108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 --2- 172.21.15.28:0/2139653122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928101c80 0x7f5928102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 --2- 172.21.15.28:0/2139653122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928100a90 0x7f5928100eb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 -- 172.21.15.28:0/2139653122 >> 172.21.15.28:0/2139653122 conn(0x7f59280fc020 msgr2=0x7f59280fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 -- 172.21.15.28:0/2139653122 shutdown_connections 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 -- 172.21.15.28:0/2139653122 wait complete. 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.281+0000 7f592db19640 1 Processor -- start 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 -- start start 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 0x7f5928197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 0x7f5928198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f592810c6e0 con 0x7f5928102640 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f592810c560 con 0x7f5928100a90 2024-03-22T02:11:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f592810c860 con 0x7f5928101c80 2024-03-22T02:11:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49502/0 (socket says 172.21.15.28:49502) 2024-03-22T02:11:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 -- 172.21.15.28:0/789420949 learned_addr learned my addr 172.21.15.28:0/789420949 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f59277fe640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 0x7f5928197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:31.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5926ffd640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 0x7f5928198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 -- 172.21.15.28:0/789420949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 msgr2=0x7f5928198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 0x7f5928198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 -- 172.21.15.28:0/789420949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 msgr2=0x7f5928197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 0x7f5928197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 -- 172.21.15.28:0/789420949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592819fa90 con 0x7f5928102640 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5926ffd640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 0x7f5928198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f59277fe640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 0x7f5928197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5927fff640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5920009fd0 tx=0x7f592000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5920017070 con 0x7f5928102640 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5920003e50 con 0x7f5928102640 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.282+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592819fd80 con 0x7f5928102640 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.283+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59200106c0 con 0x7f5928102640 2024-03-22T02:11:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.283+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59281a0220 con 0x7f5928102640 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.284+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59281a0450 con 0x7f5928102640 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.284+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f5920003970 con 0x7f5928102640 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.285+0000 7f5924ff9640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 0x7f59100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.285+0000 7f59277fe640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 0x7f59100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.285+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f59200cedc0 con 0x7f5928102640 2024-03-22T02:11:31.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.286+0000 7f59277fe640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 0x7f59100a53e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f591802c7b0 tx=0x7f5918005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.290+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f592009c090 con 0x7f5928102640 2024-03-22T02:11:31.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.485+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f59281a07e0 con 0x7f5928102640 2024-03-22T02:11:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.886+0000 7f5924ff9640 1 -- 172.21.15.28:0/789420949 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f592009b990 con 0x7f5928102640 2024-03-22T02:11:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-03-22T02:11:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 msgr2=0x7f59100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 0x7f59100a53e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f591802c7b0 tx=0x7f5918005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 msgr2=0x7f592819f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:31.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5920009fd0 tx=0x7f592000b260 comp rx=0 tx=0).stop 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 shutdown_connections 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59100a2f20 0x7f59100a53e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5928102640 0x7f592819f420 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5928101c80 0x7f5928198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 --2- 172.21.15.28:0/789420949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5928100a90 0x7f5928197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 >> 172.21.15.28:0/789420949 conn(0x7f59280fc020 msgr2=0x7f59280fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.888+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 shutdown_connections 2024-03-22T02:11:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:31.889+0000 7f592db19640 1 -- 172.21.15.28:0/789420949 wait complete. 2024-03-22T02:11:31.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-22T02:11:32.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/4173861821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb04100600 msgr2=0x7ffb0410a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:32.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/4173861821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb04100600 0x7ffb0410a1b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009ca0 tx=0x7ffafc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/4173861821 shutdown_connections 2024-03-22T02:11:32.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/4173861821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb04100600 0x7ffb0410a1b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/4173861821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb041000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/4173861821 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb041031d0 0x7ffb041035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/4173861821 >> 172.21.15.28:0/4173861821 conn(0x7ffb040fb7a0 msgr2=0x7ffb040fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/4173861821 shutdown_connections 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/4173861821 wait complete. 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 Processor -- start 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.014+0000 7ffb0c5dd640 1 -- start start 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb0419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb04100600 0x7ffb0419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0410d830 con 0x7ffb041031d0 2024-03-22T02:11:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0410d6b0 con 0x7ffb04100600 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0410d9b0 con 0x7ffb040ffc40 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38940/0 (socket says 172.21.15.28:38940) 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 -- 172.21.15.28:0/2119543255 learned_addr learned my addr 172.21.15.28:0/2119543255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb09b78640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb04100600 0x7ffb0419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0a379640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb0419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 -- 172.21.15.28:0/2119543255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 msgr2=0x7ffb0419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb0419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 -- 172.21.15.28:0/2119543255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb04100600 msgr2=0x7ffb0419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb04100600 0x7ffb0419a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 -- 172.21.15.28:0/2119543255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb041a1d00 con 0x7ffb041031d0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0a379640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb0419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0ab7a640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffafc037040 tx=0x7ffafc032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc041020 con 0x7ffb041031d0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffafc034cd0 con 0x7ffb041031d0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb041a1f90 con 0x7ffb041031d0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc039040 con 0x7ffb041031d0 2024-03-22T02:11:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.015+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb041a23f0 con 0x7ffb041031d0 2024-03-22T02:11:32.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.016+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb041a2650 con 0x7ffb041031d0 2024-03-22T02:11:32.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.018+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ffafc044020 con 0x7ffb041031d0 2024-03-22T02:11:32.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.018+0000 7ffafb7fe640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 0x7ffaf00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.019+0000 7ffb0a379640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 0x7ffaf00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.019+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7ffafc0f1370 con 0x7ffb041031d0 2024-03-22T02:11:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.019+0000 7ffb0a379640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 0x7ffaf00a53e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb04069e60 tx=0x7ffb00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:32.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.026+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffafc04b020 con 0x7ffb041031d0 2024-03-22T02:11:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.213+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7ffb041a2820 con 0x7ffb041031d0 2024-03-22T02:11:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1968806573' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T02:11:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T02:11:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:32.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1968806573' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T02:11:32.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1968806573' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:32.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:32.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.892+0000 7ffafb7fe640 1 -- 172.21.15.28:0/2119543255 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) ==== 171+0+0 (secure 0 0 0) 0x7ffafc0be560 con 0x7ffb041031d0 2024-03-22T02:11:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-03-22T02:11:32.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 msgr2=0x7ffaf00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:32.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 0x7ffaf00a53e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb04069e60 tx=0x7ffb00005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:32.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 msgr2=0x7ffb041a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffafc037040 tx=0x7ffafc032f40 comp rx=0 tx=0).stop 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 shutdown_connections 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffaf00a2f20 0x7ffaf00a53e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb041031d0 0x7ffb041a1690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffb04100600 0x7ffb0419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 --2- 172.21.15.28:0/2119543255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffb040ffc40 0x7ffb0419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.895+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 >> 172.21.15.28:0/2119543255 conn(0x7ffb040fb7a0 msgr2=0x7ffb04100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.896+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 shutdown_connections 2024-03-22T02:11:32.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:32.896+0000 7ffb0c5dd640 1 -- 172.21.15.28:0/2119543255 wait complete. 2024-03-22T02:11:32.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-22T02:11:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- 172.21.15.28:0/679306913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c102d90 msgr2=0x7f772c103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 --2- 172.21.15.28:0/679306913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c102d90 0x7f772c103210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7720009900 tx=0x7f772002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- 172.21.15.28:0/679306913 shutdown_connections 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 --2- 172.21.15.28:0/679306913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 --2- 172.21.15.28:0/679306913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c102d90 0x7f772c103210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 --2- 172.21.15.28:0/679306913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c101ba0 0x7f772c101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- 172.21.15.28:0/679306913 >> 172.21.15.28:0/679306913 conn(0x7f772c0fd130 msgr2=0x7f772c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- 172.21.15.28:0/679306913 shutdown_connections 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- 172.21.15.28:0/679306913 wait complete. 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 Processor -- start 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.026+0000 7f7732902640 1 -- start start 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:33.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c102d90 0x7f772c198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c19f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10d800 con 0x7f772c101ba0 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10d680 con 0x7f772c103750 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f772c10d980 con 0x7f772c102d90 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38958/0 (socket says 172.21.15.28:38958) 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 -- 172.21.15.28:0/2289712581 learned_addr learned my addr 172.21.15.28:0/2289712581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772b7fe640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c102d90 0x7f772c198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7730e9f640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c19f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 -- 172.21.15.28:0/2289712581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c102d90 msgr2=0x7f772c198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c102d90 0x7f772c198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 -- 172.21.15.28:0/2289712581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 msgr2=0x7f772c19f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:33.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c19f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 -- 172.21.15.28:0/2289712581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c19fa00 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7730e9f640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c19f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f772bfff640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c970 tx=0x7f771c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c007c10 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772c19fcf0 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f771c007db0 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c00f8a0 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.027+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772c1a01b0 con 0x7f772c101ba0 2024-03-22T02:11:33.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.028+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c108e60 con 0x7f772c101ba0 2024-03-22T02:11:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.029+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f771c00fa60 con 0x7f772c101ba0 2024-03-22T02:11:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.030+0000 7f77297fa640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 0x7f77140a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.030+0000 7f772b7fe640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 0x7f77140a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.030+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f771c0cf910 con 0x7f772c101ba0 2024-03-22T02:11:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.031+0000 7f772b7fe640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 0x7f77140a53e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7720006010 tx=0x7f7720000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:33.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.034+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f771c09c9f0 con 0x7f772c101ba0 2024-03-22T02:11:33.065 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:32 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T02:11:33.065 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:32 smithi120 ceph-mon[36803]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T02:11:33.065 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:33.065 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:32 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T02:11:33.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:32 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T02:11:33.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:32 smithi028 ceph-mon[31916]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T02:11:33.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:33.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:32 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T02:11:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.257+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f772c02d770 con 0x7f772c101ba0 2024-03-22T02:11:33.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:32 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/789420949' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-22T02:11:33.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:32 smithi083 ceph-mon[36340]: osdmap e83: 12 total, 12 up, 12 in 2024-03-22T02:11:33.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:11:33.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:32 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-22T02:11:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.893+0000 7f77297fa640 1 -- 172.21.15.28:0/2289712581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f771c09c2f0 con 0x7f772c101ba0 2024-03-22T02:11:33.894 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.895+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 msgr2=0x7f77140a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.895+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 0x7f77140a53e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7720006010 tx=0x7f7720000bc0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.895+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 msgr2=0x7f772c197de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.895+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c970 tx=0x7f771c00ce30 comp rx=0 tx=0).stop 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 shutdown_connections 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77140a2f20 0x7f77140a53e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f772c103750 0x7f772c19f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f772c102d90 0x7f772c198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 --2- 172.21.15.28:0/2289712581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f772c101ba0 0x7f772c197de0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 >> 172.21.15.28:0/2289712581 conn(0x7f772c0fd130 msgr2=0x7f772c106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 shutdown_connections 2024-03-22T02:11:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:33.896+0000 7f7732902640 1 -- 172.21.15.28:0/2289712581 wait complete. 2024-03-22T02:11:33.910 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-22T02:11:33.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-22T02:11:34.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 -- 172.21.15.28:0/1207701726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb0100b30 msgr2=0x7f9fb0100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 --2- 172.21.15.28:0/1207701726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb0100b30 0x7f9fb0100f50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9fac009900 tx=0x7f9fac02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 -- 172.21.15.28:0/1207701726 shutdown_connections 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 --2- 172.21.15.28:0/1207701726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb01026e0 0x7f9fb0108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 --2- 172.21.15.28:0/1207701726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0101d20 0x7f9fb01021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 --2- 172.21.15.28:0/1207701726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb0100b30 0x7f9fb0100f50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 -- 172.21.15.28:0/1207701726 >> 172.21.15.28:0/1207701726 conn(0x7f9fb00fc0a0 msgr2=0x7f9fb00fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 -- 172.21.15.28:0/1207701726 shutdown_connections 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.019+0000 7f9fb753e640 1 -- 172.21.15.28:0/1207701726 wait complete. 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 Processor -- start 2024-03-22T02:11:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 -- start start 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0100b30 0x7f9fb006ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb0101d20 0x7f9fb019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb010c710 con 0x7f9fb01026e0 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb010c590 con 0x7f9fb0101d20 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb753e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb010c890 con 0x7f9fb0100b30 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38980/0 (socket says 172.21.15.28:38980) 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 -- 172.21.15.28:0/936045573 learned_addr learned my addr 172.21.15.28:0/936045573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb4ad9640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb0101d20 0x7f9fb019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb52da640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0100b30 0x7f9fb006ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 -- 172.21.15.28:0/936045573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0100b30 msgr2=0x7f9fb006ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0100b30 0x7f9fb006ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 -- 172.21.15.28:0/936045573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb0101d20 msgr2=0x7f9fb019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb0101d20 0x7f9fb019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 -- 172.21.15.28:0/936045573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb019fb70 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.020+0000 7f9fb5adb640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800eb10 tx=0x7f9fa800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.021+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8004b20 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.021+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb019fe60 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.021+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fa8004cc0 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.021+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8010760 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.021+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb01a0330 con 0x7f9fb01026e0 2024-03-22T02:11:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.022+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb0107ec0 con 0x7f9fb01026e0 2024-03-22T02:11:34.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.023+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9fa8010900 con 0x7f9fb01026e0 2024-03-22T02:11:34.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.023+0000 7f9fa67fc640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 0x7f9f9c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.024+0000 7f9fb52da640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 0x7f9f9c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.024+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f9fa80cfa00 con 0x7f9fb01026e0 2024-03-22T02:11:34.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.024+0000 7f9fb52da640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 0x7f9f9c0a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9fac000c00 tx=0x7f9fac005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:34.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.027+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fa8053740 con 0x7f9fb01026e0 2024-03-22T02:11:34.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.219+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f9fb002d3b0 con 0x7f9fb01026e0 2024-03-22T02:11:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: pgmap v286: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T02:11:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T02:11:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:34.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: pgmap v286: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:34.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:34.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: pgmap v286: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-22T02:11:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2119543255' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: osdmap e84: 12 total, 12 up, 12 in 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: osdmap e85: 12 total, 12 up, 12 in 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:11:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:11:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.629+0000 7f9fa67fc640 1 -- 172.21.15.28:0/936045573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) ==== 157+0+0 (secure 0 0 0) 0x7f9fa809c980 con 0x7f9fb01026e0 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 msgr2=0x7f9f9c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 0x7f9f9c0a53e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9fac000c00 tx=0x7f9fac005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 msgr2=0x7f9fb019f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800eb10 tx=0x7f9fa800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 shutdown_connections 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9f9c0a2f20 0x7f9f9c0a53e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fb01026e0 0x7f9fb019f2b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fb0101d20 0x7f9fb019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 --2- 172.21.15.28:0/936045573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fb0100b30 0x7f9fb006ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 >> 172.21.15.28:0/936045573 conn(0x7f9fb00fc0a0 msgr2=0x7f9fb0105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:34.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 shutdown_connections 2024-03-22T02:11:34.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.632+0000 7f9fb753e640 1 -- 172.21.15.28:0/936045573 wait complete. 2024-03-22T02:11:34.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 -- 172.21.15.28:0/548877163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 msgr2=0x7fbb50103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 --2- 172.21.15.28:0/548877163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50103210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009900 tx=0x7fbb4402bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 -- 172.21.15.28:0/548877163 shutdown_connections 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 --2- 172.21.15.28:0/548877163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50103750 0x7fbb50109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 --2- 172.21.15.28:0/548877163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50103210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 --2- 172.21.15.28:0/548877163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50101ba0 0x7fbb50101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 -- 172.21.15.28:0/548877163 >> 172.21.15.28:0/548877163 conn(0x7fbb500fd170 msgr2=0x7fbb500ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 -- 172.21.15.28:0/548877163 shutdown_connections 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.760+0000 7fbb56513640 1 -- 172.21.15.28:0/548877163 wait complete. 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 Processor -- start 2024-03-22T02:11:34.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 -- start start 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101ba0 0x7fbb50197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50103750 0x7fbb5019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010d800 con 0x7fbb50102d90 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010d680 con 0x7fbb50103750 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb56513640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5010d980 con 0x7fbb50101ba0 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38986/0 (socket says 172.21.15.28:38986) 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/4204834582 learned_addr learned my addr 172.21.15.28:0/4204834582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb54ab0640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50103750 0x7fbb5019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4ffff640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101ba0 0x7fbb50197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/4204834582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101ba0 msgr2=0x7fbb50197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101ba0 0x7fbb50197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/4204834582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50103750 msgr2=0x7fbb5019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50103750 0x7fbb5019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 -- 172.21.15.28:0/4204834582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5019fb20 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.761+0000 7fbb4f7fe640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009230 tx=0x7fbb44005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.762+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44041070 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.762+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5019fdb0 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.762+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb440044d0 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.762+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44034440 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.762+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb501a0240 con 0x7fbb50102d90 2024-03-22T02:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.763+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50108f40 con 0x7fbb50102d90 2024-03-22T02:11:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.764+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fbb44044070 con 0x7fbb50102d90 2024-03-22T02:11:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.764+0000 7fbb4d7fa640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 0x7fbb380a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.764+0000 7fbb4ffff640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 0x7fbb380a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.765+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8277+0+0 (secure 0 0 0) 0x7fbb440f1000 con 0x7fbb50102d90 2024-03-22T02:11:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.765+0000 7fbb4ffff640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 0x7fbb380a53e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb50102bf0 tx=0x7fbb40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.769+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb440bdea0 con 0x7fbb50102d90 2024-03-22T02:11:34.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:34.954+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbb5002d3b0 con 0x7fbb50102d90 2024-03-22T02:11:35.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T02:11:35.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T02:11:35.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T02:11:35.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T02:11:35.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T02:11:35.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:34 smithi120 ceph-mon[36803]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T02:11:35.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T02:11:35.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T02:11:35.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T02:11:35.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T02:11:35.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T02:11:35.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:34 smithi083 ceph-mon[36340]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T02:11:35.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2289712581' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-22T02:11:35.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-22T02:11:35.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-22T02:11:35.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-22T02:11:35.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/936045573' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-22T02:11:35.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:34 smithi028 ceph-mon[31916]: osdmap e86: 12 total, 12 up, 12 in 2024-03-22T02:11:36.161 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: pgmap v289: 97 pgs: 22 creating+peering, 35 active+clean, 40 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-22T02:11:36.161 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:36.161 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: Cluster is now healthy 2024-03-22T02:11:36.162 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:active 2024-03-22T02:11:36.162 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.162 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.162 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:35 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T02:11:36.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: pgmap v289: 97 pgs: 22 creating+peering, 35 active+clean, 40 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: Cluster is now healthy 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:active 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:35 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T02:11:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: pgmap v289: 97 pgs: 22 creating+peering, 35 active+clean, 40 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2024-03-22T02:11:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: Cluster is now healthy 2024-03-22T02:11:36.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] up:active 2024-03-22T02:11:36.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:36.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:35 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-22T02:11:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.915+0000 7fbb4d7fa640 1 -- 172.21.15.28:0/4204834582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) ==== 228+0+0 (secure 0 0 0) 0x7fbb440bd7a0 con 0x7fbb50102d90 2024-03-22T02:11:36.915 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-03-22T02:11:36.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 msgr2=0x7fbb380a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:36.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 0x7fbb380a53e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb50102bf0 tx=0x7fbb40005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:36.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 msgr2=0x7fbb50198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009230 tx=0x7fbb44005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 shutdown_connections 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb380a2f20 0x7fbb380a53e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb50103750 0x7fbb5019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb50102d90 0x7fbb50198440 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 --2- 172.21.15.28:0/4204834582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb50101ba0 0x7fbb50197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 >> 172.21.15.28:0/4204834582 conn(0x7fbb500fd170 msgr2=0x7fbb50106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.918+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 shutdown_connections 2024-03-22T02:11:36.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:36.919+0000 7fbb56513640 1 -- 172.21.15.28:0/4204834582 wait complete. 2024-03-22T02:11:36.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- 172.21.15.28:0/875895485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 msgr2=0x7fb00c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 --2- 172.21.15.28:0/875895485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c102120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faffc009900 tx=0x7faffc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- 172.21.15.28:0/875895485 shutdown_connections 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 --2- 172.21.15.28:0/875895485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 0x7fb00c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 --2- 172.21.15.28:0/875895485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c102120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 --2- 172.21.15.28:0/875895485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 0x7fb00c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- 172.21.15.28:0/875895485 >> 172.21.15.28:0/875895485 conn(0x7fb00c0fc040 msgr2=0x7fb00c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:37.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- 172.21.15.28:0/875895485 shutdown_connections 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- 172.21.15.28:0/875895485 wait complete. 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 Processor -- start 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.049+0000 7fb013ce0640 1 -- start start 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 0x7fb00c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 0x7fb00c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c10c730 con 0x7fb00c101ca0 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c10c5b0 con 0x7fb00c100ab0 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c10c8b0 con 0x7fb00c102660 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39006/0 (socket says 172.21.15.28:39006) 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 -- 172.21.15.28:0/2732618971 learned_addr learned my addr 172.21.15.28:0/2732618971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb011a7c640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 0x7fb00c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01227d640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 0x7fb00c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 -- 172.21.15.28:0/2732618971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 msgr2=0x7fb00c19f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 0x7fb00c19f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 -- 172.21.15.28:0/2732618971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 msgr2=0x7fb00c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 0x7fb00c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 -- 172.21.15.28:0/2732618971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c19fb10 con 0x7fb00c101ca0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb01127b640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faffc009230 tx=0x7faffc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc041070 con 0x7fb00c101ca0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faffc004450 con 0x7fb00c101ca0 2024-03-22T02:11:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00c19fda0 con 0x7fb00c101ca0 2024-03-22T02:11:37.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc003e20 con 0x7fb00c101ca0 2024-03-22T02:11:37.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.050+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00c1a0230 con 0x7fb00c101ca0 2024-03-22T02:11:37.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.051+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00c1a0460 con 0x7fb00c101ca0 2024-03-22T02:11:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.053+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7faffc044070 con 0x7fb00c101ca0 2024-03-22T02:11:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.053+0000 7fb002ffd640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 0x7faff80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.053+0000 7fb011a7c640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 0x7faff80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.054+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7faffc0f14c0 con 0x7fb00c101ca0 2024-03-22T02:11:37.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.054+0000 7fb011a7c640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 0x7faff80a53e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0080095f0 tx=0x7fb008005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:37.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.058+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faffc0bda80 con 0x7fb00c101ca0 2024-03-22T02:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.241+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fb00c1a0630 con 0x7fb00c101ca0 2024-03-22T02:11:37.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:36 smithi120 ceph-mon[36803]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T02:11:37.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:36 smithi083 ceph-mon[36340]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T02:11:37.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:36 smithi028 ceph-mon[31916]: osdmap e87: 12 total, 12 up, 12 in 2024-03-22T02:11:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.917+0000 7fb002ffd640 1 -- 172.21.15.28:0/2732618971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) ==== 169+0+0 (secure 0 0 0) 0x7faffc0c1b20 con 0x7fb00c101ca0 2024-03-22T02:11:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 msgr2=0x7faff80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 0x7faff80a53e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb0080095f0 tx=0x7fb008005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 msgr2=0x7fb00c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7faffc009230 tx=0x7faffc02c650 comp rx=0 tx=0).stop 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 shutdown_connections 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faff80a2f20 0x7faff80a53e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb00c102660 0x7fb00c19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb00c101ca0 0x7fb00c198430 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 --2- 172.21.15.28:0/2732618971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb00c100ab0 0x7fb00c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.920+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 >> 172.21.15.28:0/2732618971 conn(0x7fb00c0fc040 msgr2=0x7fb00c06a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.921+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 shutdown_connections 2024-03-22T02:11:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:37.921+0000 7fb013ce0640 1 -- 172.21.15.28:0/2732618971 wait complete. 2024-03-22T02:11:37.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 -- 172.21.15.28:0/22544538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940101ba0 msgr2=0x7f9940101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 --2- 172.21.15.28:0/22544538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940101ba0 0x7f9940101fc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f993c009900 tx=0x7f993c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 -- 172.21.15.28:0/22544538 shutdown_connections 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 --2- 172.21.15.28:0/22544538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940103750 0x7f9940109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 --2- 172.21.15.28:0/22544538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940102d90 0x7f9940103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 --2- 172.21.15.28:0/22544538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940101ba0 0x7f9940101fc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 -- 172.21.15.28:0/22544538 >> 172.21.15.28:0/22544538 conn(0x7f99400fd130 msgr2=0x7f99400ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 -- 172.21.15.28:0/22544538 shutdown_connections 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.055+0000 7f99484af640 1 -- 172.21.15.28:0/22544538 wait complete. 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 Processor -- start 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 -- start start 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940101ba0 0x7f9940197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 0x7f99401983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f994010d7f0 con 0x7f9940103750 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f994010d670 con 0x7f9940102d90 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f99484af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f994010d970 con 0x7f9940101ba0 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39026/0 (socket says 172.21.15.28:39026) 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9945a4a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 0x7f99401983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 -- 172.21.15.28:0/2932595602 learned_addr learned my addr 172.21.15.28:0/2932595602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f994624b640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940101ba0 0x7f9940197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 -- 172.21.15.28:0/2932595602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940101ba0 msgr2=0x7f9940197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940101ba0 0x7f9940197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 -- 172.21.15.28:0/2932595602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 msgr2=0x7f99401983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 0x7f99401983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 -- 172.21.15.28:0/2932595602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994019fab0 con 0x7f9940103750 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9945a4a640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 0x7f99401983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.056+0000 7f9946a4c640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f993800eb10 tx=0x7f993800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.057+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993800ce20 con 0x7f9940103750 2024-03-22T02:11:38.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.057+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9938005020 con 0x7f9940103750 2024-03-22T02:11:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.057+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994019fda0 con 0x7f9940103750 2024-03-22T02:11:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.057+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9938010690 con 0x7f9940103750 2024-03-22T02:11:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.057+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99401a0260 con 0x7f9940103750 2024-03-22T02:11:38.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.058+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9940108ec0 con 0x7f9940103750 2024-03-22T02:11:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.059+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9938005540 con 0x7f9940103750 2024-03-22T02:11:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.059+0000 7f99377fe640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 0x7f992c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.060+0000 7f994624b640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 0x7f992c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.060+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f99380cfd80 con 0x7f9940103750 2024-03-22T02:11:38.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.060+0000 7f994624b640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 0x7f992c0a5460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:38.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.063+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9938053740 con 0x7f9940103750 2024-03-22T02:11:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:37 smithi120 ceph-mon[36803]: pgmap v292: 97 pgs: 22 creating+peering, 62 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-03-22T02:11:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:37 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T02:11:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:37 smithi120 ceph-mon[36803]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T02:11:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:37 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T02:11:38.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:37 smithi083 ceph-mon[36340]: pgmap v292: 97 pgs: 22 creating+peering, 62 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-03-22T02:11:38.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:37 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T02:11:38.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:37 smithi083 ceph-mon[36340]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T02:11:38.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:37 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T02:11:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:38.286+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f994002d3b0 con 0x7f9940103750 2024-03-22T02:11:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:37 smithi028 ceph-mon[31916]: pgmap v292: 97 pgs: 22 creating+peering, 62 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 7 op/s 2024-03-22T02:11:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:37 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4204834582' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-22T02:11:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:37 smithi028 ceph-mon[31916]: osdmap e88: 12 total, 12 up, 12 in 2024-03-22T02:11:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:37 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-22T02:11:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.188+0000 7f99377fe640 1 -- 172.21.15.28:0/2932595602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f993809c930 con 0x7f9940103750 2024-03-22T02:11:39.189 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-03-22T02:11:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 msgr2=0x7f992c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 0x7f992c0a5460 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f993c000c00 tx=0x7f993c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 msgr2=0x7f994019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f993800eb10 tx=0x7f993800ef10 comp rx=0 tx=0).stop 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 shutdown_connections 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f992c0a2fa0 0x7f992c0a5460 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9940103750 0x7f994019f440 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9940102d90 0x7f99401983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 --2- 172.21.15.28:0/2932595602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9940101ba0 0x7f9940197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.191+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 >> 172.21.15.28:0/2932595602 conn(0x7f99400fd130 msgr2=0x7f9940106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.192+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 shutdown_connections 2024-03-22T02:11:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.192+0000 7f99484af640 1 -- 172.21.15.28:0/2932595602 wait complete. 2024-03-22T02:11:39.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:38 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T02:11:39.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:38 smithi028 ceph-mon[31916]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T02:11:39.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:38 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T02:11:39.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:39.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:38 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T02:11:39.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:38 smithi120 ceph-mon[36803]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T02:11:39.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:38 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T02:11:39.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:38 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2732618971' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-22T02:11:39.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:38 smithi083 ceph-mon[36340]: osdmap e89: 12 total, 12 up, 12 in 2024-03-22T02:11:39.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:38 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-22T02:11:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- 172.21.15.28:0/2724881106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 msgr2=0x7fdc1c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 --2- 172.21.15.28:0/2724881106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c100ed0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009900 tx=0x7fdc1002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- 172.21.15.28:0/2724881106 shutdown_connections 2024-03-22T02:11:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 --2- 172.21.15.28:0/2724881106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 0x7fdc1c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 --2- 172.21.15.28:0/2724881106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 0x7fdc1c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 --2- 172.21.15.28:0/2724881106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c100ed0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- 172.21.15.28:0/2724881106 >> 172.21.15.28:0/2724881106 conn(0x7fdc1c0fc020 msgr2=0x7fdc1c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- 172.21.15.28:0/2724881106 shutdown_connections 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- 172.21.15.28:0/2724881106 wait complete. 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 Processor -- start 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.314+0000 7fdc2317a640 1 -- start start 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 0x7fdc1c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 0x7fdc1c19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10d850 con 0x7fdc1c100ab0 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10d6d0 con 0x7fdc1c102660 2024-03-22T02:11:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c10d9d0 con 0x7fdc1c101ca0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39038/0 (socket says 172.21.15.28:39038) 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 -- 172.21.15.28:0/3764648869 learned_addr learned my addr 172.21.15.28:0/3764648869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc21717640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 0x7fdc1c19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc1bfff640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 0x7fdc1c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 -- 172.21.15.28:0/3764648869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 msgr2=0x7fdc1c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 0x7fdc1c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 -- 172.21.15.28:0/3764648869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 msgr2=0x7fdc1c19f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 0x7fdc1c19f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 -- 172.21.15.28:0/3764648869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c19fb30 con 0x7fdc1c100ab0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc20f16640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdc10000c00 tx=0x7fdc10005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10041070 con 0x7fdc1c100ab0 2024-03-22T02:11:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc100044d0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c19fdc0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.315+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10034440 con 0x7fdc1c100ab0 2024-03-22T02:11:39.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.316+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1a01f0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.317+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc1c107df0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.318+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fdc10044070 con 0x7fdc1c100ab0 2024-03-22T02:11:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.318+0000 7fdc19ffb640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 0x7fdc0c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.319+0000 7fdc1bfff640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 0x7fdc0c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.319+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fdc100f11d0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.319+0000 7fdc1bfff640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 0x7fdc0c0a53e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc08005fd0 tx=0x7fdc08005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.323+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdc100bddc0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.547+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc1c062860 con 0x7fdc1c100ab0 2024-03-22T02:11:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.547+0000 7fdc19ffb640 1 -- 172.21.15.28:0/3764648869 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7fdc100bd6c0 con 0x7fdc1c100ab0 2024-03-22T02:11:39.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:39.548 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:38.287916+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-22T02:11:39.548 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-03-22T02:11:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 msgr2=0x7fdc0c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 0x7fdc0c0a53e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdc08005fd0 tx=0x7fdc08005850 comp rx=0 tx=0).stop 2024-03-22T02:11:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 msgr2=0x7fdc1c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdc10000c00 tx=0x7fdc10005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 shutdown_connections 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc0c0a2f20 0x7fdc0c0a53e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc1c102660 0x7fdc1c19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc1c101ca0 0x7fdc1c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 --2- 172.21.15.28:0/3764648869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c100ab0 0x7fdc1c197f10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 >> 172.21.15.28:0/3764648869 conn(0x7fdc1c0fc020 msgr2=0x7fdc1c104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 shutdown_connections 2024-03-22T02:11:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.550+0000 7fdc2317a640 1 -- 172.21.15.28:0/3764648869 wait complete. 2024-03-22T02:11:39.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 -- 172.21.15.28:0/3024260526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4104fc0 msgr2=0x7f64f41073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 --2- 172.21.15.28:0/3024260526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4104fc0 0x7f64f41073b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009900 tx=0x7f64e802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 -- 172.21.15.28:0/3024260526 shutdown_connections 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 --2- 172.21.15.28:0/3024260526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 --2- 172.21.15.28:0/3024260526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4104fc0 0x7f64f41073b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 --2- 172.21.15.28:0/3024260526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f4102690 0x7f64f4104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 -- 172.21.15.28:0/3024260526 >> 172.21.15.28:0/3024260526 conn(0x7f64f40fc020 msgr2=0x7f64f40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 -- 172.21.15.28:0/3024260526 shutdown_connections 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.663+0000 7f64fc508640 1 -- 172.21.15.28:0/3024260526 wait complete. 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 Processor -- start 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 -- start start 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f4104fc0 0x7f64f41984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64f410d800 con 0x7f64f4102690 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f64f410d680 con 0x7f64f41078f0 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fc508640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f64f410d980 con 0x7f64f4104fc0 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39054/0 (socket says 172.21.15.28:39054) 2024-03-22T02:11:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 -- 172.21.15.28:0/256979308 learned_addr learned my addr 172.21.15.28:0/256979308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64faaa5640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 -- 172.21.15.28:0/256979308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f4104fc0 msgr2=0x7f64f41984b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f4104fc0 0x7f64f41984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 -- 172.21.15.28:0/256979308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 msgr2=0x7f64f419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.664+0000 7f64fa2a4640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64fa2a4640 1 -- 172.21.15.28:0/256979308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f419fb40 con 0x7f64f4102690 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64faaa5640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64fa2a4640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00d950 tx=0x7f64ec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec017070 con 0x7f64f4102690 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ec009c30 con 0x7f64f4102690 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00f6b0 con 0x7f64f4102690 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f419fe30 con 0x7f64f4102690 2024-03-22T02:11:39.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.665+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f41a02f0 con 0x7f64f4102690 2024-03-22T02:11:39.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.666+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f40fdb10 con 0x7f64f4102690 2024-03-22T02:11:39.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.668+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f64ec0109d0 con 0x7f64f4102690 2024-03-22T02:11:39.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.668+0000 7f64f37fe640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 0x7f64dc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.668+0000 7f64f9aa3640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 0x7f64dc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.668+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f64ec0cf260 con 0x7f64f4102690 2024-03-22T02:11:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.669+0000 7f64f9aa3640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 0x7f64dc0a53e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64e802c170 tx=0x7f64e8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.672+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f64ec009de0 con 0x7f64f4102690 2024-03-22T02:11:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.857+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64f402d620 con 0x7f64f4102690 2024-03-22T02:11:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.858+0000 7f64f37fe640 1 -- 172.21.15.28:0/256979308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f64ec09be50 con 0x7f64f4102690 2024-03-22T02:11:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:39.859 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 msgr2=0x7f64dc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 0x7f64dc0a53e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64e802c170 tx=0x7f64e8000bc0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 msgr2=0x7f64f4197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f64ec00d950 tx=0x7f64ec00de10 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 shutdown_connections 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64dc0a2f20 0x7f64dc0a53e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f64f41078f0 0x7f64f419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f64f4104fc0 0x7f64f41984b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 --2- 172.21.15.28:0/256979308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f4102690 0x7f64f4197f70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 >> 172.21.15.28:0/256979308 conn(0x7f64f40fc020 msgr2=0x7f64f4102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 shutdown_connections 2024-03-22T02:11:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.860+0000 7f64fc508640 1 -- 172.21.15.28:0/256979308 wait complete. 2024-03-22T02:11:39.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.976+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1390684892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4102670 msgr2=0x7f92e4108f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.976+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1390684892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4102670 0x7f92e4108f00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f92d4009ca0 tx=0x7f92d402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1390684892 shutdown_connections 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1390684892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4102670 0x7f92e4108f00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1390684892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4101cb0 0x7f92e4102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1390684892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 0x7f92e4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1390684892 >> 172.21.15.28:0/1390684892 conn(0x7f92e40fc070 msgr2=0x7f92e40fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1390684892 shutdown_connections 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1390684892 wait complete. 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 Processor -- start 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 -- start start 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 0x7f92e4197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.977+0000 7f92e8fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4102670 0x7f92e419f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410d820 con 0x7f92e4101cb0 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410d6a0 con 0x7f92e4102670 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f92e410d9a0 con 0x7f92e4100ac0 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39078/0 (socket says 172.21.15.28:39078) 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 -- 172.21.15.28:0/1196889606 learned_addr learned my addr 172.21.15.28:0/1196889606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e2d9d640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4102670 0x7f92e419f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e259c640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 0x7f92e4197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 -- 172.21.15.28:0/1196889606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 msgr2=0x7f92e4197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 0x7f92e4197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 -- 172.21.15.28:0/1196889606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4102670 msgr2=0x7f92e419f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4102670 0x7f92e419f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 -- 172.21.15.28:0/1196889606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e419f8b0 con 0x7f92e4101cb0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e1d9b640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00e9f0 tx=0x7f92cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc00cd80 con 0x7f92e4101cb0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e419fba0 con 0x7f92e4101cb0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92cc010910 con 0x7f92e4101cb0 2024-03-22T02:11:39.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92cc018990 con 0x7f92e4101cb0 2024-03-22T02:11:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.978+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e41a00f0 con 0x7f92e4101cb0 2024-03-22T02:11:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.979+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e4107d10 con 0x7f92e4101cb0 2024-03-22T02:11:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.981+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f92cc010ac0 con 0x7f92e4101cb0 2024-03-22T02:11:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.981+0000 7f92db7fe640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 0x7f92d00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.981+0000 7f92e259c640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 0x7f92d00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.981+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f92cc0cfcc0 con 0x7f92e4101cb0 2024-03-22T02:11:39.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.982+0000 7f92e259c640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 0x7f92d00a53e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f92e4198b40 tx=0x7f92dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:39.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:39.985+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f92cc09c260 con 0x7f92e4101cb0 2024-03-22T02:11:40.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.168+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92e4062830 con 0x7f92e4101cb0 2024-03-22T02:11:40.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.170+0000 7f92db7fe640 1 -- 172.21.15.28:0/1196889606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f92cc0a0300 con 0x7f92e4101cb0 2024-03-22T02:11:40.170 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:40.170 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:40.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 msgr2=0x7f92d00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 0x7f92d00a53e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f92e4198b40 tx=0x7f92dc00b410 comp rx=0 tx=0).stop 2024-03-22T02:11:40.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 msgr2=0x7f92e41981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f92cc00e9f0 tx=0x7f92cc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 shutdown_connections 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92d00a2f20 0x7f92d00a53e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92e4102670 0x7f92e419f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e4101cb0 0x7f92e41981b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 --2- 172.21.15.28:0/1196889606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92e4100ac0 0x7f92e4197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 >> 172.21.15.28:0/1196889606 conn(0x7f92e40fc070 msgr2=0x7f92e4104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 shutdown_connections 2024-03-22T02:11:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.172+0000 7f92e8fd6640 1 -- 172.21.15.28:0/1196889606 wait complete. 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: pgmap v295: 129 pgs: 10 creating+peering, 106 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:40.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3764648869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.183 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:39 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/256979308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: pgmap v295: 129 pgs: 10 creating+peering, 106 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-22T02:11:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T02:11:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T02:11:40.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:40.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3764648869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:39 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/256979308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.264 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: 2024-03-22T02:11:39.833+0000 7f8161474700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-22T02:11:40.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: pgmap v295: 129 pgs: 10 creating+peering, 106 active+clean, 13 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 5 op/s 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: osdmap e90: 12 total, 12 up, 12 in 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2932595602' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3764648869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:39 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/256979308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 -- 172.21.15.28:0/2002176290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820100b30 msgr2=0x7f2820100f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 --2- 172.21.15.28:0/2002176290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820100b30 0x7f2820100f50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f281c009900 tx=0x7f281c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 -- 172.21.15.28:0/2002176290 shutdown_connections 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 --2- 172.21.15.28:0/2002176290 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f28201026e0 0x7f2820108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 --2- 172.21.15.28:0/2002176290 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2820101d20 0x7f28201021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 --2- 172.21.15.28:0/2002176290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820100b30 0x7f2820100f50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 -- 172.21.15.28:0/2002176290 >> 172.21.15.28:0/2002176290 conn(0x7f28200fc0e0 msgr2=0x7f28200fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 -- 172.21.15.28:0/2002176290 shutdown_connections 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 -- 172.21.15.28:0/2002176290 wait complete. 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.301+0000 7f28274b5640 1 Processor -- start 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 -- start start 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 0x7f2820197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 0x7f282019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f282010c730 con 0x7f2820101d20 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f282010c5b0 con 0x7f2820100b30 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28274b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f282010c8b0 con 0x7f28201026e0 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39100/0 (socket says 172.21.15.28:39100) 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 -- 172.21.15.28:0/302050150 learned_addr learned my addr 172.21.15.28:0/302050150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2825251640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 0x7f2820197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2825a52640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 0x7f282019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 -- 172.21.15.28:0/302050150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 msgr2=0x7f282019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 0x7f282019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 -- 172.21.15.28:0/302050150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 msgr2=0x7f2820197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 0x7f2820197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 -- 172.21.15.28:0/302050150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282019fb10 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2825a52640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 0x7f282019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2825251640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 0x7f2820197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f2824a50640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f281000d770 tx=0x7f281000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.302+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810017070 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.303+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282019fe00 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.303+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2810005500 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.303+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810004e00 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.303+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28201a0270 con 0x7f2820101d20 2024-03-22T02:11:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.304+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28201a04a0 con 0x7f2820101d20 2024-03-22T02:11:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.305+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f2810004fa0 con 0x7f2820101d20 2024-03-22T02:11:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.306+0000 7f28167fc640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 0x7f280c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.306+0000 7f2825251640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 0x7f280c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.306+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f28100cf470 con 0x7f2820101d20 2024-03-22T02:11:40.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.306+0000 7f2825251640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 0x7f280c0a53e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.310+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f281009c000 con 0x7f2820101d20 2024-03-22T02:11:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.530+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28201a0670 con 0x7f2820101d20 2024-03-22T02:11:40.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.531+0000 7f28167fc640 1 -- 172.21.15.28:0/302050150 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11794 (secure 0 0 0) 0x7f281009b900 con 0x7f2820101d20 2024-03-22T02:11:40.531 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:40.532 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24466,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24478,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:38.287916+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24509},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24509,"qdb_cluster":[24509]},"id":1}]} 2024-03-22T02:11:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 17 2024-03-22T02:11:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 msgr2=0x7f280c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 0x7f280c0a53e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f281c000c00 tx=0x7f281c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 msgr2=0x7f2820198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f281000d770 tx=0x7f281000dc30 comp rx=0 tx=0).stop 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 shutdown_connections 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f280c0a2f20 0x7f280c0a53e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f28201026e0 0x7f282019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2820101d20 0x7f2820198430 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 --2- 172.21.15.28:0/302050150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2820100b30 0x7f2820197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 >> 172.21.15.28:0/302050150 conn(0x7f28200fc0e0 msgr2=0x7f282006a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 shutdown_connections 2024-03-22T02:11:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.533+0000 7f28274b5640 1 -- 172.21.15.28:0/302050150 wait complete. 2024-03-22T02:11:40.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 -- 172.21.15.28:0/1090860542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 msgr2=0x7fc484102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 --2- 172.21.15.28:0/1090860542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc484102120 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc478009900 tx=0x7fc47802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 -- 172.21.15.28:0/1090860542 shutdown_connections 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 --2- 172.21.15.28:0/1090860542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc484108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 --2- 172.21.15.28:0/1090860542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc484102120 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 --2- 172.21.15.28:0/1090860542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 -- 172.21.15.28:0/1090860542 >> 172.21.15.28:0/1090860542 conn(0x7fc4840fc020 msgr2=0x7fc4840fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 -- 172.21.15.28:0/1090860542 shutdown_connections 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.648+0000 7fc489919640 1 -- 172.21.15.28:0/1090860542 wait complete. 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 Processor -- start 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 -- start start 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc48419f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc48410d850 con 0x7fc484101ca0 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc48410d6d0 con 0x7fc484102660 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc489919640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc48410d9d0 con 0x7fc484100ab0 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39122/0 (socket says 172.21.15.28:39122) 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 -- 172.21.15.28:0/1662424308 learned_addr learned my addr 172.21.15.28:0/1662424308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4837fe640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc48419f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc482ffd640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 -- 172.21.15.28:0/1662424308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 msgr2=0x7fc484197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 -- 172.21.15.28:0/1662424308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 msgr2=0x7fc48419f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc48419f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4827fc640 1 -- 172.21.15.28:0/1662424308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc48419fa90 con 0x7fc484101ca0 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc4837fe640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc48419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.649+0000 7fc482ffd640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc4827fc640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc478006010 tx=0x7fc478005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478041070 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc48419fd20 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4780054e0 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc478034440 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.650+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4841a01b0 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.651+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc484107d70 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.652+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc478044070 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.652+0000 7fc488917640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 0x7fc46c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.652+0000 7fc482ffd640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 0x7fc46c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.652+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc4780f1550 con 0x7fc484101ca0 2024-03-22T02:11:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.653+0000 7fc482ffd640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 0x7fc46c0a53e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc484101b00 tx=0x7fc47c007870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.656+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc4780be140 con 0x7fc484101ca0 2024-03-22T02:11:40.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.839+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc48402d580 con 0x7fc484101ca0 2024-03-22T02:11:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.841+0000 7fc488917640 1 -- 172.21.15.28:0/1662424308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7fc4780bda40 con 0x7fc484101ca0 2024-03-22T02:11:40.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:40.842 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 msgr2=0x7fc46c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 0x7fc46c0a53e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc484101b00 tx=0x7fc47c007870 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 msgr2=0x7fc4841983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc478006010 tx=0x7fc478005dd0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 shutdown_connections 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc46c0a2f20 0x7fc46c0a53e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc484102660 0x7fc48419f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc484101ca0 0x7fc4841983b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 --2- 172.21.15.28:0/1662424308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc484100ab0 0x7fc484197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 >> 172.21.15.28:0/1662424308 conn(0x7fc4840fc020 msgr2=0x7fc484104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 shutdown_connections 2024-03-22T02:11:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.843+0000 7fc489919640 1 -- 172.21.15.28:0/1662424308 wait complete. 2024-03-22T02:11:40.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-22T02:11:33.259569+0000', 'modified': '2024-03-22T02:11:38.287916+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24509}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6839/2818900206', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 2818900206}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 2818900206}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24509, 'qdb_cluster': [24509]} 2024-03-22T02:11:40.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-22T02:11:40.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 -- 172.21.15.28:0/1252817142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 msgr2=0x7fad00103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 --2- 172.21.15.28:0/1252817142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00103220 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7face8009900 tx=0x7face802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 -- 172.21.15.28:0/1252817142 shutdown_connections 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 --2- 172.21.15.28:0/1252817142 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 0x7fad00109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 --2- 172.21.15.28:0/1252817142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00103220 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 --2- 172.21.15.28:0/1252817142 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 0x7fad00101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 -- 172.21.15.28:0/1252817142 >> 172.21.15.28:0/1252817142 conn(0x7fad000fd120 msgr2=0x7fad000ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 -- 172.21.15.28:0/1252817142 shutdown_connections 2024-03-22T02:11:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.963+0000 7fad050cf640 1 -- 172.21.15.28:0/1252817142 wait complete. 2024-03-22T02:11:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 Processor -- start 2024-03-22T02:11:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 -- start start 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 0x7fad00197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 0x7fad0019f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad0010d810 con 0x7fad00102da0 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fad0010d690 con 0x7fad00103760 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7fad050cf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad0010d990 con 0x7fad00101bb0 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7facfe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7facfe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39146/0 (socket says 172.21.15.28:39146) 2024-03-22T02:11:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.964+0000 7facfe59c640 1 -- 172.21.15.28:0/2562339022 learned_addr learned my addr 172.21.15.28:0/2562339022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facff59e640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 0x7fad0019f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfed9d640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 0x7fad00197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 -- 172.21.15.28:0/2562339022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 msgr2=0x7fad00197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 0x7fad00197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 -- 172.21.15.28:0/2562339022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 msgr2=0x7fad0019f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 0x7fad0019f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 -- 172.21.15.28:0/2562339022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0019fa40 con 0x7fad00102da0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facfe59c640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7face8009230 tx=0x7face8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facf7fff640 1 -- 172.21.15.28:0/2562339022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8041070 con 0x7fad00102da0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0019fcd0 con 0x7fad00102da0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facf7fff640 1 -- 172.21.15.28:0/2562339022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7face80044d0 con 0x7fad00102da0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7facf7fff640 1 -- 172.21.15.28:0/2562339022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face8034440 con 0x7fad00102da0 2024-03-22T02:11:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.965+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad001a01c0 con 0x7fad00102da0 2024-03-22T02:11:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.967+0000 7facf7fff640 1 -- 172.21.15.28:0/2562339022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7face8044070 con 0x7fad00102da0 2024-03-22T02:11:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 Processor -- start 2024-03-22T02:11:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 -- start start 2024-03-22T02:11:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd8e7e0 0x558aacd90c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facf7fff640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 0x7facec0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aacced210 con 0x558aacd89670 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aacc95c80 con 0x558aacd88c80 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7fad07d8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aacce91c0 con 0x558aacd8e7e0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facfed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facff59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facfed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39162/0 (socket says 172.21.15.28:39162) 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facfed9d640 1 -- 172.21.15.28:0/4098016432 learned_addr learned my addr 172.21.15.28:0/4098016432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facff59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43454/0 (socket says 172.21.15.28:43454) 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facf7fff640 1 -- 172.21.15.28:0/2562339022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7face80f1210 con 0x7fad00102da0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facfe59c640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 0x7facec0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.968+0000 7facff59e640 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd8e7e0 0x558aacd90c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 -- 172.21.15.28:0/4098016432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd8e7e0 msgr2=0x558aacd90c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd8e7e0 0x558aacd90c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 -- 172.21.15.28:0/4098016432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 msgr2=0x558aacd890a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 -- 172.21.15.28:0/4098016432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aaca012d0 con 0x558aacd89670 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facff59e640 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfe59c640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 0x7facec0a53e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7face80c5c80 tx=0x7face80a8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facfed9d640 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7facf800cc40 tx=0x7facf800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948c15401f60486a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facf57fa640 1 -- 172.21.15.28:0/4098016432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf8014070 con 0x558aacd89670 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facf57fa640 1 -- 172.21.15.28:0/4098016432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facf8005500 con 0x558aacd89670 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.969+0000 7facf57fa640 1 -- 172.21.15.28:0/4098016432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf8004c60 con 0x558aacd89670 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- 172.21.15.28:0/4098016432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 msgr2=0x558aacd8e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7facf800cc40 tx=0x7facf800e720 comp rx=0 tx=0).stop 2024-03-22T02:11:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- 172.21.15.28:0/4098016432 shutdown_connections 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd8e7e0 0x558aacd90c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd89670 0x558aacd8e170 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- 172.21.15.28:0/4098016432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd890a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- 172.21.15.28:0/4098016432 >> 172.21.15.28:0/4098016432 conn(0x558aacd841f0 msgr2=0x558aacd845f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- 172.21.15.28:0/4098016432 shutdown_connections 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- 172.21.15.28:0/4098016432 wait complete. 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 -- start start 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 Processor -- start 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd88c80 0x558aacd92e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.970+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 0x558aacd933a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd9a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aacd91790 con 0x558aacd8e7e0 2024-03-22T02:11:40.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aacca8f80 con 0x558aacd89670 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aacd91910 con 0x558aacd88c80 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 10 client.0 _ll_register_callbacks cb 0x558aacce54f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facfed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 0x558aacd933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd9a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facfed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 0x558aacd933a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43456/0 (socket says 172.21.15.28:43456) 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facfed9d640 1 -- 172.21.15.28:0/462236577 learned_addr learned my addr 172.21.15.28:0/462236577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd88c80 0x558aacd92e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 -- 172.21.15.28:0/462236577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd88c80 msgr2=0x558aacd92e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd88c80 0x558aacd92e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 -- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 msgr2=0x558aacd933a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 0x558aacd933a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 -- 172.21.15.28:0/462236577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aacacb6b0 con 0x558aacd8e7e0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7facff59e640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd9a3e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7facf0005fd0 tx=0x7facf0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c994be4f626e0a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7face67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf001a040 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7facf0004d20 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.971+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facf8005500 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.972+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7facf002c050 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.972+0000 7fad07d8a740 10 client.14763 fetch_fsmap learned FSMap version 17 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.972+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558aacd9d7c0 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.972+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facf8005500 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.972+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7facf000e480 con 0x558aacd8e7e0 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 10 client.14763 fetch_fsmap finished waiting for FSMap version 17 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 10 client.14763 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 20 client.14763 populate_metadata read hostname 'smithi028' 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aacda0c80 0x558aacda05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 4 client.14763 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aaccfa750 con 0x558aacda0c80 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7facfed9d640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aacda0c80 0x558aacda05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7facfed9d640 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aacda0c80 0x558aacda05b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99a7b0d5b9f90904 server_cookie=6e08df47d8d41bed in_seq=0 out_seq=0 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.973+0000 7face67fc640 10 client.14763 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7face67fc640 1 -- 172.21.15.28:0/462236577 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558aaccfa750 con 0x558aacda0c80 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7face67fc640 10 client.14763 handle_command_reply: tid=0 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 client.14763 shutdown 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 msgr2=0x558aacd9a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd9a3e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7facf0005fd0 tx=0x7facf0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 shutdown_connections 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aacda0c80 0x558aacda05b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd9a3e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd89670 0x558aacd933a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7face67fc640 0 client.14763 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 --2- 172.21.15.28:0/462236577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd88c80 0x558aacd92e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 >> 172.21.15.28:0/462236577 conn(0x558aacd841f0 msgr2=0x558aacd834f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 shutdown_connections 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 1 -- 172.21.15.28:0/462236577 wait complete. 2024-03-22T02:11:40.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.974+0000 7fad07d8a740 20 client.14763 trim_cache size 0 max 16384 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.992+0000 7fad07d8a740 1 Processor -- start 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.992+0000 7fad07d8a740 1 -- start start 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd92130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aacd7f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aacd9f690 con 0x558aacd8e7e0 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aacd9fd60 con 0x558aacd88c80 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7fad07d8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aacda0430 con 0x558aacd89670 2024-03-22T02:11:40.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39188/0 (socket says 172.21.15.28:39188) 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 -- 172.21.15.28:0/2396462419 learned_addr learned my addr 172.21.15.28:0/2396462419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd92130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facfed9d640 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aacd7f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 -- 172.21.15.28:0/2396462419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 msgr2=0x558aacd7f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aacd7f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 -- 172.21.15.28:0/2396462419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 msgr2=0x558aacd92130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd92130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 -- 172.21.15.28:0/2396462419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aacd89a50 con 0x558aacd8e7e0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facff59e640 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7facf0012d60 tx=0x7facf0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=798a37935c66500d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facf4ff9640 1 -- 172.21.15.28:0/2396462419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0036070 con 0x558aacd8e7e0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facf4ff9640 1 -- 172.21.15.28:0/2396462419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facf000e680 con 0x558aacd8e7e0 2024-03-22T02:11:41.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.993+0000 7facf4ff9640 1 -- 172.21.15.28:0/2396462419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0021990 con 0x558aacd8e7e0 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- 172.21.15.28:0/2396462419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 msgr2=0x558aacd83370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7facf0012d60 tx=0x7facf0004a40 comp rx=0 tx=0).stop 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- 172.21.15.28:0/2396462419 shutdown_connections 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aacd83370 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aacd7f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 --2- 172.21.15.28:0/2396462419 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aacd92130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- 172.21.15.28:0/2396462419 >> 172.21.15.28:0/2396462419 conn(0x558aacd841f0 msgr2=0x558aacd82c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- 172.21.15.28:0/2396462419 shutdown_connections 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- 172.21.15.28:0/2396462419 wait complete. 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 -- start start 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.994+0000 7fad07d8a740 1 Processor -- start 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aace01df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:41.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aace02330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aacda1170 con 0x558aacd8e7e0 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aacda0b00 con 0x558aacd88c80 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aacda1810 con 0x558aacd89670 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 10 client.0 _ll_register_callbacks cb 0x7facf0012850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39200/0 (socket says 172.21.15.28:39200) 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 -- 172.21.15.28:0/1558156320 learned_addr learned my addr 172.21.15.28:0/1558156320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aace01df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facfed9d640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aace02330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 -- 172.21.15.28:0/1558156320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 msgr2=0x558aace02330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:41.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aace02330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 -- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 msgr2=0x558aace01df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aace01df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 -- 172.21.15.28:0/1558156320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aace097f0 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facfed9d640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aace02330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7facff59e640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7facf0009b50 tx=0x7facf0021450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a952dcdc58fb2677 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7face67fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf0036030 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facf000e680 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.995+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7facf000ed10 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7facf0021b50 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 10 client.14769 fetch_fsmap learned FSMap version 17 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x558aace0ab20 con 0x558aacd8e7e0 2024-03-22T02:11:41.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facf000e680 con 0x558aacd8e7e0 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7facf00315d0 con 0x558aacd8e7e0 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 10 client.14769 fetch_fsmap finished waiting for FSMap version 17 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 10 client.14769 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 20 client.14769 populate_metadata read hostname 'smithi028' 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aace0f320 0x558aace0e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 4 client.14769 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "status"}) -- 0x558aaccf4aa0 con 0x558aace0f320 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7facfed9d640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aace0f320 0x558aace0e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7facfed9d640 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aace0f320 0x558aace0e030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d98bcc376becde2e server_cookie=fa01991395cf0cb7 in_seq=0 out_seq=0 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.996+0000 7face67fc640 10 client.14769 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7face67fc640 1 -- 172.21.15.28:0/1558156320 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x558aaccf4aa0 con 0x558aace0f320 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7face67fc640 10 client.14769 handle_command_reply: tid=0 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 client.14769 shutdown 2024-03-22T02:11:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 msgr2=0x558aace091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7facf0009b50 tx=0x7facf0021450 comp rx=0 tx=0).stop 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 shutdown_connections 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aace0f320 0x558aace0e030 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aacd8e7e0 0x558aace091e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aacd89670 0x558aace02330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7face67fc640 0 client.14769 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 --2- 172.21.15.28:0/1558156320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aacd88c80 0x558aace01df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 >> 172.21.15.28:0/1558156320 conn(0x558aacd841f0 msgr2=0x558aacd834f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 shutdown_connections 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 1 -- 172.21.15.28:0/1558156320 wait complete. 2024-03-22T02:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad07d8a740 20 client.14769 trim_cache size 0 max 16384 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 7.1018909240000001, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 17, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:11:41.006 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.31753992 2024-03-22T02:11:41.007 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 msgr2=0x7facec0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 0x7facec0a53e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7face80c5c80 tx=0x7face80a8040 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 msgr2=0x7fad00198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7face8009230 tx=0x7face8005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 shutdown_connections 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7facec0a2f20 0x7facec0a53e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad00103760 0x7fad0019f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad00102da0 0x7fad00198360 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 --2- 172.21.15.28:0/2562339022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad00101bb0 0x7fad00197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.997+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 >> 172.21.15.28:0/2562339022 conn(0x7fad000fd120 msgr2=0x7fad001060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.998+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 shutdown_connections 2024-03-22T02:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:40.998+0000 7fad050cf640 1 -- 172.21.15.28:0/2562339022 wait complete. 2024-03-22T02:11:41.009 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T02:11:41.009 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:41.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T02:11:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1196889606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/302050150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1662424308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.250 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:11:41.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1196889606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/302050150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1662424308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1196889606' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/302050150' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:41.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1662424308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:41 smithi120 ceph-mon[36803]: pgmap v297: 129 pgs: 6 creating+peering, 121 active+clean, 2 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-22T02:11:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:41 smithi120 ceph-mon[36803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:41 smithi120 ceph-mon[36803]: Cluster is now healthy 2024-03-22T02:11:42.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:41 smithi083 ceph-mon[36340]: pgmap v297: 129 pgs: 6 creating+peering, 121 active+clean, 2 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-22T02:11:42.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:41 smithi083 ceph-mon[36340]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:42.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:41 smithi083 ceph-mon[36340]: Cluster is now healthy 2024-03-22T02:11:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:41 smithi028 ceph-mon[31916]: pgmap v297: 129 pgs: 6 creating+peering, 121 active+clean, 2 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 3 op/s 2024-03-22T02:11:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:41 smithi028 ceph-mon[31916]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-22T02:11:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:41 smithi028 ceph-mon[31916]: Cluster is now healthy 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 -- 172.21.15.28:0/1070057781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 msgr2=0x7f92cc1078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 --2- 172.21.15.28:0/1070057781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc1078d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92c800b0d0 tx=0x7f92c800b3e0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 -- 172.21.15.28:0/1070057781 shutdown_connections 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 --2- 172.21.15.28:0/1070057781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc1078d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 --2- 172.21.15.28:0/1070057781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc10f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 --2- 172.21.15.28:0/1070057781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 0x7f92cc102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 -- 172.21.15.28:0/1070057781 >> 172.21.15.28:0/1070057781 conn(0x7f92cc0797b0 msgr2=0x7f92cc079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 -- 172.21.15.28:0/1070057781 shutdown_connections 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.019+0000 7f92d3662700 1 -- 172.21.15.28:0/1070057781 wait complete. 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 Processor -- start 2024-03-22T02:11:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 -- start start 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 0x7f92cc198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc112960 con 0x7f92cc1074f0 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc1127e0 con 0x7f92cc103090 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.020+0000 7f92d3662700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f92cc112ae0 con 0x7f92cc1026d0 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d0bfd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56534/0 (socket says 172.21.15.28:56534) 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d0bfd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43648/0 (socket says 172.21.15.28:43648) 2024-03-22T02:11:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 -- 172.21.15.28:0/3964631427 learned_addr learned my addr 172.21.15.28:0/3964631427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d13fe700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 0x7f92cc198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 -- 172.21.15.28:0/3964631427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 msgr2=0x7f92cc198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 0x7f92cc198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 -- 172.21.15.28:0/3964631427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 msgr2=0x7f92cc199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 -- 172.21.15.28:0/3964631427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92cc19de10 con 0x7f92cc1074f0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d0bfd700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d1bff700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f92c400e940 tx=0x7f92c400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c400cbd0 con 0x7f92cc1074f0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92c400ee90 con 0x7f92cc1074f0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92cc19e100 con 0x7f92cc1074f0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92c40056c0 con 0x7f92cc1074f0 2024-03-22T02:11:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.021+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92cc1a5980 con 0x7f92cc1074f0 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.022+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92cc069340 con 0x7f92cc1074f0 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.023+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f92c4004750 con 0x7f92cc1074f0 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.024+0000 7f92c27fc700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 0x7f92b80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.024+0000 7f92d13fe700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 0x7f92b80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.024+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f92c40cfad0 con 0x7f92cc1074f0 2024-03-22T02:11:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.025+0000 7f92d13fe700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 0x7f92b80a5470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009fd0 tx=0x7f92c800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.028+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f92c4053740 con 0x7f92cc1074f0 2024-03-22T02:11:43.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.283+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f92cc02dd70 con 0x7f92cc1074f0 2024-03-22T02:11:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.283+0000 7f92c27fc700 1 -- 172.21.15.28:0/3964631427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7f92c409c660 con 0x7f92cc1074f0 2024-03-22T02:11:43.284 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-22T02:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 msgr2=0x7f92b80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 0x7f92b80a5470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f92c8009fd0 tx=0x7f92c800b040 comp rx=0 tx=0).stop 2024-03-22T02:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 msgr2=0x7f92cc19d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f92c400e940 tx=0x7f92c400ec50 comp rx=0 tx=0).stop 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 shutdown_connections 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92b80a2fb0 0x7f92b80a5470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92cc1074f0 0x7f92cc19d6d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92cc1026d0 0x7f92cc198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 --2- 172.21.15.28:0/3964631427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f92cc103090 0x7f92cc199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 >> 172.21.15.28:0/3964631427 conn(0x7f92cc0797b0 msgr2=0x7f92cc1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 shutdown_connections 2024-03-22T02:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:43.286+0000 7f92d3662700 1 -- 172.21.15.28:0/3964631427 wait complete. 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-22T02:11:43.969 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T02:11:43.970 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T02:11:43.970 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T02:11:43.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:43.970 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-22T02:11:43.986 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T02:11:43.986 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T02:11:43.986 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-22T02:11:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42204sec preferred_lft 42204sec 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-22T02:11:43.988 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-22T02:11:43.988 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T02:11:43.989 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:11:44.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:11:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:43 smithi028 ceph-mon[31916]: pgmap v298: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-22T02:11:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:43 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3964631427' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:11:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:44.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:44.241 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T02:11:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:43 smithi120 ceph-mon[36803]: pgmap v298: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-22T02:11:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:43 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3964631427' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:11:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:43 smithi083 ceph-mon[36340]: pgmap v298: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 2 op/s 2024-03-22T02:11:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:43 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3964631427' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:11:44.267 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-22T02:11:44.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:44.270 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-22T02:11:44.336 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-22T02:11:44.336 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T02:11:44.336 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-22T02:11:44.336 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-22T02:11:44.336 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-22T02:11:44.337 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:11:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:11:44.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:44.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:44.658 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-22T02:11:44.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:44.675 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-03-22T02:11:44.729 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:11:44.730 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:11:44.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:11:44.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-22T02:11:44.730 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:11:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:11:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:44.852 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-22T02:11:44.853 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:11:44.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:11:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:44.983 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:11:44.983 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:11:44.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-22T02:11:44.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T02:11:44.983 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:11:45.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:11:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:45.065 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-22T02:11:45.065 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:11:45.065 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:45.079 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-22T02:11:45.080 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:11:45.080 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:45.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-22T02:11:45.200 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:11:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-22T02:11:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-22T02:11:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-22T02:11:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-22T02:11:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-22T02:11:45.257 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6f1caabab526e71b3fc428fbf2e847f95f3dde5730d40da67115880d895825dc/merged 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/1f601a3aed248d10090c89074447595338ece00a44501be8a770af69586f3022/merged 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-03-22T02:11:45.258 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/ff6409fabf3eeadb514e0809d8400c496eca09218d4d3b24e677d8f831018895/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4ea0d29f2e044dec497e550077e3b231af55636157d5ced8515d1ca2ff367335/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/1f2206dcdf6d05f030c4ff98ba6dba843111cbf44d62cdf36d9ce698f716a756/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c9be67f14815f1f4be29ea762661ed804e97dd9a18c9863fd239c620806ec518/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/604bf38168f6f834924c53f9d4dde71c47f0d02c49e7801d1361483c1acaeb5f/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5d860371553f85407d63138275a687de7b40963672ce6a3f408ade69697d3f59/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c24b8adf61dca058702e3e89092f493bc7f390836c9680ed40b4d0be41b4415c/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0f299028b0672383609856c5a8350976bc903784a105eccfd538c7cf75d03dab/merged 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:11:45.259 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:11:45.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:45.260 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-22T02:11:45.312 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-22T02:11:45.312 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-22T02:11:45.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-03-22T02:11:45.381 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-03-22T02:11:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-03-22T02:11:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-03-22T02:11:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-03-22T02:11:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-03-22T02:11:45.436 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-03-22T02:11:45.437 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/6f1caabab526e71b3fc428fbf2e847f95f3dde5730d40da67115880d895825dc/merged 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/1f601a3aed248d10090c89074447595338ece00a44501be8a770af69586f3022/merged 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/0 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-03-22T02:11:45.438 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/ff6409fabf3eeadb514e0809d8400c496eca09218d4d3b24e677d8f831018895/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/4ea0d29f2e044dec497e550077e3b231af55636157d5ced8515d1ca2ff367335/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/1f2206dcdf6d05f030c4ff98ba6dba843111cbf44d62cdf36d9ce698f716a756/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c9be67f14815f1f4be29ea762661ed804e97dd9a18c9863fd239c620806ec518/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/604bf38168f6f834924c53f9d4dde71c47f0d02c49e7801d1361483c1acaeb5f/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/5d860371553f85407d63138275a687de7b40963672ce6a3f408ade69697d3f59/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/c24b8adf61dca058702e3e89092f493bc7f390836c9680ed40b4d0be41b4415c/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/containers/storage/overlay/0f299028b0672383609856c5a8350976bc903784a105eccfd538c7cf75d03dab/merged 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:11:45.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:45.440 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-03-22T02:11:45.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-03-22T02:11:45.445+0000 7f1d61e4b480 -1 init, newargv = 0x55e2a9cbd7e0 newargc=15 2024-03-22T02:11:45.445 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[76447]: starting ceph client 2024-03-22T02:11:45.451 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[76447]: starting fuse 2024-03-22T02:11:45.460 INFO:teuthology.orchestra.run.smithi028.stdout:119 2024-03-22T02:11:45.460 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-03-22T02:11:45.461 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-03-22T02:11:45.462 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-03-22T02:11:45.583 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.76447.asok 2024-03-22T02:11:45.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:45.593 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76447.asok 2024-03-22T02:11:45.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:45.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76447.asok status 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e", 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-2344-gc3a04c0e (c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e) squid (dev)", 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-22T02:11:45.731 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "76447", 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 14781, 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 14781 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-22T02:11:45.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1042624623 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1042624623 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.14781 192.168.0.1:0/1042624623", 2024-03-22T02:11:45.733 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/1042624623", 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 17, 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 90, 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-03-22T02:11:45.734 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:11:45.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:45.739 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-03-22T02:11:45.739 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:45.754 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-03-22T02:11:45.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:45.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:45.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:45.822 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-22T02:11:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T02:11:45.938 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-03-22T02:11:45.939 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-22T02:11:45.939 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:45.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:11:45.945 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:11:45.946 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:11:45.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:46.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:45 smithi028 ceph-mon[31916]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1195819967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 msgr2=0x7f4d441021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1195819967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d441021a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009900 tx=0x7f4d3802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1195819967 shutdown_connections 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1195819967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d441026e0 0x7f4d44108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1195819967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d441021a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1195819967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d44100b30 0x7f4d44100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1195819967 >> 172.21.15.28:0/1195819967 conn(0x7f4d440fc0e0 msgr2=0x7f4d440fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1195819967 shutdown_connections 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.103+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1195819967 wait complete. 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 Processor -- start 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 -- start start 2024-03-22T02:11:46.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d44100b30 0x7f4d44197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 0x7f4d4419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c790 con 0x7f4d44101d20 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c610 con 0x7f4d44100b30 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d4b1d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c910 con 0x7f4d441026e0 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56542/0 (socket says 172.21.15.28:56542) 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 -- 172.21.15.28:0/1722625573 learned_addr learned my addr 172.21.15.28:0/1722625573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d48f73640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d44100b30 0x7f4d44197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d49774640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 0x7f4d4419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 -- 172.21.15.28:0/1722625573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 msgr2=0x7f4d4419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 0x7f4d4419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 -- 172.21.15.28:0/1722625573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d44100b30 msgr2=0x7f4d44197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d44100b30 0x7f4d44197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d43fff640 1 -- 172.21.15.28:0/1722625573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4419fb40 con 0x7f4d44101d20 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.104+0000 7f4d49774640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 0x7f4d4419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:46.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d43fff640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009230 tx=0x7f4d38005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38041070 con 0x7f4d44101d20 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4419fdd0 con 0x7f4d44101d20 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d380044d0 con 0x7f4d44101d20 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38034440 con 0x7f4d44101d20 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.105+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d441a0280 con 0x7f4d44101d20 2024-03-22T02:11:46.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.106+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d441a04b0 con 0x7f4d44101d20 2024-03-22T02:11:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.107+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4d38044070 con 0x7f4d44101d20 2024-03-22T02:11:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.108+0000 7f4d41ffb640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 0x7f4d2c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.108+0000 7f4d48f73640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 0x7f4d2c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.108+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4d380f15f0 con 0x7f4d44101d20 2024-03-22T02:11:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.108+0000 7f4d48f73640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 0x7f4d2c0a53e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4d44101b80 tx=0x7f4d3c00a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.112+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d380be180 con 0x7f4d44101d20 2024-03-22T02:11:46.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:45 smithi120 ceph-mon[36803]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:46.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:45 smithi083 ceph-mon[36340]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.296+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d441a0840 con 0x7f4d44101d20 2024-03-22T02:11:46.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.298+0000 7f4d41ffb640 1 -- 172.21.15.28:0/1722625573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f4d380bda80 con 0x7f4d44101d20 2024-03-22T02:11:46.298 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:46.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 msgr2=0x7f4d2c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 0x7f4d2c0a53e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4d44101b80 tx=0x7f4d3c00a590 comp rx=0 tx=0).stop 2024-03-22T02:11:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 msgr2=0x7f4d44198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009230 tx=0x7f4d38005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 shutdown_connections 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d2c0a2f20 0x7f4d2c0a53e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d441026e0 0x7f4d4419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d44101d20 0x7f4d44198460 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 --2- 172.21.15.28:0/1722625573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d44100b30 0x7f4d44197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 >> 172.21.15.28:0/1722625573 conn(0x7f4d440fc0e0 msgr2=0x7f4d440fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 shutdown_connections 2024-03-22T02:11:46.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.300+0000 7f4d4b1d7640 1 -- 172.21.15.28:0/1722625573 wait complete. 2024-03-22T02:11:46.311 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:11:46.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:11:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 -- 172.21.15.28:0/4195911741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 msgr2=0x7f77c4102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4195911741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4102110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f77b8009860 tx=0x7f77b802bb70 comp rx=0 tx=0).stop 2024-03-22T02:11:46.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 -- 172.21.15.28:0/4195911741 shutdown_connections 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4195911741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c4108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4195911741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4102110 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4195911741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 0x7f77c4100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 -- 172.21.15.28:0/4195911741 >> 172.21.15.28:0/4195911741 conn(0x7f77c40fc030 msgr2=0x7f77c40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 -- 172.21.15.28:0/4195911741 shutdown_connections 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 -- 172.21.15.28:0/4195911741 wait complete. 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.417+0000 7f77ca02a640 1 Processor -- start 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 -- start start 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 0x7f77c4196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77c410c700 con 0x7f77c4102650 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f77c410c580 con 0x7f77c4101c90 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77ca02a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f77c410c880 con 0x7f77c4100aa0 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56548/0 (socket says 172.21.15.28:56548) 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 -- 172.21.15.28:0/4161407823 learned_addr learned my addr 172.21.15.28:0/4161407823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c37fe640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 0x7f77c4196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c2ffd640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 -- 172.21.15.28:0/4161407823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 msgr2=0x7f77c4196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 0x7f77c4196e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 -- 172.21.15.28:0/4161407823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 msgr2=0x7f77c4197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 -- 172.21.15.28:0/4161407823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c419ea20 con 0x7f77c4102650 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c3fff640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00efc0 tx=0x7f77bc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.418+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc003970 con 0x7f77c4102650 2024-03-22T02:11:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.419+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c419ed10 con 0x7f77c4102650 2024-03-22T02:11:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.419+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77bc004020 con 0x7f77c4102650 2024-03-22T02:11:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.419+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00b9f0 con 0x7f77c4102650 2024-03-22T02:11:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.419+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c419f1d0 con 0x7f77c4102650 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.420+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77c4107e30 con 0x7f77c4102650 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.421+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f77bc01d020 con 0x7f77c4102650 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.421+0000 7f77c0ff9640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 0x7f77ac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.421+0000 7f77c37fe640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 0x7f77ac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.421+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f77bc0cf420 con 0x7f77c4102650 2024-03-22T02:11:46.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.422+0000 7f77c37fe640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 0x7f77ac0a53e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77b4005fd0 tx=0x7f77b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.426+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f77bc09c010 con 0x7f77c4102650 2024-03-22T02:11:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.609+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77c4062790 con 0x7f77c4102650 2024-03-22T02:11:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.609+0000 7f77c0ff9640 1 -- 172.21.15.28:0/4161407823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f77bc09b910 con 0x7f77c4102650 2024-03-22T02:11:46.609 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:11:46.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 msgr2=0x7f77ac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 0x7f77ac0a53e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f77b4005fd0 tx=0x7f77b400b410 comp rx=0 tx=0).stop 2024-03-22T02:11:46.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 msgr2=0x7f77c419e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00efc0 tx=0x7f77bc00c800 comp rx=0 tx=0).stop 2024-03-22T02:11:46.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 shutdown_connections 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77ac0a2f20 0x7f77ac0a53e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77c4102650 0x7f77c419e3b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77c4101c90 0x7f77c4197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 --2- 172.21.15.28:0/4161407823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77c4100aa0 0x7f77c4196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 >> 172.21.15.28:0/4161407823 conn(0x7f77c40fc030 msgr2=0x7f77c4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 shutdown_connections 2024-03-22T02:11:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.611+0000 7f77ca02a640 1 -- 172.21.15.28:0/4161407823 wait complete. 2024-03-22T02:11:46.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:11:46.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 -- 172.21.15.28:0/3965531439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100df0 msgr2=0x7fd52c101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3965531439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100df0 0x7fd52c101250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd514009900 tx=0x7fd51402bcd0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 -- 172.21.15.28:0/3965531439 shutdown_connections 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3965531439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 0x7fd52c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3965531439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100df0 0x7fd52c101250 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3965531439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100490 0x7fd52c1008b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 -- 172.21.15.28:0/3965531439 >> 172.21.15.28:0/3965531439 conn(0x7fd52c0fc030 msgr2=0x7fd52c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 -- 172.21.15.28:0/3965531439 shutdown_connections 2024-03-22T02:11:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.727+0000 7fd5323d0640 1 -- 172.21.15.28:0/3965531439 wait complete. 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 Processor -- start 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 -- start start 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100df0 0x7fd52c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 0x7fd52c1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10fa20 con 0x7fd52c100490 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10f8a0 con 0x7fd52c101790 2024-03-22T02:11:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd5323d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd52c10fba0 con 0x7fd52c100df0 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd52bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56562/0 (socket says 172.21.15.28:56562) 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd52bfff640 1 -- 172.21.15.28:0/3377701623 learned_addr learned my addr 172.21.15.28:0/3377701623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd53096d640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 0x7fd52c1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.728+0000 7fd52b7fe640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100df0 0x7fd52c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 -- 172.21.15.28:0/3377701623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100df0 msgr2=0x7fd52c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100df0 0x7fd52c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 -- 172.21.15.28:0/3377701623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 msgr2=0x7fd52c1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 0x7fd52c1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 -- 172.21.15.28:0/3377701623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd52c1a1d10 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd52bfff640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00d770 tx=0x7fd51c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c017070 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd52c1a1fa0 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd51c005500 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51c004e00 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.729+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd52c1a2410 con 0x7fd52c100490 2024-03-22T02:11:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.730+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd52c10b3d0 con 0x7fd52c100490 2024-03-22T02:11:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.731+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fd51c004fa0 con 0x7fd52c100490 2024-03-22T02:11:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.731+0000 7fd5297fa640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 0x7fd5180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.732+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd51c0cefd0 con 0x7fd52c100490 2024-03-22T02:11:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.732+0000 7fd52b7fe640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 0x7fd5180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.732+0000 7fd52b7fe640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 0x7fd5180a53e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd51402c850 tx=0x7fd514000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.736+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd51c09b570 con 0x7fd52c100490 2024-03-22T02:11:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.921+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd52c062790 con 0x7fd52c100490 2024-03-22T02:11:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.921+0000 7fd5297fa640 1 -- 172.21.15.28:0/3377701623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fd51c09f610 con 0x7fd52c100490 2024-03-22T02:11:47.048 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.923+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 msgr2=0x7fd5180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.923+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 0x7fd5180a53e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd51402c850 tx=0x7fd514000bc0 comp rx=0 tx=0).stop 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.923+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 msgr2=0x7fd52c19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.923+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd51c00d770 tx=0x7fd51c00dc30 comp rx=0 tx=0).stop 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 shutdown_connections 2024-03-22T02:11:47.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5180a2f20 0x7fd5180a53e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd52c101790 0x7fd52c1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd52c100df0 0x7fd52c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 --2- 172.21.15.28:0/3377701623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd52c100490 0x7fd52c19a0f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 >> 172.21.15.28:0/3377701623 conn(0x7fd52c0fc030 msgr2=0x7fd52c0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 shutdown_connections 2024-03-22T02:11:47.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:46.924+0000 7fd5323d0640 1 -- 172.21.15.28:0/3377701623 wait complete. 2024-03-22T02:11:47.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:11:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- 172.21.15.28:0/1488702436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 msgr2=0x7fc3e0109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/1488702436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e0109ce0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009ca0 tx=0x7fc3d802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- 172.21.15.28:0/1488702436 shutdown_connections 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/1488702436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e0109ce0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/1488702436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 0x7fc3e0107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/1488702436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 0x7fc3e0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- 172.21.15.28:0/1488702436 >> 172.21.15.28:0/1488702436 conn(0x7fc3e00fc040 msgr2=0x7fc3e00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- 172.21.15.28:0/1488702436 shutdown_connections 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- 172.21.15.28:0/1488702436 wait complete. 2024-03-22T02:11:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 Processor -- start 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.166+0000 7fc3e61f7640 1 -- start start 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 0x7fc3e0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 0x7fc3e01983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e010d850 con 0x7fc3e0107890 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e010d6d0 con 0x7fc3e0102630 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e010d9d0 con 0x7fc3e0104f60 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56582/0 (socket says 172.21.15.28:56582) 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 -- 172.21.15.28:0/585573482 learned_addr learned my addr 172.21.15.28:0/585573482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3df7fe640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 0x7fc3e0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3deffd640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 0x7fc3e01983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 -- 172.21.15.28:0/585573482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 msgr2=0x7fc3e01983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 0x7fc3e01983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 -- 172.21.15.28:0/585573482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 msgr2=0x7fc3e0197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 0x7fc3e0197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 -- 172.21.15.28:0/585573482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e019fab0 con 0x7fc3e0107890 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dffff640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8037040 tx=0x7fc3d8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8041020 con 0x7fc3e0107890 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3d8034d90 con 0x7fc3e0107890 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e019fd40 con 0x7fc3e0107890 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8039040 con 0x7fc3e0107890 2024-03-22T02:11:48.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.167+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e01a01d0 con 0x7fc3e0107890 2024-03-22T02:11:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.168+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3e00fdb30 con 0x7fc3e0107890 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1722625573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4161407823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:46 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3377701623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1722625573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4161407823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.123 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:46 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3377701623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.170+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc3d8044020 con 0x7fc3e0107890 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.171+0000 7fc3dcff9640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 0x7fc3c80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.171+0000 7fc3df7fe640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 0x7fc3c80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.171+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc3d80f1a20 con 0x7fc3e0107890 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.171+0000 7fc3df7fe640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 0x7fc3c80a53e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009bd0 tx=0x7fc3d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.174+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc3d804b020 con 0x7fc3e0107890 2024-03-22T02:11:48.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1722625573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:48.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4161407823' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:46 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3377701623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.357+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc3e0062830 con 0x7fc3e0107890 2024-03-22T02:11:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.358+0000 7fc3dcff9640 1 -- 172.21.15.28:0/585573482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fc3d80be5b0 con 0x7fc3e0107890 2024-03-22T02:11:48.126 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:11:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 msgr2=0x7fc3c80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 0x7fc3c80a53e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc3d0009bd0 tx=0x7fc3d0005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 msgr2=0x7fc3e019f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8037040 tx=0x7fc3d8032f20 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 shutdown_connections 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3c80a2f20 0x7fc3c80a53e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3e0107890 0x7fc3e019f440 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc3e0104f60 0x7fc3e01983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 --2- 172.21.15.28:0/585573482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc3e0102630 0x7fc3e0197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 >> 172.21.15.28:0/585573482 conn(0x7fc3e00fc040 msgr2=0x7fc3e0102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 shutdown_connections 2024-03-22T02:11:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:47.360+0000 7fc3e61f7640 1 -- 172.21.15.28:0/585573482 wait complete. 2024-03-22T02:11:48.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:11:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:47 smithi120 ceph-mon[36803]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:48.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:47 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/585573482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 -- 172.21.15.28:0/3559404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30104f50 msgr2=0x7f3c30107340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 --2- 172.21.15.28:0/3559404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30104f50 0x7f3c30107340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009900 tx=0x7f3c1802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 -- 172.21.15.28:0/3559404018 shutdown_connections 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 --2- 172.21.15.28:0/3559404018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30107880 0x7f3c30109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 --2- 172.21.15.28:0/3559404018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30104f50 0x7f3c30107340 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 --2- 172.21.15.28:0/3559404018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 -- 172.21.15.28:0/3559404018 >> 172.21.15.28:0/3559404018 conn(0x7f3c300fc030 msgr2=0x7f3c300fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 -- 172.21.15.28:0/3559404018 shutdown_connections 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.246+0000 7f3c351a0640 1 -- 172.21.15.28:0/3559404018 wait complete. 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 Processor -- start 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 -- start start 2024-03-22T02:11:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30104f50 0x7f3c30198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010d810 con 0x7f3c30107880 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010d690 con 0x7f3c30102620 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.247+0000 7f3c351a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010d990 con 0x7f3c30104f50 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43730/0 (socket says 172.21.15.28:43730) 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56612/0 (socket says 172.21.15.28:56612) 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 -- 172.21.15.28:0/1189595548 learned_addr learned my addr 172.21.15.28:0/1189595548 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c27fff640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30104f50 0x7f3c30198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 -- 172.21.15.28:0/1189595548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30104f50 msgr2=0x7f3c30198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30104f50 0x7f3c30198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 -- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 msgr2=0x7f3c3019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 -- 172.21.15.28:0/1189595548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3019fb30 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2f59e640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2ed9d640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000e9f0 tx=0x7f3c2000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2000cd80 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c20010910 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c20018990 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3019fe20 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.248+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c301a0280 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.249+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c300fdb20 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.250+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f3c20010ac0 con 0x7f3c30102620 2024-03-22T02:11:48.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.251+0000 7f3c2cd99640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 0x7f3c1c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.251+0000 7f3c27fff640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 0x7f3c1c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.251+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f3c200cf9d0 con 0x7f3c30102620 2024-03-22T02:11:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.251+0000 7f3c27fff640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 0x7f3c1c0a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3c1802c8e0 tx=0x7f3c18005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.256+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c2009c5c0 con 0x7f3c30102620 2024-03-22T02:11:48.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:47 smithi083 ceph-mon[36340]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:48.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:47 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/585573482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:47 smithi028 ceph-mon[31916]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 1 op/s 2024-03-22T02:11:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:47 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/585573482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:48.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.440+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3c30062890 con 0x7f3c30102620 2024-03-22T02:11:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.440+0000 7f3c2cd99640 1 -- 172.21.15.28:0/1189595548 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f3c2009bec0 con 0x7f3c30102620 2024-03-22T02:11:48.441 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 msgr2=0x7f3c1c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 0x7f3c1c0a53e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3c1802c8e0 tx=0x7f3c18005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 msgr2=0x7f3c30197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3c2000e9f0 tx=0x7f3c2000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 shutdown_connections 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c1c0a2f20 0x7f3c1c0a53e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c30107880 0x7f3c3019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.442+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c30104f50 0x7f3c30198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.443+0000 7f3c351a0640 1 --2- 172.21.15.28:0/1189595548 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c30102620 0x7f3c30197f10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.443+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 >> 172.21.15.28:0/1189595548 conn(0x7f3c300fc030 msgr2=0x7f3c301030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.443+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 shutdown_connections 2024-03-22T02:11:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.443+0000 7f3c351a0640 1 -- 172.21.15.28:0/1189595548 wait complete. 2024-03-22T02:11:48.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- 172.21.15.28:0/3838211490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc104fe0 msgr2=0x7f17dc1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- 172.21.15.28:0/3838211490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc104fe0 0x7f17dc1073d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009900 tx=0x7f17cc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- 172.21.15.28:0/3838211490 shutdown_connections 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- 172.21.15.28:0/3838211490 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc107910 0x7f17dc109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- 172.21.15.28:0/3838211490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc104fe0 0x7f17dc1073d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- 172.21.15.28:0/3838211490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc1026b0 0x7f17dc104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- 172.21.15.28:0/3838211490 >> 172.21.15.28:0/3838211490 conn(0x7f17dc0fc020 msgr2=0x7f17dc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- 172.21.15.28:0/3838211490 shutdown_connections 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- 172.21.15.28:0/3838211490 wait complete. 2024-03-22T02:11:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 Processor -- start 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- start start 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc107910 0x7f17dc19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.560+0000 7f17e3caf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10d800 con 0x7f17dc1026b0 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e3caf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10d680 con 0x7f17dc104fe0 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e3caf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10d980 con 0x7f17dc107910 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e124a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56620/0 (socket says 172.21.15.28:56620) 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e124a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43748/0 (socket says 172.21.15.28:43748) 2024-03-22T02:11:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 -- 172.21.15.28:0/4251137539 learned_addr learned my addr 172.21.15.28:0/4251137539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e224c640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc107910 0x7f17dc19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 -- 172.21.15.28:0/4251137539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc107910 msgr2=0x7f17dc19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc107910 0x7f17dc19f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 -- 172.21.15.28:0/4251137539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 msgr2=0x7f17dc198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 -- 172.21.15.28:0/4251137539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc19fb50 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e124a640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e1a4b640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d770 tx=0x7f17d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8017070 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d8005500 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17dc19fe40 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8004e00 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.561+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17dc1a0250 con 0x7f17dc1026b0 2024-03-22T02:11:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.563+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17dc0fdb10 con 0x7f17dc1026b0 2024-03-22T02:11:48.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.564+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f17d8004fa0 con 0x7f17dc1026b0 2024-03-22T02:11:48.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.564+0000 7f17d2ffd640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 0x7f17c80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.564+0000 7f17e124a640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 0x7f17c80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.565+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f17d80cf080 con 0x7f17dc1026b0 2024-03-22T02:11:48.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.565+0000 7f17e124a640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 0x7f17c80a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17cc000c00 tx=0x7f17cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.569+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17d809bc70 con 0x7f17dc1026b0 2024-03-22T02:11:48.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.772+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f17dc062860 con 0x7f17dc1026b0 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.773+0000 7f17d2ffd640 1 -- 172.21.15.28:0/4251137539 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+587 (secure 0 0 0) 0x7f17d809b570 con 0x7f17dc1026b0 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2329372296 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:48.773 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1957817657 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/548967036 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/30661343 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2074246765 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4059471246 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:48.774 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-03-22T02:11:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 msgr2=0x7f17c80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 0x7f17c80a53e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17cc000c00 tx=0x7f17cc005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 msgr2=0x7f17dc197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f17d800d770 tx=0x7f17d800dc30 comp rx=0 tx=0).stop 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 shutdown_connections 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17c80a2f20 0x7f17c80a53e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17dc107910 0x7f17dc19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17dc104fe0 0x7f17dc198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 --2- 172.21.15.28:0/4251137539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17dc1026b0 0x7f17dc197f30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 >> 172.21.15.28:0/4251137539 conn(0x7f17dc0fc020 msgr2=0x7f17dc103150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 shutdown_connections 2024-03-22T02:11:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.775+0000 7f17e3caf640 1 -- 172.21.15.28:0/4251137539 wait complete. 2024-03-22T02:11:48.786 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:11:48.787 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:11:48.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 -- 172.21.15.28:0/2421546186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8102690 msgr2=0x7f7ff8108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/2421546186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8102690 0x7f7ff8108f20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009be0 tx=0x7f7ff002bef0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 -- 172.21.15.28:0/2421546186 shutdown_connections 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/2421546186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8102690 0x7f7ff8108f20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/2421546186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8101cd0 0x7f7ff8102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/2421546186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8100fc0 0x7f7ff81013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 -- 172.21.15.28:0/2421546186 >> 172.21.15.28:0/2421546186 conn(0x7f7ff80fb790 msgr2=0x7f7ff80fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 -- 172.21.15.28:0/2421546186 shutdown_connections 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.953+0000 7f7fff9ee640 1 -- 172.21.15.28:0/2421546186 wait complete. 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 Processor -- start 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 -- start start 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8100fc0 0x7f7ff8100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 0x7f7ff819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff810c780 con 0x7f7ff8102690 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff810c600 con 0x7f7ff8101cd0 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7fff9ee640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff810c900 con 0x7f7ff8100fc0 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffcf89640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffdf8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 0x7f7ff819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffcf89640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43752/0 (socket says 172.21.15.28:43752) 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffdf8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 0x7f7ff819f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56646/0 (socket says 172.21.15.28:56646) 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffcf89640 1 -- 172.21.15.28:0/3402884255 learned_addr learned my addr 172.21.15.28:0/3402884255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.954+0000 7f7ffd78a640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8100fc0 0x7f7ff8100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 -- 172.21.15.28:0/3402884255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8100fc0 msgr2=0x7f7ff8100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8100fc0 0x7f7ff8100680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 -- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 msgr2=0x7f7ff819f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 0x7f7ff819f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 -- 172.21.15.28:0/3402884255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff819fa80 con 0x7f7ff8101cd0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7ffcf89640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800e9f0 tx=0x7f7fe800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe800cd80 con 0x7f7ff8101cd0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fe8005500 con 0x7f7ff8101cd0 2024-03-22T02:11:48.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff819fd10 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8010690 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.955+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff81a01b0 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.956+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff8107da0 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.957+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f7fe80046c0 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.957+0000 7f7fee7fc640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 0x7f7fe40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.957+0000 7f7ffd78a640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 0x7f7fe40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.957+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7fe80cf9f0 con 0x7f7ff8101cd0 2024-03-22T02:11:48.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.958+0000 7f7ffd78a640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 0x7f7fe40a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009530 tx=0x7f7ff4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:48.966+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7fe809c5e0 con 0x7f7ff8101cd0 2024-03-22T02:11:49.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.172+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ff8062830 con 0x7f7ff8101cd0 2024-03-22T02:11:49.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.173+0000 7f7fee7fc640 1 -- 172.21.15.28:0/3402884255 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f7fe80bc0d0 con 0x7f7ff8101cd0 2024-03-22T02:11:49.174 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:49.175 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.175+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 msgr2=0x7f7fe40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.175+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 0x7f7fe40a53e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009530 tx=0x7f7ff4009260 comp rx=0 tx=0).stop 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.175+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 msgr2=0x7f7ff819cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800e9f0 tx=0x7f7fe800edf0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 shutdown_connections 2024-03-22T02:11:49.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fe40a2f20 0x7f7fe40a53e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff8102690 0x7f7ff819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ff8101cd0 0x7f7ff819cb50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 --2- 172.21.15.28:0/3402884255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ff8100fc0 0x7f7ff8100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 >> 172.21.15.28:0/3402884255 conn(0x7f7ff80fb790 msgr2=0x7f7ff8104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 shutdown_connections 2024-03-22T02:11:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.176+0000 7f7fff9ee640 1 -- 172.21.15.28:0/3402884255 wait complete. 2024-03-22T02:11:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1189595548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:49.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4251137539' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:49.188 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:11:49.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:11:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1189595548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:49.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4251137539' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:49.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1189595548' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:49.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4251137539' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:49.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 -- 172.21.15.28:0/804799743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48101c80 msgr2=0x7f5c48102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/804799743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48101c80 0x7f5c48102100 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c009860 tx=0x7f5c3c02bb70 comp rx=0 tx=0).stop 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 -- 172.21.15.28:0/804799743 shutdown_connections 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/804799743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48102640 0x7f5c48108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/804799743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48101c80 0x7f5c48102100 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/804799743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48100a90 0x7f5c48100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 -- 172.21.15.28:0/804799743 >> 172.21.15.28:0/804799743 conn(0x7f5c480fc020 msgr2=0x7f5c480fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 -- 172.21.15.28:0/804799743 shutdown_connections 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.295+0000 7f5c4e53e640 1 -- 172.21.15.28:0/804799743 wait complete. 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 Processor -- start 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 -- start start 2024-03-22T02:11:49.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 0x7f5c48197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48101c80 0x7f5c48198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810c710 con 0x7f5c48102640 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810c590 con 0x7f5c48100a90 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4e53e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4810c890 con 0x7f5c48101c80 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 0x7f5c48197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56662/0 (socket says 172.21.15.28:56662) 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 -- 172.21.15.28:0/3866192967 learned_addr learned my addr 172.21.15.28:0/3866192967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c477fe640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48101c80 0x7f5c48198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 -- 172.21.15.28:0/3866192967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48101c80 msgr2=0x7f5c48198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48101c80 0x7f5c48198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 -- 172.21.15.28:0/3866192967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 msgr2=0x7f5c48197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 0x7f5c48197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c4cadb640 1 -- 172.21.15.28:0/3866192967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4819fae0 con 0x7f5c48102640 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.296+0000 7f5c47fff640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 0x7f5c48197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c4cadb640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000eb10 tx=0x7f5c4000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c40004b20 con 0x7f5c48102640 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4819fdd0 con 0x7f5c48102640 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c40004cc0 con 0x7f5c48102640 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c40010760 con 0x7f5c48102640 2024-03-22T02:11:49.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.297+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c481a0290 con 0x7f5c48102640 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.298+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c481a0520 con 0x7f5c48102640 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.299+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f5c40005000 con 0x7f5c48102640 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.300+0000 7f5c457fa640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 0x7f5c300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.300+0000 7f5c47fff640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 0x7f5c300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.300+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f5c400d00e0 con 0x7f5c48102640 2024-03-22T02:11:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.300+0000 7f5c47fff640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 0x7f5c300a53e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5c380096c0 tx=0x7f5c38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.304+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c4009ccd0 con 0x7f5c48102640 2024-03-22T02:11:49.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.487+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5c481a06f0 con 0x7f5c48102640 2024-03-22T02:11:49.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.488+0000 7f5c457fa640 1 -- 172.21.15.28:0/3866192967 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f5c4009c5d0 con 0x7f5c48102640 2024-03-22T02:11:49.488 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 msgr2=0x7f5c300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 0x7f5c300a53e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5c380096c0 tx=0x7f5c38009260 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 msgr2=0x7f5c4819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000eb10 tx=0x7f5c4000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 shutdown_connections 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c300a2f20 0x7f5c300a53e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c48102640 0x7f5c4819f470 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c48101c80 0x7f5c48198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 --2- 172.21.15.28:0/3866192967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c48100a90 0x7f5c48197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 >> 172.21.15.28:0/3866192967 conn(0x7f5c480fc020 msgr2=0x7f5c480fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 shutdown_connections 2024-03-22T02:11:49.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.490+0000 7f5c4e53e640 1 -- 172.21.15.28:0/3866192967 wait complete. 2024-03-22T02:11:49.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- 172.21.15.28:0/2652139883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc100ab0 msgr2=0x7efdfc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 --2- 172.21.15.28:0/2652139883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc100ab0 0x7efdfc100ed0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009900 tx=0x7efdf802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- 172.21.15.28:0/2652139883 shutdown_connections 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 --2- 172.21.15.28:0/2652139883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc102660 0x7efdfc108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 --2- 172.21.15.28:0/2652139883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc101ca0 0x7efdfc102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 --2- 172.21.15.28:0/2652139883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc100ab0 0x7efdfc100ed0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- 172.21.15.28:0/2652139883 >> 172.21.15.28:0/2652139883 conn(0x7efdfc0fc040 msgr2=0x7efdfc0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- 172.21.15.28:0/2652139883 shutdown_connections 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- 172.21.15.28:0/2652139883 wait complete. 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 Processor -- start 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.619+0000 7efe040e0640 1 -- start start 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc100ab0 0x7efdfc196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc102660 0x7efdfc19e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc10c700 con 0x7efdfc101ca0 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc10c580 con 0x7efdfc102660 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdfc10c880 con 0x7efdfc100ab0 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56674/0 (socket says 172.21.15.28:56674) 2024-03-22T02:11:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 -- 172.21.15.28:0/476823444 learned_addr learned my addr 172.21.15.28:0/476823444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0267d640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc102660 0x7efdfc19e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe01e7c640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc100ab0 0x7efdfc196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 -- 172.21.15.28:0/476823444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc100ab0 msgr2=0x7efdfc196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc100ab0 0x7efdfc196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 -- 172.21.15.28:0/476823444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc102660 msgr2=0x7efdfc19e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc102660 0x7efdfc19e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 -- 172.21.15.28:0/476823444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdfc19ea40 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe0167b640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdec00ea60 tx=0x7efdec00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec00cde0 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdec0035f0 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdfc19ed30 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdec010670 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.620+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdfc19f220 con 0x7efdfc101ca0 2024-03-22T02:11:49.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.621+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdfc107ef0 con 0x7efdfc101ca0 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.622+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7efdec0037a0 con 0x7efdfc101ca0 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.623+0000 7efdf2ffd640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 0x7efde80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.623+0000 7efe01e7c640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 0x7efde80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.623+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7efdec0cfa80 con 0x7efdfc101ca0 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.623+0000 7efe01e7c640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 0x7efde80a53e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efdf802c7b0 tx=0x7efdf8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.626+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efdec09c700 con 0x7efdfc101ca0 2024-03-22T02:11:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.811+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efdfc02d600 con 0x7efdfc101ca0 2024-03-22T02:11:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.811+0000 7efdf2ffd640 1 -- 172.21.15.28:0/476823444 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7efdec09c000 con 0x7efdfc101ca0 2024-03-22T02:11:49.812 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 msgr2=0x7efde80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 0x7efde80a53e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efdf802c7b0 tx=0x7efdf8005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 msgr2=0x7efdfc197360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efdec00ea60 tx=0x7efdec00ef20 comp rx=0 tx=0).stop 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 shutdown_connections 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efde80a2f20 0x7efde80a53e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdfc102660 0x7efdfc19e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdfc101ca0 0x7efdfc197360 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 --2- 172.21.15.28:0/476823444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdfc100ab0 0x7efdfc196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 >> 172.21.15.28:0/476823444 conn(0x7efdfc0fc040 msgr2=0x7efdfc1050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 shutdown_connections 2024-03-22T02:11:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.813+0000 7efe040e0640 1 -- 172.21.15.28:0/476823444 wait complete. 2024-03-22T02:11:49.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/762720055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c101c80 msgr2=0x7fa29c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/762720055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c101c80 0x7fa29c102100 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa290009900 tx=0x7fa29002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/762720055 shutdown_connections 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/762720055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c102640 0x7fa29c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/762720055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c101c80 0x7fa29c102100 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/762720055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c100a90 0x7fa29c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/762720055 >> 172.21.15.28:0/762720055 conn(0x7fa29c0fc020 msgr2=0x7fa29c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/762720055 shutdown_connections 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.930+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/762720055 wait complete. 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 Processor -- start 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 -- start start 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 0x7fa29c1972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c102640 0x7fa29c19e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa29c10c710 con 0x7fa29c100a90 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa29c10c590 con 0x7fa29c102640 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa2a0d3e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa29c10c890 con 0x7fa29c101c80 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa29a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa29a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56688/0 (socket says 172.21.15.28:56688) 2024-03-22T02:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa29a59c640 1 -- 172.21.15.28:0/1269656594 learned_addr learned my addr 172.21.15.28:0/1269656594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa29ad9d640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c102640 0x7fa29c19e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa299d9b640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 0x7fa29c1972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.931+0000 7fa29a59c640 1 -- 172.21.15.28:0/1269656594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 msgr2=0x7fa29c1972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa29a59c640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 0x7fa29c1972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa29a59c640 1 -- 172.21.15.28:0/1269656594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c102640 msgr2=0x7fa29c19e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa29a59c640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c102640 0x7fa29c19e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa29a59c640 1 -- 172.21.15.28:0/1269656594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa29c19e9d0 con 0x7fa29c100a90 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa299d9b640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 0x7fa29c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa29a59c640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e9c0 tx=0x7fa28c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00cde0 con 0x7fa29c100a90 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa28c010910 con 0x7fa29c100a90 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa29c19ecc0 con 0x7fa29c100a90 2024-03-22T02:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c018a10 con 0x7fa29c100a90 2024-03-22T02:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.932+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa29c19f1c0 con 0x7fa29c100a90 2024-03-22T02:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.933+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa29c107dd0 con 0x7fa29c100a90 2024-03-22T02:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.934+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa28c01f020 con 0x7fa29c100a90 2024-03-22T02:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.934+0000 7fa28b7fe640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 0x7fa2840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.935+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa28c0cf8c0 con 0x7fa29c100a90 2024-03-22T02:11:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.935+0000 7fa299d9b640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 0x7fa2840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.935+0000 7fa299d9b640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 0x7fa2840a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa290036040 tx=0x7fa290000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:49.939+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa28c09c4b0 con 0x7fa29c100a90 2024-03-22T02:11:50.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.122+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa29c062830 con 0x7fa29c100a90 2024-03-22T02:11:50.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.123+0000 7fa28b7fe640 1 -- 172.21.15.28:0/1269656594 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fa28c09bdb0 con 0x7fa29c100a90 2024-03-22T02:11:50.123 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 msgr2=0x7fa2840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 0x7fa2840a53e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa290036040 tx=0x7fa290000bc0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 msgr2=0x7fa29c196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e9c0 tx=0x7fa28c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 shutdown_connections 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2840a2f20 0x7fa2840a53e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa29c102640 0x7fa29c19e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c101c80 0x7fa29c1972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 --2- 172.21.15.28:0/1269656594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa29c100a90 0x7fa29c196db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 >> 172.21.15.28:0/1269656594 conn(0x7fa29c0fc020 msgr2=0x7fa29c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 shutdown_connections 2024-03-22T02:11:50.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.125+0000 7fa2a0d3e640 1 -- 172.21.15.28:0/1269656594 wait complete. 2024-03-22T02:11:50.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:49 smithi028 ceph-mon[31916]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-03-22T02:11:50.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:49 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3402884255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:50.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:49 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3866192967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:49 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/476823444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 -- 172.21.15.28:0/2007861914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 msgr2=0x7f6138102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2007861914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138102120 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6120009900 tx=0x7f612002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 -- 172.21.15.28:0/2007861914 shutdown_connections 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2007861914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138102660 0x7f6138108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2007861914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138102120 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2007861914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138100ab0 0x7f6138100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 -- 172.21.15.28:0/2007861914 >> 172.21.15.28:0/2007861914 conn(0x7f61380fc040 msgr2=0x7f61380fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 -- 172.21.15.28:0/2007861914 shutdown_connections 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.242+0000 7f613d9f7640 1 -- 172.21.15.28:0/2007861914 wait complete. 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 Processor -- start 2024-03-22T02:11:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 -- start start 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138100ab0 0x7f6138104b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138102660 0x7f6138103690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f613810c730 con 0x7f6138101ca0 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f613810c5b0 con 0x7f6138102660 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f613d9f7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f613810c8b0 con 0x7f6138100ab0 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56710/0 (socket says 172.21.15.28:56710) 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 -- 172.21.15.28:0/2223465788 learned_addr learned my addr 172.21.15.28:0/2223465788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f6136ffd640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138100ab0 0x7f6138104b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f61377fe640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138102660 0x7f6138103690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 -- 172.21.15.28:0/2223465788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138100ab0 msgr2=0x7f6138104b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138100ab0 0x7f6138104b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 -- 172.21.15.28:0/2223465788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138102660 msgr2=0x7f6138103690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138102660 0x7f6138103690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 -- 172.21.15.28:0/2223465788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6138103f20 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.243+0000 7f612ffff640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6120009230 tx=0x7f6120005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.244+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120041070 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.244+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61200044d0 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.244+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381a1fa0 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.244+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120034440 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.244+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61381a2400 con 0x7f6138101ca0 2024-03-22T02:11:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.245+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61381a2690 con 0x7f6138101ca0 2024-03-22T02:11:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:49 smithi120 ceph-mon[36803]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-03-22T02:11:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:49 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3402884255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:49 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3866192967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:49 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/476823444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.246+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f6120044070 con 0x7f6138101ca0 2024-03-22T02:11:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.246+0000 7f613c9f5640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.246+0000 7f6136ffd640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.246+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f61200f1190 con 0x7f6138101ca0 2024-03-22T02:11:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.247+0000 7f6136ffd640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 0x7f61240a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6138101b00 tx=0x7f6128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:50.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.251+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61200bd730 con 0x7f6138101ca0 2024-03-22T02:11:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:49 smithi083 ceph-mon[36340]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 818 B/s wr, 0 op/s 2024-03-22T02:11:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:49 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3402884255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:49 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3866192967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:49 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/476823444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:11:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.433+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f61381a2fd0 con 0x7f6138101ca0 2024-03-22T02:11:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.434+0000 7f613c9f5640 1 -- 172.21.15.28:0/2223465788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7f61200c17d0 con 0x7f6138101ca0 2024-03-22T02:11:50.434 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:11:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 msgr2=0x7f61240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 0x7f61240a53e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6138101b00 tx=0x7f6128005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 msgr2=0x7f6138103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6120009230 tx=0x7f6120005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 shutdown_connections 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61240a2f20 0x7f61240a53e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6138102660 0x7f6138103690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6138101ca0 0x7f6138103150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 --2- 172.21.15.28:0/2223465788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6138100ab0 0x7f6138104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 >> 172.21.15.28:0/2223465788 conn(0x7f61380fc040 msgr2=0x7f613806a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 shutdown_connections 2024-03-22T02:11:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.436+0000 7f613d9f7640 1 -- 172.21.15.28:0/2223465788 wait complete. 2024-03-22T02:11:50.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 -- 172.21.15.28:0/1570923449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c100ab0 msgr2=0x7f4a9c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 --2- 172.21.15.28:0/1570923449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c100ed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009ca0 tx=0x7f4a9402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 -- 172.21.15.28:0/1570923449 shutdown_connections 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 --2- 172.21.15.28:0/1570923449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c102660 0x7f4a9c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 --2- 172.21.15.28:0/1570923449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 --2- 172.21.15.28:0/1570923449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c100ed0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 -- 172.21.15.28:0/1570923449 >> 172.21.15.28:0/1570923449 conn(0x7f4a9c0fc020 msgr2=0x7f4a9c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 -- 172.21.15.28:0/1570923449 shutdown_connections 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.550+0000 7f4aa3088640 1 -- 172.21.15.28:0/1570923449 wait complete. 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 Processor -- start 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 -- start start 2024-03-22T02:11:50.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c102660 0x7f4a9c19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c10d850 con 0x7f4a9c101ca0 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c10d6d0 con 0x7f4a9c102660 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa3088640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c10d9d0 con 0x7f4a9c100ab0 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56730/0 (socket says 172.21.15.28:56730) 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 -- 172.21.15.28:0/578370243 learned_addr learned my addr 172.21.15.28:0/578370243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa0e24640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4aa1625640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c102660 0x7f4a9c19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 -- 172.21.15.28:0/578370243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c100ab0 msgr2=0x7f4a9c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 -- 172.21.15.28:0/578370243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c102660 msgr2=0x7f4a9c19f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c102660 0x7f4a9c19f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.551+0000 7f4a9bfff640 1 -- 172.21.15.28:0/578370243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c19fb80 con 0x7f4a9c101ca0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4a9bfff640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000e9f0 tx=0x7f4a9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000cd80 con 0x7f4a9c101ca0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a90010910 con 0x7f4a9c101ca0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c19fe70 con 0x7f4a9c101ca0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90018990 con 0x7f4a9c101ca0 2024-03-22T02:11:50.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.552+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c1a0380 con 0x7f4a9c101ca0 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.553+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9c107ef0 con 0x7f4a9c101ca0 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.554+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f4a90010ac0 con 0x7f4a9c101ca0 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.555+0000 7f4a99ffb640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 0x7f4a840a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.555+0000 7f4aa0e24640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 0x7f4a840a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.555+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4a900cf8c0 con 0x7f4a9c101ca0 2024-03-22T02:11:50.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.555+0000 7f4aa0e24640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 0x7f4a840a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4a9402c700 tx=0x7f4a94005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:50.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.559+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a9009be00 con 0x7f4a9c101ca0 2024-03-22T02:11:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.755+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4a9c062830 con 0x7f4a9c101ca0 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.755+0000 7f4a99ffb640 1 -- 172.21.15.28:0/578370243 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) ==== 81+0+587 (secure 0 0 0) 0x7f4a9009fea0 con 0x7f4a9c101ca0 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2329372296 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1957817657 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:50.756 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/548967036 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:50.757 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/30661343 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:50.757 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2074246765 2024-03-23T02:04:26.467767+0000 2024-03-22T02:11:50.757 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4059471246 2024-03-23T02:03:59.710695+0000 2024-03-22T02:11:50.757 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 msgr2=0x7f4a840a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 0x7f4a840a5460 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4a9402c700 tx=0x7f4a94005f70 comp rx=0 tx=0).stop 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 msgr2=0x7f4a9c1984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000e9f0 tx=0x7f4a9000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 shutdown_connections 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4a840a2fa0 0x7f4a840a5460 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4a9c102660 0x7f4a9c19f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9c101ca0 0x7f4a9c1984a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 --2- 172.21.15.28:0/578370243 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4a9c100ab0 0x7f4a9c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 >> 172.21.15.28:0/578370243 conn(0x7f4a9c0fc020 msgr2=0x7f4a9c1050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.757+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 shutdown_connections 2024-03-22T02:11:50.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:50.758+0000 7f4aa3088640 1 -- 172.21.15.28:0/578370243 wait complete. 2024-03-22T02:11:50.768 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-03-22T02:11:50.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:50.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-22T02:11:50.793 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[76447]: fuse finished with error 0 and tester_r 0 2024-03-22T02:11:50.798 INFO:teuthology.orchestra.run:waiting for 300 2024-03-22T02:11:50.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:11:50.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:11:50.835 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-22T02:11:50.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:51.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 -- 172.21.15.28:0/3287021993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0101be0 msgr2=0x7fd5a0102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 --2- 172.21.15.28:0/3287021993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0101be0 0x7fd5a0102000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd590009ca0 tx=0x7fd59002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 -- 172.21.15.28:0/3287021993 shutdown_connections 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 --2- 172.21.15.28:0/3287021993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0103790 0x7fd5a010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 --2- 172.21.15.28:0/3287021993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 --2- 172.21.15.28:0/3287021993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0101be0 0x7fd5a0102000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 -- 172.21.15.28:0/3287021993 >> 172.21.15.28:0/3287021993 conn(0x7fd5a00fd130 msgr2=0x7fd5a00ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 -- 172.21.15.28:0/3287021993 shutdown_connections 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.043+0000 7fd5a609a640 1 -- 172.21.15.28:0/3287021993 wait complete. 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 Processor -- start 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 -- start start 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0101be0 0x7fd5a0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0103790 0x7fd5a019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010d860 con 0x7fd5a0102dd0 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010d6e0 con 0x7fd5a0103790 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd5a609a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd5a010d9e0 con 0x7fd5a0101be0 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd59effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd59effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56754/0 (socket says 172.21.15.28:56754) 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.044+0000 7fd59effd640 1 -- 172.21.15.28:0/1179598433 learned_addr learned my addr 172.21.15.28:0/1179598433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59ffff640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0103790 0x7fd5a019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59f7fe640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0101be0 0x7fd5a0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 -- 172.21.15.28:0/1179598433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0101be0 msgr2=0x7fd5a0197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0101be0 0x7fd5a0197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 -- 172.21.15.28:0/1179598433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0103790 msgr2=0x7fd5a019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0103790 0x7fd5a019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 -- 172.21.15.28:0/1179598433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5a019faf0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59effd640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd58800ea60 tx=0x7fd58800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58800cde0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5880035f0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5a019fde0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588010670 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.045+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5a01a02a0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.046+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5a0108f00 con 0x7fd5a0102dd0 2024-03-22T02:11:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.047+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fd5880037a0 con 0x7fd5a0102dd0 2024-03-22T02:11:51.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.048+0000 7fd59cff9640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 0x7fd58c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.048+0000 7fd59f7fe640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 0x7fd58c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.048+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd5880cf970 con 0x7fd5a0102dd0 2024-03-22T02:11:51.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.048+0000 7fd59f7fe640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 0x7fd58c0a53e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c700 tx=0x7fd590009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.053+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd58809c560 con 0x7fd5a0102dd0 2024-03-22T02:11:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:50 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1269656594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:50 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2223465788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:50 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/578370243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:51.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.237+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5a0062830 con 0x7fd5a0102dd0 2024-03-22T02:11:51.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.239+0000 7fd59cff9640 1 -- 172.21.15.28:0/1179598433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7fd58809be60 con 0x7fd5a0102dd0 2024-03-22T02:11:51.239 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:51.240 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:51.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 msgr2=0x7fd58c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 0x7fd58c0a53e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd59002c700 tx=0x7fd590009360 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 msgr2=0x7fd5a0198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd58800ea60 tx=0x7fd58800ef20 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 shutdown_connections 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd58c0a2f20 0x7fd58c0a53e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd5a0103790 0x7fd5a019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5a0102dd0 0x7fd5a0198410 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 --2- 172.21.15.28:0/1179598433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd5a0101be0 0x7fd5a0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 >> 172.21.15.28:0/1179598433 conn(0x7fd5a00fd130 msgr2=0x7fd5a01060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 shutdown_connections 2024-03-22T02:11:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.241+0000 7fd5a609a640 1 -- 172.21.15.28:0/1179598433 wait complete. 2024-03-22T02:11:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:50 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1269656594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:50 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2223465788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:50 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/578370243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:51.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:51.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:50 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1269656594' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:50 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2223465788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:11:51.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:50 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/578370243' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:11:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 -- 172.21.15.28:0/3559477409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0101ca0 msgr2=0x7f59d0102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 --2- 172.21.15.28:0/3559477409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0101ca0 0x7f59d0102120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59c0009900 tx=0x7f59c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 -- 172.21.15.28:0/3559477409 shutdown_connections 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 --2- 172.21.15.28:0/3559477409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0102660 0x7f59d0108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 --2- 172.21.15.28:0/3559477409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0101ca0 0x7f59d0102120 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 --2- 172.21.15.28:0/3559477409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 0x7f59d0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 -- 172.21.15.28:0/3559477409 >> 172.21.15.28:0/3559477409 conn(0x7f59d00fc040 msgr2=0x7f59d00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 -- 172.21.15.28:0/3559477409 shutdown_connections 2024-03-22T02:11:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.355+0000 7f59d822a640 1 -- 172.21.15.28:0/3559477409 wait complete. 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 Processor -- start 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 -- start start 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 0x7f59d006ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 0x7f59d019cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59d010c730 con 0x7f59d0102660 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f59d010c5b0 con 0x7f59d0100ab0 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d822a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f59d010c8b0 con 0x7f59d0101ca0 2024-03-22T02:11:51.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56774/0 (socket says 172.21.15.28:56774) 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d5fc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 0x7f59d006ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 -- 172.21.15.28:0/3115925281 learned_addr learned my addr 172.21.15.28:0/3115925281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d57c5640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 0x7f59d019cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 -- 172.21.15.28:0/3115925281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 msgr2=0x7f59d019cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 0x7f59d019cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 -- 172.21.15.28:0/3115925281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 msgr2=0x7f59d006ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 0x7f59d006ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d67c7640 1 -- 172.21.15.28:0/3115925281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d019fae0 con 0x7f59d0102660 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.356+0000 7f59d57c5640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 0x7f59d019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59d67c7640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f59c800eb10 tx=0x7f59c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8004b20 con 0x7f59d0102660 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59c8004cc0 con 0x7f59d0102660 2024-03-22T02:11:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d019fdd0 con 0x7f59d0102660 2024-03-22T02:11:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c8010760 con 0x7f59d0102660 2024-03-22T02:11:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.357+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d01a02a0 con 0x7f59d0102660 2024-03-22T02:11:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.358+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d0107dd0 con 0x7f59d0102660 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.359+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f59c8010900 con 0x7f59d0102660 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.360+0000 7f59c6ffd640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 0x7f59bc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.360+0000 7f59d5fc6640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 0x7f59bc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.360+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f59c80d00e0 con 0x7f59d0102660 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.360+0000 7f59d5fc6640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 0x7f59bc0a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f59d0101b00 tx=0x7f59cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.363+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f59c809cc70 con 0x7f59d0102660 2024-03-22T02:11:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.549+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59d002d530 con 0x7f59d0102660 2024-03-22T02:11:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.550+0000 7f59c6ffd640 1 -- 172.21.15.28:0/3115925281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f59c809c570 con 0x7f59d0102660 2024-03-22T02:11:51.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:51.551 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 msgr2=0x7f59bc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 0x7f59bc0a53e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f59d0101b00 tx=0x7f59cc005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 msgr2=0x7f59d019f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f59c800eb10 tx=0x7f59c800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 shutdown_connections 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f59bc0a2f20 0x7f59bc0a53e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d0102660 0x7f59d019f220 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59d0101ca0 0x7f59d019cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 --2- 172.21.15.28:0/3115925281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f59d0100ab0 0x7f59d006ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 >> 172.21.15.28:0/3115925281 conn(0x7f59d00fc040 msgr2=0x7f59d0104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 shutdown_connections 2024-03-22T02:11:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.553+0000 7f59d822a640 1 -- 172.21.15.28:0/3115925281 wait complete. 2024-03-22T02:11:51.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-22T02:11:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- 172.21.15.28:0/760530007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4100a90 msgr2=0x7fc0c4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/760530007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4100a90 0x7fc0c4100eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc009ca0 tx=0x7fc0bc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- 172.21.15.28:0/760530007 shutdown_connections 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/760530007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4102640 0x7fc0c4108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/760530007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4101c80 0x7fc0c4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/760530007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4100a90 0x7fc0c4100eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- 172.21.15.28:0/760530007 >> 172.21.15.28:0/760530007 conn(0x7fc0c40fc000 msgr2=0x7fc0c40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- 172.21.15.28:0/760530007 shutdown_connections 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- 172.21.15.28:0/760530007 wait complete. 2024-03-22T02:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 Processor -- start 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.671+0000 7fc0caf5e640 1 -- start start 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4100a90 0x7fc0c4197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4101c80 0x7fc0c41983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c410d830 con 0x7fc0c4102640 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c410d6b0 con 0x7fc0c4100a90 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c410d9b0 con 0x7fc0c4101c80 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c8cfa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4100a90 0x7fc0c4197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56796/0 (socket says 172.21.15.28:56796) 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 -- 172.21.15.28:0/429924656 learned_addr learned my addr 172.21.15.28:0/429924656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:51.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c3fff640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4101c80 0x7fc0c41983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 -- 172.21.15.28:0/429924656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4101c80 msgr2=0x7fc0c41983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4101c80 0x7fc0c41983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 -- 172.21.15.28:0/429924656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4100a90 msgr2=0x7fc0c4197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4100a90 0x7fc0c4197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 -- 172.21.15.28:0/429924656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c419fad0 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c94fb640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400d750 tx=0x7fc0b400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4017070 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b4005500 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.672+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c419fdc0 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.673+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4004d90 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.673+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c41a0230 con 0x7fc0c4102640 2024-03-22T02:11:51.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.674+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c4107e80 con 0x7fc0c4102640 2024-03-22T02:11:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.675+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc0b4004f30 con 0x7fc0c4102640 2024-03-22T02:11:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.675+0000 7fc0c1ffb640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 0x7fc0b00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.676+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc0b40cf2e0 con 0x7fc0c4102640 2024-03-22T02:11:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.676+0000 7fc0c8cfa640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 0x7fc0b00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:51.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.676+0000 7fc0c8cfa640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 0x7fc0b00a53e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc02c700 tx=0x7fc0bc032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:51.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.679+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0b409b880 con 0x7fc0c4102640 2024-03-22T02:11:51.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.901+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc0c402d450 con 0x7fc0c4102640 2024-03-22T02:11:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.960+0000 7fc0c1ffb640 1 -- 172.21.15.28:0/429924656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fc0b409f920 con 0x7fc0c4102640 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 msgr2=0x7fc0b00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 0x7fc0b00a53e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc02c700 tx=0x7fc0bc032590 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 msgr2=0x7fc0c419f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400d750 tx=0x7fc0b400dc10 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 shutdown_connections 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0b00a2f20 0x7fc0b00a53e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c4102640 0x7fc0c419f460 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c4101c80 0x7fc0c41983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 --2- 172.21.15.28:0/429924656 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c4100a90 0x7fc0c4197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 >> 172.21.15.28:0/429924656 conn(0x7fc0c40fc000 msgr2=0x7fc0c4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 shutdown_connections 2024-03-22T02:11:51.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:51.963+0000 7fc0caf5e640 1 -- 172.21.15.28:0/429924656 wait complete. 2024-03-22T02:11:51.975 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-22T02:11:51.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-22T02:11:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 -- 172.21.15.28:0/3257823931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec107930 msgr2=0x7f9fec109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 --2- 172.21.15.28:0/3257823931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec107930 0x7f9fec109d20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4009ca0 tx=0x7f9fe402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 -- 172.21.15.28:0/3257823931 shutdown_connections 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 --2- 172.21.15.28:0/3257823931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec107930 0x7f9fec109d20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 --2- 172.21.15.28:0/3257823931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec105000 0x7f9fec1073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 --2- 172.21.15.28:0/3257823931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec1026d0 0x7f9fec104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 -- 172.21.15.28:0/3257823931 >> 172.21.15.28:0/3257823931 conn(0x7f9fec0fc000 msgr2=0x7f9fec0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 -- 172.21.15.28:0/3257823931 shutdown_connections 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.080+0000 7f9ff220e640 1 -- 172.21.15.28:0/3257823931 wait complete. 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 Processor -- start 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 -- start start 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec105000 0x7f9fec100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec107930 0x7f9fec100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec10d820 con 0x7f9fec1026d0 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec10d6a0 con 0x7f9fec105000 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9ff220e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec10d9a0 con 0x7f9fec107930 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.081+0000 7f9feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49922/0 (socket says 172.21.15.28:49922) 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 -- 172.21.15.28:0/4183855509 learned_addr learned my addr 172.21.15.28:0/4183855509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feaffd640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec105000 0x7f9fec100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9febfff640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec107930 0x7f9fec100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 -- 172.21.15.28:0/4183855509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec107930 msgr2=0x7f9fec100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec107930 0x7f9fec100bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 -- 172.21.15.28:0/4183855509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec105000 msgr2=0x7f9fec100680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec105000 0x7f9fec100680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 -- 172.21.15.28:0/4183855509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec101450 con 0x7f9fec1026d0 2024-03-22T02:11:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9feb7fe640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9f0 tx=0x7f9fdc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00ccc0 con 0x7f9fec1026d0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fdc010910 con 0x7f9fec1026d0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec1a1ef0 con 0x7f9fec1026d0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc0188a0 con 0x7f9fec1026d0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.082+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec1a2350 con 0x7f9fec1026d0 2024-03-22T02:11:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.083+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec0fdaf0 con 0x7f9fec1026d0 2024-03-22T02:11:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.084+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9fdc010ac0 con 0x7f9fec1026d0 2024-03-22T02:11:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.085+0000 7f9fe8ff9640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 0x7f9fd40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.085+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f9fdc0cfa00 con 0x7f9fec1026d0 2024-03-22T02:11:52.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.085+0000 7f9feaffd640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 0x7f9fd40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.085+0000 7f9feaffd640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 0x7f9fd40a5460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fec101db0 tx=0x7f9fe000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:52.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.090+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fdc09c5f0 con 0x7f9fec1026d0 2024-03-22T02:11:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:51 smithi120 ceph-mon[36803]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s wr, 0 op/s 2024-03-22T02:11:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:51 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1179598433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:51 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3115925281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:51 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T02:11:52.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:51 smithi083 ceph-mon[36340]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s wr, 0 op/s 2024-03-22T02:11:52.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:51 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1179598433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:51 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3115925281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:51 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T02:11:52.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.311+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f9fec061fd0 con 0x7f9fec1026d0 2024-03-22T02:11:52.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:51 smithi028 ceph-mon[31916]: pgmap v302: 129 pgs: 129 active+clean; 579 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 88 B/s wr, 0 op/s 2024-03-22T02:11:52.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1179598433' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3115925281' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:52.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-22T02:11:52.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.968+0000 7f9fe8ff9640 1 -- 172.21.15.28:0/4183855509 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f9fdc09bef0 con 0x7f9fec1026d0 2024-03-22T02:11:52.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 msgr2=0x7f9fd40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 0x7f9fd40a5460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9fec101db0 tx=0x7f9fe000a2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 msgr2=0x7f9fec102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e9f0 tx=0x7f9fdc00edf0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 shutdown_connections 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9fd40a2fa0 0x7f9fd40a5460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fec107930 0x7f9fec100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9fec105000 0x7f9fec100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 --2- 172.21.15.28:0/4183855509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9fec1026d0 0x7f9fec102040 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 >> 172.21.15.28:0/4183855509 conn(0x7f9fec0fc000 msgr2=0x7f9fec103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 shutdown_connections 2024-03-22T02:11:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:52.971+0000 7f9ff220e640 1 -- 172.21.15.28:0/4183855509 wait complete. 2024-03-22T02:11:52.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-22T02:11:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- 172.21.15.28:0/2796212294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0102640 msgr2=0x7f62b0108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2796212294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0102640 0x7f62b0108ed0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009ca0 tx=0x7f62a802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- 172.21.15.28:0/2796212294 shutdown_connections 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2796212294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0102640 0x7f62b0108ed0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2796212294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0101c80 0x7f62b0102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2796212294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0100a90 0x7f62b0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- 172.21.15.28:0/2796212294 >> 172.21.15.28:0/2796212294 conn(0x7f62b00fc000 msgr2=0x7f62b00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- 172.21.15.28:0/2796212294 shutdown_connections 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- 172.21.15.28:0/2796212294 wait complete. 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 Processor -- start 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 -- start start 2024-03-22T02:11:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0100a90 0x7f62b0069880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.100+0000 7f62b7fac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0102640 0x7f62b019f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b7fac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62b010c710 con 0x7f62b0101c80 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b7fac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62b010c590 con 0x7f62b0100a90 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b7fac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62b010c890 con 0x7f62b0102640 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49942/0 (socket says 172.21.15.28:49942) 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 -- 172.21.15.28:0/2645452281 learned_addr learned my addr 172.21.15.28:0/2645452281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5d48640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0100a90 0x7f62b0069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b6549640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0102640 0x7f62b019f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 -- 172.21.15.28:0/2645452281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0102640 msgr2=0x7f62b019f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0102640 0x7f62b019f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 -- 172.21.15.28:0/2645452281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0100a90 msgr2=0x7f62b0069880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0100a90 0x7f62b0069880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 -- 172.21.15.28:0/2645452281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b019fab0 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b5547640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a000cd80 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a0010910 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b019fda0 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0018990 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.101+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b01a0260 con 0x7f62b0101c80 2024-03-22T02:11:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.102+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b0107db0 con 0x7f62b0101c80 2024-03-22T02:11:53.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.103+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f62a0010ac0 con 0x7f62b0101c80 2024-03-22T02:11:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.104+0000 7f62a6ffd640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 0x7f629c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.104+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f62a00cfa20 con 0x7f62b0101c80 2024-03-22T02:11:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.104+0000 7f62b5d48640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 0x7f629c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.104+0000 7f62b5d48640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 0x7f629c0a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b0101ae0 tx=0x7f62ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:53.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.109+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62a009c5b0 con 0x7f62b0101c80 2024-03-22T02:11:53.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:52 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T02:11:53.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:52 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:53.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:52 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-22T02:11:53.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:52 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T02:11:53.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:52 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:53.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:52 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-22T02:11:53.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/429924656' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-22T02:11:53.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:52 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:53.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-22T02:11:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.333+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f62b00628d0 con 0x7f62b0101c80 2024-03-22T02:11:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.974+0000 7f62a6ffd640 1 -- 172.21.15.28:0/2645452281 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f62a009beb0 con 0x7f62b0101c80 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 msgr2=0x7f629c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 0x7f629c0a53e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b0101ae0 tx=0x7f62ac005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 msgr2=0x7f62b019cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 shutdown_connections 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f629c0a2f20 0x7f629c0a53e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62b0102640 0x7f62b019f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b0101c80 0x7f62b019cbb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 --2- 172.21.15.28:0/2645452281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62b0100a90 0x7f62b0069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 >> 172.21.15.28:0/2645452281 conn(0x7f62b00fc000 msgr2=0x7f62b0104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 shutdown_connections 2024-03-22T02:11:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:53.977+0000 7f62b7fac640 1 -- 172.21.15.28:0/2645452281 wait complete. 2024-03-22T02:11:53.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-22T02:11:54.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 -- 172.21.15.28:0/3422288213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228101bc0 msgr2=0x7ff228101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:54.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 --2- 172.21.15.28:0/3422288213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228101bc0 0x7ff228101fe0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff220009900 tx=0x7ff22002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 -- 172.21.15.28:0/3422288213 shutdown_connections 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 --2- 172.21.15.28:0/3422288213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228103770 0x7ff22810a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 --2- 172.21.15.28:0/3422288213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 0x7ff228103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.152+0000 7ff22d583640 1 --2- 172.21.15.28:0/3422288213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228101bc0 0x7ff228101fe0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- 172.21.15.28:0/3422288213 >> 172.21.15.28:0/3422288213 conn(0x7ff2280fd130 msgr2=0x7ff2280ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- 172.21.15.28:0/3422288213 shutdown_connections 2024-03-22T02:11:54.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- 172.21.15.28:0/3422288213 wait complete. 2024-03-22T02:11:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 Processor -- start 2024-03-22T02:11:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- start start 2024-03-22T02:11:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 0x7ff228075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 0x7ff228073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff22810d830 con 0x7ff228103770 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff22810d6b0 con 0x7ff228101bc0 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.153+0000 7ff22d583640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff22810d9b0 con 0x7ff228102db0 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49954/0 (socket says 172.21.15.28:49954) 2024-03-22T02:11:54.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 -- 172.21.15.28:0/3352895298 learned_addr learned my addr 172.21.15.28:0/3352895298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff226ffd640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 0x7ff228075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2267fc640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 0x7ff228073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 -- 172.21.15.28:0/3352895298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 msgr2=0x7ff228073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 0x7ff228073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 -- 172.21.15.28:0/3352895298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 msgr2=0x7ff228075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 0x7ff228075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 -- 172.21.15.28:0/3352895298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff228074d50 con 0x7ff228103770 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff226ffd640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 0x7ff228075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff2277fe640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff21800efd0 tx=0x7ff21800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218003b70 con 0x7ff228103770 2024-03-22T02:11:54.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2281a1ff0 con 0x7ff228103770 2024-03-22T02:11:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff218003d10 con 0x7ff228103770 2024-03-22T02:11:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2180109f0 con 0x7ff228103770 2024-03-22T02:11:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.154+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2281a2460 con 0x7ff228103770 2024-03-22T02:11:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.155+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff228108fb0 con 0x7ff228103770 2024-03-22T02:11:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.157+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7ff218010b90 con 0x7ff228103770 2024-03-22T02:11:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.157+0000 7ff21ffff640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 0x7ff2140a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.158+0000 7ff226ffd640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 0x7ff2140a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:54.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.158+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff2180cfec0 con 0x7ff228103770 2024-03-22T02:11:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.158+0000 7ff226ffd640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 0x7ff2140a53e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff220000c00 tx=0x7ff220005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.161+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff21809ca50 con 0x7ff228103770 2024-03-22T02:11:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T02:11:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:11:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-22T02:11:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-22T02:11:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:54.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-22T02:11:54.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-22T02:11:54.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T02:11:54.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-22T02:11:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T02:11:54.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-22T02:11:54.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-22T02:11:54.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4183855509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-22T02:11:54.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-22T02:11:54.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-22T02:11:54.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-22T02:11:54.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-22T02:11:54.269 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-22T02:11:54.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.385+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7ff22802d3b0 con 0x7ff228103770 2024-03-22T02:11:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.983+0000 7ff21ffff640 1 -- 172.21.15.28:0/3352895298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7ff21809c350 con 0x7ff228103770 2024-03-22T02:11:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 msgr2=0x7ff2140a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 0x7ff2140a53e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff220000c00 tx=0x7ff220005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 msgr2=0x7ff2280744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff21800efd0 tx=0x7ff21800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 shutdown_connections 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2140a2f20 0x7ff2140a53e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff228103770 0x7ff2280744c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff228102db0 0x7ff228073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 --2- 172.21.15.28:0/3352895298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff228101bc0 0x7ff228075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 >> 172.21.15.28:0/3352895298 conn(0x7ff2280fd130 msgr2=0x7ff228106180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 shutdown_connections 2024-03-22T02:11:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:54.986+0000 7ff22d583640 1 -- 172.21.15.28:0/3352895298 wait complete. 2024-03-22T02:11:55.001 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-22T02:11:55.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-22T02:11:55.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1724838492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58100a90 msgr2=0x7f7d58100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1724838492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58100a90 0x7f7d58100eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009900 tx=0x7f7d4802bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1724838492 shutdown_connections 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1724838492 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58102640 0x7f7d58108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1724838492 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58101c80 0x7f7d58102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1724838492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58100a90 0x7f7d58100eb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1724838492 >> 172.21.15.28:0/1724838492 conn(0x7f7d580fc020 msgr2=0x7f7d580fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1724838492 shutdown_connections 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.113+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1724838492 wait complete. 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 Processor -- start 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 -- start start 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58100a90 0x7f7d58197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58101c80 0x7f7d581983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d830 con 0x7f7d58102640 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d6b0 con 0x7f7d58101c80 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d5d39a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810d9b0 con 0x7f7d58100a90 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49974/0 (socket says 172.21.15.28:49974) 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d577fe640 1 -- 172.21.15.28:0/1988574205 learned_addr learned my addr 172.21.15.28:0/1988574205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d56ffd640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58100a90 0x7f7d58197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.114+0000 7f7d567fc640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58101c80 0x7f7d581983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 -- 172.21.15.28:0/1988574205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58100a90 msgr2=0x7f7d58197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58100a90 0x7f7d58197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 -- 172.21.15.28:0/1988574205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58101c80 msgr2=0x7f7d581983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58101c80 0x7f7d581983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 -- 172.21.15.28:0/1988574205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5819fad0 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d577fe640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000eb10 tx=0x7f7d5000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50004b20 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d50004cc0 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5819fdc0 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50010760 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.115+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d581a0280 con 0x7f7d58102640 2024-03-22T02:11:55.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.116+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d58107db0 con 0x7f7d58102640 2024-03-22T02:11:55.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.117+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f7d50005020 con 0x7f7d58102640 2024-03-22T02:11:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.117+0000 7f7d3ffff640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 0x7f7d400a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.118+0000 7f7d56ffd640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 0x7f7d400a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.118+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7d500cfb20 con 0x7f7d58102640 2024-03-22T02:11:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.118+0000 7f7d56ffd640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 0x7f7d400a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7d48000c00 tx=0x7f7d48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.122+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d5009c710 con 0x7f7d58102640 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:active 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: Cluster is now healthy 2024-03-22T02:11:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-22T02:11:55.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-22T02:11:55.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-22T02:11:55.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T02:11:55.264 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: 2024-03-22T02:11:54.835+0000 7f8161474700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-22T02:11:55.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:active 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: Cluster is now healthy 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-22T02:11:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T02:11:55.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2645452281' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-22T02:11:55.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] up:active 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: Cluster is now healthy 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-22T02:11:55.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-22T02:11:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.330+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7d58061fd0 con 0x7f7d400a2fa0 2024-03-22T02:11:55.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.354+0000 7f7d3ffff640 1 -- 172.21.15.28:0/1988574205 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7d58061fd0 con 0x7f7d400a2fa0 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 msgr2=0x7f7d400a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 0x7f7d400a5460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7d48000c00 tx=0x7f7d48005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 msgr2=0x7f7d5819f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000eb10 tx=0x7f7d5000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 shutdown_connections 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7d400a2fa0 0x7f7d400a5460 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d58102640 0x7f7d5819f460 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7d58101c80 0x7f7d581983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 --2- 172.21.15.28:0/1988574205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7d58100a90 0x7f7d58197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.356+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 >> 172.21.15.28:0/1988574205 conn(0x7f7d580fc020 msgr2=0x7f7d58104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:55.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.357+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 shutdown_connections 2024-03-22T02:11:55.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.357+0000 7f7d5d39a640 1 -- 172.21.15.28:0/1988574205 wait complete. 2024-03-22T02:11:55.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-22T02:11:55.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.475+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3072688288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 msgr2=0x7fa148100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.475+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3072688288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa148100eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa138009860 tx=0x7fa13802bb70 comp rx=0 tx=0).stop 2024-03-22T02:11:55.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3072688288 shutdown_connections 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3072688288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 0x7fa148108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3072688288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3072688288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa148100eb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3072688288 >> 172.21.15.28:0/3072688288 conn(0x7fa1480fc020 msgr2=0x7fa1480fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3072688288 shutdown_connections 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3072688288 wait complete. 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 Processor -- start 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- start start 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa14810b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 0x7fa148109f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d830 con 0x7fa148101c80 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d6b0 con 0x7fa148100a90 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.476+0000 7fa14ca2a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d9b0 con 0x7fa148102640 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa14659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa14810b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50000/0 (socket says 172.21.15.28:50000) 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 -- 172.21.15.28:0/3352121736 learned_addr learned my addr 172.21.15.28:0/3352121736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa146d9d640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 0x7fa148109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 -- 172.21.15.28:0/3352121736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 msgr2=0x7fa148109f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 0x7fa148109f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 -- 172.21.15.28:0/3352121736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 msgr2=0x7fa14810b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa14810b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 -- 172.21.15.28:0/3352121736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14810a820 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa14659c640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa14810b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa145d9b640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e9c0 tx=0x7fa13c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c00ccc0 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c005500 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1481a1fb0 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c010430 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.477+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1481a2470 con 0x7fa148101c80 2024-03-22T02:11:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.478+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa148107e30 con 0x7fa148101c80 2024-03-22T02:11:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.479+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fa13c0046c0 con 0x7fa148101c80 2024-03-22T02:11:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.480+0000 7fa1377fe640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 0x7fa1300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.480+0000 7fa14659c640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 0x7fa1300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.480+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fa13c0cf990 con 0x7fa148101c80 2024-03-22T02:11:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.480+0000 7fa14659c640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 0x7fa1300a53e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa138009830 tx=0x7fa13802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.484+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa13c09c520 con 0x7fa148101c80 2024-03-22T02:11:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.695+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa148062060 con 0x7fa1300a2f20 2024-03-22T02:11:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.721+0000 7fa1377fe640 1 -- 172.21.15.28:0/3352121736 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa148062060 con 0x7fa1300a2f20 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 msgr2=0x7fa1300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 0x7fa1300a53e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa138009830 tx=0x7fa13802e040 comp rx=0 tx=0).stop 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 msgr2=0x7fa148109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fa13c00e9c0 tx=0x7fa13c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 shutdown_connections 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1300a2f20 0x7fa1300a53e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa148102640 0x7fa148109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa148101c80 0x7fa148109a50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 --2- 172.21.15.28:0/3352121736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa148100a90 0x7fa14810b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.723+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 >> 172.21.15.28:0/3352121736 conn(0x7fa1480fc020 msgr2=0x7fa148105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.724+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 shutdown_connections 2024-03-22T02:11:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.724+0000 7fa14ca2a640 1 -- 172.21.15.28:0/3352121736 wait complete. 2024-03-22T02:11:55.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:11:55.335+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:11:55.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:11:55.335+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:11:55.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:11:55.335+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:11:55.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:11:55.335+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:11:55.734 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:11:55.335+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:11:55.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-22T02:11:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- 172.21.15.28:0/3970276663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 msgr2=0x7f448010a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 --2- 172.21.15.28:0/3970276663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f448010a0b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4478009cd0 tx=0x7f447802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- 172.21.15.28:0/3970276663 shutdown_connections 2024-03-22T02:11:55.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 --2- 172.21.15.28:0/3970276663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f448010a0b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 --2- 172.21.15.28:0/3970276663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 0x7f44801032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 --2- 172.21.15.28:0/3970276663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- 172.21.15.28:0/3970276663 >> 172.21.15.28:0/3970276663 conn(0x7f44800fb7c0 msgr2=0x7f44800fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- 172.21.15.28:0/3970276663 shutdown_connections 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- 172.21.15.28:0/3970276663 wait complete. 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 Processor -- start 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.843+0000 7f4487b25640 1 -- start start 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 0x7f4480100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f4480101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f448010d870 con 0x7f44800ffe90 2024-03-22T02:11:55.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f448010d6f0 con 0x7f4480102e60 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f448010d9f0 con 0x7f4480103820 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50010/0 (socket says 172.21.15.28:50010) 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 -- 172.21.15.28:0/3539217041 learned_addr learned my addr 172.21.15.28:0/3539217041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44860c2640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f4480101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44850c0640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 0x7f4480100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 -- 172.21.15.28:0/3539217041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 msgr2=0x7f4480101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f4480101350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 -- 172.21.15.28:0/3539217041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 msgr2=0x7f4480100e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 0x7f4480100e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 -- 172.21.15.28:0/3539217041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480101be0 con 0x7f44800ffe90 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f44858c1640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f447c00e9c0 tx=0x7f447c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c00cd80 con 0x7f44800ffe90 2024-03-22T02:11:55.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447c0035f0 con 0x7f44800ffe90 2024-03-22T02:11:55.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44801a1fb0 con 0x7f44800ffe90 2024-03-22T02:11:55.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c010690 con 0x7f44800ffe90 2024-03-22T02:11:55.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.844+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44801a23c0 con 0x7f44800ffe90 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.845+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4480108f30 con 0x7f44800ffe90 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.846+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f447c0037a0 con 0x7f44800ffe90 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.847+0000 7f4476ffd640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 0x7f446c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.847+0000 7f44850c0640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 0x7f446c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.847+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f447c0cf9b0 con 0x7f44800ffe90 2024-03-22T02:11:55.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.847+0000 7f44850c0640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 0x7f446c0a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4480102540 tx=0x7f4470005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:55.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:55.851+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f447c09c540 con 0x7f44800ffe90 2024-03-22T02:11:56.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.068+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f4480062060 con 0x7f446c0a2f20 2024-03-22T02:11:56.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.074+0000 7f4476ffd640 1 -- 172.21.15.28:0/3539217041 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4480062060 con 0x7f446c0a2f20 2024-03-22T02:11:56.074 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/998c68fe-61a0-43f2-8132-915abab5cf45 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 msgr2=0x7f446c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 0x7f446c0a53e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4480102540 tx=0x7f4470005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 msgr2=0x7f44801027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f447c00e9c0 tx=0x7f447c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 shutdown_connections 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f446c0a2f20 0x7f446c0a53e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4480103820 0x7f4480101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4480102e60 0x7f4480100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 --2- 172.21.15.28:0/3539217041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44800ffe90 0x7f44801027d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 >> 172.21.15.28:0/3539217041 conn(0x7f44800fb7c0 msgr2=0x7f4480106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 shutdown_connections 2024-03-22T02:11:56.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.076+0000 7f4487b25640 1 -- 172.21.15.28:0/3539217041 wait complete. 2024-03-22T02:11:56.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-22T02:11:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 -- 172.21.15.28:0/778178434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c102620 msgr2=0x7fee6c104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 --2- 172.21.15.28:0/778178434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c102620 0x7fee6c104a10 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009900 tx=0x7fee5c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 -- 172.21.15.28:0/778178434 shutdown_connections 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 --2- 172.21.15.28:0/778178434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c107880 0x7fee6c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 --2- 172.21.15.28:0/778178434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 0x7fee6c107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 --2- 172.21.15.28:0/778178434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c102620 0x7fee6c104a10 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 -- 172.21.15.28:0/778178434 >> 172.21.15.28:0/778178434 conn(0x7fee6c0fc030 msgr2=0x7fee6c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 -- 172.21.15.28:0/778178434 shutdown_connections 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 -- 172.21.15.28:0/778178434 wait complete. 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.195+0000 7fee71fd6640 1 Processor -- start 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 -- start start 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c102620 0x7fee6c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 0x7fee6c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10d850 con 0x7fee6c107880 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10d6d0 con 0x7fee6c102620 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee71fd6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10d9d0 con 0x7fee6c104f50 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50024/0 (socket says 172.21.15.28:50024) 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 -- 172.21.15.28:0/3895888527 learned_addr learned my addr 172.21.15.28:0/3895888527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 0x7fee6c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6b7fe640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c102620 0x7fee6c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 -- 172.21.15.28:0/3895888527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 msgr2=0x7fee6c198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 0x7fee6c198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 -- 172.21.15.28:0/3895888527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c102620 msgr2=0x7fee6c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c102620 0x7fee6c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 -- 172.21.15.28:0/3895888527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c19fb70 con 0x7fee6c107880 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee6bfff640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee6400eb10 tx=0x7fee6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.196+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64004b20 con 0x7fee6c107880 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.197+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee64004cc0 con 0x7fee6c107880 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.197+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c19fe60 con 0x7fee6c107880 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.197+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64010760 con 0x7fee6c107880 2024-03-22T02:11:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.197+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c1a0320 con 0x7fee6c107880 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.198+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6c0fdb20 con 0x7fee6c107880 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.199+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fee64005020 con 0x7fee6c107880 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.199+0000 7fee68ff9640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 0x7fee540a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.200+0000 7fee6b7fe640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 0x7fee540a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.200+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fee640cfb20 con 0x7fee6c107880 2024-03-22T02:11:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.200+0000 7fee6b7fe640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 0x7fee540a5460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fee5c02c7b0 tx=0x7fee5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.204+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee6409c710 con 0x7fee6c107880 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:11:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:55 smithi120 ceph-mon[36803]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-22T02:11:56.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T02:11:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:11:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-22T02:11:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:11:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:55 smithi083 ceph-mon[36340]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: pgmap v304: 129 pgs: 129 active+clean; 582 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 3 op/s 2024-03-22T02:11:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3352895298' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-22T02:11:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:11:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-22T02:11:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:11:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:55 smithi028 ceph-mon[31916]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:56.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.414+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fee6c061fd0 con 0x7fee540a2fa0 2024-03-22T02:11:56.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.439+0000 7fee68ff9640 1 -- 172.21.15.28:0/3895888527 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fee6c061fd0 con 0x7fee540a2fa0 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.440+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 msgr2=0x7fee540a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.440+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 0x7fee540a5460 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fee5c02c7b0 tx=0x7fee5c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.440+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 msgr2=0x7fee6c19f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.440+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee6400eb10 tx=0x7fee6400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 shutdown_connections 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee540a2fa0 0x7fee540a5460 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee6c107880 0x7fee6c19f500 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee6c104f50 0x7fee6c198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 --2- 172.21.15.28:0/3895888527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee6c102620 0x7fee6c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 >> 172.21.15.28:0/3895888527 conn(0x7fee6c0fc030 msgr2=0x7fee6c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 shutdown_connections 2024-03-22T02:11:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.441+0000 7fee71fd6640 1 -- 172.21.15.28:0/3895888527 wait complete. 2024-03-22T02:11:56.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-22T02:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/2142577001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 msgr2=0x7f0cf4103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/2142577001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4103230 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009900 tx=0x7f0ce402bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/2142577001 shutdown_connections 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/2142577001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf410a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/2142577001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4103230 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/2142577001 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 0x7f0cf4101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.559+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/2142577001 >> 172.21.15.28:0/2142577001 conn(0x7f0cf40fd130 msgr2=0x7f0cf40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/2142577001 shutdown_connections 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/2142577001 wait complete. 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 Processor -- start 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- start start 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 0x7f0cf4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf410d820 con 0x7f0cf4102db0 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf410d6a0 con 0x7f0cf4101bc0 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cfc0c2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf410d9a0 con 0x7f0cf4103770 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cf965d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cf965d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50048/0 (socket says 172.21.15.28:50048) 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cf965d640 1 -- 172.21.15.28:0/3809876456 learned_addr learned my addr 172.21.15.28:0/3809876456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.560+0000 7f0cf9e5e640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 0x7f0cf4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cfa65f640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 -- 172.21.15.28:0/3809876456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 msgr2=0x7f0cf419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 -- 172.21.15.28:0/3809876456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 msgr2=0x7f0cf4197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 0x7f0cf4197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 -- 172.21.15.28:0/3809876456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf419fb10 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cfa65f640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cf965d640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009230 tx=0x7f0ce4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4041070 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ce40044d0 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf419fda0 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4034440 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.561+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf41a0230 con 0x7f0cf4102db0 2024-03-22T02:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.562+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cf4108f80 con 0x7f0cf4102db0 2024-03-22T02:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.563+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f0ce4044070 con 0x7f0cf4102db0 2024-03-22T02:11:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.564+0000 7f0ceaffd640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 0x7f0ce00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.564+0000 7f0cf9e5e640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 0x7f0ce00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.564+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f0ce40f1310 con 0x7f0cf4102db0 2024-03-22T02:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.564+0000 7f0cf9e5e640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 0x7f0ce00a53e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4102c10 tx=0x7f0cf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.568+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ce40bdea0 con 0x7f0cf4102db0 2024-03-22T02:11:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.778+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f0cf4062060 con 0x7f0ce00a2f20 2024-03-22T02:11:56.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.783+0000 7f0ceaffd640 1 -- 172.21.15.28:0/3809876456 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0cf4062060 con 0x7f0ce00a2f20 2024-03-22T02:11:56.783 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 2024-03-22T02:11:56.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 msgr2=0x7f0ce00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 0x7f0ce00a53e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4102c10 tx=0x7f0cf0005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 msgr2=0x7f0cf4198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009230 tx=0x7f0ce4005d50 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 shutdown_connections 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ce00a2f20 0x7f0ce00a53e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0cf4103770 0x7f0cf419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cf4102db0 0x7f0cf4198430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 --2- 172.21.15.28:0/3809876456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0cf4101bc0 0x7f0cf4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 >> 172.21.15.28:0/3809876456 conn(0x7f0cf40fd130 msgr2=0x7f0cf4106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 shutdown_connections 2024-03-22T02:11:56.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.785+0000 7f0cfc0c2640 1 -- 172.21.15.28:0/3809876456 wait complete. 2024-03-22T02:11:56.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 -- 172.21.15.28:0/2710025943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630107890 msgr2=0x7f9630109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 --2- 172.21.15.28:0/2710025943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630107890 0x7f9630109ce0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9620009ca0 tx=0x7f962002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 -- 172.21.15.28:0/2710025943 shutdown_connections 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 --2- 172.21.15.28:0/2710025943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630107890 0x7f9630109ce0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 --2- 172.21.15.28:0/2710025943 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 0x7f9630107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 --2- 172.21.15.28:0/2710025943 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630102630 0x7f9630104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 -- 172.21.15.28:0/2710025943 >> 172.21.15.28:0/2710025943 conn(0x7f96300fc020 msgr2=0x7f96300fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:56.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 -- 172.21.15.28:0/2710025943 shutdown_connections 2024-03-22T02:11:56.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.899+0000 7f9635637640 1 -- 172.21.15.28:0/2710025943 wait complete. 2024-03-22T02:11:56.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 Processor -- start 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 -- start start 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 0x7f96301983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 0x7f963019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f963010d850 con 0x7f9630102630 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f963010d6d0 con 0x7f9630104f60 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.900+0000 7f9635637640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f963010d9d0 con 0x7f9630107890 2024-03-22T02:11:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962e7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 0x7f96301983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 0x7f963019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50066/0 (socket says 172.21.15.28:50066) 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962effd640 1 -- 172.21.15.28:0/513118119 learned_addr learned my addr 172.21.15.28:0/513118119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 -- 172.21.15.28:0/513118119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 msgr2=0x7f96301983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 0x7f96301983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 -- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 msgr2=0x7f9630197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 -- 172.21.15.28:0/513118119 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963019fa80 con 0x7f9630107890 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962effd640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f962f7fe640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 0x7f963019f410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f962002c630 tx=0x7f9620032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9620049070 con 0x7f9630107890 2024-03-22T02:11:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9630100420 con 0x7f9630107890 2024-03-22T02:11:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9620003970 con 0x7f9630107890 2024-03-22T02:11:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.901+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96200044a0 con 0x7f9630107890 2024-03-22T02:11:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.902+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9630100930 con 0x7f9630107890 2024-03-22T02:11:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.903+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96300fdb10 con 0x7f9630107890 2024-03-22T02:11:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.904+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f962004c070 con 0x7f9630107890 2024-03-22T02:11:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.904+0000 7f9617fff640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 0x7f96180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.904+0000 7f962effd640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 0x7f96180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.904+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f96200f4fd0 con 0x7f9630107890 2024-03-22T02:11:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.905+0000 7f962effd640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 0x7f96180a53e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9628011500 tx=0x7f9628005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:56.912+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9620041070 con 0x7f9630107890 2024-03-22T02:11:57.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:56 smithi028 ceph-mon[31916]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:56 smithi028 ceph-mon[31916]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:56 smithi028 ceph-mon[31916]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.133+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9630101d10 con 0x7f9630107890 2024-03-22T02:11:57.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.134+0000 7f9617fff640 1 -- 172.21.15.28:0/513118119 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7f96200c1600 con 0x7f9630107890 2024-03-22T02:11:57.135 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:57.135 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:11:57.135 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 msgr2=0x7f96180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 0x7f96180a53e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9628011500 tx=0x7f9628005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 msgr2=0x7f963019f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 0x7f963019f410 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f962002c630 tx=0x7f9620032f20 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 shutdown_connections 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a2f20 0x7f96180a53e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9630107890 0x7f963019f410 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9630104f60 0x7f96301983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 --2- 172.21.15.28:0/513118119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9630102630 0x7f9630197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 >> 172.21.15.28:0/513118119 conn(0x7f96300fc020 msgr2=0x7f9630102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 shutdown_connections 2024-03-22T02:11:57.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.136+0000 7f9635637640 1 -- 172.21.15.28:0/513118119 wait complete. 2024-03-22T02:11:57.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:11:57.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:56 smithi120 ceph-mon[36803]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:56 smithi120 ceph-mon[36803]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:56 smithi120 ceph-mon[36803]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 -- 172.21.15.28:0/1083476700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 msgr2=0x7f08cc103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 --2- 172.21.15.28:0/1083476700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc103220 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009900 tx=0x7f08c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 -- 172.21.15.28:0/1083476700 shutdown_connections 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 --2- 172.21.15.28:0/1083476700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 --2- 172.21.15.28:0/1083476700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc103220 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 --2- 172.21.15.28:0/1083476700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 0x7f08cc101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 -- 172.21.15.28:0/1083476700 >> 172.21.15.28:0/1083476700 conn(0x7f08cc0fd140 msgr2=0x7f08cc0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 -- 172.21.15.28:0/1083476700 shutdown_connections 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 -- 172.21.15.28:0/1083476700 wait complete. 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.251+0000 7f08d3e39640 1 Processor -- start 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 -- start start 2024-03-22T02:11:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 0x7f08cc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc10d830 con 0x7f08cc102da0 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc10d6b0 con 0x7f08cc103760 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc10d9b0 con 0x7f08cc101bb0 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50074/0 (socket says 172.21.15.28:50074) 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 -- 172.21.15.28:0/941152186 learned_addr learned my addr 172.21.15.28:0/941152186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d1bd5640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 0x7f08cc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d23d6640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 -- 172.21.15.28:0/941152186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 msgr2=0x7f08cc197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 0x7f08cc197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 -- 172.21.15.28:0/941152186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 msgr2=0x7f08cc19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 -- 172.21.15.28:0/941152186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc19faf0 con 0x7f08cc102da0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d23d6640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d13d4640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009230 tx=0x7f08c002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0041070 con 0x7f08cc102da0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c0004450 con 0x7f08cc102da0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08cc19fd80 con 0x7f08cc102da0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.252+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0003e20 con 0x7f08cc102da0 2024-03-22T02:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.253+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08cc1a0260 con 0x7f08cc102da0 2024-03-22T02:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.254+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08cc108ed0 con 0x7f08cc102da0 2024-03-22T02:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.255+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f08c0044070 con 0x7f08cc102da0 2024-03-22T02:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.255+0000 7f08caffd640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 0x7f08b40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.255+0000 7f08d1bd5640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 0x7f08b40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.255+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f08c00f1200 con 0x7f08cc102da0 2024-03-22T02:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.256+0000 7f08d1bd5640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 0x7f08b40a5460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08cc102c00 tx=0x7f08c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.260+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f08c00bdd90 con 0x7f08cc102da0 2024-03-22T02:11:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:56 smithi083 ceph-mon[36340]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:56 smithi083 ceph-mon[36340]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:56 smithi083 ceph-mon[36340]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:57.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.443+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08cc062830 con 0x7f08cc102da0 2024-03-22T02:11:57.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.444+0000 7f08caffd640 1 -- 172.21.15.28:0/941152186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f08c00bd690 con 0x7f08cc102da0 2024-03-22T02:11:57.445 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:57.445 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 msgr2=0x7f08b40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 0x7f08b40a5460 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f08cc102c00 tx=0x7f08c4009500 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 msgr2=0x7f08cc198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009230 tx=0x7f08c002c650 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 shutdown_connections 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08b40a2fa0 0x7f08b40a5460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08cc103760 0x7f08cc19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08cc102da0 0x7f08cc198410 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 --2- 172.21.15.28:0/941152186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08cc101bb0 0x7f08cc197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 >> 172.21.15.28:0/941152186 conn(0x7f08cc0fd140 msgr2=0x7f08cc1060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 shutdown_connections 2024-03-22T02:11:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.446+0000 7f08d3e39640 1 -- 172.21.15.28:0/941152186 wait complete. 2024-03-22T02:11:57.458 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-22T02:11:57.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- 172.21.15.28:0/1927790647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c101cc0 msgr2=0x7f949c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 --2- 172.21.15.28:0/1927790647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c101cc0 0x7f949c102140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f948c009900 tx=0x7f948c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- 172.21.15.28:0/1927790647 shutdown_connections 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 --2- 172.21.15.28:0/1927790647 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c102680 0x7f949c108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 --2- 172.21.15.28:0/1927790647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c101cc0 0x7f949c102140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 --2- 172.21.15.28:0/1927790647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c100ad0 0x7f949c100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- 172.21.15.28:0/1927790647 >> 172.21.15.28:0/1927790647 conn(0x7f949c0fc040 msgr2=0x7f949c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- 172.21.15.28:0/1927790647 shutdown_connections 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- 172.21.15.28:0/1927790647 wait complete. 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 Processor -- start 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.561+0000 7f94a47d6640 1 -- start start 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c100ad0 0x7f949c06ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c101cc0 0x7f949c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c730 con 0x7f949c102680 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c5b0 con 0x7f949c100ad0 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f949c10c8b0 con 0x7f949c101cc0 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50096/0 (socket says 172.21.15.28:50096) 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 -- 172.21.15.28:0/3978142562 learned_addr learned my addr 172.21.15.28:0/3978142562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:57.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a1d71640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c101cc0 0x7f949c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2572640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c100ad0 0x7f949c06ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 -- 172.21.15.28:0/3978142562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c101cc0 msgr2=0x7f949c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c101cc0 0x7f949c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 -- 172.21.15.28:0/3978142562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c100ad0 msgr2=0x7f949c06ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c100ad0 0x7f949c06ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 -- 172.21.15.28:0/3978142562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949c19fb00 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a2d73640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f949400eb10 tx=0x7f949400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494004b20 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9494004cc0 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494010760 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f949c19fdf0 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.562+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f949c1a02c0 con 0x7f949c102680 2024-03-22T02:11:57.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.563+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f949c068dd0 con 0x7f949c102680 2024-03-22T02:11:57.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.565+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7f9494005020 con 0x7f949c102680 2024-03-22T02:11:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.565+0000 7f94937fe640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 0x7f94880a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.565+0000 7f94a2572640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 0x7f94880a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.565+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f94940cfb70 con 0x7f949c102680 2024-03-22T02:11:57.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.566+0000 7f94a2572640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 0x7f94880a53e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f94980095f0 tx=0x7f9498005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.570+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f949409c760 con 0x7f949c102680 2024-03-22T02:11:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.752+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f949c061fd0 con 0x7f94880a2f20 2024-03-22T02:11:57.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.762+0000 7f94937fe640 1 -- 172.21.15.28:0/3978142562 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340572 (secure 0 0 0) 0x7f949c061fd0 con 0x7f94880a2f20 2024-03-22T02:11:57.767 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:57.769 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 msgr2=0x7f94880a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 0x7f94880a53e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f94980095f0 tx=0x7f9498005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 msgr2=0x7f949c19f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f949400eb10 tx=0x7f949400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 shutdown_connections 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94880a2f20 0x7f94880a53e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f949c102680 0x7f949c19f240 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f949c101cc0 0x7f949c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 --2- 172.21.15.28:0/3978142562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f949c100ad0 0x7f949c06ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.765+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 >> 172.21.15.28:0/3978142562 conn(0x7f949c0fc040 msgr2=0x7f949c104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.766+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 shutdown_connections 2024-03-22T02:11:57.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.766+0000 7f94a47d6640 1 -- 172.21.15.28:0/3978142562 wait complete. 2024-03-22T02:11:57.777 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-22T02:11:56.555121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615825,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":131,"num_read_kb":116,"num_write":295,"num_write_kb":2999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365044,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576780,"statfs":{"total":1151940427776,"available":1151566622720,"internally_reserved":0,"allocated":11976704,"data_stored":7552490,"data_compressed":284580,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23047,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":69,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003458"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854847+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:11:39.854847+0000","last_peered":"2024-03-22T02:11:39.854847+0000","last_clean":"2024-03-22T02:11:39.854847+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:11:39.854847+0000","last_undegraded":"2024-03-22T02:11:39.854847+0000","last_fullsized":"2024-03-22T02:11:39.854847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918858+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:11:39.918858+0000","last_peered":"2024-03-22T02:11:39.918858+0000","last_clean":"2024-03-22T02:11:39.918858+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:11:39.918858+0000","last_undegraded":"2024-03-22T02:11:39.918858+0000","last_fullsized":"2024-03-22T02:11:39.918858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918743+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:11:39.918743+0000","last_peered":"2024-03-22T02:11:39.918743+0000","last_clean":"2024-03-22T02:11:39.918743+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:11:39.918743+0000","last_undegraded":"2024-03-22T02:11:39.918743+0000","last_fullsized":"2024-03-22T02:11:39.918743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152289+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:11:40.152289+0000","last_peered":"2024-03-22T02:11:40.152289+0000","last_clean":"2024-03-22T02:11:40.152289+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:11:40.152289+0000","last_undegraded":"2024-03-22T02:11:40.152289+0000","last_fullsized":"2024-03-22T02:11:40.152289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853871+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:11:39.853871+0000","last_peered":"2024-03-22T02:11:39.853871+0000","last_clean":"2024-03-22T02:11:39.853871+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:11:39.853871+0000","last_undegraded":"2024-03-22T02:11:39.853871+0000","last_fullsized":"2024-03-22T02:11:39.853871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853000+0000","last_change":"2024-03-22T02:11:33.630093+0000","last_active":"2024-03-22T02:11:39.853000+0000","last_peered":"2024-03-22T02:11:39.853000+0000","last_clean":"2024-03-22T02:11:39.853000+0000","last_became_active":"2024-03-22T02:11:33.629784+0000","last_became_peered":"2024-03-22T02:11:33.629784+0000","last_unstale":"2024-03-22T02:11:39.853000+0000","last_undegraded":"2024-03-22T02:11:39.853000+0000","last_fullsized":"2024-03-22T02:11:39.853000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:39.324939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933263+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:11:38.933263+0000","last_peered":"2024-03-22T02:11:38.933263+0000","last_clean":"2024-03-22T02:11:38.933263+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:11:38.933263+0000","last_undegraded":"2024-03-22T02:11:38.933263+0000","last_fullsized":"2024-03-22T02:11:38.933263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137726+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:11:39.137726+0000","last_peered":"2024-03-22T02:11:39.137726+0000","last_clean":"2024-03-22T02:11:39.137726+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:11:39.137726+0000","last_undegraded":"2024-03-22T02:11:39.137726+0000","last_fullsized":"2024-03-22T02:11:39.137726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638809+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:11:39.638809+0000","last_peered":"2024-03-22T02:11:39.638809+0000","last_clean":"2024-03-22T02:11:39.638809+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:11:39.638809+0000","last_undegraded":"2024-03-22T02:11:39.638809+0000","last_fullsized":"2024-03-22T02:11:39.638809+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853141+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:11:39.853141+0000","last_peered":"2024-03-22T02:11:39.853141+0000","last_clean":"2024-03-22T02:11:39.853141+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:11:39.853141+0000","last_undegraded":"2024-03-22T02:11:39.853141+0000","last_fullsized":"2024-03-22T02:11:39.853141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933829+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:11:38.933829+0000","last_peered":"2024-03-22T02:11:38.933829+0000","last_clean":"2024-03-22T02:11:38.933829+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:11:38.933829+0000","last_undegraded":"2024-03-22T02:11:38.933829+0000","last_fullsized":"2024-03-22T02:11:38.933829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152193+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:11:40.152193+0000","last_peered":"2024-03-22T02:11:40.152193+0000","last_clean":"2024-03-22T02:11:40.152193+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:11:40.152193+0000","last_undegraded":"2024-03-22T02:11:40.152193+0000","last_fullsized":"2024-03-22T02:11:40.152193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931289+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:11:38.931289+0000","last_peered":"2024-03-22T02:11:38.931289+0000","last_clean":"2024-03-22T02:11:38.931289+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:11:38.931289+0000","last_undegraded":"2024-03-22T02:11:38.931289+0000","last_fullsized":"2024-03-22T02:11:38.931289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918413+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:11:39.918413+0000","last_peered":"2024-03-22T02:11:39.918413+0000","last_clean":"2024-03-22T02:11:39.918413+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:11:39.918413+0000","last_undegraded":"2024-03-22T02:11:39.918413+0000","last_fullsized":"2024-03-22T02:11:39.918413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918482+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:11:39.918482+0000","last_peered":"2024-03-22T02:11:39.918482+0000","last_clean":"2024-03-22T02:11:39.918482+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:11:39.918482+0000","last_undegraded":"2024-03-22T02:11:39.918482+0000","last_fullsized":"2024-03-22T02:11:39.918482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638474+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:11:39.638474+0000","last_peered":"2024-03-22T02:11:39.638474+0000","last_clean":"2024-03-22T02:11:39.638474+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:11:39.638474+0000","last_undegraded":"2024-03-22T02:11:39.638474+0000","last_fullsized":"2024-03-22T02:11:39.638474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854728+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:11:39.854728+0000","last_peered":"2024-03-22T02:11:39.854728+0000","last_clean":"2024-03-22T02:11:39.854728+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:11:39.854728+0000","last_undegraded":"2024-03-22T02:11:39.854728+0000","last_fullsized":"2024-03-22T02:11:39.854728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974322+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:11:40.974322+0000","last_peered":"2024-03-22T02:11:40.974322+0000","last_clean":"2024-03-22T02:11:40.974322+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:11:40.974322+0000","last_undegraded":"2024-03-22T02:11:40.974322+0000","last_fullsized":"2024-03-22T02:11:40.974322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974097+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:11:40.974097+0000","last_peered":"2024-03-22T02:11:40.974097+0000","last_clean":"2024-03-22T02:11:40.974097+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:11:40.974097+0000","last_undegraded":"2024-03-22T02:11:40.974097+0000","last_fullsized":"2024-03-22T02:11:40.974097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974257+0000","last_change":"2024-03-22T02:11:37.944045+0000","last_active":"2024-03-22T02:11:40.974257+0000","last_peered":"2024-03-22T02:11:40.974257+0000","last_clean":"2024-03-22T02:11:40.974257+0000","last_became_active":"2024-03-22T02:11:37.943598+0000","last_became_peered":"2024-03-22T02:11:37.943598+0000","last_unstale":"2024-03-22T02:11:40.974257+0000","last_undegraded":"2024-03-22T02:11:40.974257+0000","last_fullsized":"2024-03-22T02:11:40.974257+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:46:43.350701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.991750+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:11:53.991750+0000","last_peered":"2024-03-22T02:11:53.991750+0000","last_clean":"2024-03-22T02:11:53.991750+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:11:53.991750+0000","last_undegraded":"2024-03-22T02:11:53.991750+0000","last_fullsized":"2024-03-22T02:11:53.991750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138425+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:11:39.138425+0000","last_peered":"2024-03-22T02:11:39.138425+0000","last_clean":"2024-03-22T02:11:39.138425+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:11:39.138425+0000","last_undegraded":"2024-03-22T02:11:39.138425+0000","last_fullsized":"2024-03-22T02:11:39.138425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.151921+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:11:40.151921+0000","last_peered":"2024-03-22T02:11:40.151921+0000","last_clean":"2024-03-22T02:11:40.151921+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:11:40.151921+0000","last_undegraded":"2024-03-22T02:11:40.151921+0000","last_fullsized":"2024-03-22T02:11:40.151921+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138625+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:11:39.138625+0000","last_peered":"2024-03-22T02:11:39.138625+0000","last_clean":"2024-03-22T02:11:39.138625+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:11:39.138625+0000","last_undegraded":"2024-03-22T02:11:39.138625+0000","last_fullsized":"2024-03-22T02:11:39.138625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918901+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:11:39.918901+0000","last_peered":"2024-03-22T02:11:39.918901+0000","last_clean":"2024-03-22T02:11:39.918901+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:11:39.918901+0000","last_undegraded":"2024-03-22T02:11:39.918901+0000","last_fullsized":"2024-03-22T02:11:39.918901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152035+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:11:40.152035+0000","last_peered":"2024-03-22T02:11:40.152035+0000","last_clean":"2024-03-22T02:11:40.152035+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:11:40.152035+0000","last_undegraded":"2024-03-22T02:11:40.152035+0000","last_fullsized":"2024-03-22T02:11:40.152035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989534+0000","last_change":"2024-03-22T02:11:32.917851+0000","last_active":"2024-03-22T02:11:53.989534+0000","last_peered":"2024-03-22T02:11:53.989534+0000","last_clean":"2024-03-22T02:11:53.989534+0000","last_became_active":"2024-03-22T02:11:32.917526+0000","last_became_peered":"2024-03-22T02:11:32.917526+0000","last_unstale":"2024-03-22T02:11:53.989534+0000","last_undegraded":"2024-03-22T02:11:53.989534+0000","last_fullsized":"2024-03-22T02:11:53.989534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:20.509077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638819+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:11:39.638819+0000","last_peered":"2024-03-22T02:11:39.638819+0000","last_clean":"2024-03-22T02:11:39.638819+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:11:39.638819+0000","last_undegraded":"2024-03-22T02:11:39.638819+0000","last_fullsized":"2024-03-22T02:11:39.638819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152119+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:11:40.152119+0000","last_peered":"2024-03-22T02:11:40.152119+0000","last_clean":"2024-03-22T02:11:40.152119+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:11:40.152119+0000","last_undegraded":"2024-03-22T02:11:40.152119+0000","last_fullsized":"2024-03-22T02:11:40.152119+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853959+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:11:39.853959+0000","last_peered":"2024-03-22T02:11:39.853959+0000","last_clean":"2024-03-22T02:11:39.853959+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:11:39.853959+0000","last_undegraded":"2024-03-22T02:11:39.853959+0000","last_fullsized":"2024-03-22T02:11:39.853959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934924+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:11:38.934924+0000","last_peered":"2024-03-22T02:11:38.934924+0000","last_clean":"2024-03-22T02:11:38.934924+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:11:38.934924+0000","last_undegraded":"2024-03-22T02:11:38.934924+0000","last_fullsized":"2024-03-22T02:11:38.934924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973995+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:11:40.973995+0000","last_peered":"2024-03-22T02:11:40.973995+0000","last_clean":"2024-03-22T02:11:40.973995+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:11:40.973995+0000","last_undegraded":"2024-03-22T02:11:40.973995+0000","last_fullsized":"2024-03-22T02:11:40.973995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.986001+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:11:52.986001+0000","last_peered":"2024-03-22T02:11:52.986001+0000","last_clean":"2024-03-22T02:11:52.986001+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:11:52.986001+0000","last_undegraded":"2024-03-22T02:11:52.986001+0000","last_fullsized":"2024-03-22T02:11:52.986001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854523+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:11:39.854523+0000","last_peered":"2024-03-22T02:11:39.854523+0000","last_clean":"2024-03-22T02:11:39.854523+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:11:39.854523+0000","last_undegraded":"2024-03-22T02:11:39.854523+0000","last_fullsized":"2024-03-22T02:11:39.854523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974010+0000","last_change":"2024-03-22T02:11:37.931900+0000","last_active":"2024-03-22T02:11:40.974010+0000","last_peered":"2024-03-22T02:11:40.974010+0000","last_clean":"2024-03-22T02:11:40.974010+0000","last_became_active":"2024-03-22T02:11:37.931400+0000","last_became_peered":"2024-03-22T02:11:37.931400+0000","last_unstale":"2024-03-22T02:11:40.974010+0000","last_undegraded":"2024-03-22T02:11:40.974010+0000","last_fullsized":"2024-03-22T02:11:40.974010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:55.837790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989397+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:11:53.989397+0000","last_peered":"2024-03-22T02:11:53.989397+0000","last_clean":"2024-03-22T02:11:53.989397+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:11:53.989397+0000","last_undegraded":"2024-03-22T02:11:53.989397+0000","last_fullsized":"2024-03-22T02:11:53.989397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934845+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:11:38.934845+0000","last_peered":"2024-03-22T02:11:38.934845+0000","last_clean":"2024-03-22T02:11:38.934845+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:11:38.934845+0000","last_undegraded":"2024-03-22T02:11:38.934845+0000","last_fullsized":"2024-03-22T02:11:38.934845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854782+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:11:39.854782+0000","last_peered":"2024-03-22T02:11:39.854782+0000","last_clean":"2024-03-22T02:11:39.854782+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:11:39.854782+0000","last_undegraded":"2024-03-22T02:11:39.854782+0000","last_fullsized":"2024-03-22T02:11:39.854782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931062+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:11:38.931062+0000","last_peered":"2024-03-22T02:11:38.931062+0000","last_clean":"2024-03-22T02:11:38.931062+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:11:38.931062+0000","last_undegraded":"2024-03-22T02:11:38.931062+0000","last_fullsized":"2024-03-22T02:11:38.931062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137957+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:11:39.137957+0000","last_peered":"2024-03-22T02:11:39.137957+0000","last_clean":"2024-03-22T02:11:39.137957+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:11:39.137957+0000","last_undegraded":"2024-03-22T02:11:39.137957+0000","last_fullsized":"2024-03-22T02:11:39.137957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918998+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:11:39.918998+0000","last_peered":"2024-03-22T02:11:39.918998+0000","last_clean":"2024-03-22T02:11:39.918998+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:11:39.918998+0000","last_undegraded":"2024-03-22T02:11:39.918998+0000","last_fullsized":"2024-03-22T02:11:39.918998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934525+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:11:38.934525+0000","last_peered":"2024-03-22T02:11:38.934525+0000","last_clean":"2024-03-22T02:11:38.934525+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:11:38.934525+0000","last_undegraded":"2024-03-22T02:11:38.934525+0000","last_fullsized":"2024-03-22T02:11:38.934525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973832+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:11:40.973832+0000","last_peered":"2024-03-22T02:11:40.973832+0000","last_clean":"2024-03-22T02:11:40.973832+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:11:40.973832+0000","last_undegraded":"2024-03-22T02:11:40.973832+0000","last_fullsized":"2024-03-22T02:11:40.973832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137918+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:11:39.137918+0000","last_peered":"2024-03-22T02:11:39.137918+0000","last_clean":"2024-03-22T02:11:39.137918+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:11:39.137918+0000","last_undegraded":"2024-03-22T02:11:39.137918+0000","last_fullsized":"2024-03-22T02:11:39.137918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984427+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:11:52.984427+0000","last_peered":"2024-03-22T02:11:52.984427+0000","last_clean":"2024-03-22T02:11:52.984427+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:11:52.984427+0000","last_undegraded":"2024-03-22T02:11:52.984427+0000","last_fullsized":"2024-03-22T02:11:52.984427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638739+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:11:39.638739+0000","last_peered":"2024-03-22T02:11:39.638739+0000","last_clean":"2024-03-22T02:11:39.638739+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:11:39.638739+0000","last_undegraded":"2024-03-22T02:11:39.638739+0000","last_fullsized":"2024-03-22T02:11:39.638739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973606+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:11:40.973606+0000","last_peered":"2024-03-22T02:11:40.973606+0000","last_clean":"2024-03-22T02:11:40.973606+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:11:40.973606+0000","last_undegraded":"2024-03-22T02:11:40.973606+0000","last_fullsized":"2024-03-22T02:11:40.973606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.983908+0000","last_change":"2024-03-22T02:11:32.910432+0000","last_active":"2024-03-22T02:11:52.983908+0000","last_peered":"2024-03-22T02:11:52.983908+0000","last_clean":"2024-03-22T02:11:52.983908+0000","last_became_active":"2024-03-22T02:11:32.910114+0000","last_became_peered":"2024-03-22T02:11:32.910114+0000","last_unstale":"2024-03-22T02:11:52.983908+0000","last_undegraded":"2024-03-22T02:11:52.983908+0000","last_fullsized":"2024-03-22T02:11:52.983908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:54:21.399399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918825+0000","last_change":"2024-03-22T02:11:33.648236+0000","last_active":"2024-03-22T02:11:39.918825+0000","last_peered":"2024-03-22T02:11:39.918825+0000","last_clean":"2024-03-22T02:11:39.918825+0000","last_became_active":"2024-03-22T02:11:33.647896+0000","last_became_peered":"2024-03-22T02:11:33.647896+0000","last_unstale":"2024-03-22T02:11:39.918825+0000","last_undegraded":"2024-03-22T02:11:39.918825+0000","last_fullsized":"2024-03-22T02:11:39.918825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:33:14.366903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152290+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:11:40.152290+0000","last_peered":"2024-03-22T02:11:40.152290+0000","last_clean":"2024-03-22T02:11:40.152290+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:11:40.152290+0000","last_undegraded":"2024-03-22T02:11:40.152290+0000","last_fullsized":"2024-03-22T02:11:40.152290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934936+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:11:38.934936+0000","last_peered":"2024-03-22T02:11:38.934936+0000","last_clean":"2024-03-22T02:11:38.934936+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:11:38.934936+0000","last_undegraded":"2024-03-22T02:11:38.934936+0000","last_fullsized":"2024-03-22T02:11:38.934936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638904+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:11:39.638904+0000","last_peered":"2024-03-22T02:11:39.638904+0000","last_clean":"2024-03-22T02:11:39.638904+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:11:39.638904+0000","last_undegraded":"2024-03-22T02:11:39.638904+0000","last_fullsized":"2024-03-22T02:11:39.638904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973898+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:11:40.973898+0000","last_peered":"2024-03-22T02:11:40.973898+0000","last_clean":"2024-03-22T02:11:40.973898+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:11:40.973898+0000","last_undegraded":"2024-03-22T02:11:40.973898+0000","last_fullsized":"2024-03-22T02:11:40.973898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'19","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:55.712522+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:11:55.712522+0000","last_peered":"2024-03-22T02:11:55.712522+0000","last_clean":"2024-03-22T02:11:55.712522+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:11:55.712522+0000","last_undegraded":"2024-03-22T02:11:55.712522+0000","last_fullsized":"2024-03-22T02:11:55.712522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":20,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854340+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:11:39.854340+0000","last_peered":"2024-03-22T02:11:39.854340+0000","last_clean":"2024-03-22T02:11:39.854340+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:11:39.854340+0000","last_undegraded":"2024-03-22T02:11:39.854340+0000","last_fullsized":"2024-03-22T02:11:39.854340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935285+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:11:38.935285+0000","last_peered":"2024-03-22T02:11:38.935285+0000","last_clean":"2024-03-22T02:11:38.935285+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:11:38.935285+0000","last_undegraded":"2024-03-22T02:11:38.935285+0000","last_fullsized":"2024-03-22T02:11:38.935285+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984583+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:11:52.984583+0000","last_peered":"2024-03-22T02:11:52.984583+0000","last_clean":"2024-03-22T02:11:52.984583+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:11:52.984583+0000","last_undegraded":"2024-03-22T02:11:52.984583+0000","last_fullsized":"2024-03-22T02:11:52.984583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931053+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:11:38.931053+0000","last_peered":"2024-03-22T02:11:38.931053+0000","last_clean":"2024-03-22T02:11:38.931053+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:11:38.931053+0000","last_undegraded":"2024-03-22T02:11:38.931053+0000","last_fullsized":"2024-03-22T02:11:38.931053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854013+0000","last_change":"2024-03-22T02:11:37.941467+0000","last_active":"2024-03-22T02:11:39.854013+0000","last_peered":"2024-03-22T02:11:39.854013+0000","last_clean":"2024-03-22T02:11:39.854013+0000","last_became_active":"2024-03-22T02:11:37.941067+0000","last_became_peered":"2024-03-22T02:11:37.941067+0000","last_unstale":"2024-03-22T02:11:39.854013+0000","last_undegraded":"2024-03-22T02:11:39.854013+0000","last_fullsized":"2024-03-22T02:11:39.854013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:35.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984908+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:11:52.984908+0000","last_peered":"2024-03-22T02:11:52.984908+0000","last_clean":"2024-03-22T02:11:52.984908+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:11:52.984908+0000","last_undegraded":"2024-03-22T02:11:52.984908+0000","last_fullsized":"2024-03-22T02:11:52.984908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933648+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:11:38.933648+0000","last_peered":"2024-03-22T02:11:38.933648+0000","last_clean":"2024-03-22T02:11:38.933648+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:11:38.933648+0000","last_undegraded":"2024-03-22T02:11:38.933648+0000","last_fullsized":"2024-03-22T02:11:38.933648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974283+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:11:40.974283+0000","last_peered":"2024-03-22T02:11:40.974283+0000","last_clean":"2024-03-22T02:11:40.974283+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:11:40.974283+0000","last_undegraded":"2024-03-22T02:11:40.974283+0000","last_fullsized":"2024-03-22T02:11:40.974283+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.151949+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:11:40.151949+0000","last_peered":"2024-03-22T02:11:40.151949+0000","last_clean":"2024-03-22T02:11:40.151949+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:11:40.151949+0000","last_undegraded":"2024-03-22T02:11:40.151949+0000","last_fullsized":"2024-03-22T02:11:40.151949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933282+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:11:38.933282+0000","last_peered":"2024-03-22T02:11:38.933282+0000","last_clean":"2024-03-22T02:11:38.933282+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:11:38.933282+0000","last_undegraded":"2024-03-22T02:11:38.933282+0000","last_fullsized":"2024-03-22T02:11:38.933282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931608+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:11:38.931608+0000","last_peered":"2024-03-22T02:11:38.931608+0000","last_clean":"2024-03-22T02:11:38.931608+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:11:38.931608+0000","last_undegraded":"2024-03-22T02:11:38.931608+0000","last_fullsized":"2024-03-22T02:11:38.931608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989836+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:11:53.989836+0000","last_peered":"2024-03-22T02:11:53.989836+0000","last_clean":"2024-03-22T02:11:53.989836+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:11:53.989836+0000","last_undegraded":"2024-03-22T02:11:53.989836+0000","last_fullsized":"2024-03-22T02:11:53.989836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934550+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:11:38.934550+0000","last_peered":"2024-03-22T02:11:38.934550+0000","last_clean":"2024-03-22T02:11:38.934550+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:11:38.934550+0000","last_undegraded":"2024-03-22T02:11:38.934550+0000","last_fullsized":"2024-03-22T02:11:38.934550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934559+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:11:38.934559+0000","last_peered":"2024-03-22T02:11:38.934559+0000","last_clean":"2024-03-22T02:11:38.934559+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:11:38.934559+0000","last_undegraded":"2024-03-22T02:11:38.934559+0000","last_fullsized":"2024-03-22T02:11:38.934559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854196+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:11:39.854196+0000","last_peered":"2024-03-22T02:11:39.854196+0000","last_clean":"2024-03-22T02:11:39.854196+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:11:39.854196+0000","last_undegraded":"2024-03-22T02:11:39.854196+0000","last_fullsized":"2024-03-22T02:11:39.854196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854722+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:11:39.854722+0000","last_peered":"2024-03-22T02:11:39.854722+0000","last_clean":"2024-03-22T02:11:39.854722+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:11:39.854722+0000","last_undegraded":"2024-03-22T02:11:39.854722+0000","last_fullsized":"2024-03-22T02:11:39.854722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854326+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:11:39.854326+0000","last_peered":"2024-03-22T02:11:39.854326+0000","last_clean":"2024-03-22T02:11:39.854326+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:11:39.854326+0000","last_undegraded":"2024-03-22T02:11:39.854326+0000","last_fullsized":"2024-03-22T02:11:39.854326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973260+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:11:40.973260+0000","last_peered":"2024-03-22T02:11:40.973260+0000","last_clean":"2024-03-22T02:11:40.973260+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:11:40.973260+0000","last_undegraded":"2024-03-22T02:11:40.973260+0000","last_fullsized":"2024-03-22T02:11:40.973260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853076+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:11:39.853076+0000","last_peered":"2024-03-22T02:11:39.853076+0000","last_clean":"2024-03-22T02:11:39.853076+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:11:39.853076+0000","last_undegraded":"2024-03-22T02:11:39.853076+0000","last_fullsized":"2024-03-22T02:11:39.853076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931353+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:11:38.931353+0000","last_peered":"2024-03-22T02:11:38.931353+0000","last_clean":"2024-03-22T02:11:38.931353+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:11:38.931353+0000","last_undegraded":"2024-03-22T02:11:38.931353+0000","last_fullsized":"2024-03-22T02:11:38.931353+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933333+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:11:38.933333+0000","last_peered":"2024-03-22T02:11:38.933333+0000","last_clean":"2024-03-22T02:11:38.933333+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:11:38.933333+0000","last_undegraded":"2024-03-22T02:11:38.933333+0000","last_fullsized":"2024-03-22T02:11:38.933333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.852832+0000","last_change":"2024-03-22T02:11:33.643607+0000","last_active":"2024-03-22T02:11:39.852832+0000","last_peered":"2024-03-22T02:11:39.852832+0000","last_clean":"2024-03-22T02:11:39.852832+0000","last_became_active":"2024-03-22T02:11:33.642717+0000","last_became_peered":"2024-03-22T02:11:33.642717+0000","last_unstale":"2024-03-22T02:11:39.852832+0000","last_undegraded":"2024-03-22T02:11:39.852832+0000","last_fullsized":"2024-03-22T02:11:39.852832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:22.727788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931781+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:11:38.931781+0000","last_peered":"2024-03-22T02:11:38.931781+0000","last_clean":"2024-03-22T02:11:38.931781+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:11:38.931781+0000","last_undegraded":"2024-03-22T02:11:38.931781+0000","last_fullsized":"2024-03-22T02:11:38.931781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934609+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:11:38.934609+0000","last_peered":"2024-03-22T02:11:38.934609+0000","last_clean":"2024-03-22T02:11:38.934609+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:11:38.934609+0000","last_undegraded":"2024-03-22T02:11:38.934609+0000","last_fullsized":"2024-03-22T02:11:38.934609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.930886+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:11:38.930886+0000","last_peered":"2024-03-22T02:11:38.930886+0000","last_clean":"2024-03-22T02:11:38.930886+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:11:38.930886+0000","last_undegraded":"2024-03-22T02:11:38.930886+0000","last_fullsized":"2024-03-22T02:11:38.930886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933767+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:11:38.933767+0000","last_peered":"2024-03-22T02:11:38.933767+0000","last_clean":"2024-03-22T02:11:38.933767+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:11:38.933767+0000","last_undegraded":"2024-03-22T02:11:38.933767+0000","last_fullsized":"2024-03-22T02:11:38.933767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137948+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:11:39.137948+0000","last_peered":"2024-03-22T02:11:39.137948+0000","last_clean":"2024-03-22T02:11:39.137948+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:11:39.137948+0000","last_undegraded":"2024-03-22T02:11:39.137948+0000","last_fullsized":"2024-03-22T02:11:39.137948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.982550+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:11:52.982550+0000","last_peered":"2024-03-22T02:11:52.982550+0000","last_clean":"2024-03-22T02:11:52.982550+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:11:52.982550+0000","last_undegraded":"2024-03-22T02:11:52.982550+0000","last_fullsized":"2024-03-22T02:11:52.982550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":345,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933171+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:11:38.933171+0000","last_peered":"2024-03-22T02:11:38.933171+0000","last_clean":"2024-03-22T02:11:38.933171+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:11:38.933171+0000","last_undegraded":"2024-03-22T02:11:38.933171+0000","last_fullsized":"2024-03-22T02:11:38.933171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935104+0000","last_change":"2024-03-22T02:11:37.942403+0000","last_active":"2024-03-22T02:11:38.935104+0000","last_peered":"2024-03-22T02:11:38.935104+0000","last_clean":"2024-03-22T02:11:38.935104+0000","last_became_active":"2024-03-22T02:11:37.942040+0000","last_became_peered":"2024-03-22T02:11:37.942040+0000","last_unstale":"2024-03-22T02:11:38.935104+0000","last_undegraded":"2024-03-22T02:11:38.935104+0000","last_fullsized":"2024-03-22T02:11:38.935104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:12:02.337725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.930886+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:11:38.930886+0000","last_peered":"2024-03-22T02:11:38.930886+0000","last_clean":"2024-03-22T02:11:38.930886+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:11:38.930886+0000","last_undegraded":"2024-03-22T02:11:38.930886+0000","last_fullsized":"2024-03-22T02:11:38.930886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933591+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:11:38.933591+0000","last_peered":"2024-03-22T02:11:38.933591+0000","last_clean":"2024-03-22T02:11:38.933591+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:11:38.933591+0000","last_undegraded":"2024-03-22T02:11:38.933591+0000","last_fullsized":"2024-03-22T02:11:38.933591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854108+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:11:39.854108+0000","last_peered":"2024-03-22T02:11:39.854108+0000","last_clean":"2024-03-22T02:11:39.854108+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:11:39.854108+0000","last_undegraded":"2024-03-22T02:11:39.854108+0000","last_fullsized":"2024-03-22T02:11:39.854108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854438+0000","last_change":"2024-03-22T02:11:32.915749+0000","last_active":"2024-03-22T02:11:39.854438+0000","last_peered":"2024-03-22T02:11:39.854438+0000","last_clean":"2024-03-22T02:11:39.854438+0000","last_became_active":"2024-03-22T02:11:32.915455+0000","last_became_peered":"2024-03-22T02:11:32.915455+0000","last_unstale":"2024-03-22T02:11:39.854438+0000","last_undegraded":"2024-03-22T02:11:39.854438+0000","last_fullsized":"2024-03-22T02:11:39.854438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:49.835132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931478+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:11:38.931478+0000","last_peered":"2024-03-22T02:11:38.931478+0000","last_clean":"2024-03-22T02:11:38.931478+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:11:38.931478+0000","last_undegraded":"2024-03-22T02:11:38.931478+0000","last_fullsized":"2024-03-22T02:11:38.931478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137899+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:11:39.137899+0000","last_peered":"2024-03-22T02:11:39.137899+0000","last_clean":"2024-03-22T02:11:39.137899+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:11:39.137899+0000","last_undegraded":"2024-03-22T02:11:39.137899+0000","last_fullsized":"2024-03-22T02:11:39.137899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989334+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:11:53.989334+0000","last_peered":"2024-03-22T02:11:53.989334+0000","last_clean":"2024-03-22T02:11:53.989334+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:11:53.989334+0000","last_undegraded":"2024-03-22T02:11:53.989334+0000","last_fullsized":"2024-03-22T02:11:53.989334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138496+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:11:39.138496+0000","last_peered":"2024-03-22T02:11:39.138496+0000","last_clean":"2024-03-22T02:11:39.138496+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:11:39.138496+0000","last_undegraded":"2024-03-22T02:11:39.138496+0000","last_fullsized":"2024-03-22T02:11:39.138496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973842+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:11:40.973842+0000","last_peered":"2024-03-22T02:11:40.973842+0000","last_clean":"2024-03-22T02:11:40.973842+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:11:40.973842+0000","last_undegraded":"2024-03-22T02:11:40.973842+0000","last_fullsized":"2024-03-22T02:11:40.973842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.983612+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:11:52.983612+0000","last_peered":"2024-03-22T02:11:52.983612+0000","last_clean":"2024-03-22T02:11:52.983612+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:11:52.983612+0000","last_undegraded":"2024-03-22T02:11:52.983612+0000","last_fullsized":"2024-03-22T02:11:52.983612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918967+0000","last_change":"2024-03-22T02:11:33.650506+0000","last_active":"2024-03-22T02:11:39.918967+0000","last_peered":"2024-03-22T02:11:39.918967+0000","last_clean":"2024-03-22T02:11:39.918967+0000","last_became_active":"2024-03-22T02:11:33.649812+0000","last_became_peered":"2024-03-22T02:11:33.649812+0000","last_unstale":"2024-03-22T02:11:39.918967+0000","last_undegraded":"2024-03-22T02:11:39.918967+0000","last_fullsized":"2024-03-22T02:11:39.918967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:08.991454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152246+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:11:40.152246+0000","last_peered":"2024-03-22T02:11:40.152246+0000","last_clean":"2024-03-22T02:11:40.152246+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:11:40.152246+0000","last_undegraded":"2024-03-22T02:11:40.152246+0000","last_fullsized":"2024-03-22T02:11:40.152246+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.985850+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:11:52.985850+0000","last_peered":"2024-03-22T02:11:52.985850+0000","last_clean":"2024-03-22T02:11:52.985850+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:11:52.985850+0000","last_undegraded":"2024-03-22T02:11:52.985850+0000","last_fullsized":"2024-03-22T02:11:52.985850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853828+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:11:39.853828+0000","last_peered":"2024-03-22T02:11:39.853828+0000","last_clean":"2024-03-22T02:11:39.853828+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:11:39.853828+0000","last_undegraded":"2024-03-22T02:11:39.853828+0000","last_fullsized":"2024-03-22T02:11:39.853828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974343+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:11:40.974343+0000","last_peered":"2024-03-22T02:11:40.974343+0000","last_clean":"2024-03-22T02:11:40.974343+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:11:40.974343+0000","last_undegraded":"2024-03-22T02:11:40.974343+0000","last_fullsized":"2024-03-22T02:11:40.974343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984866+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:11:52.984866+0000","last_peered":"2024-03-22T02:11:52.984866+0000","last_clean":"2024-03-22T02:11:52.984866+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:11:52.984866+0000","last_undegraded":"2024-03-22T02:11:52.984866+0000","last_fullsized":"2024-03-22T02:11:52.984866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853723+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:11:39.853723+0000","last_peered":"2024-03-22T02:11:39.853723+0000","last_clean":"2024-03-22T02:11:39.853723+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:11:39.853723+0000","last_undegraded":"2024-03-22T02:11:39.853723+0000","last_fullsized":"2024-03-22T02:11:39.853723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152163+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:11:40.152163+0000","last_peered":"2024-03-22T02:11:40.152163+0000","last_clean":"2024-03-22T02:11:40.152163+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:11:40.152163+0000","last_undegraded":"2024-03-22T02:11:40.152163+0000","last_fullsized":"2024-03-22T02:11:40.152163+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973269+0000","last_change":"2024-03-22T02:11:32.907985+0000","last_active":"2024-03-22T02:11:40.973269+0000","last_peered":"2024-03-22T02:11:40.973269+0000","last_clean":"2024-03-22T02:11:40.973269+0000","last_became_active":"2024-03-22T02:11:32.907686+0000","last_became_peered":"2024-03-22T02:11:32.907686+0000","last_unstale":"2024-03-22T02:11:40.973269+0000","last_undegraded":"2024-03-22T02:11:40.973269+0000","last_fullsized":"2024-03-22T02:11:40.973269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:30:40.945298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138108+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:11:39.138108+0000","last_peered":"2024-03-22T02:11:39.138108+0000","last_clean":"2024-03-22T02:11:39.138108+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:11:39.138108+0000","last_undegraded":"2024-03-22T02:11:39.138108+0000","last_fullsized":"2024-03-22T02:11:39.138108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152196+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:11:40.152196+0000","last_peered":"2024-03-22T02:11:40.152196+0000","last_clean":"2024-03-22T02:11:40.152196+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:11:40.152196+0000","last_undegraded":"2024-03-22T02:11:40.152196+0000","last_fullsized":"2024-03-22T02:11:40.152196+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.985865+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:11:52.985865+0000","last_peered":"2024-03-22T02:11:52.985865+0000","last_clean":"2024-03-22T02:11:52.985865+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:11:52.985865+0000","last_undegraded":"2024-03-22T02:11:52.985865+0000","last_fullsized":"2024-03-22T02:11:52.985865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933446+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:11:38.933446+0000","last_peered":"2024-03-22T02:11:38.933446+0000","last_clean":"2024-03-22T02:11:38.933446+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:11:38.933446+0000","last_undegraded":"2024-03-22T02:11:38.933446+0000","last_fullsized":"2024-03-22T02:11:38.933446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638625+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:11:39.638625+0000","last_peered":"2024-03-22T02:11:39.638625+0000","last_clean":"2024-03-22T02:11:39.638625+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:11:39.638625+0000","last_undegraded":"2024-03-22T02:11:39.638625+0000","last_fullsized":"2024-03-22T02:11:39.638625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973318+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:11:40.973318+0000","last_peered":"2024-03-22T02:11:40.973318+0000","last_clean":"2024-03-22T02:11:40.973318+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:11:40.973318+0000","last_undegraded":"2024-03-22T02:11:40.973318+0000","last_fullsized":"2024-03-22T02:11:40.973318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933876+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:11:38.933876+0000","last_peered":"2024-03-22T02:11:38.933876+0000","last_clean":"2024-03-22T02:11:38.933876+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:11:38.933876+0000","last_undegraded":"2024-03-22T02:11:38.933876+0000","last_fullsized":"2024-03-22T02:11:38.933876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918634+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:11:39.918634+0000","last_peered":"2024-03-22T02:11:39.918634+0000","last_clean":"2024-03-22T02:11:39.918634+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:11:39.918634+0000","last_undegraded":"2024-03-22T02:11:39.918634+0000","last_fullsized":"2024-03-22T02:11:39.918634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935013+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:11:38.935013+0000","last_peered":"2024-03-22T02:11:38.935013+0000","last_clean":"2024-03-22T02:11:38.935013+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:11:38.935013+0000","last_undegraded":"2024-03-22T02:11:38.935013+0000","last_fullsized":"2024-03-22T02:11:38.935013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933530+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:11:38.933530+0000","last_peered":"2024-03-22T02:11:38.933530+0000","last_clean":"2024-03-22T02:11:38.933530+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:11:38.933530+0000","last_undegraded":"2024-03-22T02:11:38.933530+0000","last_fullsized":"2024-03-22T02:11:38.933530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934312+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:11:38.934312+0000","last_peered":"2024-03-22T02:11:38.934312+0000","last_clean":"2024-03-22T02:11:38.934312+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:11:38.934312+0000","last_undegraded":"2024-03-22T02:11:38.934312+0000","last_fullsized":"2024-03-22T02:11:38.934312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931561+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:11:38.931561+0000","last_peered":"2024-03-22T02:11:38.931561+0000","last_clean":"2024-03-22T02:11:38.931561+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:11:38.931561+0000","last_undegraded":"2024-03-22T02:11:38.931561+0000","last_fullsized":"2024-03-22T02:11:38.931561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933755+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:11:38.933755+0000","last_peered":"2024-03-22T02:11:38.933755+0000","last_clean":"2024-03-22T02:11:38.933755+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:11:38.933755+0000","last_undegraded":"2024-03-22T02:11:38.933755+0000","last_fullsized":"2024-03-22T02:11:38.933755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853808+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:11:39.853808+0000","last_peered":"2024-03-22T02:11:39.853808+0000","last_clean":"2024-03-22T02:11:39.853808+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:11:39.853808+0000","last_undegraded":"2024-03-22T02:11:39.853808+0000","last_fullsized":"2024-03-22T02:11:39.853808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974310+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:11:40.974310+0000","last_peered":"2024-03-22T02:11:40.974310+0000","last_clean":"2024-03-22T02:11:40.974310+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:11:40.974310+0000","last_undegraded":"2024-03-22T02:11:40.974310+0000","last_fullsized":"2024-03-22T02:11:40.974310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973816+0000","last_change":"2024-03-22T02:11:33.635594+0000","last_active":"2024-03-22T02:11:40.973816+0000","last_peered":"2024-03-22T02:11:40.973816+0000","last_clean":"2024-03-22T02:11:40.973816+0000","last_became_active":"2024-03-22T02:11:33.635298+0000","last_became_peered":"2024-03-22T02:11:33.635298+0000","last_unstale":"2024-03-22T02:11:40.973816+0000","last_undegraded":"2024-03-22T02:11:40.973816+0000","last_fullsized":"2024-03-22T02:11:40.973816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:39:51.824729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918700+0000","last_change":"2024-03-22T02:11:33.643684+0000","last_active":"2024-03-22T02:11:39.918700+0000","last_peered":"2024-03-22T02:11:39.918700+0000","last_clean":"2024-03-22T02:11:39.918700+0000","last_became_active":"2024-03-22T02:11:33.643339+0000","last_became_peered":"2024-03-22T02:11:33.643339+0000","last_unstale":"2024-03-22T02:11:39.918700+0000","last_undegraded":"2024-03-22T02:11:39.918700+0000","last_fullsized":"2024-03-22T02:11:39.918700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:46:34.059019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931359+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:11:38.931359+0000","last_peered":"2024-03-22T02:11:38.931359+0000","last_clean":"2024-03-22T02:11:38.931359+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:11:38.931359+0000","last_undegraded":"2024-03-22T02:11:38.931359+0000","last_fullsized":"2024-03-22T02:11:38.931359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854696+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:11:39.854696+0000","last_peered":"2024-03-22T02:11:39.854696+0000","last_clean":"2024-03-22T02:11:39.854696+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:11:39.854696+0000","last_undegraded":"2024-03-22T02:11:39.854696+0000","last_fullsized":"2024-03-22T02:11:39.854696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974199+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:11:40.974199+0000","last_peered":"2024-03-22T02:11:40.974199+0000","last_clean":"2024-03-22T02:11:40.974199+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:11:40.974199+0000","last_undegraded":"2024-03-22T02:11:40.974199+0000","last_fullsized":"2024-03-22T02:11:40.974199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138612+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:11:39.138612+0000","last_peered":"2024-03-22T02:11:39.138612+0000","last_clean":"2024-03-22T02:11:39.138612+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:11:39.138612+0000","last_undegraded":"2024-03-22T02:11:39.138612+0000","last_fullsized":"2024-03-22T02:11:39.138612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934497+0000","last_change":"2024-03-22T02:11:33.640871+0000","last_active":"2024-03-22T02:11:38.934497+0000","last_peered":"2024-03-22T02:11:38.934497+0000","last_clean":"2024-03-22T02:11:38.934497+0000","last_became_active":"2024-03-22T02:11:33.640579+0000","last_became_peered":"2024-03-22T02:11:33.640579+0000","last_unstale":"2024-03-22T02:11:38.934497+0000","last_undegraded":"2024-03-22T02:11:38.934497+0000","last_fullsized":"2024-03-22T02:11:38.934497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:03.492201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138121+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:11:39.138121+0000","last_peered":"2024-03-22T02:11:39.138121+0000","last_clean":"2024-03-22T02:11:39.138121+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:11:39.138121+0000","last_undegraded":"2024-03-22T02:11:39.138121+0000","last_fullsized":"2024-03-22T02:11:39.138121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638760+0000","last_change":"2024-03-22T02:11:33.642009+0000","last_active":"2024-03-22T02:11:39.638760+0000","last_peered":"2024-03-22T02:11:39.638760+0000","last_clean":"2024-03-22T02:11:39.638760+0000","last_became_active":"2024-03-22T02:11:33.641673+0000","last_became_peered":"2024-03-22T02:11:33.641673+0000","last_unstale":"2024-03-22T02:11:39.638760+0000","last_undegraded":"2024-03-22T02:11:39.638760+0000","last_fullsized":"2024-03-22T02:11:39.638760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:01.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931435+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:11:38.931435+0000","last_peered":"2024-03-22T02:11:38.931435+0000","last_clean":"2024-03-22T02:11:38.931435+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:11:38.931435+0000","last_undegraded":"2024-03-22T02:11:38.931435+0000","last_fullsized":"2024-03-22T02:11:38.931435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854378+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:11:39.854378+0000","last_peered":"2024-03-22T02:11:39.854378+0000","last_clean":"2024-03-22T02:11:39.854378+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:11:39.854378+0000","last_undegraded":"2024-03-22T02:11:39.854378+0000","last_fullsized":"2024-03-22T02:11:39.854378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":25457,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":114,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":38989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350992,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1076054,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481084,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480892,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1071838,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484108,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481744,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":484764,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":480370,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":480288,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":24719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:11:57.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T02:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 -- 172.21.15.28:0/2552989572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 msgr2=0x7fc26c10a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 --2- 172.21.15.28:0/2552989572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c10a0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc264009cd0 tx=0x7fc26402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 -- 172.21.15.28:0/2552989572 shutdown_connections 2024-03-22T02:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 --2- 172.21.15.28:0/2552989572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c10a0c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 --2- 172.21.15.28:0/2552989572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 0x7fc26c1032f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 --2- 172.21.15.28:0/2552989572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 0x7fc26c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 -- 172.21.15.28:0/2552989572 >> 172.21.15.28:0/2552989572 conn(0x7fc26c0fb7d0 msgr2=0x7fc26c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 -- 172.21.15.28:0/2552989572 shutdown_connections 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.919+0000 7fc27269a640 1 -- 172.21.15.28:0/2552989572 wait complete. 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 Processor -- start 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 -- start start 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 0x7fc26c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 0x7fc26c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc26c10d880 con 0x7fc26c103830 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc26c10d700 con 0x7fc26c0ffea0 2024-03-22T02:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc27269a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc26c10da00 con 0x7fc26c102e70 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50114/0 (socket says 172.21.15.28:50114) 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 -- 172.21.15.28:0/2760477149 learned_addr learned my addr 172.21.15.28:0/2760477149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc26bfff640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 0x7fc26c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc26b7fe640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 0x7fc26c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 -- 172.21.15.28:0/2760477149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 msgr2=0x7fc26c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 0x7fc26c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 -- 172.21.15.28:0/2760477149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 msgr2=0x7fc26c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 0x7fc26c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.920+0000 7fc270c37640 1 -- 172.21.15.28:0/2760477149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26c19fae0 con 0x7fc26c103830 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc270c37640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc264009ca0 tx=0x7fc2640346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264041020 con 0x7fc26c103830 2024-03-22T02:11:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc264032890 con 0x7fc26c103830 2024-03-22T02:11:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc26c19fd70 con 0x7fc26c103830 2024-03-22T02:11:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc264039040 con 0x7fc26c103830 2024-03-22T02:11:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.921+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc26c1a0200 con 0x7fc26c103830 2024-03-22T02:11:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.922+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc26c108fa0 con 0x7fc26c103830 2024-03-22T02:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.923+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 137854+0+0 (secure 0 0 0) 0x7fc264044020 con 0x7fc26c103830 2024-03-22T02:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.924+0000 7fc2697fa640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 0x7fc2540a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.924+0000 7fc26bfff640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 0x7fc2540a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.924+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fc2640f17a0 con 0x7fc26c103830 2024-03-22T02:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.924+0000 7fc26bfff640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 0x7fc2540a53e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc2600095f0 tx=0x7fc260005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:11:57.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:57.928+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc26404b020 con 0x7fc26c103830 2024-03-22T02:11:58.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.002+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fc2640ba300 con 0x7fc26c103830 2024-03-22T02:11:58.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:58 smithi028 ceph-mon[31916]: pgmap v305: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 3.9 KiB/s wr, 6 op/s 2024-03-22T02:11:58.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:58 smithi028 ceph-mon[31916]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:58.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/513118119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:58.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/941152186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:58.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.113+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc26c061fd0 con 0x7fc2540a2f20 2024-03-22T02:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.122+0000 7fc2697fa640 1 -- 172.21.15.28:0/2760477149 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340572 (secure 0 0 0) 0x7fc26c061fd0 con 0x7fc2540a2f20 2024-03-22T02:11:58.126 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:11:58.128 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.125+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 msgr2=0x7fc2540a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.125+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 0x7fc2540a53e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc2600095f0 tx=0x7fc260005f90 comp rx=0 tx=0).stop 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 msgr2=0x7fc26c19f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc264009ca0 tx=0x7fc2640346d0 comp rx=0 tx=0).stop 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 shutdown_connections 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2540a2f20 0x7fc2540a53e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc26c103830 0x7fc26c19f470 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc26c102e70 0x7fc26c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:58.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 --2- 172.21.15.28:0/2760477149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc26c0ffea0 0x7fc26c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:11:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 >> 172.21.15.28:0/2760477149 conn(0x7fc26c0fb7d0 msgr2=0x7fc26c106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:11:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 shutdown_connections 2024-03-22T02:11:58.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:11:58.126+0000 7fc27269a640 1 -- 172.21.15.28:0/2760477149 wait complete. 2024-03-22T02:11:58.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-03-22T02:11:56.555121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":615825,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":131,"num_read_kb":116,"num_write":295,"num_write_kb":2999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365044,"kb_used_data":11696,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576780,"statfs":{"total":1151940427776,"available":1151566622720,"internally_reserved":0,"allocated":11976704,"data_stored":7552490,"data_compressed":284580,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23047,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":69,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003458"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854847+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:11:39.854847+0000","last_peered":"2024-03-22T02:11:39.854847+0000","last_clean":"2024-03-22T02:11:39.854847+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:11:39.854847+0000","last_undegraded":"2024-03-22T02:11:39.854847+0000","last_fullsized":"2024-03-22T02:11:39.854847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918858+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:11:39.918858+0000","last_peered":"2024-03-22T02:11:39.918858+0000","last_clean":"2024-03-22T02:11:39.918858+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:11:39.918858+0000","last_undegraded":"2024-03-22T02:11:39.918858+0000","last_fullsized":"2024-03-22T02:11:39.918858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918743+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:11:39.918743+0000","last_peered":"2024-03-22T02:11:39.918743+0000","last_clean":"2024-03-22T02:11:39.918743+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:11:39.918743+0000","last_undegraded":"2024-03-22T02:11:39.918743+0000","last_fullsized":"2024-03-22T02:11:39.918743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152289+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:11:40.152289+0000","last_peered":"2024-03-22T02:11:40.152289+0000","last_clean":"2024-03-22T02:11:40.152289+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:11:40.152289+0000","last_undegraded":"2024-03-22T02:11:40.152289+0000","last_fullsized":"2024-03-22T02:11:40.152289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853871+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:11:39.853871+0000","last_peered":"2024-03-22T02:11:39.853871+0000","last_clean":"2024-03-22T02:11:39.853871+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:11:39.853871+0000","last_undegraded":"2024-03-22T02:11:39.853871+0000","last_fullsized":"2024-03-22T02:11:39.853871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853000+0000","last_change":"2024-03-22T02:11:33.630093+0000","last_active":"2024-03-22T02:11:39.853000+0000","last_peered":"2024-03-22T02:11:39.853000+0000","last_clean":"2024-03-22T02:11:39.853000+0000","last_became_active":"2024-03-22T02:11:33.629784+0000","last_became_peered":"2024-03-22T02:11:33.629784+0000","last_unstale":"2024-03-22T02:11:39.853000+0000","last_undegraded":"2024-03-22T02:11:39.853000+0000","last_fullsized":"2024-03-22T02:11:39.853000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:39.324939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933263+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:11:38.933263+0000","last_peered":"2024-03-22T02:11:38.933263+0000","last_clean":"2024-03-22T02:11:38.933263+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:11:38.933263+0000","last_undegraded":"2024-03-22T02:11:38.933263+0000","last_fullsized":"2024-03-22T02:11:38.933263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137726+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:11:39.137726+0000","last_peered":"2024-03-22T02:11:39.137726+0000","last_clean":"2024-03-22T02:11:39.137726+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:11:39.137726+0000","last_undegraded":"2024-03-22T02:11:39.137726+0000","last_fullsized":"2024-03-22T02:11:39.137726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638809+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:11:39.638809+0000","last_peered":"2024-03-22T02:11:39.638809+0000","last_clean":"2024-03-22T02:11:39.638809+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:11:39.638809+0000","last_undegraded":"2024-03-22T02:11:39.638809+0000","last_fullsized":"2024-03-22T02:11:39.638809+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853141+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:11:39.853141+0000","last_peered":"2024-03-22T02:11:39.853141+0000","last_clean":"2024-03-22T02:11:39.853141+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:11:39.853141+0000","last_undegraded":"2024-03-22T02:11:39.853141+0000","last_fullsized":"2024-03-22T02:11:39.853141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933829+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:11:38.933829+0000","last_peered":"2024-03-22T02:11:38.933829+0000","last_clean":"2024-03-22T02:11:38.933829+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:11:38.933829+0000","last_undegraded":"2024-03-22T02:11:38.933829+0000","last_fullsized":"2024-03-22T02:11:38.933829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152193+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:11:40.152193+0000","last_peered":"2024-03-22T02:11:40.152193+0000","last_clean":"2024-03-22T02:11:40.152193+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:11:40.152193+0000","last_undegraded":"2024-03-22T02:11:40.152193+0000","last_fullsized":"2024-03-22T02:11:40.152193+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931289+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:11:38.931289+0000","last_peered":"2024-03-22T02:11:38.931289+0000","last_clean":"2024-03-22T02:11:38.931289+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:11:38.931289+0000","last_undegraded":"2024-03-22T02:11:38.931289+0000","last_fullsized":"2024-03-22T02:11:38.931289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918413+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:11:39.918413+0000","last_peered":"2024-03-22T02:11:39.918413+0000","last_clean":"2024-03-22T02:11:39.918413+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:11:39.918413+0000","last_undegraded":"2024-03-22T02:11:39.918413+0000","last_fullsized":"2024-03-22T02:11:39.918413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918482+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:11:39.918482+0000","last_peered":"2024-03-22T02:11:39.918482+0000","last_clean":"2024-03-22T02:11:39.918482+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:11:39.918482+0000","last_undegraded":"2024-03-22T02:11:39.918482+0000","last_fullsized":"2024-03-22T02:11:39.918482+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638474+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:11:39.638474+0000","last_peered":"2024-03-22T02:11:39.638474+0000","last_clean":"2024-03-22T02:11:39.638474+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:11:39.638474+0000","last_undegraded":"2024-03-22T02:11:39.638474+0000","last_fullsized":"2024-03-22T02:11:39.638474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854728+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:11:39.854728+0000","last_peered":"2024-03-22T02:11:39.854728+0000","last_clean":"2024-03-22T02:11:39.854728+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:11:39.854728+0000","last_undegraded":"2024-03-22T02:11:39.854728+0000","last_fullsized":"2024-03-22T02:11:39.854728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974322+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:11:40.974322+0000","last_peered":"2024-03-22T02:11:40.974322+0000","last_clean":"2024-03-22T02:11:40.974322+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:11:40.974322+0000","last_undegraded":"2024-03-22T02:11:40.974322+0000","last_fullsized":"2024-03-22T02:11:40.974322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974097+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:11:40.974097+0000","last_peered":"2024-03-22T02:11:40.974097+0000","last_clean":"2024-03-22T02:11:40.974097+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:11:40.974097+0000","last_undegraded":"2024-03-22T02:11:40.974097+0000","last_fullsized":"2024-03-22T02:11:40.974097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974257+0000","last_change":"2024-03-22T02:11:37.944045+0000","last_active":"2024-03-22T02:11:40.974257+0000","last_peered":"2024-03-22T02:11:40.974257+0000","last_clean":"2024-03-22T02:11:40.974257+0000","last_became_active":"2024-03-22T02:11:37.943598+0000","last_became_peered":"2024-03-22T02:11:37.943598+0000","last_unstale":"2024-03-22T02:11:40.974257+0000","last_undegraded":"2024-03-22T02:11:40.974257+0000","last_fullsized":"2024-03-22T02:11:40.974257+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:46:43.350701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.991750+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:11:53.991750+0000","last_peered":"2024-03-22T02:11:53.991750+0000","last_clean":"2024-03-22T02:11:53.991750+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:11:53.991750+0000","last_undegraded":"2024-03-22T02:11:53.991750+0000","last_fullsized":"2024-03-22T02:11:53.991750+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138425+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:11:39.138425+0000","last_peered":"2024-03-22T02:11:39.138425+0000","last_clean":"2024-03-22T02:11:39.138425+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:11:39.138425+0000","last_undegraded":"2024-03-22T02:11:39.138425+0000","last_fullsized":"2024-03-22T02:11:39.138425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.151921+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:11:40.151921+0000","last_peered":"2024-03-22T02:11:40.151921+0000","last_clean":"2024-03-22T02:11:40.151921+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:11:40.151921+0000","last_undegraded":"2024-03-22T02:11:40.151921+0000","last_fullsized":"2024-03-22T02:11:40.151921+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138625+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:11:39.138625+0000","last_peered":"2024-03-22T02:11:39.138625+0000","last_clean":"2024-03-22T02:11:39.138625+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:11:39.138625+0000","last_undegraded":"2024-03-22T02:11:39.138625+0000","last_fullsized":"2024-03-22T02:11:39.138625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918901+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:11:39.918901+0000","last_peered":"2024-03-22T02:11:39.918901+0000","last_clean":"2024-03-22T02:11:39.918901+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:11:39.918901+0000","last_undegraded":"2024-03-22T02:11:39.918901+0000","last_fullsized":"2024-03-22T02:11:39.918901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152035+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:11:40.152035+0000","last_peered":"2024-03-22T02:11:40.152035+0000","last_clean":"2024-03-22T02:11:40.152035+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:11:40.152035+0000","last_undegraded":"2024-03-22T02:11:40.152035+0000","last_fullsized":"2024-03-22T02:11:40.152035+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989534+0000","last_change":"2024-03-22T02:11:32.917851+0000","last_active":"2024-03-22T02:11:53.989534+0000","last_peered":"2024-03-22T02:11:53.989534+0000","last_clean":"2024-03-22T02:11:53.989534+0000","last_became_active":"2024-03-22T02:11:32.917526+0000","last_became_peered":"2024-03-22T02:11:32.917526+0000","last_unstale":"2024-03-22T02:11:53.989534+0000","last_undegraded":"2024-03-22T02:11:53.989534+0000","last_fullsized":"2024-03-22T02:11:53.989534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:20.509077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638819+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:11:39.638819+0000","last_peered":"2024-03-22T02:11:39.638819+0000","last_clean":"2024-03-22T02:11:39.638819+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:11:39.638819+0000","last_undegraded":"2024-03-22T02:11:39.638819+0000","last_fullsized":"2024-03-22T02:11:39.638819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152119+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:11:40.152119+0000","last_peered":"2024-03-22T02:11:40.152119+0000","last_clean":"2024-03-22T02:11:40.152119+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:11:40.152119+0000","last_undegraded":"2024-03-22T02:11:40.152119+0000","last_fullsized":"2024-03-22T02:11:40.152119+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853959+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:11:39.853959+0000","last_peered":"2024-03-22T02:11:39.853959+0000","last_clean":"2024-03-22T02:11:39.853959+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:11:39.853959+0000","last_undegraded":"2024-03-22T02:11:39.853959+0000","last_fullsized":"2024-03-22T02:11:39.853959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934924+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:11:38.934924+0000","last_peered":"2024-03-22T02:11:38.934924+0000","last_clean":"2024-03-22T02:11:38.934924+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:11:38.934924+0000","last_undegraded":"2024-03-22T02:11:38.934924+0000","last_fullsized":"2024-03-22T02:11:38.934924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973995+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:11:40.973995+0000","last_peered":"2024-03-22T02:11:40.973995+0000","last_clean":"2024-03-22T02:11:40.973995+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:11:40.973995+0000","last_undegraded":"2024-03-22T02:11:40.973995+0000","last_fullsized":"2024-03-22T02:11:40.973995+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.986001+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:11:52.986001+0000","last_peered":"2024-03-22T02:11:52.986001+0000","last_clean":"2024-03-22T02:11:52.986001+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:11:52.986001+0000","last_undegraded":"2024-03-22T02:11:52.986001+0000","last_fullsized":"2024-03-22T02:11:52.986001+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854523+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:11:39.854523+0000","last_peered":"2024-03-22T02:11:39.854523+0000","last_clean":"2024-03-22T02:11:39.854523+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:11:39.854523+0000","last_undegraded":"2024-03-22T02:11:39.854523+0000","last_fullsized":"2024-03-22T02:11:39.854523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974010+0000","last_change":"2024-03-22T02:11:37.931900+0000","last_active":"2024-03-22T02:11:40.974010+0000","last_peered":"2024-03-22T02:11:40.974010+0000","last_clean":"2024-03-22T02:11:40.974010+0000","last_became_active":"2024-03-22T02:11:37.931400+0000","last_became_peered":"2024-03-22T02:11:37.931400+0000","last_unstale":"2024-03-22T02:11:40.974010+0000","last_undegraded":"2024-03-22T02:11:40.974010+0000","last_fullsized":"2024-03-22T02:11:40.974010+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:55.837790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989397+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:11:53.989397+0000","last_peered":"2024-03-22T02:11:53.989397+0000","last_clean":"2024-03-22T02:11:53.989397+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:11:53.989397+0000","last_undegraded":"2024-03-22T02:11:53.989397+0000","last_fullsized":"2024-03-22T02:11:53.989397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934845+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:11:38.934845+0000","last_peered":"2024-03-22T02:11:38.934845+0000","last_clean":"2024-03-22T02:11:38.934845+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:11:38.934845+0000","last_undegraded":"2024-03-22T02:11:38.934845+0000","last_fullsized":"2024-03-22T02:11:38.934845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854782+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:11:39.854782+0000","last_peered":"2024-03-22T02:11:39.854782+0000","last_clean":"2024-03-22T02:11:39.854782+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:11:39.854782+0000","last_undegraded":"2024-03-22T02:11:39.854782+0000","last_fullsized":"2024-03-22T02:11:39.854782+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931062+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:11:38.931062+0000","last_peered":"2024-03-22T02:11:38.931062+0000","last_clean":"2024-03-22T02:11:38.931062+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:11:38.931062+0000","last_undegraded":"2024-03-22T02:11:38.931062+0000","last_fullsized":"2024-03-22T02:11:38.931062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137957+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:11:39.137957+0000","last_peered":"2024-03-22T02:11:39.137957+0000","last_clean":"2024-03-22T02:11:39.137957+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:11:39.137957+0000","last_undegraded":"2024-03-22T02:11:39.137957+0000","last_fullsized":"2024-03-22T02:11:39.137957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918998+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:11:39.918998+0000","last_peered":"2024-03-22T02:11:39.918998+0000","last_clean":"2024-03-22T02:11:39.918998+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:11:39.918998+0000","last_undegraded":"2024-03-22T02:11:39.918998+0000","last_fullsized":"2024-03-22T02:11:39.918998+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934525+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:11:38.934525+0000","last_peered":"2024-03-22T02:11:38.934525+0000","last_clean":"2024-03-22T02:11:38.934525+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:11:38.934525+0000","last_undegraded":"2024-03-22T02:11:38.934525+0000","last_fullsized":"2024-03-22T02:11:38.934525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973832+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:11:40.973832+0000","last_peered":"2024-03-22T02:11:40.973832+0000","last_clean":"2024-03-22T02:11:40.973832+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:11:40.973832+0000","last_undegraded":"2024-03-22T02:11:40.973832+0000","last_fullsized":"2024-03-22T02:11:40.973832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137918+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:11:39.137918+0000","last_peered":"2024-03-22T02:11:39.137918+0000","last_clean":"2024-03-22T02:11:39.137918+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:11:39.137918+0000","last_undegraded":"2024-03-22T02:11:39.137918+0000","last_fullsized":"2024-03-22T02:11:39.137918+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984427+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:11:52.984427+0000","last_peered":"2024-03-22T02:11:52.984427+0000","last_clean":"2024-03-22T02:11:52.984427+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:11:52.984427+0000","last_undegraded":"2024-03-22T02:11:52.984427+0000","last_fullsized":"2024-03-22T02:11:52.984427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638739+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:11:39.638739+0000","last_peered":"2024-03-22T02:11:39.638739+0000","last_clean":"2024-03-22T02:11:39.638739+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:11:39.638739+0000","last_undegraded":"2024-03-22T02:11:39.638739+0000","last_fullsized":"2024-03-22T02:11:39.638739+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973606+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:11:40.973606+0000","last_peered":"2024-03-22T02:11:40.973606+0000","last_clean":"2024-03-22T02:11:40.973606+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:11:40.973606+0000","last_undegraded":"2024-03-22T02:11:40.973606+0000","last_fullsized":"2024-03-22T02:11:40.973606+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.983908+0000","last_change":"2024-03-22T02:11:32.910432+0000","last_active":"2024-03-22T02:11:52.983908+0000","last_peered":"2024-03-22T02:11:52.983908+0000","last_clean":"2024-03-22T02:11:52.983908+0000","last_became_active":"2024-03-22T02:11:32.910114+0000","last_became_peered":"2024-03-22T02:11:32.910114+0000","last_unstale":"2024-03-22T02:11:52.983908+0000","last_undegraded":"2024-03-22T02:11:52.983908+0000","last_fullsized":"2024-03-22T02:11:52.983908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:54:21.399399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918825+0000","last_change":"2024-03-22T02:11:33.648236+0000","last_active":"2024-03-22T02:11:39.918825+0000","last_peered":"2024-03-22T02:11:39.918825+0000","last_clean":"2024-03-22T02:11:39.918825+0000","last_became_active":"2024-03-22T02:11:33.647896+0000","last_became_peered":"2024-03-22T02:11:33.647896+0000","last_unstale":"2024-03-22T02:11:39.918825+0000","last_undegraded":"2024-03-22T02:11:39.918825+0000","last_fullsized":"2024-03-22T02:11:39.918825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:33:14.366903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152290+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:11:40.152290+0000","last_peered":"2024-03-22T02:11:40.152290+0000","last_clean":"2024-03-22T02:11:40.152290+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:11:40.152290+0000","last_undegraded":"2024-03-22T02:11:40.152290+0000","last_fullsized":"2024-03-22T02:11:40.152290+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934936+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:11:38.934936+0000","last_peered":"2024-03-22T02:11:38.934936+0000","last_clean":"2024-03-22T02:11:38.934936+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:11:38.934936+0000","last_undegraded":"2024-03-22T02:11:38.934936+0000","last_fullsized":"2024-03-22T02:11:38.934936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638904+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:11:39.638904+0000","last_peered":"2024-03-22T02:11:39.638904+0000","last_clean":"2024-03-22T02:11:39.638904+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:11:39.638904+0000","last_undegraded":"2024-03-22T02:11:39.638904+0000","last_fullsized":"2024-03-22T02:11:39.638904+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973898+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:11:40.973898+0000","last_peered":"2024-03-22T02:11:40.973898+0000","last_clean":"2024-03-22T02:11:40.973898+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:11:40.973898+0000","last_undegraded":"2024-03-22T02:11:40.973898+0000","last_fullsized":"2024-03-22T02:11:40.973898+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'19","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:55.712522+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:11:55.712522+0000","last_peered":"2024-03-22T02:11:55.712522+0000","last_clean":"2024-03-22T02:11:55.712522+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:11:55.712522+0000","last_undegraded":"2024-03-22T02:11:55.712522+0000","last_fullsized":"2024-03-22T02:11:55.712522+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":20,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854340+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:11:39.854340+0000","last_peered":"2024-03-22T02:11:39.854340+0000","last_clean":"2024-03-22T02:11:39.854340+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:11:39.854340+0000","last_undegraded":"2024-03-22T02:11:39.854340+0000","last_fullsized":"2024-03-22T02:11:39.854340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935285+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:11:38.935285+0000","last_peered":"2024-03-22T02:11:38.935285+0000","last_clean":"2024-03-22T02:11:38.935285+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:11:38.935285+0000","last_undegraded":"2024-03-22T02:11:38.935285+0000","last_fullsized":"2024-03-22T02:11:38.935285+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984583+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:11:52.984583+0000","last_peered":"2024-03-22T02:11:52.984583+0000","last_clean":"2024-03-22T02:11:52.984583+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:11:52.984583+0000","last_undegraded":"2024-03-22T02:11:52.984583+0000","last_fullsized":"2024-03-22T02:11:52.984583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931053+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:11:38.931053+0000","last_peered":"2024-03-22T02:11:38.931053+0000","last_clean":"2024-03-22T02:11:38.931053+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:11:38.931053+0000","last_undegraded":"2024-03-22T02:11:38.931053+0000","last_fullsized":"2024-03-22T02:11:38.931053+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854013+0000","last_change":"2024-03-22T02:11:37.941467+0000","last_active":"2024-03-22T02:11:39.854013+0000","last_peered":"2024-03-22T02:11:39.854013+0000","last_clean":"2024-03-22T02:11:39.854013+0000","last_became_active":"2024-03-22T02:11:37.941067+0000","last_became_peered":"2024-03-22T02:11:37.941067+0000","last_unstale":"2024-03-22T02:11:39.854013+0000","last_undegraded":"2024-03-22T02:11:39.854013+0000","last_fullsized":"2024-03-22T02:11:39.854013+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:25:35.808781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984908+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:11:52.984908+0000","last_peered":"2024-03-22T02:11:52.984908+0000","last_clean":"2024-03-22T02:11:52.984908+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:11:52.984908+0000","last_undegraded":"2024-03-22T02:11:52.984908+0000","last_fullsized":"2024-03-22T02:11:52.984908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933648+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:11:38.933648+0000","last_peered":"2024-03-22T02:11:38.933648+0000","last_clean":"2024-03-22T02:11:38.933648+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:11:38.933648+0000","last_undegraded":"2024-03-22T02:11:38.933648+0000","last_fullsized":"2024-03-22T02:11:38.933648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974283+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:11:40.974283+0000","last_peered":"2024-03-22T02:11:40.974283+0000","last_clean":"2024-03-22T02:11:40.974283+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:11:40.974283+0000","last_undegraded":"2024-03-22T02:11:40.974283+0000","last_fullsized":"2024-03-22T02:11:40.974283+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.151949+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:11:40.151949+0000","last_peered":"2024-03-22T02:11:40.151949+0000","last_clean":"2024-03-22T02:11:40.151949+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:11:40.151949+0000","last_undegraded":"2024-03-22T02:11:40.151949+0000","last_fullsized":"2024-03-22T02:11:40.151949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933282+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:11:38.933282+0000","last_peered":"2024-03-22T02:11:38.933282+0000","last_clean":"2024-03-22T02:11:38.933282+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:11:38.933282+0000","last_undegraded":"2024-03-22T02:11:38.933282+0000","last_fullsized":"2024-03-22T02:11:38.933282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931608+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:11:38.931608+0000","last_peered":"2024-03-22T02:11:38.931608+0000","last_clean":"2024-03-22T02:11:38.931608+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:11:38.931608+0000","last_undegraded":"2024-03-22T02:11:38.931608+0000","last_fullsized":"2024-03-22T02:11:38.931608+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989836+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:11:53.989836+0000","last_peered":"2024-03-22T02:11:53.989836+0000","last_clean":"2024-03-22T02:11:53.989836+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:11:53.989836+0000","last_undegraded":"2024-03-22T02:11:53.989836+0000","last_fullsized":"2024-03-22T02:11:53.989836+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934550+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:11:38.934550+0000","last_peered":"2024-03-22T02:11:38.934550+0000","last_clean":"2024-03-22T02:11:38.934550+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:11:38.934550+0000","last_undegraded":"2024-03-22T02:11:38.934550+0000","last_fullsized":"2024-03-22T02:11:38.934550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934559+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:11:38.934559+0000","last_peered":"2024-03-22T02:11:38.934559+0000","last_clean":"2024-03-22T02:11:38.934559+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:11:38.934559+0000","last_undegraded":"2024-03-22T02:11:38.934559+0000","last_fullsized":"2024-03-22T02:11:38.934559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854196+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:11:39.854196+0000","last_peered":"2024-03-22T02:11:39.854196+0000","last_clean":"2024-03-22T02:11:39.854196+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:11:39.854196+0000","last_undegraded":"2024-03-22T02:11:39.854196+0000","last_fullsized":"2024-03-22T02:11:39.854196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854722+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:11:39.854722+0000","last_peered":"2024-03-22T02:11:39.854722+0000","last_clean":"2024-03-22T02:11:39.854722+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:11:39.854722+0000","last_undegraded":"2024-03-22T02:11:39.854722+0000","last_fullsized":"2024-03-22T02:11:39.854722+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854326+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:11:39.854326+0000","last_peered":"2024-03-22T02:11:39.854326+0000","last_clean":"2024-03-22T02:11:39.854326+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:11:39.854326+0000","last_undegraded":"2024-03-22T02:11:39.854326+0000","last_fullsized":"2024-03-22T02:11:39.854326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973260+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:11:40.973260+0000","last_peered":"2024-03-22T02:11:40.973260+0000","last_clean":"2024-03-22T02:11:40.973260+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:11:40.973260+0000","last_undegraded":"2024-03-22T02:11:40.973260+0000","last_fullsized":"2024-03-22T02:11:40.973260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853076+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:11:39.853076+0000","last_peered":"2024-03-22T02:11:39.853076+0000","last_clean":"2024-03-22T02:11:39.853076+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:11:39.853076+0000","last_undegraded":"2024-03-22T02:11:39.853076+0000","last_fullsized":"2024-03-22T02:11:39.853076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931353+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:11:38.931353+0000","last_peered":"2024-03-22T02:11:38.931353+0000","last_clean":"2024-03-22T02:11:38.931353+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:11:38.931353+0000","last_undegraded":"2024-03-22T02:11:38.931353+0000","last_fullsized":"2024-03-22T02:11:38.931353+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933333+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:11:38.933333+0000","last_peered":"2024-03-22T02:11:38.933333+0000","last_clean":"2024-03-22T02:11:38.933333+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:11:38.933333+0000","last_undegraded":"2024-03-22T02:11:38.933333+0000","last_fullsized":"2024-03-22T02:11:38.933333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.852832+0000","last_change":"2024-03-22T02:11:33.643607+0000","last_active":"2024-03-22T02:11:39.852832+0000","last_peered":"2024-03-22T02:11:39.852832+0000","last_clean":"2024-03-22T02:11:39.852832+0000","last_became_active":"2024-03-22T02:11:33.642717+0000","last_became_peered":"2024-03-22T02:11:33.642717+0000","last_unstale":"2024-03-22T02:11:39.852832+0000","last_undegraded":"2024-03-22T02:11:39.852832+0000","last_fullsized":"2024-03-22T02:11:39.852832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:05:22.727788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931781+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:11:38.931781+0000","last_peered":"2024-03-22T02:11:38.931781+0000","last_clean":"2024-03-22T02:11:38.931781+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:11:38.931781+0000","last_undegraded":"2024-03-22T02:11:38.931781+0000","last_fullsized":"2024-03-22T02:11:38.931781+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934609+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:11:38.934609+0000","last_peered":"2024-03-22T02:11:38.934609+0000","last_clean":"2024-03-22T02:11:38.934609+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:11:38.934609+0000","last_undegraded":"2024-03-22T02:11:38.934609+0000","last_fullsized":"2024-03-22T02:11:38.934609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.930886+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:11:38.930886+0000","last_peered":"2024-03-22T02:11:38.930886+0000","last_clean":"2024-03-22T02:11:38.930886+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:11:38.930886+0000","last_undegraded":"2024-03-22T02:11:38.930886+0000","last_fullsized":"2024-03-22T02:11:38.930886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933767+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:11:38.933767+0000","last_peered":"2024-03-22T02:11:38.933767+0000","last_clean":"2024-03-22T02:11:38.933767+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:11:38.933767+0000","last_undegraded":"2024-03-22T02:11:38.933767+0000","last_fullsized":"2024-03-22T02:11:38.933767+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137948+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:11:39.137948+0000","last_peered":"2024-03-22T02:11:39.137948+0000","last_clean":"2024-03-22T02:11:39.137948+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:11:39.137948+0000","last_undegraded":"2024-03-22T02:11:39.137948+0000","last_fullsized":"2024-03-22T02:11:39.137948+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.982550+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:11:52.982550+0000","last_peered":"2024-03-22T02:11:52.982550+0000","last_clean":"2024-03-22T02:11:52.982550+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:11:52.982550+0000","last_undegraded":"2024-03-22T02:11:52.982550+0000","last_fullsized":"2024-03-22T02:11:52.982550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":345,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933171+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:11:38.933171+0000","last_peered":"2024-03-22T02:11:38.933171+0000","last_clean":"2024-03-22T02:11:38.933171+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:11:38.933171+0000","last_undegraded":"2024-03-22T02:11:38.933171+0000","last_fullsized":"2024-03-22T02:11:38.933171+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935104+0000","last_change":"2024-03-22T02:11:37.942403+0000","last_active":"2024-03-22T02:11:38.935104+0000","last_peered":"2024-03-22T02:11:38.935104+0000","last_clean":"2024-03-22T02:11:38.935104+0000","last_became_active":"2024-03-22T02:11:37.942040+0000","last_became_peered":"2024-03-22T02:11:37.942040+0000","last_unstale":"2024-03-22T02:11:38.935104+0000","last_undegraded":"2024-03-22T02:11:38.935104+0000","last_fullsized":"2024-03-22T02:11:38.935104+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:12:02.337725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.930886+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:11:38.930886+0000","last_peered":"2024-03-22T02:11:38.930886+0000","last_clean":"2024-03-22T02:11:38.930886+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:11:38.930886+0000","last_undegraded":"2024-03-22T02:11:38.930886+0000","last_fullsized":"2024-03-22T02:11:38.930886+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933591+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:11:38.933591+0000","last_peered":"2024-03-22T02:11:38.933591+0000","last_clean":"2024-03-22T02:11:38.933591+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:11:38.933591+0000","last_undegraded":"2024-03-22T02:11:38.933591+0000","last_fullsized":"2024-03-22T02:11:38.933591+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854108+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:11:39.854108+0000","last_peered":"2024-03-22T02:11:39.854108+0000","last_clean":"2024-03-22T02:11:39.854108+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:11:39.854108+0000","last_undegraded":"2024-03-22T02:11:39.854108+0000","last_fullsized":"2024-03-22T02:11:39.854108+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854438+0000","last_change":"2024-03-22T02:11:32.915749+0000","last_active":"2024-03-22T02:11:39.854438+0000","last_peered":"2024-03-22T02:11:39.854438+0000","last_clean":"2024-03-22T02:11:39.854438+0000","last_became_active":"2024-03-22T02:11:32.915455+0000","last_became_peered":"2024-03-22T02:11:32.915455+0000","last_unstale":"2024-03-22T02:11:39.854438+0000","last_undegraded":"2024-03-22T02:11:39.854438+0000","last_fullsized":"2024-03-22T02:11:39.854438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:49.835132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931478+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:11:38.931478+0000","last_peered":"2024-03-22T02:11:38.931478+0000","last_clean":"2024-03-22T02:11:38.931478+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:11:38.931478+0000","last_undegraded":"2024-03-22T02:11:38.931478+0000","last_fullsized":"2024-03-22T02:11:38.931478+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.137899+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:11:39.137899+0000","last_peered":"2024-03-22T02:11:39.137899+0000","last_clean":"2024-03-22T02:11:39.137899+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:11:39.137899+0000","last_undegraded":"2024-03-22T02:11:39.137899+0000","last_fullsized":"2024-03-22T02:11:39.137899+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:53.989334+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:11:53.989334+0000","last_peered":"2024-03-22T02:11:53.989334+0000","last_clean":"2024-03-22T02:11:53.989334+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:11:53.989334+0000","last_undegraded":"2024-03-22T02:11:53.989334+0000","last_fullsized":"2024-03-22T02:11:53.989334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138496+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:11:39.138496+0000","last_peered":"2024-03-22T02:11:39.138496+0000","last_clean":"2024-03-22T02:11:39.138496+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:11:39.138496+0000","last_undegraded":"2024-03-22T02:11:39.138496+0000","last_fullsized":"2024-03-22T02:11:39.138496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973842+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:11:40.973842+0000","last_peered":"2024-03-22T02:11:40.973842+0000","last_clean":"2024-03-22T02:11:40.973842+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:11:40.973842+0000","last_undegraded":"2024-03-22T02:11:40.973842+0000","last_fullsized":"2024-03-22T02:11:40.973842+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.983612+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:11:52.983612+0000","last_peered":"2024-03-22T02:11:52.983612+0000","last_clean":"2024-03-22T02:11:52.983612+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:11:52.983612+0000","last_undegraded":"2024-03-22T02:11:52.983612+0000","last_fullsized":"2024-03-22T02:11:52.983612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918967+0000","last_change":"2024-03-22T02:11:33.650506+0000","last_active":"2024-03-22T02:11:39.918967+0000","last_peered":"2024-03-22T02:11:39.918967+0000","last_clean":"2024-03-22T02:11:39.918967+0000","last_became_active":"2024-03-22T02:11:33.649812+0000","last_became_peered":"2024-03-22T02:11:33.649812+0000","last_unstale":"2024-03-22T02:11:39.918967+0000","last_undegraded":"2024-03-22T02:11:39.918967+0000","last_fullsized":"2024-03-22T02:11:39.918967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:29:08.991454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152246+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:11:40.152246+0000","last_peered":"2024-03-22T02:11:40.152246+0000","last_clean":"2024-03-22T02:11:40.152246+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:11:40.152246+0000","last_undegraded":"2024-03-22T02:11:40.152246+0000","last_fullsized":"2024-03-22T02:11:40.152246+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.985850+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:11:52.985850+0000","last_peered":"2024-03-22T02:11:52.985850+0000","last_clean":"2024-03-22T02:11:52.985850+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:11:52.985850+0000","last_undegraded":"2024-03-22T02:11:52.985850+0000","last_fullsized":"2024-03-22T02:11:52.985850+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853828+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:11:39.853828+0000","last_peered":"2024-03-22T02:11:39.853828+0000","last_clean":"2024-03-22T02:11:39.853828+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:11:39.853828+0000","last_undegraded":"2024-03-22T02:11:39.853828+0000","last_fullsized":"2024-03-22T02:11:39.853828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974343+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:11:40.974343+0000","last_peered":"2024-03-22T02:11:40.974343+0000","last_clean":"2024-03-22T02:11:40.974343+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:11:40.974343+0000","last_undegraded":"2024-03-22T02:11:40.974343+0000","last_fullsized":"2024-03-22T02:11:40.974343+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.984866+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:11:52.984866+0000","last_peered":"2024-03-22T02:11:52.984866+0000","last_clean":"2024-03-22T02:11:52.984866+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:11:52.984866+0000","last_undegraded":"2024-03-22T02:11:52.984866+0000","last_fullsized":"2024-03-22T02:11:52.984866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853723+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:11:39.853723+0000","last_peered":"2024-03-22T02:11:39.853723+0000","last_clean":"2024-03-22T02:11:39.853723+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:11:39.853723+0000","last_undegraded":"2024-03-22T02:11:39.853723+0000","last_fullsized":"2024-03-22T02:11:39.853723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152163+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:11:40.152163+0000","last_peered":"2024-03-22T02:11:40.152163+0000","last_clean":"2024-03-22T02:11:40.152163+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:11:40.152163+0000","last_undegraded":"2024-03-22T02:11:40.152163+0000","last_fullsized":"2024-03-22T02:11:40.152163+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973269+0000","last_change":"2024-03-22T02:11:32.907985+0000","last_active":"2024-03-22T02:11:40.973269+0000","last_peered":"2024-03-22T02:11:40.973269+0000","last_clean":"2024-03-22T02:11:40.973269+0000","last_became_active":"2024-03-22T02:11:32.907686+0000","last_became_peered":"2024-03-22T02:11:32.907686+0000","last_unstale":"2024-03-22T02:11:40.973269+0000","last_undegraded":"2024-03-22T02:11:40.973269+0000","last_fullsized":"2024-03-22T02:11:40.973269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:30:40.945298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138108+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:11:39.138108+0000","last_peered":"2024-03-22T02:11:39.138108+0000","last_clean":"2024-03-22T02:11:39.138108+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:11:39.138108+0000","last_undegraded":"2024-03-22T02:11:39.138108+0000","last_fullsized":"2024-03-22T02:11:39.138108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.152196+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:11:40.152196+0000","last_peered":"2024-03-22T02:11:40.152196+0000","last_clean":"2024-03-22T02:11:40.152196+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:11:40.152196+0000","last_undegraded":"2024-03-22T02:11:40.152196+0000","last_fullsized":"2024-03-22T02:11:40.152196+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:52.985865+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:11:52.985865+0000","last_peered":"2024-03-22T02:11:52.985865+0000","last_clean":"2024-03-22T02:11:52.985865+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:11:52.985865+0000","last_undegraded":"2024-03-22T02:11:52.985865+0000","last_fullsized":"2024-03-22T02:11:52.985865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933446+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:11:38.933446+0000","last_peered":"2024-03-22T02:11:38.933446+0000","last_clean":"2024-03-22T02:11:38.933446+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:11:38.933446+0000","last_undegraded":"2024-03-22T02:11:38.933446+0000","last_fullsized":"2024-03-22T02:11:38.933446+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638625+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:11:39.638625+0000","last_peered":"2024-03-22T02:11:39.638625+0000","last_clean":"2024-03-22T02:11:39.638625+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:11:39.638625+0000","last_undegraded":"2024-03-22T02:11:39.638625+0000","last_fullsized":"2024-03-22T02:11:39.638625+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973318+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:11:40.973318+0000","last_peered":"2024-03-22T02:11:40.973318+0000","last_clean":"2024-03-22T02:11:40.973318+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:11:40.973318+0000","last_undegraded":"2024-03-22T02:11:40.973318+0000","last_fullsized":"2024-03-22T02:11:40.973318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933876+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:11:38.933876+0000","last_peered":"2024-03-22T02:11:38.933876+0000","last_clean":"2024-03-22T02:11:38.933876+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:11:38.933876+0000","last_undegraded":"2024-03-22T02:11:38.933876+0000","last_fullsized":"2024-03-22T02:11:38.933876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918634+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:11:39.918634+0000","last_peered":"2024-03-22T02:11:39.918634+0000","last_clean":"2024-03-22T02:11:39.918634+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:11:39.918634+0000","last_undegraded":"2024-03-22T02:11:39.918634+0000","last_fullsized":"2024-03-22T02:11:39.918634+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.935013+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:11:38.935013+0000","last_peered":"2024-03-22T02:11:38.935013+0000","last_clean":"2024-03-22T02:11:38.935013+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:11:38.935013+0000","last_undegraded":"2024-03-22T02:11:38.935013+0000","last_fullsized":"2024-03-22T02:11:38.935013+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933530+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:11:38.933530+0000","last_peered":"2024-03-22T02:11:38.933530+0000","last_clean":"2024-03-22T02:11:38.933530+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:11:38.933530+0000","last_undegraded":"2024-03-22T02:11:38.933530+0000","last_fullsized":"2024-03-22T02:11:38.933530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934312+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:11:38.934312+0000","last_peered":"2024-03-22T02:11:38.934312+0000","last_clean":"2024-03-22T02:11:38.934312+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:11:38.934312+0000","last_undegraded":"2024-03-22T02:11:38.934312+0000","last_fullsized":"2024-03-22T02:11:38.934312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931561+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:11:38.931561+0000","last_peered":"2024-03-22T02:11:38.931561+0000","last_clean":"2024-03-22T02:11:38.931561+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:11:38.931561+0000","last_undegraded":"2024-03-22T02:11:38.931561+0000","last_fullsized":"2024-03-22T02:11:38.931561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.933755+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:11:38.933755+0000","last_peered":"2024-03-22T02:11:38.933755+0000","last_clean":"2024-03-22T02:11:38.933755+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:11:38.933755+0000","last_undegraded":"2024-03-22T02:11:38.933755+0000","last_fullsized":"2024-03-22T02:11:38.933755+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.853808+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:11:39.853808+0000","last_peered":"2024-03-22T02:11:39.853808+0000","last_clean":"2024-03-22T02:11:39.853808+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:11:39.853808+0000","last_undegraded":"2024-03-22T02:11:39.853808+0000","last_fullsized":"2024-03-22T02:11:39.853808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974310+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:11:40.974310+0000","last_peered":"2024-03-22T02:11:40.974310+0000","last_clean":"2024-03-22T02:11:40.974310+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:11:40.974310+0000","last_undegraded":"2024-03-22T02:11:40.974310+0000","last_fullsized":"2024-03-22T02:11:40.974310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.973816+0000","last_change":"2024-03-22T02:11:33.635594+0000","last_active":"2024-03-22T02:11:40.973816+0000","last_peered":"2024-03-22T02:11:40.973816+0000","last_clean":"2024-03-22T02:11:40.973816+0000","last_became_active":"2024-03-22T02:11:33.635298+0000","last_became_peered":"2024-03-22T02:11:33.635298+0000","last_unstale":"2024-03-22T02:11:40.973816+0000","last_undegraded":"2024-03-22T02:11:40.973816+0000","last_fullsized":"2024-03-22T02:11:40.973816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:39:51.824729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.918700+0000","last_change":"2024-03-22T02:11:33.643684+0000","last_active":"2024-03-22T02:11:39.918700+0000","last_peered":"2024-03-22T02:11:39.918700+0000","last_clean":"2024-03-22T02:11:39.918700+0000","last_became_active":"2024-03-22T02:11:33.643339+0000","last_became_peered":"2024-03-22T02:11:33.643339+0000","last_unstale":"2024-03-22T02:11:39.918700+0000","last_undegraded":"2024-03-22T02:11:39.918700+0000","last_fullsized":"2024-03-22T02:11:39.918700+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:46:34.059019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931359+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:11:38.931359+0000","last_peered":"2024-03-22T02:11:38.931359+0000","last_clean":"2024-03-22T02:11:38.931359+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:11:38.931359+0000","last_undegraded":"2024-03-22T02:11:38.931359+0000","last_fullsized":"2024-03-22T02:11:38.931359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854696+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:11:39.854696+0000","last_peered":"2024-03-22T02:11:39.854696+0000","last_clean":"2024-03-22T02:11:39.854696+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:11:39.854696+0000","last_undegraded":"2024-03-22T02:11:39.854696+0000","last_fullsized":"2024-03-22T02:11:39.854696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:40.974199+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:11:40.974199+0000","last_peered":"2024-03-22T02:11:40.974199+0000","last_clean":"2024-03-22T02:11:40.974199+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:11:40.974199+0000","last_undegraded":"2024-03-22T02:11:40.974199+0000","last_fullsized":"2024-03-22T02:11:40.974199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138612+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:11:39.138612+0000","last_peered":"2024-03-22T02:11:39.138612+0000","last_clean":"2024-03-22T02:11:39.138612+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:11:39.138612+0000","last_undegraded":"2024-03-22T02:11:39.138612+0000","last_fullsized":"2024-03-22T02:11:39.138612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.934497+0000","last_change":"2024-03-22T02:11:33.640871+0000","last_active":"2024-03-22T02:11:38.934497+0000","last_peered":"2024-03-22T02:11:38.934497+0000","last_clean":"2024-03-22T02:11:38.934497+0000","last_became_active":"2024-03-22T02:11:33.640579+0000","last_became_peered":"2024-03-22T02:11:33.640579+0000","last_unstale":"2024-03-22T02:11:38.934497+0000","last_undegraded":"2024-03-22T02:11:38.934497+0000","last_fullsized":"2024-03-22T02:11:38.934497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:30:03.492201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.138121+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:11:39.138121+0000","last_peered":"2024-03-22T02:11:39.138121+0000","last_clean":"2024-03-22T02:11:39.138121+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:11:39.138121+0000","last_undegraded":"2024-03-22T02:11:39.138121+0000","last_fullsized":"2024-03-22T02:11:39.138121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.638760+0000","last_change":"2024-03-22T02:11:33.642009+0000","last_active":"2024-03-22T02:11:39.638760+0000","last_peered":"2024-03-22T02:11:39.638760+0000","last_clean":"2024-03-22T02:11:39.638760+0000","last_became_active":"2024-03-22T02:11:33.641673+0000","last_became_peered":"2024-03-22T02:11:33.641673+0000","last_unstale":"2024-03-22T02:11:39.638760+0000","last_undegraded":"2024-03-22T02:11:39.638760+0000","last_fullsized":"2024-03-22T02:11:39.638760+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:01.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:38.931435+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:11:38.931435+0000","last_peered":"2024-03-22T02:11:38.931435+0000","last_clean":"2024-03-22T02:11:38.931435+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:11:38.931435+0000","last_undegraded":"2024-03-22T02:11:38.931435+0000","last_fullsized":"2024-03-22T02:11:38.931435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-22T02:11:39.854378+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:11:39.854378+0000","last_peered":"2024-03-22T02:11:39.854378+0000","last_clean":"2024-03-22T02:11:39.854378+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:11:39.854378+0000","last_undegraded":"2024-03-22T02:11:39.854378+0000","last_fullsized":"2024-03-22T02:11:39.854378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":25457,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":114,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":32,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":38989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350992,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612630,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32216,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712936,"statfs":{"total":95995035648,"available":95962046464,"internally_reserved":0,"allocated":1064960,"data_stored":1076054,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481084,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480892,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1071838,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462954,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484108,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481744,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":484764,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":978944,"data_stored":480370,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":974848,"data_stored":480288,"data_compressed":14692,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":50784,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":24719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:11:58.142 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-22T02:11:58.142 INFO:tasks.cephadm:Setting up client nodes... 2024-03-22T02:11:58.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T02:11:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:58 smithi083 ceph-mon[36340]: pgmap v305: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 3.9 KiB/s wr, 6 op/s 2024-03-22T02:11:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:58 smithi083 ceph-mon[36340]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:58.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/513118119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/941152186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:58.421 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:11:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:58 smithi120 ceph-mon[36803]: pgmap v305: 129 pgs: 129 active+clean; 601 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 3.9 KiB/s wr, 6 op/s 2024-03-22T02:11:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:58 smithi120 ceph-mon[36803]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:11:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/513118119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:11:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/941152186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:11:59.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:59 smithi083 ceph-mon[36340]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:11:59.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:59 smithi083 ceph-mon[36340]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T02:11:59.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:11:59 smithi083 ceph-mon[36340]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:11:59.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:59 smithi028 ceph-mon[31916]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:11:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:59 smithi028 ceph-mon[31916]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T02:11:59.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:11:59 smithi028 ceph-mon[31916]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:11:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:59 smithi120 ceph-mon[36803]: from='client.14943 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:11:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:59 smithi120 ceph-mon[36803]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-22T02:11:59.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:11:59 smithi120 ceph-mon[36803]: from='client.14949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:12:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 -- 172.21.15.28:0/921980112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 msgr2=0x7fd22c100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 --2- 172.21.15.28:0/921980112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c100dc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd2240099c0 tx=0x7fd224009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 -- 172.21.15.28:0/921980112 shutdown_connections 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 --2- 172.21.15.28:0/921980112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c100dc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 --2- 172.21.15.28:0/921980112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 0x7fd22c10ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 --2- 172.21.15.28:0/921980112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 -- 172.21.15.28:0/921980112 >> 172.21.15.28:0/921980112 conn(0x7fd22c0fc320 msgr2=0x7fd22c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 -- 172.21.15.28:0/921980112 shutdown_connections 2024-03-22T02:12:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.113+0000 7fd230a51700 1 -- 172.21.15.28:0/921980112 wait complete. 2024-03-22T02:12:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 Processor -- start 2024-03-22T02:12:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 -- start start 2024-03-22T02:12:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:00.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 0x7fd22c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10e2b0 con 0x7fd22c1009a0 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10e130 con 0x7fd22c1043c0 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd230a51700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd22c10e430 con 0x7fd22c103a20 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd22a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd22a59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50122/0 (socket says 172.21.15.28:50122) 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd22a59c700 1 -- 172.21.15.28:0/14384513 learned_addr learned my addr 172.21.15.28:0/14384513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd22ad9d700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 0x7fd22c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 -- 172.21.15.28:0/14384513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 msgr2=0x7fd22c19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 0x7fd22c19fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 -- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 msgr2=0x7fd22c198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd229d9b700 1 -- 172.21.15.28:0/14384513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd22c1a0490 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.114+0000 7fd22a59c700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd229d9b700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c198d70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd21400ea20 tx=0x7fd21400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd21400cbd0 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd22c1a0780 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd214004500 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd214005020 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.115+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd22c1a0c70 con 0x7fd22c103a20 2024-03-22T02:12:00.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.116+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd22c050b90 con 0x7fd22c103a20 2024-03-22T02:12:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.117+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fd214042b30 con 0x7fd22c103a20 2024-03-22T02:12:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.117+0000 7fd2237fe700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 0x7fd2180a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.118+0000 7fd22a59c700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 0x7fd2180a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.118+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fd2140cf650 con 0x7fd22c103a20 2024-03-22T02:12:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.118+0000 7fd22a59c700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 0x7fd2180a55b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd224000c00 tx=0x7fd224005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:00.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.126+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd21401fe20 con 0x7fd22c103a20 2024-03-22T02:12:00.264 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:11:59 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: 2024-03-22T02:11:59.835+0000 7f8161474700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-22T02:12:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:00 smithi083 ceph-mon[36340]: pgmap v306: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-22T02:12:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:00 smithi028 ceph-mon[31916]: pgmap v306: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-22T02:12:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.416+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fd22c02de40 con 0x7fd22c103a20 2024-03-22T02:12:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.424+0000 7fd2237fe700 1 -- 172.21.15.28:0/14384513 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fd22c02de40 con 0x7fd22c103a20 2024-03-22T02:12:00.425 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-03-22T02:12:00.425 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDw6Pxl848DGRAAul+ZxxCXM18X5XMUatr/6w== 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 msgr2=0x7fd2180a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 0x7fd2180a55b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd224000c00 tx=0x7fd224005fb0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 msgr2=0x7fd22c198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c198d70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd21400ea20 tx=0x7fd21400ed30 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 shutdown_connections 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd2180a30f0 0x7fd2180a55b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd22c1009a0 0x7fd22c198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd22c1043c0 0x7fd22c19fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 --2- 172.21.15.28:0/14384513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd22c103a20 0x7fd22c198d70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 >> 172.21.15.28:0/14384513 conn(0x7fd22c0fc320 msgr2=0x7fd22c106cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 shutdown_connections 2024-03-22T02:12:00.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:00.426+0000 7fd230a51700 1 -- 172.21.15.28:0/14384513 wait complete. 2024-03-22T02:12:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:00 smithi120 ceph-mon[36803]: pgmap v306: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.2 KiB/s wr, 8 op/s 2024-03-22T02:12:01.098 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:12:01.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-22T02:12:01.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-22T02:12:01.213 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-22T02:12:01.214 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-22T02:12:01.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-22T02:12:01.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:01 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/14384513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:01 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:01 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:12:01.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:01 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/14384513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:01 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.279 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:01 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:12:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:01 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/14384513' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:01 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T02:12:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:01 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T02:12:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:02 smithi028 ceph-mon[31916]: pgmap v307: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-22T02:12:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:02 smithi120 ceph-mon[36803]: pgmap v307: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-22T02:12:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:02 smithi083 ceph-mon[36340]: pgmap v307: 129 pgs: 129 active+clean; 603 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 -- 172.21.15.28:0/1066622291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff334103090 msgr2=0x7ff33410f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/1066622291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff334103090 0x7ff33410f4d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff32c00b0d0 tx=0x7ff32c00b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 -- 172.21.15.28:0/1066622291 shutdown_connections 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/1066622291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff334103090 0x7ff33410f4d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/1066622291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341074f0 0x7ff3341078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/1066622291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3341026d0 0x7ff334102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 -- 172.21.15.28:0/1066622291 >> 172.21.15.28:0/1066622291 conn(0x7ff3340797b0 msgr2=0x7ff334079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 -- 172.21.15.28:0/1066622291 shutdown_connections 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.244+0000 7ff33bd0c700 1 -- 172.21.15.28:0/1066622291 wait complete. 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 Processor -- start 2024-03-22T02:12:03.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 -- start start 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341026d0 0x7ff334198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff334103090 0x7ff334199360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff334112950 con 0x7ff3341074f0 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3341127d0 con 0x7ff334103090 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33bd0c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff334112ad0 con 0x7ff3341026d0 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33a2a9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33a2a9700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39284/0 (socket says 172.21.15.28:39284) 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff33a2a9700 1 -- 172.21.15.28:0/2878162494 learned_addr learned my addr 172.21.15.28:0/2878162494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff339aa8700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341026d0 0x7ff334198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.245+0000 7ff3392a7700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff334103090 0x7ff334199360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:03.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 -- 172.21.15.28:0/2878162494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341026d0 msgr2=0x7ff334198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341026d0 0x7ff334198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 -- 172.21.15.28:0/2878162494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff334103090 msgr2=0x7ff334199360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff334103090 0x7ff334199360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 -- 172.21.15.28:0/2878162494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33419de30 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33a2a9700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff32c006070 tx=0x7ff32c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c003c00 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff33419e0c0 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff32c022070 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32c01b6c0 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.246+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3341a59a0 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.247+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff334103d60 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.248+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7ff32c01d470 con 0x7ff3341074f0 2024-03-22T02:12:03.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.249+0000 7ff32affd700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 0x7ff3200a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.249+0000 7ff339aa8700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 0x7ff3200a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.249+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7ff32c0d0720 con 0x7ff3341074f0 2024-03-22T02:12:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.249+0000 7ff339aa8700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 0x7ff3200a5500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff324009be0 tx=0x7ff324009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.253+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff32c09d1e0 con 0x7ff3341074f0 2024-03-22T02:12:03.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.521+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff334069340 con 0x7ff3341074f0 2024-03-22T02:12:03.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.526+0000 7ff32affd700 1 -- 172.21.15.28:0/2878162494 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263551 (secure 0 0 0) 0x7ff32c09cae0 con 0x7ff3341074f0 2024-03-22T02:12:03.530 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:12:03.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 msgr2=0x7ff3200a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 0x7ff3200a5500 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff324009be0 tx=0x7ff324009260 comp rx=0 tx=0).stop 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 msgr2=0x7ff33419d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff32c006070 tx=0x7ff32c007a20 comp rx=0 tx=0).stop 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 shutdown_connections 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3200a3040 0x7ff3200a5500 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3341074f0 0x7ff33419d6f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3341026d0 0x7ff334198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 --2- 172.21.15.28:0/2878162494 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff334103090 0x7ff334199360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 >> 172.21.15.28:0/2878162494 conn(0x7ff3340797b0 msgr2=0x7ff334100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 shutdown_connections 2024-03-22T02:12:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:03.529+0000 7ff33bd0c700 1 -- 172.21.15.28:0/2878162494 wait complete. 2024-03-22T02:12:04.359 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":4205492780},{"type":"v1","addr":"172.21.15.28:6801","nonce":4205492780}]},"active_addr":"172.21.15.28:6801/4205492780","active_change":"2024-03-22T02:04:26.467876+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24118,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24124,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":526964316}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3351322426}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2120269619}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3236218180}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3460469772}]}]} 2024-03-22T02:12:04.363 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-22T02:12:04.364 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-22T02:12:04.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:12:04.613 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:04.735 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:04 smithi028 ceph-mon[31916]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-22T02:12:04.735 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:04 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2878162494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T02:12:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:04 smithi120 ceph-mon[36803]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-22T02:12:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:04 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2878162494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T02:12:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:04 smithi083 ceph-mon[36340]: pgmap v308: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 13 op/s 2024-03-22T02:12:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:04 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2878162494' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T02:12:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 -- 172.21.15.28:0/766096668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 msgr2=0x7f7ef010f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/766096668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef010f4d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800b0d0 tx=0x7f7ee800b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 -- 172.21.15.28:0/766096668 shutdown_connections 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/766096668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef010f4d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/766096668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 0x7f7ef01078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/766096668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 0x7f7ef0102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 -- 172.21.15.28:0/766096668 >> 172.21.15.28:0/766096668 conn(0x7f7ef00797b0 msgr2=0x7f7ef0079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 -- 172.21.15.28:0/766096668 shutdown_connections 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 -- 172.21.15.28:0/766096668 wait complete. 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.345+0000 7f7ef6b68700 1 Processor -- start 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 -- start start 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 0x7f7ef0198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 0x7f7ef019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0112960 con 0x7f7ef0103090 2024-03-22T02:12:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef01127e0 con 0x7f7ef01074f0 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef6b68700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0112ae0 con 0x7f7ef01026d0 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef4904700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 0x7f7ef0198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39300/0 (socket says 172.21.15.28:39300) 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 -- 172.21.15.28:0/1838747457 learned_addr learned my addr 172.21.15.28:0/1838747457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7ef5105700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 0x7f7ef019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 -- 172.21.15.28:0/1838747457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 msgr2=0x7f7ef0198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 0x7f7ef0198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 -- 172.21.15.28:0/1838747457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 msgr2=0x7f7ef019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 0x7f7ef019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 -- 172.21.15.28:0/1838747457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef019ddc0 con 0x7f7ef0103090 2024-03-22T02:12:06.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eeffff700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e9c0 tx=0x7f7edc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.346+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00cbd0 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.347+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef019e0b0 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.347+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7edc00ee60 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.347+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc005630 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.347+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef01a5930 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.348+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef0069340 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.349+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f7edc004750 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.349+0000 7f7eedffb700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 0x7f7ee00a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.350+0000 7f7ef4904700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 0x7f7ee00a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.350+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f7edc0cfa90 con 0x7f7ef0103090 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.350+0000 7f7ef4904700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 0x7f7ee00a5500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009be0 tx=0x7f7ee4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:06.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.354+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7edc09c4c0 con 0x7f7ef0103090 2024-03-22T02:12:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.570+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ef002dd70 con 0x7f7ef0103090 2024-03-22T02:12:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.572+0000 7f7eedffb700 1 -- 172.21.15.28:0/1838747457 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f7edc09bdc0 con 0x7f7ef0103090 2024-03-22T02:12:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:12:06.574 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 msgr2=0x7f7ee00a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 0x7f7ee00a5500 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4009be0 tx=0x7f7ee4009260 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 msgr2=0x7f7ef01992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e9c0 tx=0x7f7edc00ecd0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 shutdown_connections 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7ef01074f0 0x7f7ef019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ef01026d0 0x7f7ef0198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7ee00a3040 0x7f7ee00a5500 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 --2- 172.21.15.28:0/1838747457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ef0103090 0x7f7ef01992f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.574+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 >> 172.21.15.28:0/1838747457 conn(0x7f7ef00797b0 msgr2=0x7f7ef0100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:06.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.575+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 shutdown_connections 2024-03-22T02:12:06.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:06.575+0000 7f7ef6b68700 1 -- 172.21.15.28:0/1838747457 wait complete. 2024-03-22T02:12:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:06 smithi120 ceph-mon[36803]: pgmap v309: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-03-22T02:12:06.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:06 smithi083 ceph-mon[36340]: pgmap v309: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-03-22T02:12:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:06 smithi028 ceph-mon[31916]: pgmap v309: 129 pgs: 129 active+clean; 625 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.3 KiB/s wr, 14 op/s 2024-03-22T02:12:07.385 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-22T02:12:07.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T02:12:07.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:07.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:07 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1838747457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:07.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:07 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1838747457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:07.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:07 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1838747457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:08.735 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:08 smithi028 ceph-mon[31916]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-22T02:12:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:08 smithi120 ceph-mon[36803]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-22T02:12:08.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:08 smithi083 ceph-mon[36340]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 -- 172.21.15.28:0/2210061977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381075f0 msgr2=0x7f4f381079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 --2- 172.21.15.28:0/2210061977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381075f0 0x7f4f381079d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4f280099c0 tx=0x7f4f28009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 -- 172.21.15.28:0/2210061977 shutdown_connections 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 --2- 172.21.15.28:0/2210061977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381075f0 0x7f4f381079d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 --2- 172.21.15.28:0/2210061977 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 0x7f4f3810e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 --2- 172.21.15.28:0/2210061977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381015b0 0x7f4f38101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 -- 172.21.15.28:0/2210061977 >> 172.21.15.28:0/2210061977 conn(0x7f4f380775d0 msgr2=0x7f4f380779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 -- 172.21.15.28:0/2210061977 shutdown_connections 2024-03-22T02:12:09.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.293+0000 7f4f3e716700 1 -- 172.21.15.28:0/2210061977 wait complete. 2024-03-22T02:12:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 Processor -- start 2024-03-22T02:12:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 -- start start 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 0x7f4f3819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381075f0 0x7f4f381a4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f381118a0 con 0x7f4f381015b0 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38111720 con 0x7f4f38101f70 2024-03-22T02:12:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3e716700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f38111a20 con 0x7f4f381075f0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39316/0 (socket says 172.21.15.28:39316) 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 -- 172.21.15.28:0/1339176399 learned_addr learned my addr 172.21.15.28:0/1339176399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f377fe700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 0x7f4f3819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f3ccb3700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381075f0 0x7f4f381a4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 -- 172.21.15.28:0/1339176399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381075f0 msgr2=0x7f4f381a4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381075f0 0x7f4f381a4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 -- 172.21.15.28:0/1339176399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 msgr2=0x7f4f3819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 0x7f4f3819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.294+0000 7f4f37fff700 1 -- 172.21.15.28:0/1339176399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f381a5180 con 0x7f4f381015b0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f37fff700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f280098e0 tx=0x7f4f2800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28009e60 con 0x7f4f381015b0 2024-03-22T02:12:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f381979b0 con 0x7f4f381015b0 2024-03-22T02:12:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f280050d0 con 0x7f4f381015b0 2024-03-22T02:12:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f28003d90 con 0x7f4f381015b0 2024-03-22T02:12:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.295+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f38197e90 con 0x7f4f381015b0 2024-03-22T02:12:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.296+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f38109a80 con 0x7f4f381015b0 2024-03-22T02:12:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.297+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f4f2800fa50 con 0x7f4f381015b0 2024-03-22T02:12:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.298+0000 7f4f357fa700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 0x7f4f200a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.298+0000 7f4f377fe700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 0x7f4f200a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.298+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f4f280d0320 con 0x7f4f381015b0 2024-03-22T02:12:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.298+0000 7f4f377fe700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 0x7f4f200a5500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4f38077bd0 tx=0x7f4f2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.302+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f28013070 con 0x7f4f381015b0 2024-03-22T02:12:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.507+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3806b1b0 con 0x7f4f381015b0 2024-03-22T02:12:09.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.508+0000 7f4f357fa700 1 -- 172.21.15.28:0/1339176399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+24077 (secure 0 0 0) 0x7f4f2809ce40 con 0x7f4f381015b0 2024-03-22T02:12:09.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:12:09.509 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:11:38.287742+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.510+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 msgr2=0x7f4f200a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.510+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 0x7f4f200a5500 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4f38077bd0 tx=0x7f4f2c009260 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.510+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 msgr2=0x7f4f38199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.510+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f280098e0 tx=0x7f4f2800f690 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 shutdown_connections 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f200a3040 0x7f4f200a5500 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f381015b0 0x7f4f38199bc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f381075f0 0x7f4f381a4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 --2- 172.21.15.28:0/1339176399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f38101f70 0x7f4f3819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 >> 172.21.15.28:0/1339176399 conn(0x7f4f380775d0 msgr2=0x7f4f38100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:09.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 shutdown_connections 2024-03-22T02:12:09.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:09.511+0000 7f4f3e716700 1 -- 172.21.15.28:0/1339176399 wait complete. 2024-03-22T02:12:10.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-22T02:12:10.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-22T02:12:10.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-22T02:12:10.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-22T02:12:10.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-22T02:12:10.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-22T02:12:10.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-22T02:12:10.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-22T02:12:10.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-22T02:12:10.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-22T02:12:10.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-22T02:12:10.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-22T02:12:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:10 smithi028 ceph-mon[31916]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 11 op/s 2024-03-22T02:12:10.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:10 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1339176399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:10 smithi120 ceph-mon[36803]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 11 op/s 2024-03-22T02:12:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:10 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1339176399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:10.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:10 smithi083 ceph-mon[36340]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 4.1 KiB/s wr, 11 op/s 2024-03-22T02:12:10.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:10 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1339176399' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:12:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:10.951 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.257 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.798 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.799 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.803 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:11.804 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:12 smithi028 ceph-mon[31916]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-22T02:12:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:12 smithi120 ceph-mon[36803]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-22T02:12:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:12 smithi083 ceph-mon[36340]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-03-22T02:12:14.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:14 smithi028 ceph-mon[31916]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-03-22T02:12:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:14 smithi120 ceph-mon[36803]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-03-22T02:12:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:14 smithi083 ceph-mon[36340]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2024-03-22T02:12:16.714 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:16 smithi028 ceph-mon[31916]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-22T02:12:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:16 smithi120 ceph-mon[36803]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-22T02:12:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:16 smithi083 ceph-mon[36340]: pgmap v314: 129 pgs: 129 active+clean; 637 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-03-22T02:12:18.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 -- 172.21.15.28:0/665598121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d01074f0 msgr2=0x7f58d01078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 --2- 172.21.15.28:0/665598121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d01074f0 0x7f58d01078d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58c000b0d0 tx=0x7f58c000b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 -- 172.21.15.28:0/665598121 shutdown_connections 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 --2- 172.21.15.28:0/665598121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d01074f0 0x7f58d01078d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 --2- 172.21.15.28:0/665598121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d0103090 0x7f58d010f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 --2- 172.21.15.28:0/665598121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d01026d0 0x7f58d0102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.980+0000 7f58d5394700 1 -- 172.21.15.28:0/665598121 >> 172.21.15.28:0/665598121 conn(0x7f58d00797b0 msgr2=0x7f58d0079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.981+0000 7f58d5394700 1 -- 172.21.15.28:0/665598121 shutdown_connections 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.981+0000 7f58d5394700 1 -- 172.21.15.28:0/665598121 wait complete. 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.981+0000 7f58d5394700 1 Processor -- start 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.981+0000 7f58d5394700 1 -- start start 2024-03-22T02:12:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d01026d0 0x7f58d0198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 0x7f58d0199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d01074f0 0x7f58d019d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58d01129a0 con 0x7f58d0103090 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0112820 con 0x7f58d01074f0 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58d5394700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f58d0112b20 con 0x7f58d01026d0 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 0x7f58d0199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 0x7f58d0199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47642/0 (socket says 172.21.15.28:47642) 2024-03-22T02:12:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 -- 172.21.15.28:0/3014632709 learned_addr learned my addr 172.21.15.28:0/3014632709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ceffd700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d01026d0 0x7f58d0198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58cf7fe700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d01074f0 0x7f58d019d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 -- 172.21.15.28:0/3014632709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d01026d0 msgr2=0x7f58d0198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f58d01026d0 0x7f58d0198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 -- 172.21.15.28:0/3014632709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d01074f0 msgr2=0x7f58d019d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f58d01074f0 0x7f58d019d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d019de60 con 0x7f58d0103090 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.982+0000 7f58ce7fc700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 0x7f58d0199390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e9c0 tx=0x7f58c400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400cbd0 con 0x7f58d0103090 2024-03-22T02:12:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d019e150 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d01a59d0 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58c400ee50 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4005680 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.983+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f58d0062e10 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.985+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f58c4004750 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.985+0000 7f58b7fff700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58b80a30c0 0x7f58b80a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58ceffd700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58b80a30c0 0x7f58b80a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f58c40cffb0 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58b7fff700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] conn(0x7f58b80add80 0x7f58b80b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58c401ee50 con 0x7f58b80add80 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f58c40d0400 con 0x7f58d0103090 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58cf7fe700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] conn(0x7f58b80add80 0x7f58b80b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58ceffd700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58b80a30c0 0x7f58b80a5580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009fd0 tx=0x7f58c000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.986+0000 7f58cf7fe700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] conn(0x7f58b80add80 0x7f58b80b01c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:17.987+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== osd.0 v2:172.21.15.28:6802/2828048761 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f58c401ee50 con 0x7f58b80add80 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.003+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 --> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f58d0063290 con 0x7f58b80add80 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58b7fff700 1 -- 172.21.15.28:0/3014632709 <== osd.0 v2:172.21.15.28:6802/2828048761 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f58d0063290 con 0x7f58b80add80 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] conn(0x7f58b80add80 msgr2=0x7f58b80b01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] conn(0x7f58b80add80 0x7f58b80b01c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58b80a30c0 msgr2=0x7f58b80a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58b80a30c0 0x7f58b80a5580 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009fd0 tx=0x7f58c000b040 comp rx=0 tx=0).stop 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 msgr2=0x7f58d0199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 --2- 172.21.15.28:0/3014632709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58d0103090 0x7f58d0199390 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e9c0 tx=0x7f58c400ecd0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58cf7fe700 1 -- 172.21.15.28:0/3014632709 reap_dead start 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 shutdown_connections 2024-03-22T02:12:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.004+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 >> 172.21.15.28:0/3014632709 conn(0x7f58d00797b0 msgr2=0x7f58d0100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.005+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 shutdown_connections 2024-03-22T02:12:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.005+0000 7f58d5394700 1 -- 172.21.15.28:0/3014632709 wait complete. 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 -- 172.21.15.28:0/962128845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c107680 msgr2=0x7f958c107a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 --2- 172.21.15.28:0/962128845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c107680 0x7f958c107a60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f957c0098f0 tx=0x7f957c009c00 comp rx=0 tx=0).stop 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 -- 172.21.15.28:0/962128845 shutdown_connections 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 --2- 172.21.15.28:0/962128845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c107680 0x7f958c107a60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 --2- 172.21.15.28:0/962128845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101f30 0x7f958c10e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 --2- 172.21.15.28:0/962128845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101570 0x7f958c1019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 -- 172.21.15.28:0/962128845 >> 172.21.15.28:0/962128845 conn(0x7f958c0784f0 msgr2=0x7f958c078900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 -- 172.21.15.28:0/962128845 shutdown_connections 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.086+0000 7f959231f700 1 -- 172.21.15.28:0/962128845 wait complete. 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 Processor -- start 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 -- start start 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 0x7f958c19d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101f30 0x7f958c19de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c107680 0x7f958c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c111860 con 0x7f958c101570 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1116e0 con 0x7f958c101f30 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f959231f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1119e0 con 0x7f958c107680 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.087+0000 7f958bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 0x7f958c19d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 0x7f958c19d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47660/0 (socket says 172.21.15.28:47660) 2024-03-22T02:12:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 -- 172.21.15.28:0/3404363637 learned_addr learned my addr 172.21.15.28:0/3404363637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f95908bc700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c107680 0x7f958c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958b7fe700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101f30 0x7f958c19de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 -- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c107680 msgr2=0x7f958c197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c107680 0x7f958c197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 -- 172.21.15.28:0/3404363637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101f30 msgr2=0x7f958c19de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101f30 0x7f958c19de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c198220 con 0x7f958c101570 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f958bfff700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 0x7f958c19d900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f957c009230 tx=0x7f957c00fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c005180 con 0x7f958c101570 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c1984b0 con 0x7f958c101570 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f957c00b560 con 0x7f958c101570 2024-03-22T02:12:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c003a70 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.088+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1a47d0 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.089+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f958c062e10 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.090+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f957c029060 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95897fa700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a3040 0x7f95740a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f958b7fe700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a3040 0x7f95740a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f957c0d0820 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95897fa700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] conn(0x7f95740add00 0x7f95740b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f957c00bde0 con 0x7f95740add00 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f957c0d0c70 con 0x7f958c101570 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f95908bc700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] conn(0x7f95740add00 0x7f95740b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.091+0000 7f958b7fe700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a3040 0x7f95740a5500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f958c078a80 tx=0x7f9580009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.092+0000 7f95908bc700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] conn(0x7f95740add00 0x7f95740b0140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.093+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== osd.5 v2:172.21.15.120:6808/1653723326 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f957c00bde0 con 0x7f95740add00 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.109+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 --> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f958c063290 con 0x7f95740add00 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f95897fa700 1 -- 172.21.15.28:0/3404363637 <== osd.5 v2:172.21.15.120:6808/1653723326 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f958c063290 con 0x7f95740add00 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] conn(0x7f95740add00 msgr2=0x7f95740b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] conn(0x7f95740add00 0x7f95740b0140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a3040 msgr2=0x7f95740a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a3040 0x7f95740a5500 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f958c078a80 tx=0x7f9580009260 comp rx=0 tx=0).stop 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 msgr2=0x7f958c19d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 --2- 172.21.15.28:0/3404363637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101570 0x7f958c19d900 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f957c009230 tx=0x7f957c00fa70 comp rx=0 tx=0).stop 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f95908bc700 1 -- 172.21.15.28:0/3404363637 reap_dead start 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 shutdown_connections 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 >> 172.21.15.28:0/3404363637 conn(0x7f958c0784f0 msgr2=0x7f958c100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 shutdown_connections 2024-03-22T02:12:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:18.110+0000 7f959231f700 1 -- 172.21.15.28:0/3404363637 wait complete. 2024-03-22T02:12:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:18 smithi120 ceph-mon[36803]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-22T02:12:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:18 smithi083 ceph-mon[36340]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-22T02:12:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:18 smithi028 ceph-mon[31916]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-22T02:12:20.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:20 smithi028 ceph-mon[31916]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:20 smithi120 ceph-mon[36803]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:20 smithi083 ceph-mon[36340]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.993+0000 7fe0300af700 1 -- 172.21.15.28:0/279609822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028101210 msgr2=0x7fe0281015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.993+0000 7fe0300af700 1 --2- 172.21.15.28:0/279609822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028101210 0x7fe0281015f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fe0240099c0 tx=0x7fe024009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 -- 172.21.15.28:0/279609822 shutdown_connections 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 --2- 172.21.15.28:0/279609822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028101210 0x7fe0281015f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 --2- 172.21.15.28:0/279609822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028107a30 0x7fe02810f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 --2- 172.21.15.28:0/279609822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101bc0 0x7fe028107300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 -- 172.21.15.28:0/279609822 >> 172.21.15.28:0/279609822 conn(0x7fe0280768b0 msgr2=0x7fe028076cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 -- 172.21.15.28:0/279609822 shutdown_connections 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 -- 172.21.15.28:0/279609822 wait complete. 2024-03-22T02:12:21.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 Processor -- start 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 -- start start 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 0x7fe028198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.994+0000 7fe0300af700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 0x7fe0281993c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028107a30 0x7fe02819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0281129f0 con 0x7fe028107a30 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe028112870 con 0x7fe028101bc0 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe028112b70 con 0x7fe028101210 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02de4b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 0x7fe028198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 0x7fe0281993c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 0x7fe0281993c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:35872/0 (socket says 172.21.15.28:35872) 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02de4b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 0x7fe028198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58624/0 (socket says 172.21.15.28:58624) 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 -- 172.21.15.28:0/2378496971 learned_addr learned my addr 172.21.15.28:0/2378496971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02e64c700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028107a30 0x7fe02819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 -- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 msgr2=0x7fe028198e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 0x7fe028198e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 -- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028107a30 msgr2=0x7fe02819d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028107a30 0x7fe02819d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02819de40 con 0x7fe028101bc0 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02de4b700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe028101210 0x7fe028198e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02e64c700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe028107a30 0x7fe02819d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe02d64a700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 0x7fe0281993c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe01800e9c0 tx=0x7fe01800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01800cbd0 con 0x7fe028101bc0 2024-03-22T02:12:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02819e130 con 0x7fe028101bc0 2024-03-22T02:12:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01800cd70 con 0x7fe028101bc0 2024-03-22T02:12:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0180056e0 con 0x7fe028101bc0 2024-03-22T02:12:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.995+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0281020c0 con 0x7fe028101bc0 2024-03-22T02:12:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.996+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe028062e10 con 0x7fe028101bc0 2024-03-22T02:12:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.997+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe018004750 con 0x7fe028101bc0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe01effd700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0140a3040 0x7fe0140a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe02de4b700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0140a3040 0x7fe0140a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7fe0180cfe70 con 0x7fe028101bc0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe01effd700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] conn(0x7fe0140add00 0x7fe0140b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe018004500 con 0x7fe0140add00 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.998+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe0180d02c0 con 0x7fe028101bc0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.999+0000 7fe02e64c700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] conn(0x7fe0140add00 0x7fe0140b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.999+0000 7fe02de4b700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0140a3040 0x7fe0140a5500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe024000c00 tx=0x7fe024005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:21.999+0000 7fe02e64c700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] conn(0x7fe0140add00 0x7fe0140b0140 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:22.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.000+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== osd.11 v2:172.21.15.120:6824/511858328 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe018004500 con 0x7fe0140add00 2024-03-22T02:12:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.016+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 --> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe028063290 con 0x7fe0140add00 2024-03-22T02:12:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.016+0000 7fe01effd700 1 -- 172.21.15.28:0/2378496971 <== osd.11 v2:172.21.15.120:6824/511858328 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe028063290 con 0x7fe0140add00 2024-03-22T02:12:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] conn(0x7fe0140add00 msgr2=0x7fe0140b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] conn(0x7fe0140add00 0x7fe0140b0140 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0140a3040 msgr2=0x7fe0140a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0140a3040 0x7fe0140a5500 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe024000c00 tx=0x7fe024005fb0 comp rx=0 tx=0).stop 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 msgr2=0x7fe0281993c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 --2- 172.21.15.28:0/2378496971 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe028101bc0 0x7fe0281993c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe01800e9c0 tx=0x7fe01800ecd0 comp rx=0 tx=0).stop 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe02e64c700 1 -- 172.21.15.28:0/2378496971 reap_dead start 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 shutdown_connections 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 >> 172.21.15.28:0/2378496971 conn(0x7fe0280768b0 msgr2=0x7fe0280ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 shutdown_connections 2024-03-22T02:12:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:22.017+0000 7fe0300af700 1 -- 172.21.15.28:0/2378496971 wait complete. 2024-03-22T02:12:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:22 smithi120 ceph-mon[36803]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-22T02:12:22.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:22 smithi083 ceph-mon[36340]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-22T02:12:22.820 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:22 smithi028 ceph-mon[31916]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-22T02:12:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:24 smithi028 ceph-mon[31916]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-22T02:12:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:24 smithi120 ceph-mon[36803]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-22T02:12:24.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:24 smithi083 ceph-mon[36340]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 -- 172.21.15.28:0/1153823211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841087c0 msgr2=0x7f8784108ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 --2- 172.21.15.28:0/1153823211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841087c0 0x7f8784108ba0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8780009990 tx=0x7f8780009ca0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 -- 172.21.15.28:0/1153823211 shutdown_connections 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 --2- 172.21.15.28:0/1153823211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841087c0 0x7f8784108ba0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 --2- 172.21.15.28:0/1153823211 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8784101160 0x7f878410f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 --2- 172.21.15.28:0/1153823211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87841007a0 0x7f8784100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 -- 172.21.15.28:0/1153823211 >> 172.21.15.28:0/1153823211 conn(0x7f8784077750 msgr2=0x7f87840fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 -- 172.21.15.28:0/1153823211 shutdown_connections 2024-03-22T02:12:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.361+0000 7f878c794700 1 -- 172.21.15.28:0/1153823211 wait complete. 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 Processor -- start 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 -- start start 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 0x7f8784198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8784101160 0x7f87841992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87841087c0 0x7f878419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8784112920 con 0x7f87841007a0 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f87841127a0 con 0x7f8784101160 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878c794700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8784112aa0 con 0x7f87841087c0 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 0x7f8784198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 0x7f8784198d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33636/0 (socket says 172.21.15.28:33636) 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 -- 172.21.15.28:0/603214461 learned_addr learned my addr 172.21.15.28:0/603214461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878ad31700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87841087c0 0x7f878419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f8789d2f700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8784101160 0x7f87841992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 -- 172.21.15.28:0/603214461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87841087c0 msgr2=0x7f878419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f87841087c0 0x7f878419d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 -- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8784101160 msgr2=0x7f87841992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8784101160 0x7f87841992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.362+0000 7f878a530700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878419dd80 con 0x7f87841007a0 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f878a530700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 0x7f8784198d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8780000c00 tx=0x7f878000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780027070 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878419e010 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8780003d70 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878000f460 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.363+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841a58f0 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.364+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8784062e10 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.365+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f878002a070 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.365+0000 7f877b7fe700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f87700a3040 0x7f87700a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f8789d2f700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f87700a3040 0x7f87700a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f87800d0bb0 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f877b7fe700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] conn(0x7f87700add00 0x7f87700b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87700b0810 con 0x7f87700add00 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f87800d0f60 con 0x7f87841007a0 2024-03-22T02:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f878ad31700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] conn(0x7f87700add00 0x7f87700b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f8789d2f700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f87700a3040 0x7f87700a5500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8784107e20 tx=0x7f8774009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.366+0000 7f878ad31700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] conn(0x7f87700add00 0x7f87700b0140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.367+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== osd.10 v2:172.21.15.83:6824/3932179263 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87700b0810 con 0x7f87700add00 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.385+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 --> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8784063210 con 0x7f87700add00 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.385+0000 7f877b7fe700 1 -- 172.21.15.28:0/603214461 <== osd.10 v2:172.21.15.83:6824/3932179263 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8784063210 con 0x7f87700add00 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] conn(0x7f87700add00 msgr2=0x7f87700b0140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] conn(0x7f87700add00 0x7f87700b0140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f87700a3040 msgr2=0x7f87700a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f87700a3040 0x7f87700a5500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8784107e20 tx=0x7f8774009260 comp rx=0 tx=0).stop 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 msgr2=0x7f8784198d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 --2- 172.21.15.28:0/603214461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87841007a0 0x7f8784198d70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8780000c00 tx=0x7f878000bab0 comp rx=0 tx=0).stop 2024-03-22T02:12:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878ad31700 1 -- 172.21.15.28:0/603214461 reap_dead start 2024-03-22T02:12:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 shutdown_connections 2024-03-22T02:12:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 >> 172.21.15.28:0/603214461 conn(0x7f8784077750 msgr2=0x7f87840fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 shutdown_connections 2024-03-22T02:12:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:23.386+0000 7f878c794700 1 -- 172.21.15.28:0/603214461 wait complete. 2024-03-22T02:12:26.017 INFO:teuthology.orchestra.run.smithi028.stdout:167503724598 2024-03-22T02:12:26.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-22T02:12:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 -- 172.21.15.28:0/3927574554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100100ed0 msgr2=0x7f71001012b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 --2- 172.21.15.28:0/3927574554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100100ed0 0x7f71001012b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f70f00099c0 tx=0x7f70f0009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 -- 172.21.15.28:0/3927574554 shutdown_connections 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 --2- 172.21.15.28:0/3927574554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100100ed0 0x7f71001012b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 --2- 172.21.15.28:0/3927574554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100107a00 0x7f710010f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 --2- 172.21.15.28:0/3927574554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100101880 0x7f71001072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 -- 172.21.15.28:0/3927574554 >> 172.21.15.28:0/3927574554 conn(0x7f7100077b80 msgr2=0x7f71000fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 -- 172.21.15.28:0/3927574554 shutdown_connections 2024-03-22T02:12:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.200+0000 7f7107abe700 1 -- 172.21.15.28:0/3927574554 wait complete. 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 Processor -- start 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 -- start start 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100100ed0 0x7f7100101e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 0x7f7100102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100107a00 0x7f71001028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71001129a0 con 0x7f7100101880 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7100112820 con 0x7f7100100ed0 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7107abe700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7100112b20 con 0x7f7100107a00 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 0x7f7100102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 0x7f7100102390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33664/0 (socket says 172.21.15.28:33664) 2024-03-22T02:12:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 -- 172.21.15.28:0/2165791777 learned_addr learned my addr 172.21.15.28:0/2165791777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f710585a700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100100ed0 0x7f7100101e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f710605b700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100107a00 0x7f71001028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 -- 172.21.15.28:0/2165791777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100107a00 msgr2=0x7f71001028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7100107a00 0x7f71001028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 -- 172.21.15.28:0/2165791777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100100ed0 msgr2=0x7f7100101e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.201+0000 7f7105059700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100100ed0 0x7f7100101e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f7105059700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71001a7360 con 0x7f7100101880 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f710585a700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7100100ed0 0x7f7100101e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f7105059700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 0x7f7100102390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00e9a0 tx=0x7f70fc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc00ca10 con 0x7f7100101880 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71001a7650 con 0x7f7100101880 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.202+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70fc00cbb0 con 0x7f7100101880 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.203+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc018680 con 0x7f7100101880 2024-03-22T02:12:26.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.203+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71001a7be0 con 0x7f7100101880 2024-03-22T02:12:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.203+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7100062d90 con 0x7f7100101880 2024-03-22T02:12:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.205+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f70fc010430 con 0x7f7100101880 2024-03-22T02:12:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f70f6ffd700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f70ec0a30c0 0x7f70ec0a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8802+0+0 (secure 0 0 0) 0x7f70fc0cf970 con 0x7f7100101880 2024-03-22T02:12:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f710585a700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f70ec0a30c0 0x7f70ec0a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f70f6ffd700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] conn(0x7f70ec0add80 0x7f70ec0b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70fc015e80 con 0x7f70ec0add80 2024-03-22T02:12:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f70fc0d42c0 con 0x7f7100101880 2024-03-22T02:12:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.206+0000 7f710605b700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] conn(0x7f70ec0add80 0x7f70ec0b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.207+0000 7f710605b700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] conn(0x7f70ec0add80 0x7f70ec0b01c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.207+0000 7f710585a700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f70ec0a30c0 0x7f70ec0a5580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f0000c00 tx=0x7f70f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.208+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== osd.6 v2:172.21.15.28:6818/1589283942 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f70fc015e80 con 0x7f70ec0add80 2024-03-22T02:12:26.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.225+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 --> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7100069340 con 0x7f70ec0add80 2024-03-22T02:12:26.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f70f6ffd700 1 -- 172.21.15.28:0/2165791777 <== osd.6 v2:172.21.15.28:6818/1589283942 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7100069340 con 0x7f70ec0add80 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] conn(0x7f70ec0add80 msgr2=0x7f70ec0b01c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] conn(0x7f70ec0add80 0x7f70ec0b01c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f70ec0a30c0 msgr2=0x7f70ec0a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f70ec0a30c0 0x7f70ec0a5580 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f70f0000c00 tx=0x7f70f0005fb0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 msgr2=0x7f7100102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 --2- 172.21.15.28:0/2165791777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7100101880 0x7f7100102390 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00e9a0 tx=0x7f70fc00ecb0 comp rx=0 tx=0).stop 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f710605b700 1 -- 172.21.15.28:0/2165791777 reap_dead start 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 shutdown_connections 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 >> 172.21.15.28:0/2165791777 conn(0x7f7100077b80 msgr2=0x7f71000fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 shutdown_connections 2024-03-22T02:12:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:26.226+0000 7f7107abe700 1 -- 172.21.15.28:0/2165791777 wait complete. 2024-03-22T02:12:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:26 smithi028 ceph-mon[31916]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:26.623 INFO:teuthology.orchestra.run.smithi028.stdout:313532612636 2024-03-22T02:12:26.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-22T02:12:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:26 smithi120 ceph-mon[36803]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:26.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:26 smithi083 ceph-mon[36340]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:12:27.153 INFO:teuthology.orchestra.run.smithi028.stdout:347892350998 2024-03-22T02:12:27.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-22T02:12:27.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-22T02:12:27.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2024-03-22T02:12:27.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-22T02:12:27.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-22T02:12:27.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2024-03-22T02:12:27.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-22T02:12:27.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-22T02:12:27.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-03-22T02:12:27.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-22T02:12:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-03-22T02:12:27.934 INFO:teuthology.orchestra.run.smithi028.stdout:34359738446 2024-03-22T02:12:27.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-22T02:12:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- 172.21.15.28:0/3613663444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4107650 msgr2=0x7fccc4107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 --2- 172.21.15.28:0/3613663444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4107650 0x7fccc4107a30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fccb4009990 tx=0x7fccb4009ca0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- 172.21.15.28:0/3613663444 shutdown_connections 2024-03-22T02:12:28.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 --2- 172.21.15.28:0/3613663444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4107650 0x7fccc4107a30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 --2- 172.21.15.28:0/3613663444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc4102740 0x7fccc410e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 --2- 172.21.15.28:0/3613663444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccc4101d80 0x7fccc4102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- 172.21.15.28:0/3613663444 >> 172.21.15.28:0/3613663444 conn(0x7fccc4077b80 msgr2=0x7fccc40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- 172.21.15.28:0/3613663444 shutdown_connections 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- 172.21.15.28:0/3613663444 wait complete. 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 Processor -- start 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.138+0000 7fccca8d6700 1 -- start start 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 0x7fccc4198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc4102740 0x7fccc41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccc4107650 0x7fccc419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4111910 con 0x7fccc4101d80 2024-03-22T02:12:28.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4111790 con 0x7fccc4102740 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fccc4111a90 con 0x7fccc4107650 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 0x7fccc4198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 0x7fccc4198e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33706/0 (socket says 172.21.15.28:33706) 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 -- 172.21.15.28:0/1015650839 learned_addr learned my addr 172.21.15.28:0/1015650839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc8e73700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccc4107650 0x7fccc419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc37fe700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc4102740 0x7fccc41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 -- 172.21.15.28:0/1015650839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccc4107650 msgr2=0x7fccc419d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fccc4107650 0x7fccc419d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 -- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc4102740 msgr2=0x7fccc41993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fccc4102740 0x7fccc41993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc419de50 con 0x7fccc4101d80 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc3fff700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 0x7fccc4198e90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fccb4000c00 tx=0x7fccb400bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb4026070 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccb4003d70 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.139+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc419e0e0 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.140+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccb400fdf0 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.140+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41a59c0 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.140+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fccc4105670 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.141+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fccb4029070 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc17fa700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fccac0a3040 0x7fccac0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc37fe700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fccac0a3040 0x7fccac0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fccb40d0d50 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc17fa700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] conn(0x7fccac0ae120 0x7fccac0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccb4010ec0 con 0x7fccac0ae120 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fccb40d11a0 con 0x7fccc4101d80 2024-03-22T02:12:28.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc37fe700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fccac0a3040 0x7fccac0a5500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fccc4106c90 tx=0x7fccb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:28.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.142+0000 7fccc8e73700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] conn(0x7fccac0ae120 0x7fccac0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:28.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.143+0000 7fccc8e73700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] conn(0x7fccac0ae120 0x7fccac0b05c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:28.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.144+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== osd.7 v2:172.21.15.83:6816/1165824815 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fccb4010ec0 con 0x7fccac0ae120 2024-03-22T02:12:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.161+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 --> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fccc4102d90 con 0x7fccac0ae120 2024-03-22T02:12:28.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.161+0000 7fccc17fa700 1 -- 172.21.15.28:0/1015650839 <== osd.7 v2:172.21.15.83:6816/1165824815 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fccc4102d90 con 0x7fccac0ae120 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] conn(0x7fccac0ae120 msgr2=0x7fccac0b05c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] conn(0x7fccac0ae120 0x7fccac0b05c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fccac0a3040 msgr2=0x7fccac0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fccac0a3040 0x7fccac0a5500 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fccc4106c90 tx=0x7fccb8009260 comp rx=0 tx=0).stop 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 msgr2=0x7fccc4198e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 --2- 172.21.15.28:0/1015650839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccc4101d80 0x7fccc4198e90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fccb4000c00 tx=0x7fccb400bb30 comp rx=0 tx=0).stop 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccc8e73700 1 -- 172.21.15.28:0/1015650839 reap_dead start 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 shutdown_connections 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 >> 172.21.15.28:0/1015650839 conn(0x7fccc4077b80 msgr2=0x7fccc40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 shutdown_connections 2024-03-22T02:12:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:28.162+0000 7fccca8d6700 1 -- 172.21.15.28:0/1015650839 wait complete. 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-22T02:12:28.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-22T02:12:28.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2024-03-22T02:12:28.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-22T02:12:28.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-22T02:12:28.569 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:28 smithi028 ceph-mon[31916]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T02:12:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:12:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-22T02:12:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-22T02:12:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2024-03-22T02:12:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-22T02:12:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:28 smithi120 ceph-mon[36803]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T02:12:28.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 8]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6, 5, 7]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 6]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2a", "id": [1, 9]}]': finished 2024-03-22T02:12:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-22T02:12:28.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-03-22T02:12:28.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:28 smithi083 ceph-mon[36340]: osdmap e91: 12 total, 12 up, 12 in 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 -- 172.21.15.28:0/1947628596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac108640 msgr2=0x7f30ac108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/1947628596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac108640 0x7f30ac108a20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f30a80099c0 tx=0x7f30a8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 -- 172.21.15.28:0/1947628596 shutdown_connections 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/1947628596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac108640 0x7f30ac108a20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/1947628596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ac06acd0 0x7f30ac10f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/1947628596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ac06a330 0x7f30ac06a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.287+0000 7f30b3f6d700 1 -- 172.21.15.28:0/1947628596 >> 172.21.15.28:0/1947628596 conn(0x7f30ac0784b0 msgr2=0x7f30ac0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:29.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- 172.21.15.28:0/1947628596 shutdown_connections 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- 172.21.15.28:0/1947628596 wait complete. 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 Processor -- start 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- start start 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 0x7f30ac101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ac06acd0 0x7f30ac101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ac108640 0x7f30ac1020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112940 con 0x7f30ac06a330 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac1127c0 con 0x7f30ac108640 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.288+0000 7f30b3f6d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f30ac112ac0 con 0x7f30ac06acd0 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 0x7f30ac101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 0x7f30ac101650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33724/0 (socket says 172.21.15.28:33724) 2024-03-22T02:12:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 -- 172.21.15.28:0/3285702037 learned_addr learned my addr 172.21.15.28:0/3285702037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1508700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ac06acd0 0x7f30ac101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b250a700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ac108640 0x7f30ac1020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 -- 172.21.15.28:0/3285702037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ac06acd0 msgr2=0x7f30ac101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30ac06acd0 0x7f30ac101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 -- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ac108640 msgr2=0x7f30ac1020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f30ac108640 0x7f30ac1020d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ac1a72d0 con 0x7f30ac06a330 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b1d09700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 0x7f30ac101650 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006010 tx=0x7f30a800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8003680 con 0x7f30ac06a330 2024-03-22T02:12:29.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30ac1a7560 con 0x7f30ac06a330 2024-03-22T02:12:29.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30a80050d0 con 0x7f30ac06a330 2024-03-22T02:12:29.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a8003db0 con 0x7f30ac06a330 2024-03-22T02:12:29.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.289+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30ac1a7a10 con 0x7f30ac06a330 2024-03-22T02:12:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.290+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f30ac062d90 con 0x7f30ac06a330 2024-03-22T02:12:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.291+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f30a8005280 con 0x7f30ac06a330 2024-03-22T02:12:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30a2ffd700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30980a31e0 0x7f30980a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30b1508700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30980a31e0 0x7f30980a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f30a80d0a80 con 0x7f30ac06a330 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30a2ffd700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] conn(0x7f30980ae2c0 0x7f30980b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f30a802ae50 con 0x7f30980ae2c0 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f30a80d0ed0 con 0x7f30ac06a330 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30b1508700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30980a31e0 0x7f30980a56a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0789b0 tx=0x7f309c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.292+0000 7f30b250a700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] conn(0x7f30980ae2c0 0x7f30980b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.293+0000 7f30b250a700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] conn(0x7f30980ae2c0 0x7f30980b0760 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.294+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== osd.1 v2:172.21.15.83:6800/1302915006 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f30a802ae50 con 0x7f30980ae2c0 2024-03-22T02:12:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.311+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 --> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f30ac063210 con 0x7f30980ae2c0 2024-03-22T02:12:29.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.311+0000 7f30a2ffd700 1 -- 172.21.15.28:0/3285702037 <== osd.1 v2:172.21.15.83:6800/1302915006 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f30ac063210 con 0x7f30980ae2c0 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] conn(0x7f30980ae2c0 msgr2=0x7f30980b0760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] conn(0x7f30980ae2c0 0x7f30980b0760 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30980a31e0 msgr2=0x7f30980a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f30980a31e0 0x7f30980a56a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f30ac0789b0 tx=0x7f309c009260 comp rx=0 tx=0).stop 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 msgr2=0x7f30ac101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 --2- 172.21.15.28:0/3285702037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06a330 0x7f30ac101650 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f30a8006010 tx=0x7f30a800f690 comp rx=0 tx=0).stop 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b250a700 1 -- 172.21.15.28:0/3285702037 reap_dead start 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 shutdown_connections 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 >> 172.21.15.28:0/3285702037 conn(0x7f30ac0784b0 msgr2=0x7f30ac0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 shutdown_connections 2024-03-22T02:12:29.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:29.312+0000 7f30b3f6d700 1 -- 172.21.15.28:0/3285702037 wait complete. 2024-03-22T02:12:29.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 -- 172.21.15.28:0/25237906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 msgr2=0x7f9b4010f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 --2- 172.21.15.28:0/25237906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4010f5c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800b0d0 tx=0x7f9b3800b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 -- 172.21.15.28:0/25237906 shutdown_connections 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 --2- 172.21.15.28:0/25237906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4010f5c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 --2- 172.21.15.28:0/25237906 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 0x7f9b40101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 --2- 172.21.15.28:0/25237906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 0x7f9b40107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.675+0000 7f9b462d8700 1 -- 172.21.15.28:0/25237906 >> 172.21.15.28:0/25237906 conn(0x7f9b400768c0 msgr2=0x7f9b40076cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.676+0000 7f9b462d8700 1 -- 172.21.15.28:0/25237906 shutdown_connections 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.676+0000 7f9b462d8700 1 -- 172.21.15.28:0/25237906 wait complete. 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.676+0000 7f9b462d8700 1 Processor -- start 2024-03-22T02:12:29.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.676+0000 7f9b462d8700 1 -- start start 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 0x7f9b40198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 0x7f9b40199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b401129f0 con 0x7f9b40107a40 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9b40112870 con 0x7f9b40101220 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b462d8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9b40112b70 con 0x7f9b40101bd0 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b3f7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 0x7f9b40199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b3ffff700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 0x7f9b40198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4019d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33694/0 (socket says 172.21.15.28:33694) 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 -- 172.21.15.28:0/2145221687 learned_addr learned my addr 172.21.15.28:0/2145221687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 -- 172.21.15.28:0/2145221687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 msgr2=0x7f9b40199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 0x7f9b40199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 -- 172.21.15.28:0/2145221687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 msgr2=0x7f9b40198dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 0x7f9b40198dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b44875700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4019ddd0 con 0x7f9b40107a40 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b3ffff700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b40101220 0x7f9b40198dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.677+0000 7f9b3f7fe700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b40101bd0 0x7f9b40199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b44875700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4019d690 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b38006070 tx=0x7f9b38007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b38003c00 con 0x7f9b40107a40 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4019e060 con 0x7f9b40107a40 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b38022070 con 0x7f9b40107a40 2024-03-22T02:12:29.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3801b6c0 con 0x7f9b40107a40 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b401a5940 con 0x7f9b40107a40 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.678+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b40062e10 con 0x7f9b40107a40 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.680+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f9b3801d470 con 0x7f9b40107a40 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.680+0000 7f9b3d7fa700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b280a3040 0x7f9b280a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.680+0000 7f9b3ffff700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b280a3040 0x7f9b280a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.680+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f9b3800f020 con 0x7f9b40107a40 2024-03-22T02:12:29.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3d7fa700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] conn(0x7f9b280ae120 0x7f9b280b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b38028e70 con 0x7f9b280ae120 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f9b380d0a60 con 0x7f9b40107a40 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3f7fe700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] conn(0x7f9b280ae120 0x7f9b280b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3ffff700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b280a3040 0x7f9b280a5500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9b30011490 tx=0x7f9b30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.681+0000 7f9b3f7fe700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] conn(0x7f9b280ae120 0x7f9b280b05c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.682+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== osd.9 v2:172.21.15.28:6826/855775168 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b38028e70 con 0x7f9b280ae120 2024-03-22T02:12:29.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.698+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 --> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b40063210 con 0x7f9b280ae120 2024-03-22T02:12:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.698+0000 7f9b3d7fa700 1 -- 172.21.15.28:0/2145221687 <== osd.9 v2:172.21.15.28:6826/855775168 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9b40063210 con 0x7f9b280ae120 2024-03-22T02:12:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] conn(0x7f9b280ae120 msgr2=0x7f9b280b05c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] conn(0x7f9b280ae120 0x7f9b280b05c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b280a3040 msgr2=0x7f9b280a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b280a3040 0x7f9b280a5500 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9b30011490 tx=0x7f9b30009260 comp rx=0 tx=0).stop 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 msgr2=0x7f9b4019d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 --2- 172.21.15.28:0/2145221687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b40107a40 0x7f9b4019d690 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b38006070 tx=0x7f9b38007a20 comp rx=0 tx=0).stop 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b44875700 1 -- 172.21.15.28:0/2145221687 reap_dead start 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 shutdown_connections 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 >> 172.21.15.28:0/2145221687 conn(0x7f9b400768c0 msgr2=0x7f9b400fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 shutdown_connections 2024-03-22T02:12:29.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:27.699+0000 7f9b462d8700 1 -- 172.21.15.28:0/2145221687 wait complete. 2024-03-22T02:12:29.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:29 smithi120 ceph-mon[36803]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T02:12:29.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:29 smithi083 ceph-mon[36340]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T02:12:29.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:29 smithi028 ceph-mon[31916]: osdmap e92: 12 total, 12 up, 12 in 2024-03-22T02:12:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:30 smithi120 ceph-mon[36803]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:12:30.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:30 smithi083 ceph-mon[36340]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:12:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:30 smithi028 ceph-mon[31916]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:12:31.177 INFO:teuthology.orchestra.run.smithi028.stdout:292057776163 2024-03-22T02:12:31.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-22T02:12:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 -- 172.21.15.28:0/886426824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc102e50 msgr2=0x7f57dc10f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 --2- 172.21.15.28:0/886426824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc102e50 0x7f57dc10f510 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f57d400b0a0 tx=0x7f57d400b3b0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 -- 172.21.15.28:0/886426824 shutdown_connections 2024-03-22T02:12:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 --2- 172.21.15.28:0/886426824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc102e50 0x7f57dc10f510 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 --2- 172.21.15.28:0/886426824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57dc108690 0x7f57dc108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 --2- 172.21.15.28:0/886426824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc108fb0 0x7f57dc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 -- 172.21.15.28:0/886426824 >> 172.21.15.28:0/886426824 conn(0x7f57dc079740 msgr2=0x7f57dc079b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 -- 172.21.15.28:0/886426824 shutdown_connections 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.069+0000 7f57e2961700 1 -- 172.21.15.28:0/886426824 wait complete. 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 Processor -- start 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 -- start start 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 0x7f57dc198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57dc108690 0x7f57dc199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 0x7f57dc19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc1129a0 con 0x7f57dc19d6d0 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc112820 con 0x7f57dc108690 2024-03-22T02:12:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.070+0000 7f57e2961700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc112b20 con 0x7f57dc102e50 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57e0efe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 0x7f57dc19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 0x7f57dc198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57e0efe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 0x7f57dc19db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54774/0 (socket says 172.21.15.28:54774) 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 0x7f57dc198e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34520/0 (socket says 172.21.15.28:34520) 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57e0efe700 1 -- 172.21.15.28:0/371113962 learned_addr learned my addr 172.21.15.28:0/371113962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57db7fe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57dc108690 0x7f57dc199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 -- 172.21.15.28:0/371113962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57dc108690 msgr2=0x7f57dc199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f57dc108690 0x7f57dc199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 -- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 msgr2=0x7f57dc19db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 0x7f57dc19db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc19e1f0 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57e0efe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57dc19d6d0 0x7f57dc19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57dbfff700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 0x7f57dc198e50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9f0 tx=0x7f57cc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc00c9c0 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57cc00cb60 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc018680 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.071+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc19e4e0 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.072+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc074ba0 con 0x7f57dc102e50 2024-03-22T02:12:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.072+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f57dc103dc0 con 0x7f57dc102e50 2024-03-22T02:12:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.074+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f57cc010430 con 0x7f57dc102e50 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57d97fa700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57c40a30c0 0x7f57c40a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57db7fe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57c40a30c0 0x7f57c40a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f57cc0d0420 con 0x7f57dc102e50 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57d97fa700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] conn(0x7f57c40ae1a0 0x7f57c40b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f57c40b0d10 con 0x7f57c40ae1a0 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f57cc0d0870 con 0x7f57dc102e50 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57e0efe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] conn(0x7f57c40ae1a0 0x7f57c40b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.075+0000 7f57db7fe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57c40a30c0 0x7f57c40a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f57dc079db0 tx=0x7f57d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.076+0000 7f57e0efe700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] conn(0x7f57c40ae1a0 0x7f57c40b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.077+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== osd.3 v2:172.21.15.28:6810/3156433849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f57c40b0d10 con 0x7f57c40ae1a0 2024-03-22T02:12:32.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.095+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 --> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f57dc062d90 con 0x7f57c40ae1a0 2024-03-22T02:12:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57d97fa700 1 -- 172.21.15.28:0/371113962 <== osd.3 v2:172.21.15.28:6810/3156433849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f57dc062d90 con 0x7f57c40ae1a0 2024-03-22T02:12:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] conn(0x7f57c40ae1a0 msgr2=0x7f57c40b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57e2961700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] conn(0x7f57c40ae1a0 0x7f57c40b0640 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57c40a30c0 msgr2=0x7f57c40a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57e2961700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57c40a30c0 0x7f57c40a5580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f57dc079db0 tx=0x7f57d0009260 comp rx=0 tx=0).stop 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.096+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 msgr2=0x7f57dc198e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e2961700 1 --2- 172.21.15.28:0/371113962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57dc102e50 0x7f57dc198e50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00e9f0 tx=0x7f57cc00ed00 comp rx=0 tx=0).stop 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e0efe700 1 -- 172.21.15.28:0/371113962 reap_dead start 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 shutdown_connections 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 >> 172.21.15.28:0/371113962 conn(0x7f57dc079740 msgr2=0x7f57dc1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 shutdown_connections 2024-03-22T02:12:32.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.097+0000 7f57e2961700 1 -- 172.21.15.28:0/371113962 wait complete. 2024-03-22T02:12:32.185 INFO:teuthology.orchestra.run.smithi028.stdout:201863462961 2024-03-22T02:12:32.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 -- 172.21.15.28:0/129878784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc102730 msgr2=0x7f37cc102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 --2- 172.21.15.28:0/129878784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc102730 0x7f37cc102bb0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009990 tx=0x7f37bc009ca0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 -- 172.21.15.28:0/129878784 shutdown_connections 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 --2- 172.21.15.28:0/129878784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc1030f0 0x7f37cc10f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 --2- 172.21.15.28:0/129878784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37cc107650 0x7f37cc107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 --2- 172.21.15.28:0/129878784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc102730 0x7f37cc102bb0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 -- 172.21.15.28:0/129878784 >> 172.21.15.28:0/129878784 conn(0x7f37cc079750 msgr2=0x7f37cc079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 -- 172.21.15.28:0/129878784 shutdown_connections 2024-03-22T02:12:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.492+0000 7f37d3561700 1 -- 172.21.15.28:0/129878784 wait complete. 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 Processor -- start 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 -- start start 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc102730 0x7f37cc198eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 0x7f37cc1993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37cc107650 0x7f37cc19d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc1129a0 con 0x7f37cc1030f0 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc112820 con 0x7f37cc107650 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d3561700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc112b20 con 0x7f37cc102730 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d0afc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 0x7f37cc1993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d0afc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 0x7f37cc1993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54792/0 (socket says 172.21.15.28:54792) 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d0afc700 1 -- 172.21.15.28:0/3548915413 learned_addr learned my addr 172.21.15.28:0/3548915413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d12fd700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc102730 0x7f37cc198eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.493+0000 7f37d1afe700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37cc107650 0x7f37cc19d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 -- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc102730 msgr2=0x7f37cc198eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37cc102730 0x7f37cc198eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 -- 172.21.15.28:0/3548915413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37cc107650 msgr2=0x7f37cc19d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f37cc107650 0x7f37cc19d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc19dec0 con 0x7f37cc1030f0 2024-03-22T02:12:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d0afc700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 0x7f37cc1993f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f37bc000c00 tx=0x7f37bc00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc027070 con 0x7f37cc1030f0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37cc19e150 con 0x7f37cc1030f0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37bc0056d0 con 0x7f37cc1030f0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc022e10 con 0x7f37cc1030f0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.494+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37cc1a5a30 con 0x7f37cc1030f0 2024-03-22T02:12:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.495+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37cc062d90 con 0x7f37cc1030f0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.497+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f37bc02a070 con 0x7f37cc1030f0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.497+0000 7f37c27fc700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f37b80a30c0 0x7f37b80a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.497+0000 7f37d12fd700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f37b80a30c0 0x7f37b80a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.497+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f37bc0d1060 con 0x7f37cc1030f0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37c27fc700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] conn(0x7f37b80ae1a0 0x7f37b80b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37b80b0d10 con 0x7f37b80ae1a0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f37bc0d14b0 con 0x7f37cc1030f0 2024-03-22T02:12:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37d12fd700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f37b80a30c0 0x7f37b80a5580 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009be0 tx=0x7f37c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37d1afe700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] conn(0x7f37b80ae1a0 0x7f37b80b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.498+0000 7f37d1afe700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] conn(0x7f37b80ae1a0 0x7f37b80b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.499+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== osd.2 v2:172.21.15.120:6800/2212833348 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37b80b0d10 con 0x7f37b80ae1a0 2024-03-22T02:12:32.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.516+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 --> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37cc063190 con 0x7f37b80ae1a0 2024-03-22T02:12:32.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.516+0000 7f37c27fc700 1 -- 172.21.15.28:0/3548915413 <== osd.2 v2:172.21.15.120:6800/2212833348 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37cc063190 con 0x7f37b80ae1a0 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] conn(0x7f37b80ae1a0 msgr2=0x7f37b80b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] conn(0x7f37b80ae1a0 0x7f37b80b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f37b80a30c0 msgr2=0x7f37b80a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f37b80a30c0 0x7f37b80a5580 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009be0 tx=0x7f37c8009260 comp rx=0 tx=0).stop 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 msgr2=0x7f37cc1993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 --2- 172.21.15.28:0/3548915413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37cc1030f0 0x7f37cc1993f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f37bc000c00 tx=0x7f37bc00bab0 comp rx=0 tx=0).stop 2024-03-22T02:12:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d1afe700 1 -- 172.21.15.28:0/3548915413 reap_dead start 2024-03-22T02:12:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 shutdown_connections 2024-03-22T02:12:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 >> 172.21.15.28:0/3548915413 conn(0x7f37cc079750 msgr2=0x7f37cc100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 shutdown_connections 2024-03-22T02:12:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:32.517+0000 7f37d3561700 1 -- 172.21.15.28:0/3548915413 wait complete. 2024-03-22T02:12:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:32 smithi120 ceph-mon[36803]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:32.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:32 smithi083 ceph-mon[36340]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:32 smithi028 ceph-mon[31916]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 -- 172.21.15.28:0/3485242777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 msgr2=0x7f655c10e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 --2- 172.21.15.28:0/3485242777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c10e320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f654c0099c0 tx=0x7f654c009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 -- 172.21.15.28:0/3485242777 shutdown_connections 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 --2- 172.21.15.28:0/3485242777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c10e320 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 --2- 172.21.15.28:0/3485242777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c107550 0x7f655c107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 --2- 172.21.15.28:0/3485242777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f655c1014a0 0x7f655c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 -- 172.21.15.28:0/3485242777 >> 172.21.15.28:0/3485242777 conn(0x7f655c0fcf90 msgr2=0x7f655c0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 -- 172.21.15.28:0/3485242777 shutdown_connections 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 -- 172.21.15.28:0/3485242777 wait complete. 2024-03-22T02:12:33.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.308+0000 7f6561a04700 1 Processor -- start 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 -- start start 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f655c1014a0 0x7f655c19e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c19ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c107550 0x7f655c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f655c1117a0 con 0x7f655c101e60 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f655c111620 con 0x7f655c1014a0 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f6561a04700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f655c111920 con 0x7f655c107550 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c19ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c19ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54808/0 (socket says 172.21.15.28:54808) 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 -- 172.21.15.28:0/903180195 learned_addr learned my addr 172.21.15.28:0/903180195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655affd700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f655c1014a0 0x7f655c19e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655b7fe700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c107550 0x7f655c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 -- 172.21.15.28:0/903180195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c107550 msgr2=0x7f655c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f655c107550 0x7f655c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 -- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f655c1014a0 msgr2=0x7f655c19e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f655c1014a0 0x7f655c19e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c199190 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.309+0000 7f655a7fc700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c19ed60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f654400e9a0 tx=0x7f654400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.310+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654400ca10 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.310+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c199480 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.310+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f654400cbb0 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.310+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6544018680 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.310+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c1a56f0 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.311+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f655c105e90 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.312+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6544010430 con 0x7f655c101e60 2024-03-22T02:12:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.312+0000 7f6560a02700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65480a3040 0x7f65480a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.312+0000 7f655affd700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65480a3040 0x7f65480a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.312+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f65440d0310 con 0x7f655c101e60 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.312+0000 7f6560a02700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] conn(0x7f65480ae120 0x7f65480b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.313+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f654401bec0 con 0x7f65480ae120 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.313+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f65440d0760 con 0x7f655c101e60 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.313+0000 7f655affd700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65480a3040 0x7f65480a5500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f655c106620 tx=0x7f655400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.313+0000 7f655b7fe700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] conn(0x7f65480ae120 0x7f65480b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.313+0000 7f655b7fe700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] conn(0x7f65480ae120 0x7f65480b05c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:33.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.314+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== osd.4 v2:172.21.15.83:6808/3880312837 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f654401bec0 con 0x7f65480ae120 2024-03-22T02:12:33.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.330+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 --> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f655c102bc0 con 0x7f65480ae120 2024-03-22T02:12:33.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6560a02700 1 -- 172.21.15.28:0/903180195 <== osd.4 v2:172.21.15.83:6808/3880312837 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f655c102bc0 con 0x7f65480ae120 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] conn(0x7f65480ae120 msgr2=0x7f65480b05c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] conn(0x7f65480ae120 0x7f65480b05c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65480a3040 msgr2=0x7f65480a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65480a3040 0x7f65480a5500 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f655c106620 tx=0x7f655400b040 comp rx=0 tx=0).stop 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 msgr2=0x7f655c19ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 --2- 172.21.15.28:0/903180195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c101e60 0x7f655c19ed60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f654400e9a0 tx=0x7f654400ecb0 comp rx=0 tx=0).stop 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f655b7fe700 1 -- 172.21.15.28:0/903180195 reap_dead start 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 shutdown_connections 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.331+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 >> 172.21.15.28:0/903180195 conn(0x7f655c0fcf90 msgr2=0x7f655c105430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:33.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.332+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 shutdown_connections 2024-03-22T02:12:33.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:33.332+0000 7f6561a04700 1 -- 172.21.15.28:0/903180195 wait complete. 2024-03-22T02:12:34.287 INFO:teuthology.orchestra.run.smithi028.stdout:223338299437 2024-03-22T02:12:34.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-22T02:12:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:34 smithi028 ceph-mon[31916]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:12:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:34 smithi120 ceph-mon[36803]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:12:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:34 smithi083 ceph-mon[36340]: pgmap v325: 129 pgs: 1 peering, 128 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:12:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:12:35.252 INFO:teuthology.orchestra.run.smithi028.stdout:146028888126 2024-03-22T02:12:35.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-22T02:12:35.692 INFO:teuthology.orchestra.run.smithi028.stdout:77309411399 2024-03-22T02:12:35.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-22T02:12:36.393 INFO:teuthology.orchestra.run.smithi028.stdout:111669149762 2024-03-22T02:12:36.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-22T02:12:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:36 smithi120 ceph-mon[36803]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-03-22T02:12:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:36 smithi083 ceph-mon[36340]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-03-22T02:12:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:36 smithi028 ceph-mon[31916]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-03-22T02:12:36.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:12:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:55834574922 2024-03-22T02:12:36.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-22T02:12:37.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 -- 172.21.15.28:0/2301861411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281075f0 msgr2=0x7f85281079d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 --2- 172.21.15.28:0/2301861411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281075f0 0x7f85281079d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f85240099c0 tx=0x7f8524009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 -- 172.21.15.28:0/2301861411 shutdown_connections 2024-03-22T02:12:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 --2- 172.21.15.28:0/2301861411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281075f0 0x7f85281079d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 --2- 172.21.15.28:0/2301861411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f852810e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 --2- 172.21.15.28:0/2301861411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281015b0 0x7f8528101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 -- 172.21.15.28:0/2301861411 >> 172.21.15.28:0/2301861411 conn(0x7f85280775d0 msgr2=0x7f85280779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 -- 172.21.15.28:0/2301861411 shutdown_connections 2024-03-22T02:12:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.051+0000 7f852f5ed700 1 -- 172.21.15.28:0/2301861411 wait complete. 2024-03-22T02:12:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.052+0000 7f852f5ed700 1 Processor -- start 2024-03-22T02:12:37.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.052+0000 7f852f5ed700 1 -- start start 2024-03-22T02:12:37.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.052+0000 7f852f5ed700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281015b0 0x7f85281031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.052+0000 7f852f5ed700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f85281036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.052+0000 7f852f5ed700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281075f0 0x7f8528103c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852f5ed700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8528111860 con 0x7f85281015b0 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852f5ed700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f85281116e0 con 0x7f8528101f70 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852f5ed700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f85281119e0 con 0x7f85281075f0 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852cb88700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f85281036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852cb88700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f85281036f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52994/0 (socket says 172.21.15.28:52994) 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852cb88700 1 -- 172.21.15.28:0/1183082204 learned_addr learned my addr 172.21.15.28:0/1183082204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852d389700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281015b0 0x7f85281031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281075f0 0x7f8528103c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 -- 172.21.15.28:0/1183082204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 msgr2=0x7f85281036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f85281036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 -- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281015b0 msgr2=0x7f85281031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281015b0 0x7f85281031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85281a72d0 con 0x7f85281075f0 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852cb88700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8528101f70 0x7f85281036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852d389700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85281015b0 0x7f85281031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.053+0000 7f852db8a700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281075f0 0x7f8528103c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f852000efc0 tx=0x7f852000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.054+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85200092e0 con 0x7f85281075f0 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.054+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85281a7560 con 0x7f85281075f0 2024-03-22T02:12:37.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.054+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8520004bb0 con 0x7f85281075f0 2024-03-22T02:12:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.054+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85281a7af0 con 0x7f85281075f0 2024-03-22T02:12:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.054+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8520004280 con 0x7f85281075f0 2024-03-22T02:12:37.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.055+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8528105620 con 0x7f85281075f0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.057+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f852001b030 con 0x7f85281075f0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.057+0000 7f851e7fc700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f85140a30c0 0x7f85140a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.057+0000 7f852d389700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f85140a30c0 0x7f85140a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f85200cf780 con 0x7f85281075f0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f851e7fc700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] conn(0x7f85140ae1a0 0x7f85140b0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f852001ae50 con 0x7f85140ae1a0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) ==== 24+0+0 (secure 0 0 0) 0x7f85200cfbd0 con 0x7f85281075f0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f852d389700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f85140a30c0 0x7f85140a5580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8524000c00 tx=0x7f8524005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f852cb88700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] conn(0x7f85140ae1a0 0x7f85140b0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.058+0000 7f852cb88700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] conn(0x7f85140ae1a0 0x7f85140b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.060+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== osd.8 v2:172.21.15.120:6816/739561916 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f852001ae50 con 0x7f85140ae1a0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.077+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 --> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8528102c60 con 0x7f85140ae1a0 2024-03-22T02:12:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.077+0000 7f851e7fc700 1 -- 172.21.15.28:0/1183082204 <== osd.8 v2:172.21.15.120:6816/739561916 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8528102c60 con 0x7f85140ae1a0 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.077+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] conn(0x7f85140ae1a0 msgr2=0x7f85140b0640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] conn(0x7f85140ae1a0 0x7f85140b0640 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f85140a30c0 msgr2=0x7f85140a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f85140a30c0 0x7f85140a5580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8524000c00 tx=0x7f8524005fb0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281075f0 msgr2=0x7f8528103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 --2- 172.21.15.28:0/1183082204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f85281075f0 0x7f8528103c30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f852000efc0 tx=0x7f852000c3a0 comp rx=0 tx=0).stop 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852db8a700 1 -- 172.21.15.28:0/1183082204 reap_dead start 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 shutdown_connections 2024-03-22T02:12:37.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 >> 172.21.15.28:0/1183082204 conn(0x7f85280775d0 msgr2=0x7f85281008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:37.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.078+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 shutdown_connections 2024-03-22T02:12:37.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:37.079+0000 7f852f5ed700 1 -- 172.21.15.28:0/1183082204 wait complete. 2024-03-22T02:12:37.158 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:38 smithi120 ceph-mon[36803]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-22T02:12:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:38 smithi083 ceph-mon[36340]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-22T02:12:38.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:38 smithi028 ceph-mon[31916]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-22T02:12:38.811 INFO:teuthology.orchestra.run.smithi028.stdout:257698037802 2024-03-22T02:12:38.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-22T02:12:39.031 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.053 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.054 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.057 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 -- 172.21.15.28:0/265574301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c1076b0 msgr2=0x7fc55c107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 --2- 172.21.15.28:0/265574301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c1076b0 0x7fc55c107a90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc54c0098f0 tx=0x7fc54c009c00 comp rx=0 tx=0).stop 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 -- 172.21.15.28:0/265574301 shutdown_connections 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 --2- 172.21.15.28:0/265574301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 --2- 172.21.15.28:0/265574301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c1076b0 0x7fc55c107a90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 --2- 172.21.15.28:0/265574301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c101860 0x7fc55c101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 -- 172.21.15.28:0/265574301 >> 172.21.15.28:0/265574301 conn(0x7fc55c078440 msgr2=0x7fc55c078850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 -- 172.21.15.28:0/265574301 shutdown_connections 2024-03-22T02:12:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 -- 172.21.15.28:0/265574301 wait complete. 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.286+0000 7fc562147700 1 Processor -- start 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 -- start start 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c1076b0 0x7fc55c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c111860 con 0x7fc55c102220 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c1116e0 con 0x7fc55c1076b0 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc562147700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c1119e0 con 0x7fc55c101860 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34586/0 (socket says 172.21.15.28:34586) 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 -- 172.21.15.28:0/1637703450 learned_addr learned my addr 172.21.15.28:0/1637703450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54846/0 (socket says 172.21.15.28:54846) 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55bfff700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c1076b0 0x7fc55c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 -- 172.21.15.28:0/1637703450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c1076b0 msgr2=0x7fc55c19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c1076b0 0x7fc55c19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 -- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 msgr2=0x7fc55c1992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 -- 172.21.15.28:0/1637703450 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c19ddc0 con 0x7fc55c101860 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55affd700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.287+0000 7fc55b7fe700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009230 tx=0x7fc54c00ff20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c005490 con 0x7fc55c101860 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc54c005630 con 0x7fc55c101860 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55c19e050 con 0x7fc55c101860 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc54c003de0 con 0x7fc55c101860 2024-03-22T02:12:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55c1a5930 con 0x7fc55c101860 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.288+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc55c069340 con 0x7fc55c101860 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.290+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fc54c04bb30 con 0x7fc55c101860 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.290+0000 7fc558ff9700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 0x7fc5440a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.290+0000 7fc55affd700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 0x7fc5440a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.290+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc54c0d0280 con 0x7fc55c101860 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.291+0000 7fc55affd700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 0x7fc5440a5580 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0fffc0 tx=0x7fc550009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.294+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc54c013030 con 0x7fc55c101860 2024-03-22T02:12:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.509+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fc55c1a5b00 con 0x7fc55c101860 2024-03-22T02:12:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.510+0000 7fc558ff9700 1 -- 172.21.15.28:0/1637703450 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fc54c04be20 con 0x7fc55c101860 2024-03-22T02:12:39.512 INFO:teuthology.orchestra.run.smithi028.stdout:313532612639 2024-03-22T02:12:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 msgr2=0x7fc5440a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 0x7fc5440a5580 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0fffc0 tx=0x7fc550009260 comp rx=0 tx=0).stop 2024-03-22T02:12:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 msgr2=0x7fc55c198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009230 tx=0x7fc54c00ff20 comp rx=0 tx=0).stop 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 shutdown_connections 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc55c1076b0 0x7fc55c19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc55c101860 0x7fc55c198db0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5440a30c0 0x7fc5440a5580 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 --2- 172.21.15.28:0/1637703450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc55c102220 0x7fc55c1992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 >> 172.21.15.28:0/1637703450 conn(0x7fc55c078440 msgr2=0x7fc55c06af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 shutdown_connections 2024-03-22T02:12:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:39.512+0000 7fc562147700 1 -- 172.21.15.28:0/1637703450 wait complete. 2024-03-22T02:12:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:40 smithi120 ceph-mon[36803]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-22T02:12:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1637703450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T02:12:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:40 smithi083 ceph-mon[36340]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-22T02:12:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1637703450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T02:12:40.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:40 smithi028 ceph-mon[31916]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-22T02:12:40.814 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1637703450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-22T02:12:41.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:41.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:42.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:42 smithi028 ceph-mon[31916]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:42 smithi083 ceph-mon[36340]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:42 smithi120 ceph-mon[36803]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:43.435 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612636 got 313532612639 for osd.10 2024-03-22T02:12:43.435 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:44 smithi028 ceph-mon[31916]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:44 smithi083 ceph-mon[36340]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:44 smithi120 ceph-mon[36803]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:46 smithi028 ceph-mon[31916]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:46 smithi083 ceph-mon[36340]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:46 smithi120 ceph-mon[36803]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:12:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 -- 172.21.15.28:0/901106543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b41026d0 msgr2=0x7f39b4102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 --2- 172.21.15.28:0/901106543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b41026d0 0x7f39b4102b50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f39a80099c0 tx=0x7f39a8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 -- 172.21.15.28:0/901106543 shutdown_connections 2024-03-22T02:12:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 --2- 172.21.15.28:0/901106543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4103090 0x7f39b410f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 --2- 172.21.15.28:0/901106543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41074f0 0x7f39b41078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 --2- 172.21.15.28:0/901106543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b41026d0 0x7f39b4102b50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 -- 172.21.15.28:0/901106543 >> 172.21.15.28:0/901106543 conn(0x7f39b40797b0 msgr2=0x7f39b4079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 -- 172.21.15.28:0/901106543 shutdown_connections 2024-03-22T02:12:47.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.555+0000 7f39bbc66700 1 -- 172.21.15.28:0/901106543 wait complete. 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 Processor -- start 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 -- start start 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41026d0 0x7f39b4198e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 0x7f39b419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112960 con 0x7f39b4103090 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41127e0 con 0x7f39b41026d0 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39bbc66700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4112ae0 con 0x7f39b41074f0 2024-03-22T02:12:47.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39b9201700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39b9201700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39236/0 (socket says 172.21.15.28:39236) 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39b9201700 1 -- 172.21.15.28:0/476603149 learned_addr learned my addr 172.21.15.28:0/476603149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39ba203700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 0x7f39b419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.556+0000 7f39b9a02700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41026d0 0x7f39b4198e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 -- 172.21.15.28:0/476603149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41026d0 msgr2=0x7f39b4198e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41026d0 0x7f39b4198e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 -- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 msgr2=0x7f39b4199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 -- 172.21.15.28:0/476603149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b419de10 con 0x7f39b41074f0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39b9201700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39ba203700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 0x7f39b419d6d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00e970 tx=0x7f39ac00ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac004220 con 0x7f39b41074f0 2024-03-22T02:12:47.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39ac004d10 con 0x7f39b41074f0 2024-03-22T02:12:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b419e100 con 0x7f39b41074f0 2024-03-22T02:12:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac010700 con 0x7f39b41074f0 2024-03-22T02:12:47.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.557+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a5980 con 0x7f39b41074f0 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.558+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b4069340 con 0x7f39b41074f0 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.559+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f39ac004750 con 0x7f39b41074f0 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.560+0000 7f39b2ffd700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 0x7f399c0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.560+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f39ac0cf5f0 con 0x7f39b41074f0 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.560+0000 7f39b9a02700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 0x7f399c0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.561+0000 7f39b9a02700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 0x7f399c0a5500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009da0 tx=0x7f39a400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:47.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.564+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f39ac09bf80 con 0x7f39b41074f0 2024-03-22T02:12:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.772+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f39b402dd70 con 0x7f39b41074f0 2024-03-22T02:12:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.773+0000 7f39b2ffd700 1 -- 172.21.15.28:0/476603149 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f39ac09b880 con 0x7f39b41074f0 2024-03-22T02:12:47.773 INFO:teuthology.orchestra.run.smithi028.stdout:201863462965 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 msgr2=0x7f399c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 0x7f399c0a5500 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009da0 tx=0x7f39a400b040 comp rx=0 tx=0).stop 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 msgr2=0x7f39b419d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 0x7f39b419d6d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f39ac00e970 tx=0x7f39ac00ec80 comp rx=0 tx=0).stop 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 shutdown_connections 2024-03-22T02:12:47.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b41074f0 0x7f39b419d6d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39b41026d0 0x7f39b4198e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f399c0a3040 0x7f399c0a5500 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 --2- 172.21.15.28:0/476603149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b4103090 0x7f39b4199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 >> 172.21.15.28:0/476603149 conn(0x7f39b40797b0 msgr2=0x7f39b41007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 shutdown_connections 2024-03-22T02:12:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:47.775+0000 7f39bbc66700 1 -- 172.21.15.28:0/476603149 wait complete. 2024-03-22T02:12:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:48 smithi083 ceph-mon[36340]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/476603149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T02:12:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:48 smithi028 ceph-mon[31916]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/476603149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T02:12:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:48 smithi120 ceph-mon[36803]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/476603149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T02:12:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:50 smithi083 ceph-mon[36340]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:12:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:50 smithi028 ceph-mon[31916]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:12:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:50 smithi120 ceph-mon[36803]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 -- 172.21.15.28:0/1698797192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc101580 msgr2=0x7f1dcc101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/1698797192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc101580 0x7f1dcc101a00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009990 tx=0x7f1dc8009ca0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 -- 172.21.15.28:0/1698797192 shutdown_connections 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/1698797192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101f40 0x7f1dcc10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/1698797192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc107590 0x7f1dcc107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/1698797192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc101580 0x7f1dcc101a00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 -- 172.21.15.28:0/1698797192 >> 172.21.15.28:0/1698797192 conn(0x7f1dcc0784b0 msgr2=0x7f1dcc0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 -- 172.21.15.28:0/1698797192 shutdown_connections 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.083+0000 7f1dd37b2700 1 -- 172.21.15.28:0/1698797192 wait complete. 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 Processor -- start 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 -- start start 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 0x7f1dcc103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 0x7f1dcc103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc111830 con 0x7f1dcc107590 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1116b0 con 0x7f1dcc101f40 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd37b2700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1dcc1119b0 con 0x7f1dcc101580 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd154e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd1d4f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 0x7f1dcc103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd154e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49590/0 (socket says 172.21.15.28:49590) 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd154e700 1 -- 172.21.15.28:0/105494072 learned_addr learned my addr 172.21.15.28:0/105494072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.084+0000 7f1dd0d4d700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 0x7f1dcc103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 -- 172.21.15.28:0/105494072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 msgr2=0x7f1dcc103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 0x7f1dcc103700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 -- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 msgr2=0x7f1dcc103c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 0x7f1dcc103c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 -- 172.21.15.28:0/105494072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dcc1a72d0 con 0x7f1dcc101580 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd1d4f700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 0x7f1dcc103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd0d4d700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 0x7f1dcc103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd154e700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00e9a0 tx=0x7f1dbc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc00ca10 con 0x7f1dcc101580 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dbc00cbb0 con 0x7f1dcc101580 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dcc1a75c0 con 0x7f1dcc101580 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc018680 con 0x7f1dcc101580 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.085+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dcc1a7ad0 con 0x7f1dcc101580 2024-03-22T02:12:51.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.086+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dcc069340 con 0x7f1dcc101580 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.088+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f1dbc010430 con 0x7f1dcc101580 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.088+0000 7f1dc27fc700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 0x7f1db80a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.088+0000 7f1dd0d4d700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 0x7f1db80a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.088+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1dbc0cff30 con 0x7f1dcc101580 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.089+0000 7f1dd0d4d700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 0x7f1db80a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800b2e0 tx=0x7f1dc8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:51.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.092+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1dbc09c8c0 con 0x7f1dcc101580 2024-03-22T02:12:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.301+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1dcc1a7e10 con 0x7f1dcc101580 2024-03-22T02:12:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.302+0000 7f1dc27fc700 1 -- 172.21.15.28:0/105494072 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1dbc09c1c0 con 0x7f1dcc101580 2024-03-22T02:12:51.303 INFO:teuthology.orchestra.run.smithi028.stdout:167503724604 2024-03-22T02:12:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 msgr2=0x7f1db80a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 0x7f1db80a5500 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1dc800b2e0 tx=0x7f1dc8005960 comp rx=0 tx=0).stop 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 msgr2=0x7f1dcc1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00e9a0 tx=0x7f1dbc00ecb0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 shutdown_connections 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1db80a3040 0x7f1db80a5500 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1dcc107590 0x7f1dcc103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1dcc101580 0x7f1dcc1031c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 --2- 172.21.15.28:0/105494072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1dcc101f40 0x7f1dcc103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 >> 172.21.15.28:0/105494072 conn(0x7f1dcc0784b0 msgr2=0x7f1dcc100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 shutdown_connections 2024-03-22T02:12:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:51.304+0000 7f1dd37b2700 1 -- 172.21.15.28:0/105494072 wait complete. 2024-03-22T02:12:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/105494072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T02:12:51.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:51 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/105494072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T02:12:51.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:51 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/105494072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T02:12:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 -- 172.21.15.28:0/809061536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0107a40 msgr2=0x7fe1b010f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 --2- 172.21.15.28:0/809061536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0107a40 0x7fe1b010f5c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800b0d0 tx=0x7fe1a800b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 -- 172.21.15.28:0/809061536 shutdown_connections 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 --2- 172.21.15.28:0/809061536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0107a40 0x7fe1b010f5c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 --2- 172.21.15.28:0/809061536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0101220 0x7fe1b0101600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 --2- 172.21.15.28:0/809061536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0107310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 -- 172.21.15.28:0/809061536 >> 172.21.15.28:0/809061536 conn(0x7fe1b00768c0 msgr2=0x7fe1b0076cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.161+0000 7fe1b4956700 1 -- 172.21.15.28:0/809061536 shutdown_connections 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 -- 172.21.15.28:0/809061536 wait complete. 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 Processor -- start 2024-03-22T02:12:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 -- start start 2024-03-22T02:12:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0107a40 0x7fe1b019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b01129f0 con 0x7fe1b0101220 2024-03-22T02:12:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b0112870 con 0x7fe1b0101bd0 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.162+0000 7fe1b4956700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b0112b70 con 0x7fe1b0107a40 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1add9b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57234/0 (socket says 172.21.15.28:57234) 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 -- 172.21.15.28:0/2161257540 learned_addr learned my addr 172.21.15.28:0/2161257540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1add9b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57236/0 (socket says 172.21.15.28:57236) 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1aed9d700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0107a40 0x7fe1b019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 -- 172.21.15.28:0/2161257540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0107a40 msgr2=0x7fe1b019d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0107a40 0x7fe1b019d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 -- 172.21.15.28:0/2161257540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 msgr2=0x7fe1b0199300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 -- 172.21.15.28:0/2161257540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b019ddd0 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1add9b700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1ae59c700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400e9c0 tx=0x7fe1a400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a400cbd0 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b019e0c0 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b01a5940 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a400ee60 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.163+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4005630 con 0x7fe1b0101220 2024-03-22T02:12:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.165+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe190005330 con 0x7fe1b0101220 2024-03-22T02:12:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.166+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe1a4004750 con 0x7fe1b0101220 2024-03-22T02:12:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.166+0000 7fe19f7fe700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 0x7fe1980a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.167+0000 7fe1add9b700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 0x7fe1980a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.167+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe1a40cf9e0 con 0x7fe1b0101220 2024-03-22T02:12:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.167+0000 7fe1add9b700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 0x7fe1980a5580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0108d90 tx=0x7fe1a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.171+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1a409c370 con 0x7fe1b0101220 2024-03-22T02:12:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.381+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe1900058f0 con 0x7fe1b0101220 2024-03-22T02:12:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.381+0000 7fe19f7fe700 1 -- 172.21.15.28:0/2161257540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe1a409bc70 con 0x7fe1b0101220 2024-03-22T02:12:52.382 INFO:teuthology.orchestra.run.smithi028.stdout:55834574927 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 msgr2=0x7fe1980a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 0x7fe1980a5580 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0108d90 tx=0x7fe1a0009260 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 msgr2=0x7fe1b0198dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe1a400e9c0 tx=0x7fe1a400ecd0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 shutdown_connections 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1980a30c0 0x7fe1980a5580 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b0101220 0x7fe1b0198dc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1b0107a40 0x7fe1b019d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 --2- 172.21.15.28:0/2161257540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1b0101bd0 0x7fe1b0199300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 >> 172.21.15.28:0/2161257540 conn(0x7fe1b00768c0 msgr2=0x7fe1b00fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 shutdown_connections 2024-03-22T02:12:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:52.383+0000 7fe1b4956700 1 -- 172.21.15.28:0/2161257540 wait complete. 2024-03-22T02:12:52.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724598 got 167503724604 for osd.5 2024-03-22T02:12:52.587 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:52 smithi083 ceph-mon[36340]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:52 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2161257540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T02:12:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:52 smithi028 ceph-mon[31916]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2161257540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T02:12:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:52 smithi120 ceph-mon[36803]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:12:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:52 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2161257540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T02:12:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- 172.21.15.28:0/3016334206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c06b780 msgr2=0x7fe59c10f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 --2- 172.21.15.28:0/3016334206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c06b780 0x7fe59c10f5d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe59400b0d0 tx=0x7fe59400b3e0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- 172.21.15.28:0/3016334206 shutdown_connections 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 --2- 172.21.15.28:0/3016334206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c06b780 0x7fe59c10f5d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 --2- 172.21.15.28:0/3016334206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c1076f0 0x7fe59c107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 --2- 172.21.15.28:0/3016334206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06adc0 0x7fe59c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- 172.21.15.28:0/3016334206 >> 172.21.15.28:0/3016334206 conn(0x7fe59c0783e0 msgr2=0x7fe59c0787f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- 172.21.15.28:0/3016334206 shutdown_connections 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- 172.21.15.28:0/3016334206 wait complete. 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 Processor -- start 2024-03-22T02:12:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.403+0000 7fe5a247c700 1 -- start start 2024-03-22T02:12:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c06adc0 0x7fe59c101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 0x7fe59c1023b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c112a20 con 0x7fe59c1076f0 2024-03-22T02:12:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c1128a0 con 0x7fe59c06adc0 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a247c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe59c112ba0 con 0x7fe59c06b780 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 0x7fe59c1023b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe59b7fe700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59966/0 (socket says 172.21.15.28:59966) 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe59b7fe700 1 -- 172.21.15.28:0/2134717322 learned_addr learned my addr 172.21.15.28:0/2134717322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe59bfff700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c06adc0 0x7fe59c101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 -- 172.21.15.28:0/2134717322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 msgr2=0x7fe59c101e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 -- 172.21.15.28:0/2134717322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c06adc0 msgr2=0x7fe59c101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c06adc0 0x7fe59c101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe5a0a19700 1 -- 172.21.15.28:0/2134717322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59c1a72a0 con 0x7fe59c1076f0 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.404+0000 7fe59b7fe700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5a0a19700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 0x7fe59c1023b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe59400b770 tx=0x7fe594007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe594021020 con 0x7fe59c1076f0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe594004c50 con 0x7fe59c1076f0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59c1a7530 con 0x7fe59c1076f0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5940052b0 con 0x7fe59c1076f0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.405+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe59c1a7990 con 0x7fe59c1076f0 2024-03-22T02:12:53.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.406+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59c103e40 con 0x7fe59c1076f0 2024-03-22T02:12:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.408+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe594024020 con 0x7fe59c1076f0 2024-03-22T02:12:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.408+0000 7fe5997fa700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 0x7fe5880a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.408+0000 7fe59bfff700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 0x7fe5880a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.408+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe5940d0e60 con 0x7fe59c1076f0 2024-03-22T02:12:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.409+0000 7fe59bfff700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 0x7fe5880a5500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe58c007b60 tx=0x7fe58c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:53.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.412+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe59401de20 con 0x7fe59c1076f0 2024-03-22T02:12:53.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.634+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fe59c1a7b60 con 0x7fe59c1076f0 2024-03-22T02:12:53.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.634+0000 7fe5997fa700 1 -- 172.21.15.28:0/2134717322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe59402b020 con 0x7fe59c1076f0 2024-03-22T02:12:53.637 INFO:teuthology.orchestra.run.smithi028.stdout:34359738453 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 msgr2=0x7fe5880a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 0x7fe5880a5500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe58c007b60 tx=0x7fe58c0073d0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 msgr2=0x7fe59c1023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 0x7fe59c1023b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe59400b770 tx=0x7fe594007a20 comp rx=0 tx=0).stop 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 shutdown_connections 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5880a3040 0x7fe5880a5500 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59c1076f0 0x7fe59c1023b0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe59c06adc0 0x7fe59c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 --2- 172.21.15.28:0/2134717322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe59c06b780 0x7fe59c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:53.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 >> 172.21.15.28:0/2134717322 conn(0x7fe59c0783e0 msgr2=0x7fe59c1002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:53.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 shutdown_connections 2024-03-22T02:12:53.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:53.637+0000 7fe5a247c700 1 -- 172.21.15.28:0/2134717322 wait complete. 2024-03-22T02:12:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:54 smithi028 ceph-mon[31916]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2134717322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T02:12:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/687384201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T02:12:54.742 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738446 got 34359738453 for osd.0 2024-03-22T02:12:54.743 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 -- 172.21.15.28:0/890469084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 msgr2=0x7f0608108a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 --2- 172.21.15.28:0/890469084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f0608108a20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f05f80099c0 tx=0x7f05f8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 -- 172.21.15.28:0/890469084 shutdown_connections 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 --2- 172.21.15.28:0/890469084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f0608108a20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 --2- 172.21.15.28:0/890469084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806acd0 0x7f060810f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 --2- 172.21.15.28:0/890469084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806a330 0x7f060806a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 -- 172.21.15.28:0/890469084 >> 172.21.15.28:0/890469084 conn(0x7f06080784b0 msgr2=0x7f06080788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 -- 172.21.15.28:0/890469084 shutdown_connections 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.091+0000 7f060d557700 1 -- 172.21.15.28:0/890469084 wait complete. 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 Processor -- start 2024-03-22T02:12:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 -- start start 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 0x7f0608101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0608112960 con 0x7f0608108640 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f06081127e0 con 0x7f060806a330 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f060d557700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0608112ae0 con 0x7f060806acd0 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06077fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59984/0 (socket says 172.21.15.28:59984) 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f0606ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 0x7f0608101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 -- 172.21.15.28:0/687384201 learned_addr learned my addr 172.21.15.28:0/687384201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06077fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57272/0 (socket says 172.21.15.28:57272) 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 -- 172.21.15.28:0/687384201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 msgr2=0x7f0608101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 0x7f0608101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 -- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 msgr2=0x7f06081020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06067fc700 1 -- 172.21.15.28:0/687384201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06081a7380 con 0x7f060806acd0 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f06077fe700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.092+0000 7f0606ffd700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 0x7f0608101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stdout:223338299442 2024-03-22T02:12:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f06067fc700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9a0 tx=0x7f05fc00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc00ca10 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05fc00cbb0 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06081a7670 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc018680 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.093+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06081a7c00 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.094+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0608069340 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.095+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f05fc010430 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.096+0000 7f05effff700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 0x7f05f00a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.096+0000 7f0606ffd700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 0x7f05f00a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.096+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f05fc0cff60 con 0x7f060806acd0 2024-03-22T02:12:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.096+0000 7f0606ffd700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 0x7f05f00a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c00 tx=0x7f05f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.099+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f05fc09c890 con 0x7f060806acd0 2024-03-22T02:12:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.306+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f060802dd70 con 0x7f060806acd0 2024-03-22T02:12:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.306+0000 7f05effff700 1 -- 172.21.15.28:0/687384201 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f05fc09c190 con 0x7f060806acd0 2024-03-22T02:12:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 msgr2=0x7f05f00a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 0x7f05f00a5580 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f05f8000c00 tx=0x7f05f8005fb0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 msgr2=0x7f0608101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05fc00e9a0 tx=0x7f05fc00ecb0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 shutdown_connections 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f05f00a30c0 0x7f05f00a5580 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0608108640 0x7f06081020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f060806a330 0x7f0608101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 --2- 172.21.15.28:0/687384201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f060806acd0 0x7f0608101b90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 >> 172.21.15.28:0/687384201 conn(0x7f06080784b0 msgr2=0x7f06080ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 shutdown_connections 2024-03-22T02:12:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:54.308+0000 7f060d557700 1 -- 172.21.15.28:0/687384201 wait complete. 2024-03-22T02:12:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:54 smithi120 ceph-mon[36803]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2134717322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T02:12:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/687384201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T02:12:55.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:54 smithi083 ceph-mon[36340]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:12:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2134717322' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T02:12:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/687384201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T02:12:56.722 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:56 smithi028 ceph-mon[31916]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:56.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574922 got 55834574927 for osd.1 2024-03-22T02:12:56.944 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:56 smithi120 ceph-mon[36803]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:57.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:56 smithi083 ceph-mon[36340]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 -- 172.21.15.28:0/1000127133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101560 msgr2=0x7f94681019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 --2- 172.21.15.28:0/1000127133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101560 0x7f94681019e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f945c0099c0 tx=0x7f945c009cd0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 -- 172.21.15.28:0/1000127133 shutdown_connections 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 --2- 172.21.15.28:0/1000127133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101f20 0x7f946810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 --2- 172.21.15.28:0/1000127133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 0x7f94681068f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 --2- 172.21.15.28:0/1000127133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101560 0x7f94681019e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 -- 172.21.15.28:0/1000127133 >> 172.21.15.28:0/1000127133 conn(0x7f9468078590 msgr2=0x7f94680789a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 -- 172.21.15.28:0/1000127133 shutdown_connections 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.499+0000 7f946de1b700 1 -- 172.21.15.28:0/1000127133 wait complete. 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 Processor -- start 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 -- start start 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101560 0x7f9468199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 0x7f94681a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9468111880 con 0x7f9468101f20 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9468111700 con 0x7f9468106510 2024-03-22T02:12:57.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f946de1b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9468111a00 con 0x7f9468101560 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f9466ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f9466ffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57292/0 (socket says 172.21.15.28:57292) 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f9466ffd700 1 -- 172.21.15.28:0/2266130411 learned_addr learned my addr 172.21.15.28:0/2266130411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f94677fe700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101560 0x7f9468199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.500+0000 7f9467fff700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 0x7f94681a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 -- 172.21.15.28:0/2266130411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101560 msgr2=0x7f9468199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101560 0x7f9468199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 -- 172.21.15.28:0/2266130411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 msgr2=0x7f94681a4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 0x7f94681a4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 -- 172.21.15.28:0/2266130411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94681a5240 con 0x7f9468101f20 2024-03-22T02:12:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9466ffd700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f945c006010 tx=0x7f945c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c003680 con 0x7f9468101f20 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f945c0050d0 con 0x7f9468101f20 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94681a5410 con 0x7f9468101f20 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945c003db0 con 0x7f9468101f20 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.501+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94681a58c0 con 0x7f9468101f20 2024-03-22T02:12:57.037 INFO:teuthology.orchestra.run.smithi028.stdout:292057776168 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.502+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9468069340 con 0x7f9468101f20 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.503+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f945c005280 con 0x7f9468101f20 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.504+0000 7f9464ff9700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 0x7f94500a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.504+0000 7f94677fe700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 0x7f94500a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.504+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f945c0d07b0 con 0x7f9468101f20 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.504+0000 7f94677fe700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 0x7f94500a56a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9468102770 tx=0x7f9458009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.507+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f945c05d740 con 0x7f9468101f20 2024-03-22T02:12:57.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.721+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f946802dd70 con 0x7f9468101f20 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.722+0000 7f9464ff9700 1 -- 172.21.15.28:0/2266130411 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f945c013070 con 0x7f9468101f20 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 msgr2=0x7f94500a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 0x7f94500a56a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9468102770 tx=0x7f9458009260 comp rx=0 tx=0).stop 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 msgr2=0x7f946819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f945c006010 tx=0x7f945c00f690 comp rx=0 tx=0).stop 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 shutdown_connections 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9468106510 0x7f94681a4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9468101560 0x7f9468199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94500a31e0 0x7f94500a56a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 --2- 172.21.15.28:0/2266130411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9468101f20 0x7f946819a1c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.724+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 >> 172.21.15.28:0/2266130411 conn(0x7f9468078590 msgr2=0x7f94680ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:57.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.725+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 shutdown_connections 2024-03-22T02:12:57.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.725+0000 7f946de1b700 1 -- 172.21.15.28:0/2266130411 wait complete. 2024-03-22T02:12:57.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2266130411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T02:12:57.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1492778717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T02:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 -- 172.21.15.28:0/3787485995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0101f40 msgr2=0x7febe010e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 --2- 172.21.15.28:0/3787485995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0101f40 0x7febe010e3d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7febd800b0a0 tx=0x7febd800b3b0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 -- 172.21.15.28:0/3787485995 shutdown_connections 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 --2- 172.21.15.28:0/3787485995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0107590 0x7febe0107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 --2- 172.21.15.28:0/3787485995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0101f40 0x7febe010e3d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 --2- 172.21.15.28:0/3787485995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101580 0x7febe0101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 -- 172.21.15.28:0/3787485995 >> 172.21.15.28:0/3787485995 conn(0x7febe00784b0 msgr2=0x7febe00788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 -- 172.21.15.28:0/3787485995 shutdown_connections 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.617+0000 7febe764e700 1 -- 172.21.15.28:0/3787485995 wait complete. 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 Processor -- start 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 -- start start 2024-03-22T02:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 0x7febe0103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0107590 0x7febe0103c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7febe0111880 con 0x7febe0101580 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7febe0111700 con 0x7febe0101f40 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe764e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7febe0111a00 con 0x7febe0107590 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe53ea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe53ea700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57306/0 (socket says 172.21.15.28:57306) 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe4be9700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 0x7febe0103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe53ea700 1 -- 172.21.15.28:0/1492778717 learned_addr learned my addr 172.21.15.28:0/1492778717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.618+0000 7febe5beb700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0107590 0x7febe0103c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 -- 172.21.15.28:0/1492778717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0107590 msgr2=0x7febe0103c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0107590 0x7febe0103c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 -- 172.21.15.28:0/1492778717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 msgr2=0x7febe0103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 0x7febe0103700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 -- 172.21.15.28:0/1492778717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe01a7270 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe4be9700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 0x7febe0103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe53ea700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e9a0 tx=0x7febdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc00cbd0 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe01a7560 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febdc00cd70 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc018680 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.619+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe01a7af0 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.621+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe0069340 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.622+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7febdc0188e0 con 0x7febe0101580 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.622+0000 7febd67fc700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 0x7febcc0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.623+0000 7febe4be9700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 0x7febcc0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stdout:77309411404 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.623+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7febdc0cfbc0 con 0x7febe0101580 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.623+0000 7febe4be9700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 0x7febcc0a5500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7febe0078a20 tx=0x7febd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.627+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febdc053740 con 0x7febe0101580 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.833+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7febe01a7dc0 con 0x7febe0101580 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.834+0000 7febd67fc700 1 -- 172.21.15.28:0/1492778717 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7febdc09c550 con 0x7febe0101580 2024-03-22T02:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 msgr2=0x7febcc0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 0x7febcc0a5500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7febe0078a20 tx=0x7febd0009260 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 msgr2=0x7febe01031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7febdc00e9a0 tx=0x7febdc00ecb0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 shutdown_connections 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7febcc0a3040 0x7febcc0a5500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febe0101580 0x7febe01031c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febe0107590 0x7febe0103c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 --2- 172.21.15.28:0/1492778717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7febe0101f40 0x7febe0103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 >> 172.21.15.28:0/1492778717 conn(0x7febe00784b0 msgr2=0x7febe01008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 shutdown_connections 2024-03-22T02:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:12:56.836+0000 7febe764e700 1 -- 172.21.15.28:0/1492778717 wait complete. 2024-03-22T02:12:57.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2266130411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T02:12:57.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1492778717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T02:12:58.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2266130411' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-22T02:12:58.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1492778717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T02:12:58.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-03-22T02:12:58.402 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:58.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:12:58 smithi028 ceph-mon[31916]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:12:58 smithi120 ceph-mon[36803]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:12:58 smithi083 ceph-mon[36340]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:12:59.027 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462961 got 201863462965 for osd.6 2024-03-22T02:12:59.027 DEBUG:teuthology.parallel:result is None 2024-03-22T02:12:59.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776168 for osd.9 2024-03-22T02:12:59.827 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.234+0000 7f652fdef700 1 -- 172.21.15.28:0/4227103819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 msgr2=0x7f652810f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.234+0000 7f652fdef700 1 --2- 172.21.15.28:0/4227103819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f652810f4d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f652000b0d0 tx=0x7f652000b3e0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- 172.21.15.28:0/4227103819 shutdown_connections 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- 172.21.15.28:0/4227103819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f652810f4d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- 172.21.15.28:0/4227103819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 0x7f65281078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- 172.21.15.28:0/4227103819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- 172.21.15.28:0/4227103819 >> 172.21.15.28:0/4227103819 conn(0x7f65280797b0 msgr2=0x7f6528079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- 172.21.15.28:0/4227103819 shutdown_connections 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- 172.21.15.28:0/4227103819 wait complete. 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 Processor -- start 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- start start 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 0x7f6528075400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6528112980 con 0x7f6528103090 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6528112800 con 0x7f65281026d0 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.235+0000 7f652fdef700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6528112b00 con 0x7f65281074f0 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652db8b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652db8b700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57318/0 (socket says 172.21.15.28:57318) 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57328/0 (socket says 172.21.15.28:57328) 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652db8b700 1 -- 172.21.15.28:0/86085226 learned_addr learned my addr 172.21.15.28:0/86085226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652e38c700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 0x7f6528075400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 -- 172.21.15.28:0/86085226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 msgr2=0x7f6528075400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 0x7f6528075400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 -- 172.21.15.28:0/86085226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 msgr2=0x7f6528074980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528074980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 -- 172.21.15.28:0/86085226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65281a7300 con 0x7f6528103090 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652d38a700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f651800d750 tx=0x7f651800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518014030 con 0x7f6528103090 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6518004500 con 0x7f6528103090 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65281a75f0 con 0x7f6528103090 2024-03-22T02:13:00.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518005420 con 0x7f6528103090 2024-03-22T02:13:00.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.236+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65281a7b30 con 0x7f6528103090 2024-03-22T02:13:00.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.237+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6528069340 con 0x7f6528103090 2024-03-22T02:13:00.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.238+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6518004750 con 0x7f6528103090 2024-03-22T02:13:00.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.239+0000 7f651effd700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 0x7f65140a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.239+0000 7f652db8b700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 0x7f65140a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.239+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f65180cf5a0 con 0x7f6528103090 2024-03-22T02:13:00.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.240+0000 7f652db8b700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 0x7f65140a5500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6524009ca0 tx=0x7f6524009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:00.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.243+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f651809bf30 con 0x7f6528103090 2024-03-22T02:13:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.448+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f652802dd70 con 0x7f6528103090 2024-03-22T02:13:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.449+0000 7f651effd700 1 -- 172.21.15.28:0/86085226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f651809b830 con 0x7f6528103090 2024-03-22T02:13:00.449 INFO:teuthology.orchestra.run.smithi028.stdout:347892351006 2024-03-22T02:13:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 msgr2=0x7f65140a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 0x7f65140a5500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6524009ca0 tx=0x7f6524009260 comp rx=0 tx=0).stop 2024-03-22T02:13:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 msgr2=0x7f6528074ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f651800d750 tx=0x7f651800da60 comp rx=0 tx=0).stop 2024-03-22T02:13:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 shutdown_connections 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65281074f0 0x7f6528075400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65281026d0 0x7f6528074980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65140a3040 0x7f65140a5500 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 --2- 172.21.15.28:0/86085226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6528103090 0x7f6528074ec0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 >> 172.21.15.28:0/86085226 conn(0x7f65280797b0 msgr2=0x7f65281006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 shutdown_connections 2024-03-22T02:13:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.451+0000 7f652fdef700 1 -- 172.21.15.28:0/86085226 wait complete. 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 -- 172.21.15.28:0/3149142226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101580 msgr2=0x7f099c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 --2- 172.21.15.28:0/3149142226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101580 0x7f099c101a00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f098c0098f0 tx=0x7f098c009c00 comp rx=0 tx=0).stop 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 -- 172.21.15.28:0/3149142226 shutdown_connections 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 --2- 172.21.15.28:0/3149142226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101f40 0x7f099c10e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 --2- 172.21.15.28:0/3149142226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 --2- 172.21.15.28:0/3149142226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101580 0x7f099c101a00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 -- 172.21.15.28:0/3149142226 >> 172.21.15.28:0/3149142226 conn(0x7f099c0784b0 msgr2=0x7f099c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 -- 172.21.15.28:0/3149142226 shutdown_connections 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.482+0000 7f09a329a700 1 -- 172.21.15.28:0/3149142226 wait complete. 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 Processor -- start 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 -- start start 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 0x7f099c19b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f099c1118e0 con 0x7f099c101580 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f099c111760 con 0x7f099c107590 2024-03-22T02:13:00.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a329a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f099c111a60 con 0x7f099c101f40 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a1036700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a1036700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57340/0 (socket says 172.21.15.28:57340) 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a1036700 1 -- 172.21.15.28:0/1581062581 learned_addr learned my addr 172.21.15.28:0/1581062581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a0835700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 0x7f099c19b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.483+0000 7f09a1837700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 -- 172.21.15.28:0/1581062581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 msgr2=0x7f099c19b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 0x7f099c19b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 -- 172.21.15.28:0/1581062581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 msgr2=0x7f099c19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 -- 172.21.15.28:0/1581062581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c1a0020 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a0835700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 0x7f099c19b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1837700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a1036700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9a0 tx=0x7f099800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f099800ca10 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099c1a0310 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f099800cbb0 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0998018680 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.484+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099c1a7b90 con 0x7f099c101580 2024-03-22T02:13:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.485+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f099c069340 con 0x7f099c101580 2024-03-22T02:13:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.487+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f0998010430 con 0x7f099c101580 2024-03-22T02:13:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.488+0000 7f09927fc700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 0x7f09880a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.488+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f09980cff90 con 0x7f099c101580 2024-03-22T02:13:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.488+0000 7f09a0835700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 0x7f09880a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.489+0000 7f09a0835700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 0x7f09880a5500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f098c009810 tx=0x7f098c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.492+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f099809c920 con 0x7f099c101580 2024-03-22T02:13:00.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.698+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f099c1a7d60 con 0x7f099c101580 2024-03-22T02:13:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:00 smithi028 ceph-mon[31916]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:00.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:00 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/86085226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T02:13:00.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.698+0000 7f09927fc700 1 -- 172.21.15.28:0/1581062581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f099809c220 con 0x7f099c101580 2024-03-22T02:13:00.699 INFO:teuthology.orchestra.run.smithi028.stdout:146028888131 2024-03-22T02:13:00.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 msgr2=0x7f09880a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 0x7f09880a5500 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f098c009810 tx=0x7f098c019040 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 msgr2=0x7f099c19b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f099800e9a0 tx=0x7f099800ecb0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 shutdown_connections 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09880a3040 0x7f09880a5500 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f099c101580 0x7f099c19b010 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f099c107590 0x7f099c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 --2- 172.21.15.28:0/1581062581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f099c101f40 0x7f099c19b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 >> 172.21.15.28:0/1581062581 conn(0x7f099c0784b0 msgr2=0x7f099c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 shutdown_connections 2024-03-22T02:13:00.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:00.701+0000 7f09a329a700 1 -- 172.21.15.28:0/1581062581 wait complete. 2024-03-22T02:13:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:00 smithi120 ceph-mon[36803]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:00 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/86085226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T02:13:01.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:00 smithi083 ceph-mon[36340]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:01.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:00 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/86085226' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-22T02:13:01.135 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351006 for osd.11 2024-03-22T02:13:01.135 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:01.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:01 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1581062581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T02:13:01.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:01.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:01 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.986+0000 7f67cd2fb700 1 -- 172.21.15.28:0/1393686368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 msgr2=0x7f67c81068f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.986+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/1393686368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c81068f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f67b80099c0 tx=0x7f67b8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- 172.21.15.28:0/1393686368 shutdown_connections 2024-03-22T02:13:01.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/1393686368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c81068f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/1393686368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/1393686368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- 172.21.15.28:0/1393686368 >> 172.21.15.28:0/1393686368 conn(0x7f67c8078590 msgr2=0x7f67c80789a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- 172.21.15.28:0/1393686368 shutdown_connections 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- 172.21.15.28:0/1393686368 wait complete. 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 Processor -- start 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- start start 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c81036e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:01.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c8103c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67c81118a0 con 0x7f67c8106510 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67c8111720 con 0x7f67c8101f20 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.987+0000 7f67cd2fb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67c8111a20 con 0x7f67c8101560 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c8103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c8103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50206/0 (socket says 172.21.15.28:50206) 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53222/0 (socket says 172.21.15.28:53222) 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c77fe700 1 -- 172.21.15.28:0/651664015 learned_addr learned my addr 172.21.15.28:0/651664015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67bffff700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c81036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 -- 172.21.15.28:0/651664015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 msgr2=0x7f67c81036e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c81036e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 -- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 msgr2=0x7f67c8103c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:01.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c8103c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 -- 172.21.15.28:0/651664015 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c81a73f0 con 0x7f67c8101560 2024-03-22T02:13:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67bffff700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c6ffd700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b580 tx=0x7f67b8003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b8003680 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c81a7680 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b80055a0 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b800fc60 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.988+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c81a7b60 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.989+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67c8069340 con 0x7f67c8101560 2024-03-22T02:13:01.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.991+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f67b800f460 con 0x7f67c8101560 2024-03-22T02:13:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.992+0000 7f67c4ff9700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 0x7f67b40a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.992+0000 7f67bffff700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 0x7f67b40a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.992+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f67b80d0a60 con 0x7f67c8101560 2024-03-22T02:13:01.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.992+0000 7f67bffff700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 0x7f67b40a56a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009680 tx=0x7f67b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:01.995+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67b8013070 con 0x7f67c8101560 2024-03-22T02:13:01.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:01 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1581062581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T02:13:01.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:01.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:01 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:02.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:01 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1581062581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T02:13:02.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:02.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:01 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:02.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.205+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f67c802dd70 con 0x7f67c8101560 2024-03-22T02:13:02.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.205+0000 7f67c4ff9700 1 -- 172.21.15.28:0/651664015 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f67b809d480 con 0x7f67c8101560 2024-03-22T02:13:02.205 INFO:teuthology.orchestra.run.smithi028.stdout:257698037807 2024-03-22T02:13:02.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 msgr2=0x7f67b40a56a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 0x7f67b40a56a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009680 tx=0x7f67b0009260 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 msgr2=0x7f67c81031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67b800b580 tx=0x7f67b8003a90 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 shutdown_connections 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67b40a31e0 0x7f67b40a56a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c8106510 0x7f67c8103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67c8101560 0x7f67c81031a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 --2- 172.21.15.28:0/651664015 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67c8101f20 0x7f67c81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 >> 172.21.15.28:0/651664015 conn(0x7f67c8078590 msgr2=0x7f67c806b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:02.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 shutdown_connections 2024-03-22T02:13:02.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:02.207+0000 7f67cd2fb700 1 -- 172.21.15.28:0/651664015 wait complete. 2024-03-22T02:13:02.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299437 got 223338299442 for osd.7 2024-03-22T02:13:02.314 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:02 smithi028 ceph-mon[31916]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:02 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:13:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:02 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:13:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:02 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:02 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/651664015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T02:13:02.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888131 for osd.4 2024-03-22T02:13:02.948 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:02 smithi120 ceph-mon[36803]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:02 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:13:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:02 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:13:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:02 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:02 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/651664015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T02:13:03.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:02 smithi083 ceph-mon[36340]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:03.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:02 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:13:03.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:02 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:13:03.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:02 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:13:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:02 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/651664015' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-22T02:13:03.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037802 got 257698037807 for osd.8 2024-03-22T02:13:03.608 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.844+0000 7efef13a4700 1 -- 172.21.15.28:0/2171591454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec103090 msgr2=0x7efeec10f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.844+0000 7efef13a4700 1 --2- 172.21.15.28:0/2171591454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec103090 0x7efeec10f4d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efee400b0d0 tx=0x7efee400b3e0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 -- 172.21.15.28:0/2171591454 shutdown_connections 2024-03-22T02:13:03.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 --2- 172.21.15.28:0/2171591454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec103090 0x7efeec10f4d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 --2- 172.21.15.28:0/2171591454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec1074f0 0x7efeec1078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 --2- 172.21.15.28:0/2171591454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1026d0 0x7efeec102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 -- 172.21.15.28:0/2171591454 >> 172.21.15.28:0/2171591454 conn(0x7efeec0797b0 msgr2=0x7efeec079bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 -- 172.21.15.28:0/2171591454 shutdown_connections 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 -- 172.21.15.28:0/2171591454 wait complete. 2024-03-22T02:13:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 Processor -- start 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.845+0000 7efef13a4700 1 -- start start 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 0x7efeec1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1074f0 0x7efeec19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efeec112960 con 0x7efeec1026d0 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efeec1127e0 con 0x7efeec103090 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efeec112ae0 con 0x7efeec1074f0 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 0x7efeec1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeeaffd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50230/0 (socket says 172.21.15.28:50230) 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeeaffd700 1 -- 172.21.15.28:0/4265790347 learned_addr learned my addr 172.21.15.28:0/4265790347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeeb7fe700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1074f0 0x7efeec19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 -- 172.21.15.28:0/4265790347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1074f0 msgr2=0x7efeec19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1074f0 0x7efeec19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 -- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 msgr2=0x7efeec198db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 -- 172.21.15.28:0/4265790347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeec19ddc0 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeeaffd700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efeea7fc700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 0x7efeec1992f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9f0 tx=0x7efee000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0004220 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efee0004d10 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeec19e0b0 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0005020 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.846+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeec1a5930 con 0x7efeec103090 2024-03-22T02:13:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.847+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeec069340 con 0x7efeec103090 2024-03-22T02:13:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.848+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7efee0004750 con 0x7efeec103090 2024-03-22T02:13:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.849+0000 7efed3fff700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 0x7efed40a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.849+0000 7efeeaffd700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 0x7efed40a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.849+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7efee00cf840 con 0x7efeec103090 2024-03-22T02:13:03.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.849+0000 7efeeaffd700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 0x7efed40a5500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efedc011550 tx=0x7efedc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:03.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:03.853+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efee009c170 con 0x7efeec103090 2024-03-22T02:13:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.058+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7efeec02dd70 con 0x7efeec103090 2024-03-22T02:13:04.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.058+0000 7efed3fff700 1 -- 172.21.15.28:0/4265790347 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7efee009ba70 con 0x7efeec103090 2024-03-22T02:13:04.058 INFO:teuthology.orchestra.run.smithi028.stdout:111669149768 2024-03-22T02:13:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 msgr2=0x7efed40a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 0x7efed40a5500 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efedc011550 tx=0x7efedc009260 comp rx=0 tx=0).stop 2024-03-22T02:13:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 msgr2=0x7efeec1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:04.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 0x7efeec1992f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9f0 tx=0x7efee000ed00 comp rx=0 tx=0).stop 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 shutdown_connections 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efed40a3040 0x7efed40a5500 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efeec1026d0 0x7efeec198db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeec1074f0 0x7efeec19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 --2- 172.21.15.28:0/4265790347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efeec103090 0x7efeec1992f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 >> 172.21.15.28:0/4265790347 conn(0x7efeec0797b0 msgr2=0x7efeec100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 shutdown_connections 2024-03-22T02:13:04.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:04.060+0000 7efef13a4700 1 -- 172.21.15.28:0/4265790347 wait complete. 2024-03-22T02:13:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:04 smithi028 ceph-mon[31916]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:04 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4265790347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T02:13:04.902 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149762 got 111669149768 for osd.3 2024-03-22T02:13:04.902 DEBUG:teuthology.parallel:result is None 2024-03-22T02:13:04.902 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-22T02:13:04.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T02:13:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:04 smithi120 ceph-mon[36803]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:04 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4265790347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T02:13:05.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:04 smithi083 ceph-mon[36340]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:04 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4265790347' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T02:13:05.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:06.745 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:06 smithi028 ceph-mon[31916]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:06.800 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:06 smithi120 ceph-mon[36803]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 -- 172.21.15.28:0/2241878707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 msgr2=0x7eff14102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2241878707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14102bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7eff08009990 tx=0x7eff08009ca0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 -- 172.21.15.28:0/2241878707 shutdown_connections 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2241878707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff141030f0 0x7eff1410f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2241878707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff14107650 0x7eff14107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2241878707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14102bb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 -- 172.21.15.28:0/2241878707 >> 172.21.15.28:0/2241878707 conn(0x7eff14079750 msgr2=0x7eff14079b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 -- 172.21.15.28:0/2241878707 shutdown_connections 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.986+0000 7eff1aa65700 1 -- 172.21.15.28:0/2241878707 wait complete. 2024-03-22T02:13:06.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 Processor -- start 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 -- start start 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff141030f0 0x7eff14199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff14107650 0x7eff1419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff14112980 con 0x7eff14102730 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7eff14112800 con 0x7eff141030f0 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff1aa65700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7eff14112b00 con 0x7eff14107650 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff18801700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff18801700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50250/0 (socket says 172.21.15.28:50250) 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.987+0000 7eff18801700 1 -- 172.21.15.28:0/2986805437 learned_addr learned my addr 172.21.15.28:0/2986805437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff13fff700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff141030f0 0x7eff14199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff19002700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff14107650 0x7eff1419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 -- 172.21.15.28:0/2986805437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff14107650 msgr2=0x7eff1419d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff14107650 0x7eff1419d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 -- 172.21.15.28:0/2986805437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff141030f0 msgr2=0x7eff14199370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff141030f0 0x7eff14199370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 -- 172.21.15.28:0/2986805437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1419ddf0 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff18801700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00ef10 tx=0x7eff0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c00c9c0 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1419e080 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff0c019070 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff0c007e60 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.988+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff141a5960 con 0x7eff14102730 2024-03-22T02:13:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.989+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff14069340 con 0x7eff14102730 2024-03-22T02:13:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.990+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7eff0c012050 con 0x7eff14102730 2024-03-22T02:13:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.991+0000 7eff11ffb700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 0x7efefc0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.991+0000 7eff13fff700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 0x7efefc0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.991+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7eff0c0cf830 con 0x7eff14102730 2024-03-22T02:13:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.991+0000 7eff13fff700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 0x7efefc0a5500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7eff0800b2e0 tx=0x7eff08005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:06.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:06.995+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff0c010040 con 0x7eff14102730 2024-03-22T02:13:07.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:06 smithi083 ceph-mon[36340]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:07.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.195+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7eff14062ac0 con 0x7efefc0a3040 2024-03-22T02:13:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.205+0000 7eff11ffb700 1 -- 172.21.15.28:0/2986805437 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340655 (secure 0 0 0) 0x7eff14062ac0 con 0x7efefc0a3040 2024-03-22T02:13:07.210 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.208+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 msgr2=0x7efefc0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.208+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 0x7efefc0a5500 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7eff0800b2e0 tx=0x7eff08005960 comp rx=0 tx=0).stop 2024-03-22T02:13:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.208+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 msgr2=0x7eff14198e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.208+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7eff0c00ef10 tx=0x7eff0c00c3a0 comp rx=0 tx=0).stop 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 shutdown_connections 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efefc0a3040 0x7efefc0a5500 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff14102730 0x7eff14198e30 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7eff14107650 0x7eff1419d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 --2- 172.21.15.28:0/2986805437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7eff141030f0 0x7eff14199370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 >> 172.21.15.28:0/2986805437 conn(0x7eff14079750 msgr2=0x7eff14100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 shutdown_connections 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:07.209+0000 7eff1aa65700 1 -- 172.21.15.28:0/2986805437 wait complete. 2024-03-22T02:13:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:13:07.923 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-22T02:13:06.575521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":571,"num_read_kb":404,"num_write":343,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365648,"kb_used_data":12248,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576176,"statfs":{"total":1151940427776,"available":1151566004224,"internally_reserved":0,"allocated":12541952,"data_stored":7995443,"data_compressed":361053,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":40,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003422"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485912+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:12:28.485912+0000","last_peered":"2024-03-22T02:12:28.485912+0000","last_clean":"2024-03-22T02:12:28.485912+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:12:28.485912+0000","last_undegraded":"2024-03-22T02:12:28.485912+0000","last_fullsized":"2024-03-22T02:12:28.485912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487376+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:12:28.487376+0000","last_peered":"2024-03-22T02:12:28.487376+0000","last_clean":"2024-03-22T02:12:28.487376+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:12:28.487376+0000","last_undegraded":"2024-03-22T02:12:28.487376+0000","last_fullsized":"2024-03-22T02:12:28.487376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487208+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:12:28.487208+0000","last_peered":"2024-03-22T02:12:28.487208+0000","last_clean":"2024-03-22T02:12:28.487208+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:12:28.487208+0000","last_undegraded":"2024-03-22T02:12:28.487208+0000","last_fullsized":"2024-03-22T02:12:28.487208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486922+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:12:28.486922+0000","last_peered":"2024-03-22T02:12:28.486922+0000","last_clean":"2024-03-22T02:12:28.486922+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:12:28.486922+0000","last_undegraded":"2024-03-22T02:12:28.486922+0000","last_fullsized":"2024-03-22T02:12:28.486922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483723+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:12:28.483723+0000","last_peered":"2024-03-22T02:12:28.483723+0000","last_clean":"2024-03-22T02:12:28.483723+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:12:28.483723+0000","last_undegraded":"2024-03-22T02:12:28.483723+0000","last_fullsized":"2024-03-22T02:12:28.483723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521805+0000","last_change":"2024-03-22T02:12:28.521805+0000","last_active":"2024-03-22T02:12:28.521805+0000","last_peered":"2024-03-22T02:12:28.521805+0000","last_clean":"2024-03-22T02:12:28.521805+0000","last_became_active":"2024-03-22T02:12:28.521419+0000","last_became_peered":"2024-03-22T02:12:28.521419+0000","last_unstale":"2024-03-22T02:12:28.521805+0000","last_undegraded":"2024-03-22T02:12:28.521805+0000","last_fullsized":"2024-03-22T02:12:28.521805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:01:11.831486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520104+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:12:28.520104+0000","last_peered":"2024-03-22T02:12:28.520104+0000","last_clean":"2024-03-22T02:12:28.520104+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:12:28.520104+0000","last_undegraded":"2024-03-22T02:12:28.520104+0000","last_fullsized":"2024-03-22T02:12:28.520104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519299+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:12:28.519299+0000","last_peered":"2024-03-22T02:12:28.519299+0000","last_clean":"2024-03-22T02:12:28.519299+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:12:28.519299+0000","last_undegraded":"2024-03-22T02:12:28.519299+0000","last_fullsized":"2024-03-22T02:12:28.519299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519867+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:12:28.519867+0000","last_peered":"2024-03-22T02:12:28.519867+0000","last_clean":"2024-03-22T02:12:28.519867+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:12:28.519867+0000","last_undegraded":"2024-03-22T02:12:28.519867+0000","last_fullsized":"2024-03-22T02:12:28.519867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486394+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:12:28.486394+0000","last_peered":"2024-03-22T02:12:28.486394+0000","last_clean":"2024-03-22T02:12:28.486394+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:12:28.486394+0000","last_undegraded":"2024-03-22T02:12:28.486394+0000","last_fullsized":"2024-03-22T02:12:28.486394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520862+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:12:28.520862+0000","last_peered":"2024-03-22T02:12:28.520862+0000","last_clean":"2024-03-22T02:12:28.520862+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:12:28.520862+0000","last_undegraded":"2024-03-22T02:12:28.520862+0000","last_fullsized":"2024-03-22T02:12:28.520862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486836+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:12:28.486836+0000","last_peered":"2024-03-22T02:12:28.486836+0000","last_clean":"2024-03-22T02:12:28.486836+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:12:28.486836+0000","last_undegraded":"2024-03-22T02:12:28.486836+0000","last_fullsized":"2024-03-22T02:12:28.486836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485196+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:12:28.485196+0000","last_peered":"2024-03-22T02:12:28.485196+0000","last_clean":"2024-03-22T02:12:28.485196+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:12:28.485196+0000","last_undegraded":"2024-03-22T02:12:28.485196+0000","last_fullsized":"2024-03-22T02:12:28.485196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486452+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:12:28.486452+0000","last_peered":"2024-03-22T02:12:28.486452+0000","last_clean":"2024-03-22T02:12:28.486452+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:12:28.486452+0000","last_undegraded":"2024-03-22T02:12:28.486452+0000","last_fullsized":"2024-03-22T02:12:28.486452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486546+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:12:28.486546+0000","last_peered":"2024-03-22T02:12:28.486546+0000","last_clean":"2024-03-22T02:12:28.486546+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:12:28.486546+0000","last_undegraded":"2024-03-22T02:12:28.486546+0000","last_fullsized":"2024-03-22T02:12:28.486546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519119+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:12:28.519119+0000","last_peered":"2024-03-22T02:12:28.519119+0000","last_clean":"2024-03-22T02:12:28.519119+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:12:28.519119+0000","last_undegraded":"2024-03-22T02:12:28.519119+0000","last_fullsized":"2024-03-22T02:12:28.519119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485878+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:12:28.485878+0000","last_peered":"2024-03-22T02:12:28.485878+0000","last_clean":"2024-03-22T02:12:28.485878+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:12:28.485878+0000","last_undegraded":"2024-03-22T02:12:28.485878+0000","last_fullsized":"2024-03-22T02:12:28.485878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483626+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:12:28.483626+0000","last_peered":"2024-03-22T02:12:28.483626+0000","last_clean":"2024-03-22T02:12:28.483626+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:12:28.483626+0000","last_undegraded":"2024-03-22T02:12:28.483626+0000","last_fullsized":"2024-03-22T02:12:28.483626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487486+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:12:28.487486+0000","last_peered":"2024-03-22T02:12:28.487486+0000","last_clean":"2024-03-22T02:12:28.487486+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:12:28.487486+0000","last_undegraded":"2024-03-22T02:12:28.487486+0000","last_fullsized":"2024-03-22T02:12:28.487486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486669+0000","last_change":"2024-03-22T02:11:37.944045+0000","last_active":"2024-03-22T02:12:28.486669+0000","last_peered":"2024-03-22T02:12:28.486669+0000","last_clean":"2024-03-22T02:12:28.486669+0000","last_became_active":"2024-03-22T02:11:37.943598+0000","last_became_peered":"2024-03-22T02:11:37.943598+0000","last_unstale":"2024-03-22T02:12:28.486669+0000","last_undegraded":"2024-03-22T02:12:28.486669+0000","last_fullsized":"2024-03-22T02:12:28.486669+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:46:43.350701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483450+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:12:28.483450+0000","last_peered":"2024-03-22T02:12:28.483450+0000","last_clean":"2024-03-22T02:12:28.483450+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:12:28.483450+0000","last_undegraded":"2024-03-22T02:12:28.483450+0000","last_fullsized":"2024-03-22T02:12:28.483450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520357+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:12:28.520357+0000","last_peered":"2024-03-22T02:12:28.520357+0000","last_clean":"2024-03-22T02:12:28.520357+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:12:28.520357+0000","last_undegraded":"2024-03-22T02:12:28.520357+0000","last_fullsized":"2024-03-22T02:12:28.520357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486420+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:12:28.486420+0000","last_peered":"2024-03-22T02:12:28.486420+0000","last_clean":"2024-03-22T02:12:28.486420+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:12:28.486420+0000","last_undegraded":"2024-03-22T02:12:28.486420+0000","last_fullsized":"2024-03-22T02:12:28.486420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520229+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:12:28.520229+0000","last_peered":"2024-03-22T02:12:28.520229+0000","last_clean":"2024-03-22T02:12:28.520229+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:12:28.520229+0000","last_undegraded":"2024-03-22T02:12:28.520229+0000","last_fullsized":"2024-03-22T02:12:28.520229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.488163+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:12:28.488163+0000","last_peered":"2024-03-22T02:12:28.488163+0000","last_clean":"2024-03-22T02:12:28.488163+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:12:28.488163+0000","last_undegraded":"2024-03-22T02:12:28.488163+0000","last_fullsized":"2024-03-22T02:12:28.488163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486733+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:12:28.486733+0000","last_peered":"2024-03-22T02:12:28.486733+0000","last_clean":"2024-03-22T02:12:28.486733+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:12:28.486733+0000","last_undegraded":"2024-03-22T02:12:28.486733+0000","last_fullsized":"2024-03-22T02:12:28.486733+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484149+0000","last_change":"2024-03-22T02:11:32.917851+0000","last_active":"2024-03-22T02:12:28.484149+0000","last_peered":"2024-03-22T02:12:28.484149+0000","last_clean":"2024-03-22T02:12:28.484149+0000","last_became_active":"2024-03-22T02:11:32.917526+0000","last_became_peered":"2024-03-22T02:11:32.917526+0000","last_unstale":"2024-03-22T02:12:28.484149+0000","last_undegraded":"2024-03-22T02:12:28.484149+0000","last_fullsized":"2024-03-22T02:12:28.484149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:20.509077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519946+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:12:28.519946+0000","last_peered":"2024-03-22T02:12:28.519946+0000","last_clean":"2024-03-22T02:12:28.519946+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:12:28.519946+0000","last_undegraded":"2024-03-22T02:12:28.519946+0000","last_fullsized":"2024-03-22T02:12:28.519946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486485+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:12:28.486485+0000","last_peered":"2024-03-22T02:12:28.486485+0000","last_clean":"2024-03-22T02:12:28.486485+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:12:28.486485+0000","last_undegraded":"2024-03-22T02:12:28.486485+0000","last_fullsized":"2024-03-22T02:12:28.486485+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483864+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:12:28.483864+0000","last_peered":"2024-03-22T02:12:28.483864+0000","last_clean":"2024-03-22T02:12:28.483864+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:12:28.483864+0000","last_undegraded":"2024-03-22T02:12:28.483864+0000","last_fullsized":"2024-03-22T02:12:28.483864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519696+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:12:28.519696+0000","last_peered":"2024-03-22T02:12:28.519696+0000","last_clean":"2024-03-22T02:12:28.519696+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:12:28.519696+0000","last_undegraded":"2024-03-22T02:12:28.519696+0000","last_fullsized":"2024-03-22T02:12:28.519696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487376+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:12:28.487376+0000","last_peered":"2024-03-22T02:12:28.487376+0000","last_clean":"2024-03-22T02:12:28.487376+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:12:28.487376+0000","last_undegraded":"2024-03-22T02:12:28.487376+0000","last_fullsized":"2024-03-22T02:12:28.487376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:01.062572+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:13:01.062572+0000","last_peered":"2024-03-22T02:13:01.062572+0000","last_clean":"2024-03-22T02:13:01.062572+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:13:01.062572+0000","last_undegraded":"2024-03-22T02:13:01.062572+0000","last_fullsized":"2024-03-22T02:13:01.062572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483742+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:12:28.483742+0000","last_peered":"2024-03-22T02:12:28.483742+0000","last_clean":"2024-03-22T02:12:28.483742+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:12:28.483742+0000","last_undegraded":"2024-03-22T02:12:28.483742+0000","last_fullsized":"2024-03-22T02:12:28.483742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487101+0000","last_change":"2024-03-22T02:11:37.931900+0000","last_active":"2024-03-22T02:12:28.487101+0000","last_peered":"2024-03-22T02:12:28.487101+0000","last_clean":"2024-03-22T02:12:28.487101+0000","last_became_active":"2024-03-22T02:11:37.931400+0000","last_became_peered":"2024-03-22T02:11:37.931400+0000","last_unstale":"2024-03-22T02:12:28.487101+0000","last_undegraded":"2024-03-22T02:12:28.487101+0000","last_fullsized":"2024-03-22T02:12:28.487101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:55.837790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485811+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:12:28.485811+0000","last_peered":"2024-03-22T02:12:28.485811+0000","last_clean":"2024-03-22T02:12:28.485811+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:12:28.485811+0000","last_undegraded":"2024-03-22T02:12:28.485811+0000","last_fullsized":"2024-03-22T02:12:28.485811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519737+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:12:28.519737+0000","last_peered":"2024-03-22T02:12:28.519737+0000","last_clean":"2024-03-22T02:12:28.519737+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:12:28.519737+0000","last_undegraded":"2024-03-22T02:12:28.519737+0000","last_fullsized":"2024-03-22T02:12:28.519737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485810+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:12:28.485810+0000","last_peered":"2024-03-22T02:12:28.485810+0000","last_clean":"2024-03-22T02:12:28.485810+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:12:28.485810+0000","last_undegraded":"2024-03-22T02:12:28.485810+0000","last_fullsized":"2024-03-22T02:12:28.485810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484818+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:12:28.484818+0000","last_peered":"2024-03-22T02:12:28.484818+0000","last_clean":"2024-03-22T02:12:28.484818+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:12:28.484818+0000","last_undegraded":"2024-03-22T02:12:28.484818+0000","last_fullsized":"2024-03-22T02:12:28.484818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519572+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:12:28.519572+0000","last_peered":"2024-03-22T02:12:28.519572+0000","last_clean":"2024-03-22T02:12:28.519572+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:12:28.519572+0000","last_undegraded":"2024-03-22T02:12:28.519572+0000","last_fullsized":"2024-03-22T02:12:28.519572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487028+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:12:28.487028+0000","last_peered":"2024-03-22T02:12:28.487028+0000","last_clean":"2024-03-22T02:12:28.487028+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:12:28.487028+0000","last_undegraded":"2024-03-22T02:12:28.487028+0000","last_fullsized":"2024-03-22T02:12:28.487028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519193+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:12:28.519193+0000","last_peered":"2024-03-22T02:12:28.519193+0000","last_clean":"2024-03-22T02:12:28.519193+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:12:28.519193+0000","last_undegraded":"2024-03-22T02:12:28.519193+0000","last_fullsized":"2024-03-22T02:12:28.519193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483313+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:12:28.483313+0000","last_peered":"2024-03-22T02:12:28.483313+0000","last_clean":"2024-03-22T02:12:28.483313+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:12:28.483313+0000","last_undegraded":"2024-03-22T02:12:28.483313+0000","last_fullsized":"2024-03-22T02:12:28.483313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519226+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:12:28.519226+0000","last_peered":"2024-03-22T02:12:28.519226+0000","last_clean":"2024-03-22T02:12:28.519226+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:12:28.519226+0000","last_undegraded":"2024-03-22T02:12:28.519226+0000","last_fullsized":"2024-03-22T02:12:28.519226+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:04.065156+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:13:04.065156+0000","last_peered":"2024-03-22T02:13:04.065156+0000","last_clean":"2024-03-22T02:13:04.065156+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:13:04.065156+0000","last_undegraded":"2024-03-22T02:13:04.065156+0000","last_fullsized":"2024-03-22T02:13:04.065156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519537+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:12:28.519537+0000","last_peered":"2024-03-22T02:12:28.519537+0000","last_clean":"2024-03-22T02:12:28.519537+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:12:28.519537+0000","last_undegraded":"2024-03-22T02:12:28.519537+0000","last_fullsized":"2024-03-22T02:12:28.519537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.482882+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:12:28.482882+0000","last_peered":"2024-03-22T02:12:28.482882+0000","last_clean":"2024-03-22T02:12:28.482882+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:12:28.482882+0000","last_undegraded":"2024-03-22T02:12:28.482882+0000","last_fullsized":"2024-03-22T02:12:28.482882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520833+0000","last_change":"2024-03-22T02:11:32.910432+0000","last_active":"2024-03-22T02:12:28.520833+0000","last_peered":"2024-03-22T02:12:28.520833+0000","last_clean":"2024-03-22T02:12:28.520833+0000","last_became_active":"2024-03-22T02:11:32.910114+0000","last_became_peered":"2024-03-22T02:11:32.910114+0000","last_unstale":"2024-03-22T02:12:28.520833+0000","last_undegraded":"2024-03-22T02:12:28.520833+0000","last_fullsized":"2024-03-22T02:12:28.520833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:54:21.399399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521865+0000","last_change":"2024-03-22T02:12:28.521865+0000","last_active":"2024-03-22T02:12:28.521865+0000","last_peered":"2024-03-22T02:12:28.521865+0000","last_clean":"2024-03-22T02:12:28.521865+0000","last_became_active":"2024-03-22T02:12:28.521413+0000","last_became_peered":"2024-03-22T02:12:28.521413+0000","last_unstale":"2024-03-22T02:12:28.521865+0000","last_undegraded":"2024-03-22T02:12:28.521865+0000","last_fullsized":"2024-03-22T02:12:28.521865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:54:26.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487016+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:12:28.487016+0000","last_peered":"2024-03-22T02:12:28.487016+0000","last_clean":"2024-03-22T02:12:28.487016+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:12:28.487016+0000","last_undegraded":"2024-03-22T02:12:28.487016+0000","last_fullsized":"2024-03-22T02:12:28.487016+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519641+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:12:28.519641+0000","last_peered":"2024-03-22T02:12:28.519641+0000","last_clean":"2024-03-22T02:12:28.519641+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:12:28.519641+0000","last_undegraded":"2024-03-22T02:12:28.519641+0000","last_fullsized":"2024-03-22T02:12:28.519641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520013+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:12:28.520013+0000","last_peered":"2024-03-22T02:12:28.520013+0000","last_clean":"2024-03-22T02:12:28.520013+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:12:28.520013+0000","last_undegraded":"2024-03-22T02:12:28.520013+0000","last_fullsized":"2024-03-22T02:12:28.520013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483121+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:12:28.483121+0000","last_peered":"2024-03-22T02:12:28.483121+0000","last_clean":"2024-03-22T02:12:28.483121+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:12:28.483121+0000","last_undegraded":"2024-03-22T02:12:28.483121+0000","last_fullsized":"2024-03-22T02:12:28.483121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'32","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:05.065993+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:13:05.065993+0000","last_peered":"2024-03-22T02:13:05.065993+0000","last_clean":"2024-03-22T02:13:05.065993+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:13:05.065993+0000","last_undegraded":"2024-03-22T02:13:05.065993+0000","last_fullsized":"2024-03-22T02:13:05.065993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":137,"num_write":33,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483968+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:12:28.483968+0000","last_peered":"2024-03-22T02:12:28.483968+0000","last_clean":"2024-03-22T02:12:28.483968+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:12:28.483968+0000","last_undegraded":"2024-03-22T02:12:28.483968+0000","last_fullsized":"2024-03-22T02:12:28.483968+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519881+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:12:28.519881+0000","last_peered":"2024-03-22T02:12:28.519881+0000","last_clean":"2024-03-22T02:12:28.519881+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:12:28.519881+0000","last_undegraded":"2024-03-22T02:12:28.519881+0000","last_fullsized":"2024-03-22T02:12:28.519881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486937+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:12:28.486937+0000","last_peered":"2024-03-22T02:12:28.486937+0000","last_clean":"2024-03-22T02:12:28.486937+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:12:28.486937+0000","last_undegraded":"2024-03-22T02:12:28.486937+0000","last_fullsized":"2024-03-22T02:12:28.486937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485196+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:12:28.485196+0000","last_peered":"2024-03-22T02:12:28.485196+0000","last_clean":"2024-03-22T02:12:28.485196+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:12:28.485196+0000","last_undegraded":"2024-03-22T02:12:28.485196+0000","last_fullsized":"2024-03-22T02:12:28.485196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520817+0000","last_change":"2024-03-22T02:12:28.520817+0000","last_active":"2024-03-22T02:12:28.520817+0000","last_peered":"2024-03-22T02:12:28.520817+0000","last_clean":"2024-03-22T02:12:28.520817+0000","last_became_active":"2024-03-22T02:12:28.520360+0000","last_became_peered":"2024-03-22T02:12:28.520360+0000","last_unstale":"2024-03-22T02:12:28.520817+0000","last_undegraded":"2024-03-22T02:12:28.520817+0000","last_fullsized":"2024-03-22T02:12:28.520817+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:34:21.654588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519453+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:12:28.519453+0000","last_peered":"2024-03-22T02:12:28.519453+0000","last_clean":"2024-03-22T02:12:28.519453+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:12:28.519453+0000","last_undegraded":"2024-03-22T02:12:28.519453+0000","last_fullsized":"2024-03-22T02:12:28.519453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520690+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:12:28.520690+0000","last_peered":"2024-03-22T02:12:28.520690+0000","last_clean":"2024-03-22T02:12:28.520690+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:12:28.520690+0000","last_undegraded":"2024-03-22T02:12:28.520690+0000","last_fullsized":"2024-03-22T02:12:28.520690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483340+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:12:28.483340+0000","last_peered":"2024-03-22T02:12:28.483340+0000","last_clean":"2024-03-22T02:12:28.483340+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:12:28.483340+0000","last_undegraded":"2024-03-22T02:12:28.483340+0000","last_fullsized":"2024-03-22T02:12:28.483340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:02.088362+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:13:02.088362+0000","last_peered":"2024-03-22T02:13:02.088362+0000","last_clean":"2024-03-22T02:13:02.088362+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:13:02.088362+0000","last_undegraded":"2024-03-22T02:13:02.088362+0000","last_fullsized":"2024-03-22T02:13:02.088362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519872+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:12:28.519872+0000","last_peered":"2024-03-22T02:12:28.519872+0000","last_clean":"2024-03-22T02:12:28.519872+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:12:28.519872+0000","last_undegraded":"2024-03-22T02:12:28.519872+0000","last_fullsized":"2024-03-22T02:12:28.519872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485659+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:12:28.485659+0000","last_peered":"2024-03-22T02:12:28.485659+0000","last_clean":"2024-03-22T02:12:28.485659+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:12:28.485659+0000","last_undegraded":"2024-03-22T02:12:28.485659+0000","last_fullsized":"2024-03-22T02:12:28.485659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483467+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:12:28.483467+0000","last_peered":"2024-03-22T02:12:28.483467+0000","last_clean":"2024-03-22T02:12:28.483467+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:12:28.483467+0000","last_undegraded":"2024-03-22T02:12:28.483467+0000","last_fullsized":"2024-03-22T02:12:28.483467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519321+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:12:28.519321+0000","last_peered":"2024-03-22T02:12:28.519321+0000","last_clean":"2024-03-22T02:12:28.519321+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:12:28.519321+0000","last_undegraded":"2024-03-22T02:12:28.519321+0000","last_fullsized":"2024-03-22T02:12:28.519321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519235+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:12:28.519235+0000","last_peered":"2024-03-22T02:12:28.519235+0000","last_clean":"2024-03-22T02:12:28.519235+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:12:28.519235+0000","last_undegraded":"2024-03-22T02:12:28.519235+0000","last_fullsized":"2024-03-22T02:12:28.519235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483970+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:12:28.483970+0000","last_peered":"2024-03-22T02:12:28.483970+0000","last_clean":"2024-03-22T02:12:28.483970+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:12:28.483970+0000","last_undegraded":"2024-03-22T02:12:28.483970+0000","last_fullsized":"2024-03-22T02:12:28.483970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485742+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:12:28.485742+0000","last_peered":"2024-03-22T02:12:28.485742+0000","last_clean":"2024-03-22T02:12:28.485742+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:12:28.485742+0000","last_undegraded":"2024-03-22T02:12:28.485742+0000","last_fullsized":"2024-03-22T02:12:28.485742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483736+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:12:28.483736+0000","last_peered":"2024-03-22T02:12:28.483736+0000","last_clean":"2024-03-22T02:12:28.483736+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:12:28.483736+0000","last_undegraded":"2024-03-22T02:12:28.483736+0000","last_fullsized":"2024-03-22T02:12:28.483736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483260+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:12:28.483260+0000","last_peered":"2024-03-22T02:12:28.483260+0000","last_clean":"2024-03-22T02:12:28.483260+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:12:28.483260+0000","last_undegraded":"2024-03-22T02:12:28.483260+0000","last_fullsized":"2024-03-22T02:12:28.483260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486041+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:12:28.486041+0000","last_peered":"2024-03-22T02:12:28.486041+0000","last_clean":"2024-03-22T02:12:28.486041+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:12:28.486041+0000","last_undegraded":"2024-03-22T02:12:28.486041+0000","last_fullsized":"2024-03-22T02:12:28.486041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485500+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:12:28.485500+0000","last_peered":"2024-03-22T02:12:28.485500+0000","last_clean":"2024-03-22T02:12:28.485500+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:12:28.485500+0000","last_undegraded":"2024-03-22T02:12:28.485500+0000","last_fullsized":"2024-03-22T02:12:28.485500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520593+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:12:28.520593+0000","last_peered":"2024-03-22T02:12:28.520593+0000","last_clean":"2024-03-22T02:12:28.520593+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:12:28.520593+0000","last_undegraded":"2024-03-22T02:12:28.520593+0000","last_fullsized":"2024-03-22T02:12:28.520593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.488917+0000","last_change":"2024-03-22T02:12:28.488917+0000","last_active":"2024-03-22T02:12:28.488917+0000","last_peered":"2024-03-22T02:12:28.488917+0000","last_clean":"2024-03-22T02:12:28.488917+0000","last_became_active":"2024-03-22T02:12:28.488542+0000","last_became_peered":"2024-03-22T02:12:28.488542+0000","last_unstale":"2024-03-22T02:12:28.488917+0000","last_undegraded":"2024-03-22T02:12:28.488917+0000","last_fullsized":"2024-03-22T02:12:28.488917+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:41.575475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485769+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:12:28.485769+0000","last_peered":"2024-03-22T02:12:28.485769+0000","last_clean":"2024-03-22T02:12:28.485769+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:12:28.485769+0000","last_undegraded":"2024-03-22T02:12:28.485769+0000","last_fullsized":"2024-03-22T02:12:28.485769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519346+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:12:28.519346+0000","last_peered":"2024-03-22T02:12:28.519346+0000","last_clean":"2024-03-22T02:12:28.519346+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:12:28.519346+0000","last_undegraded":"2024-03-22T02:12:28.519346+0000","last_fullsized":"2024-03-22T02:12:28.519346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:01.087022+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:13:01.087022+0000","last_peered":"2024-03-22T02:13:01.087022+0000","last_clean":"2024-03-22T02:13:01.087022+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:13:01.087022+0000","last_undegraded":"2024-03-22T02:13:01.087022+0000","last_fullsized":"2024-03-22T02:13:01.087022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520576+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:12:28.520576+0000","last_peered":"2024-03-22T02:12:28.520576+0000","last_clean":"2024-03-22T02:12:28.520576+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:12:28.520576+0000","last_undegraded":"2024-03-22T02:12:28.520576+0000","last_fullsized":"2024-03-22T02:12:28.520576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519374+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:12:28.519374+0000","last_peered":"2024-03-22T02:12:28.519374+0000","last_clean":"2024-03-22T02:12:28.519374+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:12:28.519374+0000","last_undegraded":"2024-03-22T02:12:28.519374+0000","last_fullsized":"2024-03-22T02:12:28.519374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486894+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:12:28.486894+0000","last_peered":"2024-03-22T02:12:28.486894+0000","last_clean":"2024-03-22T02:12:28.486894+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:12:28.486894+0000","last_undegraded":"2024-03-22T02:12:28.486894+0000","last_fullsized":"2024-03-22T02:12:28.486894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519819+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:12:28.519819+0000","last_peered":"2024-03-22T02:12:28.519819+0000","last_clean":"2024-03-22T02:12:28.519819+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:12:28.519819+0000","last_undegraded":"2024-03-22T02:12:28.519819+0000","last_fullsized":"2024-03-22T02:12:28.519819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519771+0000","last_change":"2024-03-22T02:11:37.942403+0000","last_active":"2024-03-22T02:12:28.519771+0000","last_peered":"2024-03-22T02:12:28.519771+0000","last_clean":"2024-03-22T02:12:28.519771+0000","last_became_active":"2024-03-22T02:11:37.942040+0000","last_became_peered":"2024-03-22T02:11:37.942040+0000","last_unstale":"2024-03-22T02:12:28.519771+0000","last_undegraded":"2024-03-22T02:12:28.519771+0000","last_fullsized":"2024-03-22T02:12:28.519771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:12:02.337725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484686+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:12:28.484686+0000","last_peered":"2024-03-22T02:12:28.484686+0000","last_clean":"2024-03-22T02:12:28.484686+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:12:28.484686+0000","last_undegraded":"2024-03-22T02:12:28.484686+0000","last_fullsized":"2024-03-22T02:12:28.484686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519799+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:12:28.519799+0000","last_peered":"2024-03-22T02:12:28.519799+0000","last_clean":"2024-03-22T02:12:28.519799+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:12:28.519799+0000","last_undegraded":"2024-03-22T02:12:28.519799+0000","last_fullsized":"2024-03-22T02:12:28.519799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485396+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:12:28.485396+0000","last_peered":"2024-03-22T02:12:28.485396+0000","last_clean":"2024-03-22T02:12:28.485396+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:12:28.485396+0000","last_undegraded":"2024-03-22T02:12:28.485396+0000","last_fullsized":"2024-03-22T02:12:28.485396+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484086+0000","last_change":"2024-03-22T02:11:32.915749+0000","last_active":"2024-03-22T02:12:28.484086+0000","last_peered":"2024-03-22T02:12:28.484086+0000","last_clean":"2024-03-22T02:12:28.484086+0000","last_became_active":"2024-03-22T02:11:32.915455+0000","last_became_peered":"2024-03-22T02:11:32.915455+0000","last_unstale":"2024-03-22T02:12:28.484086+0000","last_undegraded":"2024-03-22T02:12:28.484086+0000","last_fullsized":"2024-03-22T02:12:28.484086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:49.835132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485640+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:12:28.485640+0000","last_peered":"2024-03-22T02:12:28.485640+0000","last_clean":"2024-03-22T02:12:28.485640+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:12:28.485640+0000","last_undegraded":"2024-03-22T02:12:28.485640+0000","last_fullsized":"2024-03-22T02:12:28.485640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519437+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:12:28.519437+0000","last_peered":"2024-03-22T02:12:28.519437+0000","last_clean":"2024-03-22T02:12:28.519437+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:12:28.519437+0000","last_undegraded":"2024-03-22T02:12:28.519437+0000","last_fullsized":"2024-03-22T02:12:28.519437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483633+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:12:28.483633+0000","last_peered":"2024-03-22T02:12:28.483633+0000","last_clean":"2024-03-22T02:12:28.483633+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:12:28.483633+0000","last_undegraded":"2024-03-22T02:12:28.483633+0000","last_fullsized":"2024-03-22T02:12:28.483633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520030+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:12:28.520030+0000","last_peered":"2024-03-22T02:12:28.520030+0000","last_clean":"2024-03-22T02:12:28.520030+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:12:28.520030+0000","last_undegraded":"2024-03-22T02:12:28.520030+0000","last_fullsized":"2024-03-22T02:12:28.520030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486542+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:12:28.486542+0000","last_peered":"2024-03-22T02:12:28.486542+0000","last_clean":"2024-03-22T02:12:28.486542+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:12:28.486542+0000","last_undegraded":"2024-03-22T02:12:28.486542+0000","last_fullsized":"2024-03-22T02:12:28.486542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486686+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:12:28.486686+0000","last_peered":"2024-03-22T02:12:28.486686+0000","last_clean":"2024-03-22T02:12:28.486686+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:12:28.486686+0000","last_undegraded":"2024-03-22T02:12:28.486686+0000","last_fullsized":"2024-03-22T02:12:28.486686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489608+0000","last_change":"2024-03-22T02:12:28.489608+0000","last_active":"2024-03-22T02:12:28.489608+0000","last_peered":"2024-03-22T02:12:28.489608+0000","last_clean":"2024-03-22T02:12:28.489608+0000","last_became_active":"2024-03-22T02:12:28.489005+0000","last_became_peered":"2024-03-22T02:12:28.489005+0000","last_unstale":"2024-03-22T02:12:28.489608+0000","last_undegraded":"2024-03-22T02:12:28.489608+0000","last_fullsized":"2024-03-22T02:12:28.489608+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:01.427654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486794+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:12:28.486794+0000","last_peered":"2024-03-22T02:12:28.486794+0000","last_clean":"2024-03-22T02:12:28.486794+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:12:28.486794+0000","last_undegraded":"2024-03-22T02:12:28.486794+0000","last_fullsized":"2024-03-22T02:12:28.486794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519127+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:12:28.519127+0000","last_peered":"2024-03-22T02:12:28.519127+0000","last_clean":"2024-03-22T02:12:28.519127+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:12:28.519127+0000","last_undegraded":"2024-03-22T02:12:28.519127+0000","last_fullsized":"2024-03-22T02:12:28.519127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483468+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:12:28.483468+0000","last_peered":"2024-03-22T02:12:28.483468+0000","last_clean":"2024-03-22T02:12:28.483468+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:12:28.483468+0000","last_undegraded":"2024-03-22T02:12:28.483468+0000","last_fullsized":"2024-03-22T02:12:28.483468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483425+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:12:28.483425+0000","last_peered":"2024-03-22T02:12:28.483425+0000","last_clean":"2024-03-22T02:12:28.483425+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:12:28.483425+0000","last_undegraded":"2024-03-22T02:12:28.483425+0000","last_fullsized":"2024-03-22T02:12:28.483425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519273+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:12:28.519273+0000","last_peered":"2024-03-22T02:12:28.519273+0000","last_clean":"2024-03-22T02:12:28.519273+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:12:28.519273+0000","last_undegraded":"2024-03-22T02:12:28.519273+0000","last_fullsized":"2024-03-22T02:12:28.519273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485294+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:12:28.485294+0000","last_peered":"2024-03-22T02:12:28.485294+0000","last_clean":"2024-03-22T02:12:28.485294+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:12:28.485294+0000","last_undegraded":"2024-03-22T02:12:28.485294+0000","last_fullsized":"2024-03-22T02:12:28.485294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486924+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:12:28.486924+0000","last_peered":"2024-03-22T02:12:28.486924+0000","last_clean":"2024-03-22T02:12:28.486924+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:12:28.486924+0000","last_undegraded":"2024-03-22T02:12:28.486924+0000","last_fullsized":"2024-03-22T02:12:28.486924+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486770+0000","last_change":"2024-03-22T02:11:32.907985+0000","last_active":"2024-03-22T02:12:28.486770+0000","last_peered":"2024-03-22T02:12:28.486770+0000","last_clean":"2024-03-22T02:12:28.486770+0000","last_became_active":"2024-03-22T02:11:32.907686+0000","last_became_peered":"2024-03-22T02:11:32.907686+0000","last_unstale":"2024-03-22T02:12:28.486770+0000","last_undegraded":"2024-03-22T02:12:28.486770+0000","last_fullsized":"2024-03-22T02:12:28.486770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:30:40.945298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519678+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:12:28.519678+0000","last_peered":"2024-03-22T02:12:28.519678+0000","last_clean":"2024-03-22T02:12:28.519678+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:12:28.519678+0000","last_undegraded":"2024-03-22T02:12:28.519678+0000","last_fullsized":"2024-03-22T02:12:28.519678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486680+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:12:28.486680+0000","last_peered":"2024-03-22T02:12:28.486680+0000","last_clean":"2024-03-22T02:12:28.486680+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:12:28.486680+0000","last_undegraded":"2024-03-22T02:12:28.486680+0000","last_fullsized":"2024-03-22T02:12:28.486680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483723+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:12:28.483723+0000","last_peered":"2024-03-22T02:12:28.483723+0000","last_clean":"2024-03-22T02:12:28.483723+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:12:28.483723+0000","last_undegraded":"2024-03-22T02:12:28.483723+0000","last_fullsized":"2024-03-22T02:12:28.483723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520389+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:12:28.520389+0000","last_peered":"2024-03-22T02:12:28.520389+0000","last_clean":"2024-03-22T02:12:28.520389+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:12:28.520389+0000","last_undegraded":"2024-03-22T02:12:28.520389+0000","last_fullsized":"2024-03-22T02:12:28.520389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519158+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:12:28.519158+0000","last_peered":"2024-03-22T02:12:28.519158+0000","last_clean":"2024-03-22T02:12:28.519158+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:12:28.519158+0000","last_undegraded":"2024-03-22T02:12:28.519158+0000","last_fullsized":"2024-03-22T02:12:28.519158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487234+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:12:28.487234+0000","last_peered":"2024-03-22T02:12:28.487234+0000","last_clean":"2024-03-22T02:12:28.487234+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:12:28.487234+0000","last_undegraded":"2024-03-22T02:12:28.487234+0000","last_fullsized":"2024-03-22T02:12:28.487234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520867+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:12:28.520867+0000","last_peered":"2024-03-22T02:12:28.520867+0000","last_clean":"2024-03-22T02:12:28.520867+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:12:28.520867+0000","last_undegraded":"2024-03-22T02:12:28.520867+0000","last_fullsized":"2024-03-22T02:12:28.520867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486960+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:12:28.486960+0000","last_peered":"2024-03-22T02:12:28.486960+0000","last_clean":"2024-03-22T02:12:28.486960+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:12:28.486960+0000","last_undegraded":"2024-03-22T02:12:28.486960+0000","last_fullsized":"2024-03-22T02:12:28.486960+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519757+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:12:28.519757+0000","last_peered":"2024-03-22T02:12:28.519757+0000","last_clean":"2024-03-22T02:12:28.519757+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:12:28.519757+0000","last_undegraded":"2024-03-22T02:12:28.519757+0000","last_fullsized":"2024-03-22T02:12:28.519757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520592+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:12:28.520592+0000","last_peered":"2024-03-22T02:12:28.520592+0000","last_clean":"2024-03-22T02:12:28.520592+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:12:28.520592+0000","last_undegraded":"2024-03-22T02:12:28.520592+0000","last_fullsized":"2024-03-22T02:12:28.520592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519062+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:12:28.519062+0000","last_peered":"2024-03-22T02:12:28.519062+0000","last_clean":"2024-03-22T02:12:28.519062+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:12:28.519062+0000","last_undegraded":"2024-03-22T02:12:28.519062+0000","last_fullsized":"2024-03-22T02:12:28.519062+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485857+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:12:28.485857+0000","last_peered":"2024-03-22T02:12:28.485857+0000","last_clean":"2024-03-22T02:12:28.485857+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:12:28.485857+0000","last_undegraded":"2024-03-22T02:12:28.485857+0000","last_fullsized":"2024-03-22T02:12:28.485857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521004+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:12:28.521004+0000","last_peered":"2024-03-22T02:12:28.521004+0000","last_clean":"2024-03-22T02:12:28.521004+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:12:28.521004+0000","last_undegraded":"2024-03-22T02:12:28.521004+0000","last_fullsized":"2024-03-22T02:12:28.521004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485350+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:12:28.485350+0000","last_peered":"2024-03-22T02:12:28.485350+0000","last_clean":"2024-03-22T02:12:28.485350+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:12:28.485350+0000","last_undegraded":"2024-03-22T02:12:28.485350+0000","last_fullsized":"2024-03-22T02:12:28.485350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486835+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:12:28.486835+0000","last_peered":"2024-03-22T02:12:28.486835+0000","last_clean":"2024-03-22T02:12:28.486835+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:12:28.486835+0000","last_undegraded":"2024-03-22T02:12:28.486835+0000","last_fullsized":"2024-03-22T02:12:28.486835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489359+0000","last_change":"2024-03-22T02:12:28.489359+0000","last_active":"2024-03-22T02:12:28.489359+0000","last_peered":"2024-03-22T02:12:28.489359+0000","last_clean":"2024-03-22T02:12:28.489359+0000","last_became_active":"2024-03-22T02:12:28.489038+0000","last_became_peered":"2024-03-22T02:12:28.489038+0000","last_unstale":"2024-03-22T02:12:28.489359+0000","last_undegraded":"2024-03-22T02:12:28.489359+0000","last_fullsized":"2024-03-22T02:12:28.489359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:15.886990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489164+0000","last_change":"2024-03-22T02:12:28.489164+0000","last_active":"2024-03-22T02:12:28.489164+0000","last_peered":"2024-03-22T02:12:28.489164+0000","last_clean":"2024-03-22T02:12:28.489164+0000","last_became_active":"2024-03-22T02:12:28.488707+0000","last_became_peered":"2024-03-22T02:12:28.488707+0000","last_unstale":"2024-03-22T02:12:28.489164+0000","last_undegraded":"2024-03-22T02:12:28.489164+0000","last_fullsized":"2024-03-22T02:12:28.489164+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:40.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485638+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:12:28.485638+0000","last_peered":"2024-03-22T02:12:28.485638+0000","last_clean":"2024-03-22T02:12:28.485638+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:12:28.485638+0000","last_undegraded":"2024-03-22T02:12:28.485638+0000","last_fullsized":"2024-03-22T02:12:28.485638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485679+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:12:28.485679+0000","last_peered":"2024-03-22T02:12:28.485679+0000","last_clean":"2024-03-22T02:12:28.485679+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:12:28.485679+0000","last_undegraded":"2024-03-22T02:12:28.485679+0000","last_fullsized":"2024-03-22T02:12:28.485679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487564+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:12:28.487564+0000","last_peered":"2024-03-22T02:12:28.487564+0000","last_clean":"2024-03-22T02:12:28.487564+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:12:28.487564+0000","last_undegraded":"2024-03-22T02:12:28.487564+0000","last_fullsized":"2024-03-22T02:12:28.487564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520126+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:12:28.520126+0000","last_peered":"2024-03-22T02:12:28.520126+0000","last_clean":"2024-03-22T02:12:28.520126+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:12:28.520126+0000","last_undegraded":"2024-03-22T02:12:28.520126+0000","last_fullsized":"2024-03-22T02:12:28.520126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521387+0000","last_change":"2024-03-22T02:12:28.520917+0000","last_active":"2024-03-22T02:12:28.521387+0000","last_peered":"2024-03-22T02:12:28.521387+0000","last_clean":"2024-03-22T02:12:28.521387+0000","last_became_active":"2024-03-22T02:12:28.520533+0000","last_became_peered":"2024-03-22T02:12:28.520533+0000","last_unstale":"2024-03-22T02:12:28.521387+0000","last_undegraded":"2024-03-22T02:12:28.521387+0000","last_fullsized":"2024-03-22T02:12:28.521387+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:44.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519555+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:12:28.519555+0000","last_peered":"2024-03-22T02:12:28.519555+0000","last_clean":"2024-03-22T02:12:28.519555+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:12:28.519555+0000","last_undegraded":"2024-03-22T02:12:28.519555+0000","last_fullsized":"2024-03-22T02:12:28.519555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521533+0000","last_change":"2024-03-22T02:12:28.521533+0000","last_active":"2024-03-22T02:12:28.521533+0000","last_peered":"2024-03-22T02:12:28.521533+0000","last_clean":"2024-03-22T02:12:28.521533+0000","last_became_active":"2024-03-22T02:12:28.521176+0000","last_became_peered":"2024-03-22T02:12:28.521176+0000","last_unstale":"2024-03-22T02:12:28.521533+0000","last_undegraded":"2024-03-22T02:12:28.521533+0000","last_fullsized":"2024-03-22T02:12:28.521533+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:10:11.909377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485296+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:12:28.485296+0000","last_peered":"2024-03-22T02:12:28.485296+0000","last_clean":"2024-03-22T02:12:28.485296+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:12:28.485296+0000","last_undegraded":"2024-03-22T02:12:28.485296+0000","last_fullsized":"2024-03-22T02:12:28.485296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485901+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:12:28.485901+0000","last_peered":"2024-03-22T02:12:28.485901+0000","last_clean":"2024-03-22T02:12:28.485901+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:12:28.485901+0000","last_undegraded":"2024-03-22T02:12:28.485901+0000","last_fullsized":"2024-03-22T02:12:28.485901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":445,"num_read_kb":295,"num_write":160,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351007,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":503803,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1146880,"data_stored":1154427,"data_compressed":58377,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":508135,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":503959,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1099868,"data_compressed":56750,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462969,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":499215,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":562481,"data_compressed":22285,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":500603,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":564222,"data_compressed":22285,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":499409,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":503865,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1095456,"data_compressed":56750,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65097,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64934,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:13:07.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T02:13:08.178 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:08.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:08 smithi028 ceph-mon[31916]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:08.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:08 smithi028 ceph-mon[31916]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:08.816 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:08 smithi083 ceph-mon[36340]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:08.816 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:08 smithi083 ceph-mon[36340]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:08 smithi120 ceph-mon[36803]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:08 smithi120 ceph-mon[36803]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:09.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:09 smithi028 ceph-mon[31916]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:09 smithi120 ceph-mon[36803]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 -- 172.21.15.28:0/1860527406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364107590 msgr2=0x7f4364107970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 --2- 172.21.15.28:0/1860527406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364107590 0x7f4364107970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4354009990 tx=0x7f4354009ca0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 -- 172.21.15.28:0/1860527406 shutdown_connections 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 --2- 172.21.15.28:0/1860527406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364107590 0x7f4364107970 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 --2- 172.21.15.28:0/1860527406 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364101f40 0x7f436410e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 --2- 172.21.15.28:0/1860527406 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 0x7f4364101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 -- 172.21.15.28:0/1860527406 >> 172.21.15.28:0/1860527406 conn(0x7f43640784b0 msgr2=0x7f43640788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 -- 172.21.15.28:0/1860527406 shutdown_connections 2024-03-22T02:13:10.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.000+0000 7f43693d8700 1 -- 172.21.15.28:0/1860527406 wait complete. 2024-03-22T02:13:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 Processor -- start 2024-03-22T02:13:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 -- start start 2024-03-22T02:13:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 0x7f4364199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:10.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364107590 0x7f43641a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4364111890 con 0x7f4364101f40 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4364111710 con 0x7f4364107590 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43693d8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4364111a10 con 0x7f4364101580 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f435bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f4362ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 0x7f4364199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f435bfff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50274/0 (socket says 172.21.15.28:50274) 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f435bfff700 1 -- 172.21.15.28:0/3333128179 learned_addr learned my addr 172.21.15.28:0/3333128179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.001+0000 7f43637fe700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364107590 0x7f43641a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 -- 172.21.15.28:0/3333128179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 msgr2=0x7f4364199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 0x7f4364199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 -- 172.21.15.28:0/3333128179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364107590 msgr2=0x7f43641a4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364107590 0x7f43641a4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 -- 172.21.15.28:0/3333128179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43641a51d0 con 0x7f4364101f40 2024-03-22T02:13:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f435bfff700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f434c00e9c0 tx=0x7f434c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c00cbd0 con 0x7f4364101f40 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f434c00ee60 con 0x7f4364101f40 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c005630 con 0x7f4364101f40 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43641a53a0 con 0x7f4364101f40 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.002+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43641a5890 con 0x7f4364101f40 2024-03-22T02:13:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.003+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4364069340 con 0x7f4364101f40 2024-03-22T02:13:10.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.004+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f434c004750 con 0x7f4364101f40 2024-03-22T02:13:10.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.005+0000 7f4360ff9700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 0x7f43500a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:10.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.005+0000 7f4362ffd700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 0x7f43500a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:10.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.005+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f434c0cfda0 con 0x7f4364101f40 2024-03-22T02:13:10.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.005+0000 7f4362ffd700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 0x7f43500a5500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f435400b2e0 tx=0x7f4354005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:10.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.008+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f434c053740 con 0x7f4364101f40 2024-03-22T02:13:10.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:09 smithi083 ceph-mon[36340]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:10.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.205+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4364062ac0 con 0x7f43500a3040 2024-03-22T02:13:10.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.215+0000 7f4360ff9700 1 -- 172.21.15.28:0/3333128179 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340656 (secure 0 0 0) 0x7f4364062ac0 con 0x7f43500a3040 2024-03-22T02:13:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:10.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 msgr2=0x7f43500a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 0x7f43500a5500 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f435400b2e0 tx=0x7f4354005960 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 msgr2=0x7f436419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f434c00e9c0 tx=0x7f434c00ecd0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 shutdown_connections 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4364107590 0x7f43641a4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4364101580 0x7f4364199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43500a3040 0x7f43500a5500 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 --2- 172.21.15.28:0/3333128179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4364101f40 0x7f436419a150 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 >> 172.21.15.28:0/3333128179 conn(0x7f43640784b0 msgr2=0x7f4364100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 shutdown_connections 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:10.219+0000 7f43693d8700 1 -- 172.21.15.28:0/3333128179 wait complete. 2024-03-22T02:13:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:13:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:10 smithi120 ceph-mon[36803]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:11.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:10 smithi083 ceph-mon[36340]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:11.027 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:10 smithi028 ceph-mon[31916]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:11.055 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-22T02:13:08.575978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657959,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":586,"num_read_kb":409,"num_write":343,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":278,"ondisk_log_size":278,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365648,"kb_used_data":12248,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576176,"statfs":{"total":1151940427776,"available":1151566004224,"internally_reserved":0,"allocated":12541952,"data_stored":7995443,"data_compressed":361053,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485912+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:12:28.485912+0000","last_peered":"2024-03-22T02:12:28.485912+0000","last_clean":"2024-03-22T02:12:28.485912+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:12:28.485912+0000","last_undegraded":"2024-03-22T02:12:28.485912+0000","last_fullsized":"2024-03-22T02:12:28.485912+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487376+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:12:28.487376+0000","last_peered":"2024-03-22T02:12:28.487376+0000","last_clean":"2024-03-22T02:12:28.487376+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:12:28.487376+0000","last_undegraded":"2024-03-22T02:12:28.487376+0000","last_fullsized":"2024-03-22T02:12:28.487376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487208+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:12:28.487208+0000","last_peered":"2024-03-22T02:12:28.487208+0000","last_clean":"2024-03-22T02:12:28.487208+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:12:28.487208+0000","last_undegraded":"2024-03-22T02:12:28.487208+0000","last_fullsized":"2024-03-22T02:12:28.487208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486922+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:12:28.486922+0000","last_peered":"2024-03-22T02:12:28.486922+0000","last_clean":"2024-03-22T02:12:28.486922+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:12:28.486922+0000","last_undegraded":"2024-03-22T02:12:28.486922+0000","last_fullsized":"2024-03-22T02:12:28.486922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483723+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:12:28.483723+0000","last_peered":"2024-03-22T02:12:28.483723+0000","last_clean":"2024-03-22T02:12:28.483723+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:12:28.483723+0000","last_undegraded":"2024-03-22T02:12:28.483723+0000","last_fullsized":"2024-03-22T02:12:28.483723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521805+0000","last_change":"2024-03-22T02:12:28.521805+0000","last_active":"2024-03-22T02:12:28.521805+0000","last_peered":"2024-03-22T02:12:28.521805+0000","last_clean":"2024-03-22T02:12:28.521805+0000","last_became_active":"2024-03-22T02:12:28.521419+0000","last_became_peered":"2024-03-22T02:12:28.521419+0000","last_unstale":"2024-03-22T02:12:28.521805+0000","last_undegraded":"2024-03-22T02:12:28.521805+0000","last_fullsized":"2024-03-22T02:12:28.521805+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:01:11.831486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520104+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:12:28.520104+0000","last_peered":"2024-03-22T02:12:28.520104+0000","last_clean":"2024-03-22T02:12:28.520104+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:12:28.520104+0000","last_undegraded":"2024-03-22T02:12:28.520104+0000","last_fullsized":"2024-03-22T02:12:28.520104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519299+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:12:28.519299+0000","last_peered":"2024-03-22T02:12:28.519299+0000","last_clean":"2024-03-22T02:12:28.519299+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:12:28.519299+0000","last_undegraded":"2024-03-22T02:12:28.519299+0000","last_fullsized":"2024-03-22T02:12:28.519299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519867+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:12:28.519867+0000","last_peered":"2024-03-22T02:12:28.519867+0000","last_clean":"2024-03-22T02:12:28.519867+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:12:28.519867+0000","last_undegraded":"2024-03-22T02:12:28.519867+0000","last_fullsized":"2024-03-22T02:12:28.519867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486394+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:12:28.486394+0000","last_peered":"2024-03-22T02:12:28.486394+0000","last_clean":"2024-03-22T02:12:28.486394+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:12:28.486394+0000","last_undegraded":"2024-03-22T02:12:28.486394+0000","last_fullsized":"2024-03-22T02:12:28.486394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520862+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:12:28.520862+0000","last_peered":"2024-03-22T02:12:28.520862+0000","last_clean":"2024-03-22T02:12:28.520862+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:12:28.520862+0000","last_undegraded":"2024-03-22T02:12:28.520862+0000","last_fullsized":"2024-03-22T02:12:28.520862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486836+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:12:28.486836+0000","last_peered":"2024-03-22T02:12:28.486836+0000","last_clean":"2024-03-22T02:12:28.486836+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:12:28.486836+0000","last_undegraded":"2024-03-22T02:12:28.486836+0000","last_fullsized":"2024-03-22T02:12:28.486836+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485196+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:12:28.485196+0000","last_peered":"2024-03-22T02:12:28.485196+0000","last_clean":"2024-03-22T02:12:28.485196+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:12:28.485196+0000","last_undegraded":"2024-03-22T02:12:28.485196+0000","last_fullsized":"2024-03-22T02:12:28.485196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486452+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:12:28.486452+0000","last_peered":"2024-03-22T02:12:28.486452+0000","last_clean":"2024-03-22T02:12:28.486452+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:12:28.486452+0000","last_undegraded":"2024-03-22T02:12:28.486452+0000","last_fullsized":"2024-03-22T02:12:28.486452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486546+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:12:28.486546+0000","last_peered":"2024-03-22T02:12:28.486546+0000","last_clean":"2024-03-22T02:12:28.486546+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:12:28.486546+0000","last_undegraded":"2024-03-22T02:12:28.486546+0000","last_fullsized":"2024-03-22T02:12:28.486546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519119+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:12:28.519119+0000","last_peered":"2024-03-22T02:12:28.519119+0000","last_clean":"2024-03-22T02:12:28.519119+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:12:28.519119+0000","last_undegraded":"2024-03-22T02:12:28.519119+0000","last_fullsized":"2024-03-22T02:12:28.519119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485878+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:12:28.485878+0000","last_peered":"2024-03-22T02:12:28.485878+0000","last_clean":"2024-03-22T02:12:28.485878+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:12:28.485878+0000","last_undegraded":"2024-03-22T02:12:28.485878+0000","last_fullsized":"2024-03-22T02:12:28.485878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483626+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:12:28.483626+0000","last_peered":"2024-03-22T02:12:28.483626+0000","last_clean":"2024-03-22T02:12:28.483626+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:12:28.483626+0000","last_undegraded":"2024-03-22T02:12:28.483626+0000","last_fullsized":"2024-03-22T02:12:28.483626+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487486+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:12:28.487486+0000","last_peered":"2024-03-22T02:12:28.487486+0000","last_clean":"2024-03-22T02:12:28.487486+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:12:28.487486+0000","last_undegraded":"2024-03-22T02:12:28.487486+0000","last_fullsized":"2024-03-22T02:12:28.487486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486669+0000","last_change":"2024-03-22T02:11:37.944045+0000","last_active":"2024-03-22T02:12:28.486669+0000","last_peered":"2024-03-22T02:12:28.486669+0000","last_clean":"2024-03-22T02:12:28.486669+0000","last_became_active":"2024-03-22T02:11:37.943598+0000","last_became_peered":"2024-03-22T02:11:37.943598+0000","last_unstale":"2024-03-22T02:12:28.486669+0000","last_undegraded":"2024-03-22T02:12:28.486669+0000","last_fullsized":"2024-03-22T02:12:28.486669+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:46:43.350701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":37,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483450+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:12:28.483450+0000","last_peered":"2024-03-22T02:12:28.483450+0000","last_clean":"2024-03-22T02:12:28.483450+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:12:28.483450+0000","last_undegraded":"2024-03-22T02:12:28.483450+0000","last_fullsized":"2024-03-22T02:12:28.483450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520357+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:12:28.520357+0000","last_peered":"2024-03-22T02:12:28.520357+0000","last_clean":"2024-03-22T02:12:28.520357+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:12:28.520357+0000","last_undegraded":"2024-03-22T02:12:28.520357+0000","last_fullsized":"2024-03-22T02:12:28.520357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486420+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:12:28.486420+0000","last_peered":"2024-03-22T02:12:28.486420+0000","last_clean":"2024-03-22T02:12:28.486420+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:12:28.486420+0000","last_undegraded":"2024-03-22T02:12:28.486420+0000","last_fullsized":"2024-03-22T02:12:28.486420+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520229+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:12:28.520229+0000","last_peered":"2024-03-22T02:12:28.520229+0000","last_clean":"2024-03-22T02:12:28.520229+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:12:28.520229+0000","last_undegraded":"2024-03-22T02:12:28.520229+0000","last_fullsized":"2024-03-22T02:12:28.520229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.488163+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:12:28.488163+0000","last_peered":"2024-03-22T02:12:28.488163+0000","last_clean":"2024-03-22T02:12:28.488163+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:12:28.488163+0000","last_undegraded":"2024-03-22T02:12:28.488163+0000","last_fullsized":"2024-03-22T02:12:28.488163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486733+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:12:28.486733+0000","last_peered":"2024-03-22T02:12:28.486733+0000","last_clean":"2024-03-22T02:12:28.486733+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:12:28.486733+0000","last_undegraded":"2024-03-22T02:12:28.486733+0000","last_fullsized":"2024-03-22T02:12:28.486733+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484149+0000","last_change":"2024-03-22T02:11:32.917851+0000","last_active":"2024-03-22T02:12:28.484149+0000","last_peered":"2024-03-22T02:12:28.484149+0000","last_clean":"2024-03-22T02:12:28.484149+0000","last_became_active":"2024-03-22T02:11:32.917526+0000","last_became_peered":"2024-03-22T02:11:32.917526+0000","last_unstale":"2024-03-22T02:12:28.484149+0000","last_undegraded":"2024-03-22T02:12:28.484149+0000","last_fullsized":"2024-03-22T02:12:28.484149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:20.509077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519946+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:12:28.519946+0000","last_peered":"2024-03-22T02:12:28.519946+0000","last_clean":"2024-03-22T02:12:28.519946+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:12:28.519946+0000","last_undegraded":"2024-03-22T02:12:28.519946+0000","last_fullsized":"2024-03-22T02:12:28.519946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486485+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:12:28.486485+0000","last_peered":"2024-03-22T02:12:28.486485+0000","last_clean":"2024-03-22T02:12:28.486485+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:12:28.486485+0000","last_undegraded":"2024-03-22T02:12:28.486485+0000","last_fullsized":"2024-03-22T02:12:28.486485+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483864+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:12:28.483864+0000","last_peered":"2024-03-22T02:12:28.483864+0000","last_clean":"2024-03-22T02:12:28.483864+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:12:28.483864+0000","last_undegraded":"2024-03-22T02:12:28.483864+0000","last_fullsized":"2024-03-22T02:12:28.483864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519696+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:12:28.519696+0000","last_peered":"2024-03-22T02:12:28.519696+0000","last_clean":"2024-03-22T02:12:28.519696+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:12:28.519696+0000","last_undegraded":"2024-03-22T02:12:28.519696+0000","last_fullsized":"2024-03-22T02:12:28.519696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487376+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:12:28.487376+0000","last_peered":"2024-03-22T02:12:28.487376+0000","last_clean":"2024-03-22T02:12:28.487376+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:12:28.487376+0000","last_undegraded":"2024-03-22T02:12:28.487376+0000","last_fullsized":"2024-03-22T02:12:28.487376+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'6","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:06.067434+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:13:06.067434+0000","last_peered":"2024-03-22T02:13:06.067434+0000","last_clean":"2024-03-22T02:13:06.067434+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:13:06.067434+0000","last_undegraded":"2024-03-22T02:13:06.067434+0000","last_fullsized":"2024-03-22T02:13:06.067434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483742+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:12:28.483742+0000","last_peered":"2024-03-22T02:12:28.483742+0000","last_clean":"2024-03-22T02:12:28.483742+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:12:28.483742+0000","last_undegraded":"2024-03-22T02:12:28.483742+0000","last_fullsized":"2024-03-22T02:12:28.483742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487101+0000","last_change":"2024-03-22T02:11:37.931900+0000","last_active":"2024-03-22T02:12:28.487101+0000","last_peered":"2024-03-22T02:12:28.487101+0000","last_clean":"2024-03-22T02:12:28.487101+0000","last_became_active":"2024-03-22T02:11:37.931400+0000","last_became_peered":"2024-03-22T02:11:37.931400+0000","last_unstale":"2024-03-22T02:12:28.487101+0000","last_undegraded":"2024-03-22T02:12:28.487101+0000","last_fullsized":"2024-03-22T02:12:28.487101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:55.837790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485811+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:12:28.485811+0000","last_peered":"2024-03-22T02:12:28.485811+0000","last_clean":"2024-03-22T02:12:28.485811+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:12:28.485811+0000","last_undegraded":"2024-03-22T02:12:28.485811+0000","last_fullsized":"2024-03-22T02:12:28.485811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519737+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:12:28.519737+0000","last_peered":"2024-03-22T02:12:28.519737+0000","last_clean":"2024-03-22T02:12:28.519737+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:12:28.519737+0000","last_undegraded":"2024-03-22T02:12:28.519737+0000","last_fullsized":"2024-03-22T02:12:28.519737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485810+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:12:28.485810+0000","last_peered":"2024-03-22T02:12:28.485810+0000","last_clean":"2024-03-22T02:12:28.485810+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:12:28.485810+0000","last_undegraded":"2024-03-22T02:12:28.485810+0000","last_fullsized":"2024-03-22T02:12:28.485810+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484818+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:12:28.484818+0000","last_peered":"2024-03-22T02:12:28.484818+0000","last_clean":"2024-03-22T02:12:28.484818+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:12:28.484818+0000","last_undegraded":"2024-03-22T02:12:28.484818+0000","last_fullsized":"2024-03-22T02:12:28.484818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519572+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:12:28.519572+0000","last_peered":"2024-03-22T02:12:28.519572+0000","last_clean":"2024-03-22T02:12:28.519572+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:12:28.519572+0000","last_undegraded":"2024-03-22T02:12:28.519572+0000","last_fullsized":"2024-03-22T02:12:28.519572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487028+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:12:28.487028+0000","last_peered":"2024-03-22T02:12:28.487028+0000","last_clean":"2024-03-22T02:12:28.487028+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:12:28.487028+0000","last_undegraded":"2024-03-22T02:12:28.487028+0000","last_fullsized":"2024-03-22T02:12:28.487028+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519193+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:12:28.519193+0000","last_peered":"2024-03-22T02:12:28.519193+0000","last_clean":"2024-03-22T02:12:28.519193+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:12:28.519193+0000","last_undegraded":"2024-03-22T02:12:28.519193+0000","last_fullsized":"2024-03-22T02:12:28.519193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483313+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:12:28.483313+0000","last_peered":"2024-03-22T02:12:28.483313+0000","last_clean":"2024-03-22T02:12:28.483313+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:12:28.483313+0000","last_undegraded":"2024-03-22T02:12:28.483313+0000","last_fullsized":"2024-03-22T02:12:28.483313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519226+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:12:28.519226+0000","last_peered":"2024-03-22T02:12:28.519226+0000","last_clean":"2024-03-22T02:12:28.519226+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:12:28.519226+0000","last_undegraded":"2024-03-22T02:12:28.519226+0000","last_fullsized":"2024-03-22T02:12:28.519226+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'3","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:04.065156+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:13:04.065156+0000","last_peered":"2024-03-22T02:13:04.065156+0000","last_clean":"2024-03-22T02:13:04.065156+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:13:04.065156+0000","last_undegraded":"2024-03-22T02:13:04.065156+0000","last_fullsized":"2024-03-22T02:13:04.065156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519537+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:12:28.519537+0000","last_peered":"2024-03-22T02:12:28.519537+0000","last_clean":"2024-03-22T02:12:28.519537+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:12:28.519537+0000","last_undegraded":"2024-03-22T02:12:28.519537+0000","last_fullsized":"2024-03-22T02:12:28.519537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.482882+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:12:28.482882+0000","last_peered":"2024-03-22T02:12:28.482882+0000","last_clean":"2024-03-22T02:12:28.482882+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:12:28.482882+0000","last_undegraded":"2024-03-22T02:12:28.482882+0000","last_fullsized":"2024-03-22T02:12:28.482882+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520833+0000","last_change":"2024-03-22T02:11:32.910432+0000","last_active":"2024-03-22T02:12:28.520833+0000","last_peered":"2024-03-22T02:12:28.520833+0000","last_clean":"2024-03-22T02:12:28.520833+0000","last_became_active":"2024-03-22T02:11:32.910114+0000","last_became_peered":"2024-03-22T02:11:32.910114+0000","last_unstale":"2024-03-22T02:12:28.520833+0000","last_undegraded":"2024-03-22T02:12:28.520833+0000","last_fullsized":"2024-03-22T02:12:28.520833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:54:21.399399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521865+0000","last_change":"2024-03-22T02:12:28.521865+0000","last_active":"2024-03-22T02:12:28.521865+0000","last_peered":"2024-03-22T02:12:28.521865+0000","last_clean":"2024-03-22T02:12:28.521865+0000","last_became_active":"2024-03-22T02:12:28.521413+0000","last_became_peered":"2024-03-22T02:12:28.521413+0000","last_unstale":"2024-03-22T02:12:28.521865+0000","last_undegraded":"2024-03-22T02:12:28.521865+0000","last_fullsized":"2024-03-22T02:12:28.521865+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:54:26.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487016+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:12:28.487016+0000","last_peered":"2024-03-22T02:12:28.487016+0000","last_clean":"2024-03-22T02:12:28.487016+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:12:28.487016+0000","last_undegraded":"2024-03-22T02:12:28.487016+0000","last_fullsized":"2024-03-22T02:12:28.487016+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519641+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:12:28.519641+0000","last_peered":"2024-03-22T02:12:28.519641+0000","last_clean":"2024-03-22T02:12:28.519641+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:12:28.519641+0000","last_undegraded":"2024-03-22T02:12:28.519641+0000","last_fullsized":"2024-03-22T02:12:28.519641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520013+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:12:28.520013+0000","last_peered":"2024-03-22T02:12:28.520013+0000","last_clean":"2024-03-22T02:12:28.520013+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:12:28.520013+0000","last_undegraded":"2024-03-22T02:12:28.520013+0000","last_fullsized":"2024-03-22T02:12:28.520013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483121+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:12:28.483121+0000","last_peered":"2024-03-22T02:12:28.483121+0000","last_clean":"2024-03-22T02:12:28.483121+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:12:28.483121+0000","last_undegraded":"2024-03-22T02:12:28.483121+0000","last_fullsized":"2024-03-22T02:12:28.483121+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'32","reported_seq":209,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:05.065993+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:13:05.065993+0000","last_peered":"2024-03-22T02:13:05.065993+0000","last_clean":"2024-03-22T02:13:05.065993+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:13:05.065993+0000","last_undegraded":"2024-03-22T02:13:05.065993+0000","last_fullsized":"2024-03-22T02:13:05.065993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":150,"num_read_kb":137,"num_write":33,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483968+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:12:28.483968+0000","last_peered":"2024-03-22T02:12:28.483968+0000","last_clean":"2024-03-22T02:12:28.483968+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:12:28.483968+0000","last_undegraded":"2024-03-22T02:12:28.483968+0000","last_fullsized":"2024-03-22T02:12:28.483968+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519881+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:12:28.519881+0000","last_peered":"2024-03-22T02:12:28.519881+0000","last_clean":"2024-03-22T02:12:28.519881+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:12:28.519881+0000","last_undegraded":"2024-03-22T02:12:28.519881+0000","last_fullsized":"2024-03-22T02:12:28.519881+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486937+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:12:28.486937+0000","last_peered":"2024-03-22T02:12:28.486937+0000","last_clean":"2024-03-22T02:12:28.486937+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:12:28.486937+0000","last_undegraded":"2024-03-22T02:12:28.486937+0000","last_fullsized":"2024-03-22T02:12:28.486937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485196+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:12:28.485196+0000","last_peered":"2024-03-22T02:12:28.485196+0000","last_clean":"2024-03-22T02:12:28.485196+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:12:28.485196+0000","last_undegraded":"2024-03-22T02:12:28.485196+0000","last_fullsized":"2024-03-22T02:12:28.485196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520817+0000","last_change":"2024-03-22T02:12:28.520817+0000","last_active":"2024-03-22T02:12:28.520817+0000","last_peered":"2024-03-22T02:12:28.520817+0000","last_clean":"2024-03-22T02:12:28.520817+0000","last_became_active":"2024-03-22T02:12:28.520360+0000","last_became_peered":"2024-03-22T02:12:28.520360+0000","last_unstale":"2024-03-22T02:12:28.520817+0000","last_undegraded":"2024-03-22T02:12:28.520817+0000","last_fullsized":"2024-03-22T02:12:28.520817+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:34:21.654588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519453+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:12:28.519453+0000","last_peered":"2024-03-22T02:12:28.519453+0000","last_clean":"2024-03-22T02:12:28.519453+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:12:28.519453+0000","last_undegraded":"2024-03-22T02:12:28.519453+0000","last_fullsized":"2024-03-22T02:12:28.519453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520690+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:12:28.520690+0000","last_peered":"2024-03-22T02:12:28.520690+0000","last_clean":"2024-03-22T02:12:28.520690+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:12:28.520690+0000","last_undegraded":"2024-03-22T02:12:28.520690+0000","last_fullsized":"2024-03-22T02:12:28.520690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483340+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:12:28.483340+0000","last_peered":"2024-03-22T02:12:28.483340+0000","last_clean":"2024-03-22T02:12:28.483340+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:12:28.483340+0000","last_undegraded":"2024-03-22T02:12:28.483340+0000","last_fullsized":"2024-03-22T02:12:28.483340+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":106,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:07.093564+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:13:07.093564+0000","last_peered":"2024-03-22T02:13:07.093564+0000","last_clean":"2024-03-22T02:13:07.093564+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:13:07.093564+0000","last_undegraded":"2024-03-22T02:13:07.093564+0000","last_fullsized":"2024-03-22T02:13:07.093564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519872+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:12:28.519872+0000","last_peered":"2024-03-22T02:12:28.519872+0000","last_clean":"2024-03-22T02:12:28.519872+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:12:28.519872+0000","last_undegraded":"2024-03-22T02:12:28.519872+0000","last_fullsized":"2024-03-22T02:12:28.519872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485659+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:12:28.485659+0000","last_peered":"2024-03-22T02:12:28.485659+0000","last_clean":"2024-03-22T02:12:28.485659+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:12:28.485659+0000","last_undegraded":"2024-03-22T02:12:28.485659+0000","last_fullsized":"2024-03-22T02:12:28.485659+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483467+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:12:28.483467+0000","last_peered":"2024-03-22T02:12:28.483467+0000","last_clean":"2024-03-22T02:12:28.483467+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:12:28.483467+0000","last_undegraded":"2024-03-22T02:12:28.483467+0000","last_fullsized":"2024-03-22T02:12:28.483467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519321+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:12:28.519321+0000","last_peered":"2024-03-22T02:12:28.519321+0000","last_clean":"2024-03-22T02:12:28.519321+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:12:28.519321+0000","last_undegraded":"2024-03-22T02:12:28.519321+0000","last_fullsized":"2024-03-22T02:12:28.519321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519235+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:12:28.519235+0000","last_peered":"2024-03-22T02:12:28.519235+0000","last_clean":"2024-03-22T02:12:28.519235+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:12:28.519235+0000","last_undegraded":"2024-03-22T02:12:28.519235+0000","last_fullsized":"2024-03-22T02:12:28.519235+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483970+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:12:28.483970+0000","last_peered":"2024-03-22T02:12:28.483970+0000","last_clean":"2024-03-22T02:12:28.483970+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:12:28.483970+0000","last_undegraded":"2024-03-22T02:12:28.483970+0000","last_fullsized":"2024-03-22T02:12:28.483970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485742+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:12:28.485742+0000","last_peered":"2024-03-22T02:12:28.485742+0000","last_clean":"2024-03-22T02:12:28.485742+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:12:28.485742+0000","last_undegraded":"2024-03-22T02:12:28.485742+0000","last_fullsized":"2024-03-22T02:12:28.485742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483736+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:12:28.483736+0000","last_peered":"2024-03-22T02:12:28.483736+0000","last_clean":"2024-03-22T02:12:28.483736+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:12:28.483736+0000","last_undegraded":"2024-03-22T02:12:28.483736+0000","last_fullsized":"2024-03-22T02:12:28.483736+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483260+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:12:28.483260+0000","last_peered":"2024-03-22T02:12:28.483260+0000","last_clean":"2024-03-22T02:12:28.483260+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:12:28.483260+0000","last_undegraded":"2024-03-22T02:12:28.483260+0000","last_fullsized":"2024-03-22T02:12:28.483260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486041+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:12:28.486041+0000","last_peered":"2024-03-22T02:12:28.486041+0000","last_clean":"2024-03-22T02:12:28.486041+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:12:28.486041+0000","last_undegraded":"2024-03-22T02:12:28.486041+0000","last_fullsized":"2024-03-22T02:12:28.486041+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485500+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:12:28.485500+0000","last_peered":"2024-03-22T02:12:28.485500+0000","last_clean":"2024-03-22T02:12:28.485500+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:12:28.485500+0000","last_undegraded":"2024-03-22T02:12:28.485500+0000","last_fullsized":"2024-03-22T02:12:28.485500+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520593+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:12:28.520593+0000","last_peered":"2024-03-22T02:12:28.520593+0000","last_clean":"2024-03-22T02:12:28.520593+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:12:28.520593+0000","last_undegraded":"2024-03-22T02:12:28.520593+0000","last_fullsized":"2024-03-22T02:12:28.520593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.488917+0000","last_change":"2024-03-22T02:12:28.488917+0000","last_active":"2024-03-22T02:12:28.488917+0000","last_peered":"2024-03-22T02:12:28.488917+0000","last_clean":"2024-03-22T02:12:28.488917+0000","last_became_active":"2024-03-22T02:12:28.488542+0000","last_became_peered":"2024-03-22T02:12:28.488542+0000","last_unstale":"2024-03-22T02:12:28.488917+0000","last_undegraded":"2024-03-22T02:12:28.488917+0000","last_fullsized":"2024-03-22T02:12:28.488917+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:41.575475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485769+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:12:28.485769+0000","last_peered":"2024-03-22T02:12:28.485769+0000","last_clean":"2024-03-22T02:12:28.485769+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:12:28.485769+0000","last_undegraded":"2024-03-22T02:12:28.485769+0000","last_fullsized":"2024-03-22T02:12:28.485769+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519346+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:12:28.519346+0000","last_peered":"2024-03-22T02:12:28.519346+0000","last_clean":"2024-03-22T02:12:28.519346+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:12:28.519346+0000","last_undegraded":"2024-03-22T02:12:28.519346+0000","last_fullsized":"2024-03-22T02:12:28.519346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:13:06.091952+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:13:06.091952+0000","last_peered":"2024-03-22T02:13:06.091952+0000","last_clean":"2024-03-22T02:13:06.091952+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:13:06.091952+0000","last_undegraded":"2024-03-22T02:13:06.091952+0000","last_fullsized":"2024-03-22T02:13:06.091952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520576+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:12:28.520576+0000","last_peered":"2024-03-22T02:12:28.520576+0000","last_clean":"2024-03-22T02:12:28.520576+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:12:28.520576+0000","last_undegraded":"2024-03-22T02:12:28.520576+0000","last_fullsized":"2024-03-22T02:12:28.520576+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519374+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:12:28.519374+0000","last_peered":"2024-03-22T02:12:28.519374+0000","last_clean":"2024-03-22T02:12:28.519374+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:12:28.519374+0000","last_undegraded":"2024-03-22T02:12:28.519374+0000","last_fullsized":"2024-03-22T02:12:28.519374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486894+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:12:28.486894+0000","last_peered":"2024-03-22T02:12:28.486894+0000","last_clean":"2024-03-22T02:12:28.486894+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:12:28.486894+0000","last_undegraded":"2024-03-22T02:12:28.486894+0000","last_fullsized":"2024-03-22T02:12:28.486894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":350,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519819+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:12:28.519819+0000","last_peered":"2024-03-22T02:12:28.519819+0000","last_clean":"2024-03-22T02:12:28.519819+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:12:28.519819+0000","last_undegraded":"2024-03-22T02:12:28.519819+0000","last_fullsized":"2024-03-22T02:12:28.519819+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519771+0000","last_change":"2024-03-22T02:11:37.942403+0000","last_active":"2024-03-22T02:12:28.519771+0000","last_peered":"2024-03-22T02:12:28.519771+0000","last_clean":"2024-03-22T02:12:28.519771+0000","last_became_active":"2024-03-22T02:11:37.942040+0000","last_became_peered":"2024-03-22T02:11:37.942040+0000","last_unstale":"2024-03-22T02:12:28.519771+0000","last_undegraded":"2024-03-22T02:12:28.519771+0000","last_fullsized":"2024-03-22T02:12:28.519771+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:12:02.337725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484686+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:12:28.484686+0000","last_peered":"2024-03-22T02:12:28.484686+0000","last_clean":"2024-03-22T02:12:28.484686+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:12:28.484686+0000","last_undegraded":"2024-03-22T02:12:28.484686+0000","last_fullsized":"2024-03-22T02:12:28.484686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519799+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:12:28.519799+0000","last_peered":"2024-03-22T02:12:28.519799+0000","last_clean":"2024-03-22T02:12:28.519799+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:12:28.519799+0000","last_undegraded":"2024-03-22T02:12:28.519799+0000","last_fullsized":"2024-03-22T02:12:28.519799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485396+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:12:28.485396+0000","last_peered":"2024-03-22T02:12:28.485396+0000","last_clean":"2024-03-22T02:12:28.485396+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:12:28.485396+0000","last_undegraded":"2024-03-22T02:12:28.485396+0000","last_fullsized":"2024-03-22T02:12:28.485396+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.484086+0000","last_change":"2024-03-22T02:11:32.915749+0000","last_active":"2024-03-22T02:12:28.484086+0000","last_peered":"2024-03-22T02:12:28.484086+0000","last_clean":"2024-03-22T02:12:28.484086+0000","last_became_active":"2024-03-22T02:11:32.915455+0000","last_became_peered":"2024-03-22T02:11:32.915455+0000","last_unstale":"2024-03-22T02:12:28.484086+0000","last_undegraded":"2024-03-22T02:12:28.484086+0000","last_fullsized":"2024-03-22T02:12:28.484086+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:49.835132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485640+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:12:28.485640+0000","last_peered":"2024-03-22T02:12:28.485640+0000","last_clean":"2024-03-22T02:12:28.485640+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:12:28.485640+0000","last_undegraded":"2024-03-22T02:12:28.485640+0000","last_fullsized":"2024-03-22T02:12:28.485640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519437+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:12:28.519437+0000","last_peered":"2024-03-22T02:12:28.519437+0000","last_clean":"2024-03-22T02:12:28.519437+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:12:28.519437+0000","last_undegraded":"2024-03-22T02:12:28.519437+0000","last_fullsized":"2024-03-22T02:12:28.519437+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483633+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:12:28.483633+0000","last_peered":"2024-03-22T02:12:28.483633+0000","last_clean":"2024-03-22T02:12:28.483633+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:12:28.483633+0000","last_undegraded":"2024-03-22T02:12:28.483633+0000","last_fullsized":"2024-03-22T02:12:28.483633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520030+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:12:28.520030+0000","last_peered":"2024-03-22T02:12:28.520030+0000","last_clean":"2024-03-22T02:12:28.520030+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:12:28.520030+0000","last_undegraded":"2024-03-22T02:12:28.520030+0000","last_fullsized":"2024-03-22T02:12:28.520030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486542+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:12:28.486542+0000","last_peered":"2024-03-22T02:12:28.486542+0000","last_clean":"2024-03-22T02:12:28.486542+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:12:28.486542+0000","last_undegraded":"2024-03-22T02:12:28.486542+0000","last_fullsized":"2024-03-22T02:12:28.486542+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486686+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:12:28.486686+0000","last_peered":"2024-03-22T02:12:28.486686+0000","last_clean":"2024-03-22T02:12:28.486686+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:12:28.486686+0000","last_undegraded":"2024-03-22T02:12:28.486686+0000","last_fullsized":"2024-03-22T02:12:28.486686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489608+0000","last_change":"2024-03-22T02:12:28.489608+0000","last_active":"2024-03-22T02:12:28.489608+0000","last_peered":"2024-03-22T02:12:28.489608+0000","last_clean":"2024-03-22T02:12:28.489608+0000","last_became_active":"2024-03-22T02:12:28.489005+0000","last_became_peered":"2024-03-22T02:12:28.489005+0000","last_unstale":"2024-03-22T02:12:28.489608+0000","last_undegraded":"2024-03-22T02:12:28.489608+0000","last_fullsized":"2024-03-22T02:12:28.489608+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:01.427654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486794+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:12:28.486794+0000","last_peered":"2024-03-22T02:12:28.486794+0000","last_clean":"2024-03-22T02:12:28.486794+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:12:28.486794+0000","last_undegraded":"2024-03-22T02:12:28.486794+0000","last_fullsized":"2024-03-22T02:12:28.486794+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519127+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:12:28.519127+0000","last_peered":"2024-03-22T02:12:28.519127+0000","last_clean":"2024-03-22T02:12:28.519127+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:12:28.519127+0000","last_undegraded":"2024-03-22T02:12:28.519127+0000","last_fullsized":"2024-03-22T02:12:28.519127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483468+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:12:28.483468+0000","last_peered":"2024-03-22T02:12:28.483468+0000","last_clean":"2024-03-22T02:12:28.483468+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:12:28.483468+0000","last_undegraded":"2024-03-22T02:12:28.483468+0000","last_fullsized":"2024-03-22T02:12:28.483468+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483425+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:12:28.483425+0000","last_peered":"2024-03-22T02:12:28.483425+0000","last_clean":"2024-03-22T02:12:28.483425+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:12:28.483425+0000","last_undegraded":"2024-03-22T02:12:28.483425+0000","last_fullsized":"2024-03-22T02:12:28.483425+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519273+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:12:28.519273+0000","last_peered":"2024-03-22T02:12:28.519273+0000","last_clean":"2024-03-22T02:12:28.519273+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:12:28.519273+0000","last_undegraded":"2024-03-22T02:12:28.519273+0000","last_fullsized":"2024-03-22T02:12:28.519273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485294+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:12:28.485294+0000","last_peered":"2024-03-22T02:12:28.485294+0000","last_clean":"2024-03-22T02:12:28.485294+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:12:28.485294+0000","last_undegraded":"2024-03-22T02:12:28.485294+0000","last_fullsized":"2024-03-22T02:12:28.485294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486924+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:12:28.486924+0000","last_peered":"2024-03-22T02:12:28.486924+0000","last_clean":"2024-03-22T02:12:28.486924+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:12:28.486924+0000","last_undegraded":"2024-03-22T02:12:28.486924+0000","last_fullsized":"2024-03-22T02:12:28.486924+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486770+0000","last_change":"2024-03-22T02:11:32.907985+0000","last_active":"2024-03-22T02:12:28.486770+0000","last_peered":"2024-03-22T02:12:28.486770+0000","last_clean":"2024-03-22T02:12:28.486770+0000","last_became_active":"2024-03-22T02:11:32.907686+0000","last_became_peered":"2024-03-22T02:11:32.907686+0000","last_unstale":"2024-03-22T02:12:28.486770+0000","last_undegraded":"2024-03-22T02:12:28.486770+0000","last_fullsized":"2024-03-22T02:12:28.486770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:30:40.945298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519678+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:12:28.519678+0000","last_peered":"2024-03-22T02:12:28.519678+0000","last_clean":"2024-03-22T02:12:28.519678+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:12:28.519678+0000","last_undegraded":"2024-03-22T02:12:28.519678+0000","last_fullsized":"2024-03-22T02:12:28.519678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486680+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:12:28.486680+0000","last_peered":"2024-03-22T02:12:28.486680+0000","last_clean":"2024-03-22T02:12:28.486680+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:12:28.486680+0000","last_undegraded":"2024-03-22T02:12:28.486680+0000","last_fullsized":"2024-03-22T02:12:28.486680+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.483723+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:12:28.483723+0000","last_peered":"2024-03-22T02:12:28.483723+0000","last_clean":"2024-03-22T02:12:28.483723+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:12:28.483723+0000","last_undegraded":"2024-03-22T02:12:28.483723+0000","last_fullsized":"2024-03-22T02:12:28.483723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520389+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:12:28.520389+0000","last_peered":"2024-03-22T02:12:28.520389+0000","last_clean":"2024-03-22T02:12:28.520389+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:12:28.520389+0000","last_undegraded":"2024-03-22T02:12:28.520389+0000","last_fullsized":"2024-03-22T02:12:28.520389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519158+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:12:28.519158+0000","last_peered":"2024-03-22T02:12:28.519158+0000","last_clean":"2024-03-22T02:12:28.519158+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:12:28.519158+0000","last_undegraded":"2024-03-22T02:12:28.519158+0000","last_fullsized":"2024-03-22T02:12:28.519158+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487234+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:12:28.487234+0000","last_peered":"2024-03-22T02:12:28.487234+0000","last_clean":"2024-03-22T02:12:28.487234+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:12:28.487234+0000","last_undegraded":"2024-03-22T02:12:28.487234+0000","last_fullsized":"2024-03-22T02:12:28.487234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520867+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:12:28.520867+0000","last_peered":"2024-03-22T02:12:28.520867+0000","last_clean":"2024-03-22T02:12:28.520867+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:12:28.520867+0000","last_undegraded":"2024-03-22T02:12:28.520867+0000","last_fullsized":"2024-03-22T02:12:28.520867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486960+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:12:28.486960+0000","last_peered":"2024-03-22T02:12:28.486960+0000","last_clean":"2024-03-22T02:12:28.486960+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:12:28.486960+0000","last_undegraded":"2024-03-22T02:12:28.486960+0000","last_fullsized":"2024-03-22T02:12:28.486960+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519757+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:12:28.519757+0000","last_peered":"2024-03-22T02:12:28.519757+0000","last_clean":"2024-03-22T02:12:28.519757+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:12:28.519757+0000","last_undegraded":"2024-03-22T02:12:28.519757+0000","last_fullsized":"2024-03-22T02:12:28.519757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520592+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:12:28.520592+0000","last_peered":"2024-03-22T02:12:28.520592+0000","last_clean":"2024-03-22T02:12:28.520592+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:12:28.520592+0000","last_undegraded":"2024-03-22T02:12:28.520592+0000","last_fullsized":"2024-03-22T02:12:28.520592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519062+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:12:28.519062+0000","last_peered":"2024-03-22T02:12:28.519062+0000","last_clean":"2024-03-22T02:12:28.519062+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:12:28.519062+0000","last_undegraded":"2024-03-22T02:12:28.519062+0000","last_fullsized":"2024-03-22T02:12:28.519062+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485857+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:12:28.485857+0000","last_peered":"2024-03-22T02:12:28.485857+0000","last_clean":"2024-03-22T02:12:28.485857+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:12:28.485857+0000","last_undegraded":"2024-03-22T02:12:28.485857+0000","last_fullsized":"2024-03-22T02:12:28.485857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521004+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:12:28.521004+0000","last_peered":"2024-03-22T02:12:28.521004+0000","last_clean":"2024-03-22T02:12:28.521004+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:12:28.521004+0000","last_undegraded":"2024-03-22T02:12:28.521004+0000","last_fullsized":"2024-03-22T02:12:28.521004+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485350+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:12:28.485350+0000","last_peered":"2024-03-22T02:12:28.485350+0000","last_clean":"2024-03-22T02:12:28.485350+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:12:28.485350+0000","last_undegraded":"2024-03-22T02:12:28.485350+0000","last_fullsized":"2024-03-22T02:12:28.485350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.486835+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:12:28.486835+0000","last_peered":"2024-03-22T02:12:28.486835+0000","last_clean":"2024-03-22T02:12:28.486835+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:12:28.486835+0000","last_undegraded":"2024-03-22T02:12:28.486835+0000","last_fullsized":"2024-03-22T02:12:28.486835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489359+0000","last_change":"2024-03-22T02:12:28.489359+0000","last_active":"2024-03-22T02:12:28.489359+0000","last_peered":"2024-03-22T02:12:28.489359+0000","last_clean":"2024-03-22T02:12:28.489359+0000","last_became_active":"2024-03-22T02:12:28.489038+0000","last_became_peered":"2024-03-22T02:12:28.489038+0000","last_unstale":"2024-03-22T02:12:28.489359+0000","last_undegraded":"2024-03-22T02:12:28.489359+0000","last_fullsized":"2024-03-22T02:12:28.489359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:15.886990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.489164+0000","last_change":"2024-03-22T02:12:28.489164+0000","last_active":"2024-03-22T02:12:28.489164+0000","last_peered":"2024-03-22T02:12:28.489164+0000","last_clean":"2024-03-22T02:12:28.489164+0000","last_became_active":"2024-03-22T02:12:28.488707+0000","last_became_peered":"2024-03-22T02:12:28.488707+0000","last_unstale":"2024-03-22T02:12:28.489164+0000","last_undegraded":"2024-03-22T02:12:28.489164+0000","last_fullsized":"2024-03-22T02:12:28.489164+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:40.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485638+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:12:28.485638+0000","last_peered":"2024-03-22T02:12:28.485638+0000","last_clean":"2024-03-22T02:12:28.485638+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:12:28.485638+0000","last_undegraded":"2024-03-22T02:12:28.485638+0000","last_fullsized":"2024-03-22T02:12:28.485638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485679+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:12:28.485679+0000","last_peered":"2024-03-22T02:12:28.485679+0000","last_clean":"2024-03-22T02:12:28.485679+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:12:28.485679+0000","last_undegraded":"2024-03-22T02:12:28.485679+0000","last_fullsized":"2024-03-22T02:12:28.485679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.487564+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:12:28.487564+0000","last_peered":"2024-03-22T02:12:28.487564+0000","last_clean":"2024-03-22T02:12:28.487564+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:12:28.487564+0000","last_undegraded":"2024-03-22T02:12:28.487564+0000","last_fullsized":"2024-03-22T02:12:28.487564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.520126+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:12:28.520126+0000","last_peered":"2024-03-22T02:12:28.520126+0000","last_clean":"2024-03-22T02:12:28.520126+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:12:28.520126+0000","last_undegraded":"2024-03-22T02:12:28.520126+0000","last_fullsized":"2024-03-22T02:12:28.520126+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521387+0000","last_change":"2024-03-22T02:12:28.520917+0000","last_active":"2024-03-22T02:12:28.521387+0000","last_peered":"2024-03-22T02:12:28.521387+0000","last_clean":"2024-03-22T02:12:28.521387+0000","last_became_active":"2024-03-22T02:12:28.520533+0000","last_became_peered":"2024-03-22T02:12:28.520533+0000","last_unstale":"2024-03-22T02:12:28.521387+0000","last_undegraded":"2024-03-22T02:12:28.521387+0000","last_fullsized":"2024-03-22T02:12:28.521387+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:44.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.519555+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:12:28.519555+0000","last_peered":"2024-03-22T02:12:28.519555+0000","last_clean":"2024-03-22T02:12:28.519555+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:12:28.519555+0000","last_undegraded":"2024-03-22T02:12:28.519555+0000","last_fullsized":"2024-03-22T02:12:28.519555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.521533+0000","last_change":"2024-03-22T02:12:28.521533+0000","last_active":"2024-03-22T02:12:28.521533+0000","last_peered":"2024-03-22T02:12:28.521533+0000","last_clean":"2024-03-22T02:12:28.521533+0000","last_became_active":"2024-03-22T02:12:28.521176+0000","last_became_peered":"2024-03-22T02:12:28.521176+0000","last_unstale":"2024-03-22T02:12:28.521533+0000","last_undegraded":"2024-03-22T02:12:28.521533+0000","last_fullsized":"2024-03-22T02:12:28.521533+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:10:11.909377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485296+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:12:28.485296+0000","last_peered":"2024-03-22T02:12:28.485296+0000","last_clean":"2024-03-22T02:12:28.485296+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:12:28.485296+0000","last_undegraded":"2024-03-22T02:12:28.485296+0000","last_fullsized":"2024-03-22T02:12:28.485296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-22T02:12:28.485901+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:12:28.485901+0000","last_peered":"2024-03-22T02:12:28.485901+0000","last_clean":"2024-03-22T02:12:28.485901+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:12:28.485901+0000","last_undegraded":"2024-03-22T02:12:28.485901+0000","last_fullsized":"2024-03-22T02:12:28.485901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67357,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":460,"num_read_kb":300,"num_write":160,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202071,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1003520,"data_stored":503803,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712852,"statfs":{"total":95995035648,"available":95961960448,"internally_reserved":0,"allocated":1146880,"data_stored":1154427,"data_compressed":58377,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":508135,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":503959,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1069056,"data_stored":1099868,"data_compressed":56750,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":499215,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":562481,"data_compressed":22285,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":500603,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1089536,"data_stored":564222,"data_compressed":22285,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1011712,"data_stored":499409,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":503865,"data_compressed":20658,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":1095456,"data_compressed":56750,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":65097,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64934,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:13:11.060 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-22T02:13:11.060 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-22T02:13:11.060 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-22T02:13:11.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-22T02:13:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:11.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:11 smithi028 ceph-mon[31916]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:11 smithi120 ceph-mon[36803]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:11 smithi083 ceph-mon[36340]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 -- 172.21.15.28:0/1260541591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f501042b0 msgr2=0x7f4f5010c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:13.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 --2- 172.21.15.28:0/1260541591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f501042b0 0x7f4f5010c7a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c009990 tx=0x7f4f4c009ca0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 -- 172.21.15.28:0/1260541591 shutdown_connections 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 --2- 172.21.15.28:0/1260541591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f5010cd70 0x7f4f5010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 --2- 172.21.15.28:0/1260541591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f50103d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 --2- 172.21.15.28:0/1260541591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f501042b0 0x7f4f5010c7a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 -- 172.21.15.28:0/1260541591 >> 172.21.15.28:0/1260541591 conn(0x7f4f500786b0 msgr2=0x7f4f50078ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 -- 172.21.15.28:0/1260541591 shutdown_connections 2024-03-22T02:13:13.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 -- 172.21.15.28:0/1260541591 wait complete. 2024-03-22T02:13:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.009+0000 7f4f580e1700 1 Processor -- start 2024-03-22T02:13:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 -- start start 2024-03-22T02:13:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:13.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f501042b0 0x7f4f50101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5010cd70 0x7f4f50102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f50112a30 con 0x7f4f5010cd70 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f501128b0 con 0x7f4f501042b0 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f580e1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f50112bb0 con 0x7f4f50103990 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34306/0 (socket says 172.21.15.28:34306) 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 -- 172.21.15.28:0/3213889999 learned_addr learned my addr 172.21.15.28:0/3213889999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f5567c700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f501042b0 0x7f4f50101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f5667e700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5010cd70 0x7f4f50102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 -- 172.21.15.28:0/3213889999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f501042b0 msgr2=0x7f4f50101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:13.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f501042b0 0x7f4f50101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 -- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5010cd70 msgr2=0x7f4f50102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5010cd70 0x7f4f50102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 -- 172.21.15.28:0/3213889999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f501a7420 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.010+0000 7f4f55e7d700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000eb30 tx=0x7f4f4000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.011+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f4000cbd0 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.011+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f501a7710 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.011+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f40004500 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.011+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f40010430 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.011+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f501a7c20 con 0x7f4f50103990 2024-03-22T02:13:13.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.012+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f50069340 con 0x7f4f50103990 2024-03-22T02:13:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.013+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f4f40043b30 con 0x7f4f50103990 2024-03-22T02:13:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.013+0000 7f4f46ffd700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 0x7f4f3c0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.013+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f4f400cfc20 con 0x7f4f50103990 2024-03-22T02:13:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.013+0000 7f4f5567c700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 0x7f4f3c0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:13.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.014+0000 7f4f5567c700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 0x7f4f3c0a5500 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c00b2e0 tx=0x7f4f4c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:13.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.017+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f4009c550 con 0x7f4f50103990 2024-03-22T02:13:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.266+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4f50102ad0 con 0x7f4f50103990 2024-03-22T02:13:13.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.267+0000 7f4f46ffd700 1 -- 172.21.15.28:0/3213889999 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f4f40043e20 con 0x7f4f50103990 2024-03-22T02:13:13.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:13.269 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 msgr2=0x7f4f3c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 0x7f4f3c0a5500 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c00b2e0 tx=0x7f4f4c005960 comp rx=0 tx=0).stop 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 msgr2=0x7f4f501015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000eb30 tx=0x7f4f4000ee40 comp rx=0 tx=0).stop 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 shutdown_connections 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f3c0a3040 0x7f4f3c0a5500 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f5010cd70 0x7f4f50102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f50103990 0x7f4f501015f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 --2- 172.21.15.28:0/3213889999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f501042b0 0x7f4f50101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:13.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 >> 172.21.15.28:0/3213889999 conn(0x7f4f500786b0 msgr2=0x7f4f500fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:13.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 shutdown_connections 2024-03-22T02:13:13.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:13.269+0000 7f4f580e1700 1 -- 172.21.15.28:0/3213889999 wait complete. 2024-03-22T02:13:13.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:13 smithi028 ceph-mon[31916]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:13.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:13 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3213889999' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T02:13:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:13 smithi120 ceph-mon[36803]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:13 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3213889999' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T02:13:14.005 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-22T02:13:14.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:14.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:13 smithi083 ceph-mon[36340]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:14.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:13 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3213889999' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 -- 172.21.15.28:0/1897437818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 msgr2=0x7f864c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 --2- 172.21.15.28:0/1897437818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c1032f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8640009930 tx=0x7f864002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 -- 172.21.15.28:0/1897437818 shutdown_connections 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 --2- 172.21.15.28:0/1897437818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c103830 0x7f864c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 --2- 172.21.15.28:0/1897437818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c1032f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 --2- 172.21.15.28:0/1897437818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c0ffea0 0x7f864c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 -- 172.21.15.28:0/1897437818 >> 172.21.15.28:0/1897437818 conn(0x7f864c0fb7d0 msgr2=0x7f864c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 -- 172.21.15.28:0/1897437818 shutdown_connections 2024-03-22T02:13:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.115+0000 7f86524fb640 1 -- 172.21.15.28:0/1897437818 wait complete. 2024-03-22T02:13:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 Processor -- start 2024-03-22T02:13:14.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 -- start start 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c0ffea0 0x7f864c1027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c103830 0x7f864c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f864c10d880 con 0x7f864c102e70 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f864c10d700 con 0x7f864c103830 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f86524fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f864c10da00 con 0x7f864c0ffea0 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f864b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f864b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40726/0 (socket says 172.21.15.28:40726) 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f864b7fe640 1 -- 172.21.15.28:0/2707184274 learned_addr learned my addr 172.21.15.28:0/2707184274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:14.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f864bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c0ffea0 0x7f864c1027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.116+0000 7f8650a98640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c103830 0x7f864c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 -- 172.21.15.28:0/2707184274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c0ffea0 msgr2=0x7f864c1027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c0ffea0 0x7f864c1027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 -- 172.21.15.28:0/2707184274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c103830 msgr2=0x7f864c101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c103830 0x7f864c101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 -- 172.21.15.28:0/2707184274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864c101bf0 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f864b7fe640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f864002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640041070 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f864c1a1fe0 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8640004450 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640034440 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.117+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f864c1a2440 con 0x7f864c102e70 2024-03-22T02:13:14.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.118+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f864c109090 con 0x7f864c102e70 2024-03-22T02:13:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.119+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f8640044070 con 0x7f864c102e70 2024-03-22T02:13:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.120+0000 7f86497fa640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 0x7f86340a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.120+0000 7f864bfff640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 0x7f86340a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.120+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f86400f1200 con 0x7f864c102e70 2024-03-22T02:13:14.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.120+0000 7f864bfff640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 0x7f86340a5470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f863c0095f0 tx=0x7f863c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.124+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86400bd650 con 0x7f864c102e70 2024-03-22T02:13:14.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.308+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f864c02d5f0 con 0x7f864c102e70 2024-03-22T02:13:14.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.310+0000 7f86497fa640 1 -- 172.21.15.28:0/2707184274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24518 (secure 0 0 0) 0x7f86400c16f0 con 0x7f864c102e70 2024-03-22T02:13:14.312 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:14.312 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:12:28.476410+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 msgr2=0x7f86340a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 0x7f86340a5470 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f863c0095f0 tx=0x7f863c005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 msgr2=0x7f864c100e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f864002c650 comp rx=0 tx=0).stop 2024-03-22T02:13:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 shutdown_connections 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f86340a2fb0 0x7f86340a5470 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f864c103830 0x7f864c101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f864c102e70 0x7f864c100e20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 --2- 172.21.15.28:0/2707184274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f864c0ffea0 0x7f864c1027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 >> 172.21.15.28:0/2707184274 conn(0x7f864c0fb7d0 msgr2=0x7f864c106260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 shutdown_connections 2024-03-22T02:13:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.312+0000 7f86524fb640 1 -- 172.21.15.28:0/2707184274 wait complete. 2024-03-22T02:13:14.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:14.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:14.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.429+0000 7f6b94581640 1 -- 172.21.15.28:0/410663266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 msgr2=0x7f6b8c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.429+0000 7f6b94581640 1 --2- 172.21.15.28:0/410663266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c100eb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009900 tx=0x7f6b8402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- 172.21.15.28:0/410663266 shutdown_connections 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- 172.21.15.28:0/410663266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 0x7f6b8c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- 172.21.15.28:0/410663266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 0x7f6b8c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- 172.21.15.28:0/410663266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c100eb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- 172.21.15.28:0/410663266 >> 172.21.15.28:0/410663266 conn(0x7f6b8c0fc020 msgr2=0x7f6b8c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- 172.21.15.28:0/410663266 shutdown_connections 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- 172.21.15.28:0/410663266 wait complete. 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 Processor -- start 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- start start 2024-03-22T02:13:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 0x7f6b8c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 0x7f6b8c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c069880 con 0x7f6b8c100a90 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c069700 con 0x7f6b8c102640 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b94581640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8c069a00 con 0x7f6b8c101c80 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b9231d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b9231d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40740/0 (socket says 172.21.15.28:40740) 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.430+0000 7f6b9231d640 1 -- 172.21.15.28:0/279940342 learned_addr learned my addr 172.21.15.28:0/279940342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b91b1c640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 0x7f6b8c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b92b1e640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 0x7f6b8c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 -- 172.21.15.28:0/279940342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 msgr2=0x7f6b8c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 0x7f6b8c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 -- 172.21.15.28:0/279940342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 msgr2=0x7f6b8c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 0x7f6b8c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 -- 172.21.15.28:0/279940342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c19fb50 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b9231d640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000c00 tx=0x7f6b8402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84041070 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b84004450 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8c19fde0 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84003df0 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.431+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8c1a0240 con 0x7f6b8c100a90 2024-03-22T02:13:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.432+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b8c107e20 con 0x7f6b8c100a90 2024-03-22T02:13:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.433+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6b84044070 con 0x7f6b8c100a90 2024-03-22T02:13:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.434+0000 7f6b8b7fe640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 0x7f6b740a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.434+0000 7f6b91b1c640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 0x7f6b740a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.434+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f6b840f1300 con 0x7f6b8c100a90 2024-03-22T02:13:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.434+0000 7f6b91b1c640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 0x7f6b740a54f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c199440 tx=0x7f6b80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.437+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b840bded0 con 0x7f6b8c100a90 2024-03-22T02:13:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.622+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6b8c02d540 con 0x7f6b8c100a90 2024-03-22T02:13:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.622+0000 7f6b8b7fe640 1 -- 172.21.15.28:0/279940342 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f6b840bd7d0 con 0x7f6b8c100a90 2024-03-22T02:13:14.624 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 msgr2=0x7f6b740a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 0x7f6b740a54f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c199440 tx=0x7f6b80005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 msgr2=0x7f6b8c197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000c00 tx=0x7f6b8402c640 comp rx=0 tx=0).stop 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 shutdown_connections 2024-03-22T02:13:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3030 0x7f6b740a54f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b8c102640 0x7f6b8c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b8c101c80 0x7f6b8c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 --2- 172.21.15.28:0/279940342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b8c100a90 0x7f6b8c197f30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 >> 172.21.15.28:0/279940342 conn(0x7f6b8c0fc020 msgr2=0x7f6b8c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 shutdown_connections 2024-03-22T02:13:14.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.625+0000 7f6b94581640 1 -- 172.21.15.28:0/279940342 wait complete. 2024-03-22T02:13:14.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- 172.21.15.28:0/1360652996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101ba0 msgr2=0x7fb198101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/1360652996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101ba0 0x7fb198101fc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fb190009900 tx=0x7fb19002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- 172.21.15.28:0/1360652996 shutdown_connections 2024-03-22T02:13:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/1360652996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198103750 0x7fb198109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/1360652996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198102d90 0x7fb198103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/1360652996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198101ba0 0x7fb198101fc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- 172.21.15.28:0/1360652996 >> 172.21.15.28:0/1360652996 conn(0x7fb1980fd130 msgr2=0x7fb1980ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- 172.21.15.28:0/1360652996 shutdown_connections 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- 172.21.15.28:0/1360652996 wait complete. 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 Processor -- start 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- start start 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198101ba0 0x7fb198197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198102d90 0x7fb198198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb19810d820 con 0x7fb198103750 2024-03-22T02:13:14.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb19810d6a0 con 0x7fb198101ba0 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.742+0000 7fb19fcc2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb19810d9a0 con 0x7fb198102d90 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40764/0 (socket says 172.21.15.28:40764) 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 -- 172.21.15.28:0/2515565117 learned_addr learned my addr 172.21.15.28:0/2515565117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19d25d640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198102d90 0x7fb198198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19da5e640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198101ba0 0x7fb198197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 -- 172.21.15.28:0/2515565117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198102d90 msgr2=0x7fb198198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198102d90 0x7fb198198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 -- 172.21.15.28:0/2515565117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198101ba0 msgr2=0x7fb198197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198101ba0 0x7fb198197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 -- 172.21.15.28:0/2515565117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19819fb20 con 0x7fb198103750 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19e25f640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb19400eb10 tx=0x7fb19400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194004b20 con 0x7fb198103750 2024-03-22T02:13:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19819fe10 con 0x7fb198103750 2024-03-22T02:13:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb194004cc0 con 0x7fb198103750 2024-03-22T02:13:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194010760 con 0x7fb198103750 2024-03-22T02:13:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.743+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1981a02d0 con 0x7fb198103750 2024-03-22T02:13:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.744+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb198108ee0 con 0x7fb198103750 2024-03-22T02:13:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.745+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fb194005000 con 0x7fb198103750 2024-03-22T02:13:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.746+0000 7fb18effd640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 0x7fb1840a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.746+0000 7fb19da5e640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 0x7fb1840a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.746+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb1940d02f0 con 0x7fb198103750 2024-03-22T02:13:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.747+0000 7fb19da5e640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 0x7fb1840a5470 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb190000c00 tx=0x7fb190005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:14.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.750+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb19409cc80 con 0x7fb198103750 2024-03-22T02:13:14.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.933+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb19802d600 con 0x7fb198103750 2024-03-22T02:13:14.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.934+0000 7fb18effd640 1 -- 172.21.15.28:0/2515565117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7fb19409c580 con 0x7fb198103750 2024-03-22T02:13:14.935 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 msgr2=0x7fb1840a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 0x7fb1840a5470 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb190000c00 tx=0x7fb190005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 msgr2=0x7fb19819f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb19400eb10 tx=0x7fb19400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 shutdown_connections 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1840a2fb0 0x7fb1840a5470 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198103750 0x7fb19819f4b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb198102d90 0x7fb198198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 --2- 172.21.15.28:0/2515565117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb198101ba0 0x7fb198197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 >> 172.21.15.28:0/2515565117 conn(0x7fb1980fd130 msgr2=0x7fb1981060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 shutdown_connections 2024-03-22T02:13:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:14.936+0000 7fb19fcc2640 1 -- 172.21.15.28:0/2515565117 wait complete. 2024-03-22T02:13:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:14 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2707184274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:14.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:14 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2707184274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:15.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:14 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2707184274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 -- 172.21.15.28:0/1769755660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 msgr2=0x7ff100107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 --2- 172.21.15.28:0/1769755660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff100107390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009900 tx=0x7ff0f402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 -- 172.21.15.28:0/1769755660 shutdown_connections 2024-03-22T02:13:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 --2- 172.21.15.28:0/1769755660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1001078d0 0x7ff100109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 --2- 172.21.15.28:0/1769755660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff100107390 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 --2- 172.21.15.28:0/1769755660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff100102670 0x7ff100104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 -- 172.21.15.28:0/1769755660 >> 172.21.15.28:0/1769755660 conn(0x7ff1000fc020 msgr2=0x7ff1000fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 -- 172.21.15.28:0/1769755660 shutdown_connections 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 -- 172.21.15.28:0/1769755660 wait complete. 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.053+0000 7ff105888640 1 Processor -- start 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 -- start start 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff100102670 0x7ff100197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001078d0 0x7ff10019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010d7e0 con 0x7ff100104fa0 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010d660 con 0x7ff1001078d0 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff10010d960 con 0x7ff100102670 2024-03-22T02:13:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40780/0 (socket says 172.21.15.28:40780) 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 -- 172.21.15.28:0/2940173846 learned_addr learned my addr 172.21.15.28:0/2940173846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0feffd640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff100102670 0x7ff100197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0ff7fe640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001078d0 0x7ff10019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 -- 172.21.15.28:0/2940173846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff100102670 msgr2=0x7ff100197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff100102670 0x7ff100197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 -- 172.21.15.28:0/2940173846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001078d0 msgr2=0x7ff10019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001078d0 0x7ff10019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 -- 172.21.15.28:0/2940173846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10019fb40 con 0x7ff100104fa0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff0fe7fc640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4000c00 tx=0x7ff0f402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4041070 con 0x7ff100104fa0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10019fdd0 con 0x7ff100104fa0 2024-03-22T02:13:15.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.054+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0f4004450 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.055+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4003df0 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.055+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1001a0260 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.056+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1000fdb10 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.057+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7ff0f4044070 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.057+0000 7ff104886640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 0x7ff0e80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.057+0000 7ff0feffd640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 0x7ff0e80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.057+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7ff0f40f12a0 con 0x7ff100104fa0 2024-03-22T02:13:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.058+0000 7ff0feffd640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 0x7ff0e80a5470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00095f0 tx=0x7ff0f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.062+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff0f40bddd0 con 0x7ff100104fa0 2024-03-22T02:13:15.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.245+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff10002d5d0 con 0x7ff100104fa0 2024-03-22T02:13:15.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.245+0000 7ff104886640 1 -- 172.21.15.28:0/2940173846 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7ff0f40bd6d0 con 0x7ff100104fa0 2024-03-22T02:13:15.247 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 msgr2=0x7ff0e80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 0x7ff0e80a5470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff0f00095f0 tx=0x7ff0f0005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 msgr2=0x7ff1001984b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4000c00 tx=0x7ff0f402c640 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 shutdown_connections 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0e80a2fb0 0x7ff0e80a5470 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff1001078d0 0x7ff10019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff100104fa0 0x7ff1001984b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 --2- 172.21.15.28:0/2940173846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff100102670 0x7ff100197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 >> 172.21.15.28:0/2940173846 conn(0x7ff1000fc020 msgr2=0x7ff1001030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 shutdown_connections 2024-03-22T02:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.247+0000 7ff105888640 1 -- 172.21.15.28:0/2940173846 wait complete. 2024-03-22T02:13:15.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 -- 172.21.15.28:0/983188880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8102620 msgr2=0x7f66a8104a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 --2- 172.21.15.28:0/983188880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8102620 0x7f66a8104a10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009900 tx=0x7f66a002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 -- 172.21.15.28:0/983188880 shutdown_connections 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 --2- 172.21.15.28:0/983188880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 --2- 172.21.15.28:0/983188880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8104f50 0x7f66a8107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 --2- 172.21.15.28:0/983188880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8102620 0x7f66a8104a10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 -- 172.21.15.28:0/983188880 >> 172.21.15.28:0/983188880 conn(0x7f66a80fc030 msgr2=0x7f66a80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 -- 172.21.15.28:0/983188880 shutdown_connections 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.364+0000 7f66acf3a640 1 -- 172.21.15.28:0/983188880 wait complete. 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 Processor -- start 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 -- start start 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8102620 0x7f66a8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f66a810d820 con 0x7f66a8104f50 2024-03-22T02:13:15.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f66a810d6a0 con 0x7f66a8107880 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66acf3a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f66a810d9a0 con 0x7f66a8102620 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66a5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66a5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40800/0 (socket says 172.21.15.28:40800) 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66a5d9b640 1 -- 172.21.15.28:0/2018126140 learned_addr learned my addr 172.21.15.28:0/2018126140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.365+0000 7f66a6d9d640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a659c640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8102620 0x7f66a8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 -- 172.21.15.28:0/2018126140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8102620 msgr2=0x7f66a8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8102620 0x7f66a8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 -- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 msgr2=0x7f66a8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 -- 172.21.15.28:0/2018126140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a819fb70 con 0x7f66a8107880 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a5d9b640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66a6d9d640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a819f500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f669800ec50 tx=0x7f669800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698003b70 con 0x7f66a8107880 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a819fe60 con 0x7f66a8107880 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6698003d10 con 0x7f66a8107880 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698010890 con 0x7f66a8107880 2024-03-22T02:13:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.366+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a81a02d0 con 0x7f66a8107880 2024-03-22T02:13:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.367+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a80fdb20 con 0x7f66a8107880 2024-03-22T02:13:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.368+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6698003fc0 con 0x7f66a8107880 2024-03-22T02:13:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.369+0000 7f669f7fe640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 0x7f66940a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.369+0000 7f66a659c640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 0x7f66940a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.369+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f66980cfe70 con 0x7f66a8107880 2024-03-22T02:13:15.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.369+0000 7f66a659c640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 0x7f66940a5470 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f66a002c7b0 tx=0x7f66a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.374+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f669809c800 con 0x7f66a8107880 2024-03-22T02:13:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.557+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f66a802d560 con 0x7f66a8107880 2024-03-22T02:13:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.558+0000 7f669f7fe640 1 -- 172.21.15.28:0/2018126140 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f669809c100 con 0x7f66a8107880 2024-03-22T02:13:15.559 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.559+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 msgr2=0x7f66940a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.559+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 0x7f66940a5470 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f66a002c7b0 tx=0x7f66a0005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.559+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 msgr2=0x7f66a819f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.559+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a819f500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f669800ec50 tx=0x7f669800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.559+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 shutdown_connections 2024-03-22T02:13:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f66940a2fb0 0x7f66940a5470 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f66a8107880 0x7f66a819f500 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66a8104f50 0x7f66a8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 --2- 172.21.15.28:0/2018126140 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f66a8102620 0x7f66a8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 >> 172.21.15.28:0/2018126140 conn(0x7f66a80fc030 msgr2=0x7f66a8102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 shutdown_connections 2024-03-22T02:13:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.560+0000 7f66acf3a640 1 -- 172.21.15.28:0/2018126140 wait complete. 2024-03-22T02:13:15.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 -- 172.21.15.28:0/2369822179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90100ab0 msgr2=0x7f3e90100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 --2- 172.21.15.28:0/2369822179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90100ab0 0x7f3e90100ed0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3e80009900 tx=0x7f3e8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 -- 172.21.15.28:0/2369822179 shutdown_connections 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 --2- 172.21.15.28:0/2369822179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 0x7f3e90108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 --2- 172.21.15.28:0/2369822179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90101ca0 0x7f3e90102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 --2- 172.21.15.28:0/2369822179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90100ab0 0x7f3e90100ed0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 -- 172.21.15.28:0/2369822179 >> 172.21.15.28:0/2369822179 conn(0x7f3e900fc020 msgr2=0x7f3e900fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 -- 172.21.15.28:0/2369822179 shutdown_connections 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.674+0000 7f3e953e9640 1 -- 172.21.15.28:0/2369822179 wait complete. 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 Processor -- start 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 -- start start 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90100ab0 0x7f3e90197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 0x7f3e9019f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9010d850 con 0x7f3e90101ca0 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9010d6d0 con 0x7f3e90100ab0 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e953e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9010d9d0 con 0x7f3e90102660 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40820/0 (socket says 172.21.15.28:40820) 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 -- 172.21.15.28:0/534739796 learned_addr learned my addr 172.21.15.28:0/534739796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e8effd640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90100ab0 0x7f3e90197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e8f7fe640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 0x7f3e9019f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 -- 172.21.15.28:0/534739796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 msgr2=0x7f3e9019f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 0x7f3e9019f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 -- 172.21.15.28:0/534739796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90100ab0 msgr2=0x7f3e90197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90100ab0 0x7f3e90197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.675+0000 7f3e867fc640 1 -- 172.21.15.28:0/534739796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9019fae0 con 0x7f3e90101ca0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e867fc640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800e9f0 tx=0x7f3e7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800cd80 con 0x7f3e90101ca0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e78010910 con 0x7f3e90101ca0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9019fdd0 con 0x7f3e90101ca0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78018990 con 0x7f3e90101ca0 2024-03-22T02:13:15.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.676+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e901a0290 con 0x7f3e90101ca0 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.677+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e90107dd0 con 0x7f3e90101ca0 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.678+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f3e78010ac0 con 0x7f3e90101ca0 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.678+0000 7f3e8cff9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 0x7f3e7c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.678+0000 7f3e8effd640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 0x7f3e7c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.678+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f3e780cfbb0 con 0x7f3e90101ca0 2024-03-22T02:13:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.679+0000 7f3e8effd640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 0x7f3e7c0a5470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e80000c00 tx=0x7f3e80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.683+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e7809bef0 con 0x7f3e90101ca0 2024-03-22T02:13:15.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:15 smithi028 ceph-mon[31916]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:15.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/279940342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2515565117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2940173846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2018126140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.905+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e9002d580 con 0x7f3e90101ca0 2024-03-22T02:13:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.905+0000 7f3e8cff9640 1 -- 172.21.15.28:0/534739796 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7f3e7809ff90 con 0x7f3e90101ca0 2024-03-22T02:13:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:15.908 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:13:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 msgr2=0x7f3e7c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 0x7f3e7c0a5470 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3e80000c00 tx=0x7f3e80005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 msgr2=0x7f3e90198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e7800e9f0 tx=0x7f3e7800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 shutdown_connections 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3e7c0a2fb0 0x7f3e7c0a5470 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e90102660 0x7f3e9019f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e90101ca0 0x7f3e90198400 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.907+0000 7f3e953e9640 1 --2- 172.21.15.28:0/534739796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3e90100ab0 0x7f3e90197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.908+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 >> 172.21.15.28:0/534739796 conn(0x7f3e900fc020 msgr2=0x7f3e90104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.908+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 shutdown_connections 2024-03-22T02:13:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:15.908+0000 7f3e953e9640 1 -- 172.21.15.28:0/534739796 wait complete. 2024-03-22T02:13:15.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:15 smithi120 ceph-mon[36803]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/279940342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2515565117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2940173846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2018126140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:15 smithi083 ceph-mon[36340]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:16.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/279940342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2515565117' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2940173846' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2018126140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/3065845929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 msgr2=0x7fc7d8108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/3065845929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d8108ef0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0009be0 tx=0x7fc7d002bef0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/3065845929 shutdown_connections 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/3065845929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d8108ef0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/3065845929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 0x7fc7d8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/3065845929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/3065845929 >> 172.21.15.28:0/3065845929 conn(0x7fc7d80fc020 msgr2=0x7fc7d80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/3065845929 shutdown_connections 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/3065845929 wait complete. 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 Processor -- start 2024-03-22T02:13:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- start start 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 0x7fc7d819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c730 con 0x7fc7d8102660 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c5b0 con 0x7fc7d8100ab0 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.022+0000 7fc7dfc6d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d810c8b0 con 0x7fc7d8101ca0 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40840/0 (socket says 172.21.15.28:40840) 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7dda09640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 -- 172.21.15.28:0/2999777902 learned_addr learned my addr 172.21.15.28:0/2999777902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7dd208640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 0x7fc7d819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 -- 172.21.15.28:0/2999777902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 msgr2=0x7fc7d819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 0x7fc7d819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 -- 172.21.15.28:0/2999777902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 msgr2=0x7fc7d8069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 -- 172.21.15.28:0/2999777902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d819faf0 con 0x7fc7d8102660 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7dda09640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7de20a640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0036040 tx=0x7fc7d00337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0043020 con 0x7fc7d8102660 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7d002ca60 con 0x7fc7d8102660 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d819fd20 con 0x7fc7d8102660 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0034700 con 0x7fc7d8102660 2024-03-22T02:13:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.023+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d81a0260 con 0x7fc7d8102660 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.024+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d8107dd0 con 0x7fc7d8102660 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.025+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fc7d0046020 con 0x7fc7d8102660 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.026+0000 7fc7ceffd640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 0x7fc7c40a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.026+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fc7d00f2540 con 0x7fc7d8102660 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.026+0000 7fc7dda09640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 0x7fc7c40a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.026+0000 7fc7dda09640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 0x7fc7c40a54f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40096c0 tx=0x7fc7d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.030+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc7d004d020 con 0x7fc7d8102660 2024-03-22T02:13:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.213+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7d802d530 con 0x7fc7d8102660 2024-03-22T02:13:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.215+0000 7fc7ceffd640 1 -- 172.21.15.28:0/2999777902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24518 (secure 0 0 0) 0x7fc7d00f2090 con 0x7fc7d8102660 2024-03-22T02:13:16.216 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:16.216 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:12:28.476410+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 msgr2=0x7fc7c40a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 0x7fc7c40a54f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40096c0 tx=0x7fc7d4009260 comp rx=0 tx=0).stop 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 msgr2=0x7fc7d819f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc7d0036040 tx=0x7fc7d00337b0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 shutdown_connections 2024-03-22T02:13:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc7c40a3030 0x7fc7c40a54f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8102660 0x7fc7d819f1e0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7d8101ca0 0x7fc7d819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 --2- 172.21.15.28:0/2999777902 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7d8100ab0 0x7fc7d8069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 >> 172.21.15.28:0/2999777902 conn(0x7fc7d80fc020 msgr2=0x7fc7d8104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 shutdown_connections 2024-03-22T02:13:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.217+0000 7fc7dfc6d640 1 -- 172.21.15.28:0/2999777902 wait complete. 2024-03-22T02:13:16.230 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:16.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:16.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 -- 172.21.15.28:0/1032721570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc1078f0 msgr2=0x7f27cc109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 --2- 172.21.15.28:0/1032721570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc1078f0 0x7f27cc109ce0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f27bc009ca0 tx=0x7f27bc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 -- 172.21.15.28:0/1032721570 shutdown_connections 2024-03-22T02:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 --2- 172.21.15.28:0/1032721570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc1078f0 0x7f27cc109ce0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 --2- 172.21.15.28:0/1032721570 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc104fc0 0x7f27cc1073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 --2- 172.21.15.28:0/1032721570 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc102690 0x7f27cc104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 -- 172.21.15.28:0/1032721570 >> 172.21.15.28:0/1032721570 conn(0x7f27cc0fc020 msgr2=0x7f27cc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 -- 172.21.15.28:0/1032721570 shutdown_connections 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.334+0000 7f27d35f6640 1 -- 172.21.15.28:0/1032721570 wait complete. 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 Processor -- start 2024-03-22T02:13:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 -- start start 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc104fc0 0x7f27cc198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 0x7f27cc19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10d830 con 0x7f27cc102690 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10d6b0 con 0x7f27cc1078f0 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d35f6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10d9b0 con 0x7f27cc104fc0 2024-03-22T02:13:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1392640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1392640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40856/0 (socket says 172.21.15.28:40856) 2024-03-22T02:13:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1392640 1 -- 172.21.15.28:0/2719909215 learned_addr learned my addr 172.21.15.28:0/2719909215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d0b91640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc104fc0 0x7f27cc198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1b93640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 0x7f27cc19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1392640 1 -- 172.21.15.28:0/2719909215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc104fc0 msgr2=0x7f27cc198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.335+0000 7f27d1392640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc104fc0 0x7f27cc198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d1392640 1 -- 172.21.15.28:0/2719909215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 msgr2=0x7f27cc19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d1392640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 0x7f27cc19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d1392640 1 -- 172.21.15.28:0/2719909215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc19faf0 con 0x7f27cc102690 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d1b93640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 0x7f27cc19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:13:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d1392640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f27c400e9f0 tx=0x7f27c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c400cd80 con 0x7f27cc102690 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c4005500 con 0x7f27cc102690 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27cc19fde0 con 0x7f27cc102690 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4010690 con 0x7f27cc102690 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.336+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27cc1a01f0 con 0x7f27cc102690 2024-03-22T02:13:16.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.337+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27cc0fdb10 con 0x7f27cc102690 2024-03-22T02:13:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.338+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f27c40046c0 con 0x7f27cc102690 2024-03-22T02:13:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.339+0000 7f27ca7fc640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 0x7f27b40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.339+0000 7f27d0b91640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 0x7f27b40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.339+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f27c40cffa0 con 0x7f27cc102690 2024-03-22T02:13:16.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.339+0000 7f27d0b91640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 0x7f27b40a5470 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f27cc1993e0 tx=0x7f27c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.342+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27c4053740 con 0x7f27cc102690 2024-03-22T02:13:16.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.526+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f27cc02d540 con 0x7f27cc102690 2024-03-22T02:13:16.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.526+0000 7f27ca7fc640 1 -- 172.21.15.28:0/2719909215 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) ==== 93+0+10 (secure 0 0 0) 0x7f27c409c930 con 0x7f27cc102690 2024-03-22T02:13:16.528 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:16.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.528+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 msgr2=0x7f27b40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.528+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 0x7f27b40a5470 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f27cc1993e0 tx=0x7f27c0005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:16.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.528+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 msgr2=0x7f27cc197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f27c400e9f0 tx=0x7f27c400edf0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 shutdown_connections 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f27b40a2fb0 0x7f27b40a5470 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f27cc1078f0 0x7f27cc19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f27cc104fc0 0x7f27cc198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 --2- 172.21.15.28:0/2719909215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27cc102690 0x7f27cc197ed0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 >> 172.21.15.28:0/2719909215 conn(0x7f27cc0fc020 msgr2=0x7f27cc1030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 shutdown_connections 2024-03-22T02:13:16.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.529+0000 7f27d35f6640 1 -- 172.21.15.28:0/2719909215 wait complete. 2024-03-22T02:13:16.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 -- 172.21.15.28:0/451113848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc101ba0 msgr2=0x7f54dc101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 --2- 172.21.15.28:0/451113848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc101ba0 0x7f54dc101fc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009ca0 tx=0x7f54cc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 -- 172.21.15.28:0/451113848 shutdown_connections 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 --2- 172.21.15.28:0/451113848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc103750 0x7f54dc109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 --2- 172.21.15.28:0/451113848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 0x7f54dc103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 --2- 172.21.15.28:0/451113848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc101ba0 0x7f54dc101fc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 -- 172.21.15.28:0/451113848 >> 172.21.15.28:0/451113848 conn(0x7f54dc0fd130 msgr2=0x7f54dc0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 -- 172.21.15.28:0/451113848 shutdown_connections 2024-03-22T02:13:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.646+0000 7f54e097a640 1 -- 172.21.15.28:0/451113848 wait complete. 2024-03-22T02:13:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 Processor -- start 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 -- start start 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 0x7f54dc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 0x7f54dc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d820 con 0x7f54dc103750 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d6a0 con 0x7f54dc102d90 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54e097a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f54dc10d9a0 con 0x7f54dc101ba0 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40864/0 (socket says 172.21.15.28:40864) 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54da59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 0x7f54dc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 -- 172.21.15.28:0/328613763 learned_addr learned my addr 172.21.15.28:0/328613763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54d9d9b640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 0x7f54dc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 -- 172.21.15.28:0/328613763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 msgr2=0x7f54dc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 0x7f54dc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 -- 172.21.15.28:0/328613763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 msgr2=0x7f54dc198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 0x7f54dc198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 -- 172.21.15.28:0/328613763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54dc19fb10 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54da59c640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 0x7f54dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.647+0000 7f54dad9d640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f54d400e970 tx=0x7f54d400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.648+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d400ccc0 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.648+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54d400ce60 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.648+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54dc19fe00 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.648+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4010490 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.648+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54dc1a02c0 con 0x7f54dc103750 2024-03-22T02:13:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.649+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54dc108f60 con 0x7f54dc103750 2024-03-22T02:13:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.650+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f54d4005020 con 0x7f54dc103750 2024-03-22T02:13:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.650+0000 7f54cb7fe640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 0x7f54c40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.651+0000 7f54da59c640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 0x7f54c40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.651+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f54d40cfcc0 con 0x7f54dc103750 2024-03-22T02:13:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.651+0000 7f54da59c640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 0x7f54c40a5470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f54cc02c630 tx=0x7f54cc009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.655+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f54d409c650 con 0x7f54dc103750 2024-03-22T02:13:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/534739796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2999777902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2719909215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.838+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f54dc02d600 con 0x7f54dc103750 2024-03-22T02:13:16.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.838+0000 7f54cb7fe640 1 -- 172.21.15.28:0/328613763 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) ==== 104+0+11 (secure 0 0 0) 0x7f54d409bf50 con 0x7f54dc103750 2024-03-22T02:13:16.840 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 msgr2=0x7f54c40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 0x7f54c40a5470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f54cc02c630 tx=0x7f54cc009360 comp rx=0 tx=0).stop 2024-03-22T02:13:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 msgr2=0x7f54dc19f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f54d400e970 tx=0x7f54d400ed70 comp rx=0 tx=0).stop 2024-03-22T02:13:16.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 shutdown_connections 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f54c40a2fb0 0x7f54c40a5470 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54dc103750 0x7f54dc19f4a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f54dc102d90 0x7f54dc198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 --2- 172.21.15.28:0/328613763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54dc101ba0 0x7f54dc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 >> 172.21.15.28:0/328613763 conn(0x7f54dc0fd130 msgr2=0x7f54dc106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 shutdown_connections 2024-03-22T02:13:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.840+0000 7f54e097a640 1 -- 172.21.15.28:0/328613763 wait complete. 2024-03-22T02:13:16.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 -- 172.21.15.28:0/3337477898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664100ab0 msgr2=0x7f5664100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 --2- 172.21.15.28:0/3337477898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664100ab0 0x7f5664100ed0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5660009900 tx=0x7f566002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 -- 172.21.15.28:0/3337477898 shutdown_connections 2024-03-22T02:13:16.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 --2- 172.21.15.28:0/3337477898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664102660 0x7f5664108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 --2- 172.21.15.28:0/3337477898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 0x7f5664102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 --2- 172.21.15.28:0/3337477898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664100ab0 0x7f5664100ed0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 -- 172.21.15.28:0/3337477898 >> 172.21.15.28:0/3337477898 conn(0x7f56640fc040 msgr2=0x7f56640fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 -- 172.21.15.28:0/3337477898 shutdown_connections 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.958+0000 7f566b039640 1 -- 172.21.15.28:0/3337477898 wait complete. 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 Processor -- start 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 -- start start 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664100ab0 0x7f5664197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 0x7f5664198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f566410c730 con 0x7f5664102660 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f566410c5b0 con 0x7f5664101ca0 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f566b039640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f566410c8b0 con 0x7f5664100ab0 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40880/0 (socket says 172.21.15.28:40880) 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 -- 172.21.15.28:0/317516165 learned_addr learned my addr 172.21.15.28:0/317516165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f5668dd5640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664100ab0 0x7f5664197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f565bfff640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 0x7f5664198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 -- 172.21.15.28:0/317516165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664100ab0 msgr2=0x7f5664197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664100ab0 0x7f5664197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 -- 172.21.15.28:0/317516165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 msgr2=0x7f5664198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.959+0000 7f56695d6640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 0x7f5664198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f56695d6640 1 -- 172.21.15.28:0/317516165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566419fb00 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f56695d6640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f565c00eb10 tx=0x7f565c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c004b20 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f565c004cc0 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566419fdf0 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c010760 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.960+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641a0300 con 0x7f5664102660 2024-03-22T02:13:16.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.961+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56641a05c0 con 0x7f5664102660 2024-03-22T02:13:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.962+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f565c005020 con 0x7f5664102660 2024-03-22T02:13:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.962+0000 7f5659ffb640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 0x7f564c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.963+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f565c0cfef0 con 0x7f5664102660 2024-03-22T02:13:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.963+0000 7f5668dd5640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 0x7f564c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:16.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.963+0000 7f5668dd5640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 0x7f564c0a54f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5660000c00 tx=0x7f5660005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:16.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:16.967+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f565c09c880 con 0x7f5664102660 2024-03-22T02:13:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:16 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/534739796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:16 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2999777902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:16 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2719909215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:16 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/534739796' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:16 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2999777902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:16 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2719909215' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.151+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f56641a0790 con 0x7f5664102660 2024-03-22T02:13:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.152+0000 7f5659ffb640 1 -- 172.21.15.28:0/317516165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) ==== 100+0+11 (secure 0 0 0) 0x7f565c09c180 con 0x7f5664102660 2024-03-22T02:13:17.153 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 msgr2=0x7f564c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 0x7f564c0a54f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5660000c00 tx=0x7f5660005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 msgr2=0x7f566419f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f565c00eb10 tx=0x7f565c00c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 shutdown_connections 2024-03-22T02:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f564c0a3030 0x7f564c0a54f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5664102660 0x7f566419f490 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5664101ca0 0x7f5664198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 --2- 172.21.15.28:0/317516165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5664100ab0 0x7f5664197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 >> 172.21.15.28:0/317516165 conn(0x7f56640fc040 msgr2=0x7f56640fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 shutdown_connections 2024-03-22T02:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.154+0000 7f566b039640 1 -- 172.21.15.28:0/317516165 wait complete. 2024-03-22T02:13:17.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.270+0000 7f1cbcca4640 1 -- 172.21.15.28:0/4011200099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 msgr2=0x7f1cb8108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.270+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/4011200099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb8108ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009ca0 tx=0x7f1cb002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 -- 172.21.15.28:0/4011200099 shutdown_connections 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/4011200099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb8108ed0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/4011200099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 0x7f1cb8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/4011200099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 0x7f1cb8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 -- 172.21.15.28:0/4011200099 >> 172.21.15.28:0/4011200099 conn(0x7f1cb80fc020 msgr2=0x7f1cb80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 -- 172.21.15.28:0/4011200099 shutdown_connections 2024-03-22T02:13:17.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 -- 172.21.15.28:0/4011200099 wait complete. 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 Processor -- start 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 -- start start 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.271+0000 7f1cbcca4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 0x7f1cb8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 0x7f1cb8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810c6f0 con 0x7f1cb8102640 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810c570 con 0x7f1cb8101c80 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1cb810c870 con 0x7f1cb8100a90 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40896/0 (socket says 172.21.15.28:40896) 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 -- 172.21.15.28:0/271399655 learned_addr learned my addr 172.21.15.28:0/271399655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb659c640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 0x7f1cb8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb5d9b640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 0x7f1cb8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 -- 172.21.15.28:0/271399655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 msgr2=0x7f1cb8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 0x7f1cb8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 -- 172.21.15.28:0/271399655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 msgr2=0x7f1cb8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 0x7f1cb8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 -- 172.21.15.28:0/271399655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cb819fa70 con 0x7f1cb8102640 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cb6d9d640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0037040 tx=0x7f1cb0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0041020 con 0x7f1cb8102640 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cb819fd00 con 0x7f1cb8102640 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cb0034cd0 con 0x7f1cb8102640 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0039040 con 0x7f1cb8102640 2024-03-22T02:13:17.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.272+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cb81a01a0 con 0x7f1cb8102640 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.273+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cb81a03d0 con 0x7f1cb8102640 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.274+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f1cb0044020 con 0x7f1cb8102640 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.275+0000 7f1ca77fe640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 0x7f1ca00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.275+0000 7f1cb659c640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 0x7f1ca00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.275+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f1cb00f1630 con 0x7f1cb8102640 2024-03-22T02:13:17.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.275+0000 7f1cb659c640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 0x7f1ca00a5470 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80095f0 tx=0x7f1ca8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.279+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1cb004b020 con 0x7f1cb8102640 2024-03-22T02:13:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.464+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1cb81a0760 con 0x7f1cb8102640 2024-03-22T02:13:17.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.465+0000 7f1ca77fe640 1 -- 172.21.15.28:0/271399655 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) ==== 103+0+11 (secure 0 0 0) 0x7f1cb00be100 con 0x7f1cb8102640 2024-03-22T02:13:17.466 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 msgr2=0x7f1ca00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 0x7f1ca00a5470 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80095f0 tx=0x7f1ca8005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 msgr2=0x7f1cb819f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0037040 tx=0x7f1cb0032f40 comp rx=0 tx=0).stop 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 shutdown_connections 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ca00a2fb0 0x7f1ca00a5470 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cb8102640 0x7f1cb819f400 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1cb8101c80 0x7f1cb8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 --2- 172.21.15.28:0/271399655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1cb8100a90 0x7f1cb8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 >> 172.21.15.28:0/271399655 conn(0x7f1cb80fc020 msgr2=0x7f1cb80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 shutdown_connections 2024-03-22T02:13:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.467+0000 7f1cbcca4640 1 -- 172.21.15.28:0/271399655 wait complete. 2024-03-22T02:13:17.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:17.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3280761121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0103760 msgr2=0x7f48c0109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3280761121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0103760 0x7f48c0109ff0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f48b8009900 tx=0x7f48b802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3280761121 shutdown_connections 2024-03-22T02:13:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3280761121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0103760 0x7f48c0109ff0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3280761121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0102da0 0x7f48c0103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3280761121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 0x7f48c0101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3280761121 >> 172.21.15.28:0/3280761121 conn(0x7f48c00fd140 msgr2=0x7f48c00ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3280761121 shutdown_connections 2024-03-22T02:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.581+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3280761121 wait complete. 2024-03-22T02:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 Processor -- start 2024-03-22T02:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 -- start start 2024-03-22T02:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 0x7f48c0197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0103760 0x7f48c019f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010d810 con 0x7f48c0102da0 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010d690 con 0x7f48c0103760 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48c4a8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48c010d990 con 0x7f48c0101bb0 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40920/0 (socket says 172.21.15.28:40920) 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 -- 172.21.15.28:0/3545956329 learned_addr learned my addr 172.21.15.28:0/3545956329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48be59c640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 0x7f48c0197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bed9d640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0103760 0x7f48c019f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 -- 172.21.15.28:0/3545956329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 msgr2=0x7f48c0197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 0x7f48c0197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 -- 172.21.15.28:0/3545956329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0103760 msgr2=0x7f48c019f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0103760 0x7f48c019f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 -- 172.21.15.28:0/3545956329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c019fa40 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.582+0000 7f48bdd9b640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f48a800e9f0 tx=0x7f48a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.583+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a800cd80 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.583+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c019fd30 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.583+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48a8010910 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.583+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a80188f0 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.583+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c01a0270 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.584+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c0108e70 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.585+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f48a8010ac0 con 0x7f48c0102da0 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.585+0000 7f48b77fe640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 0x7f48ac0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.585+0000 7f48be59c640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 0x7f48ac0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.585+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f48a80d0040 con 0x7f48c0102da0 2024-03-22T02:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.586+0000 7f48be59c640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 0x7f48ac0a5470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f48b00099a0 tx=0x7f48b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.589+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48a809c380 con 0x7f48c0102da0 2024-03-22T02:13:17.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.815+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48c002d3b0 con 0x7f48c0102da0 2024-03-22T02:13:17.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.816+0000 7f48b77fe640 1 -- 172.21.15.28:0/3545956329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7f48a80a0420 con 0x7f48c0102da0 2024-03-22T02:13:17.817 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:17.817 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:13:17.818 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:17.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 msgr2=0x7f48ac0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 0x7f48ac0a5470 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f48b00099a0 tx=0x7f48b0009500 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 msgr2=0x7f48c0198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f48a800e9f0 tx=0x7f48a800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 shutdown_connections 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f48ac0a2fb0 0x7f48ac0a5470 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f48c0103760 0x7f48c019f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48c0102da0 0x7f48c0198360 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 --2- 172.21.15.28:0/3545956329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48c0101bb0 0x7f48c0197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 >> 172.21.15.28:0/3545956329 conn(0x7f48c00fd140 msgr2=0x7f48c0106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 shutdown_connections 2024-03-22T02:13:17.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.818+0000 7f48c4a8a640 1 -- 172.21.15.28:0/3545956329 wait complete. 2024-03-22T02:13:17.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:17 smithi028 ceph-mon[31916]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:17.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:17 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/328613763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:17 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/317516165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:17 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/271399655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/1166376137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48101ca0 msgr2=0x7f2f48102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/1166376137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48101ca0 0x7f2f48102120 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c009900 tx=0x7f2f3c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/1166376137 shutdown_connections 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/1166376137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48102660 0x7f2f48108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/1166376137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48101ca0 0x7f2f48102120 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/1166376137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48100ab0 0x7f2f48100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/1166376137 >> 172.21.15.28:0/1166376137 conn(0x7f2f480fc020 msgr2=0x7f2f480fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/1166376137 shutdown_connections 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.932+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/1166376137 wait complete. 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 Processor -- start 2024-03-22T02:13:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 -- start start 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 0x7f2f480698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48101ca0 0x7f2f4819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4810c710 con 0x7f2f48102660 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4810c590 con 0x7f2f48100ab0 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f4dc1c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4810c890 con 0x7f2f48101ca0 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f47fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40936/0 (socket says 172.21.15.28:40936) 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f47fff640 1 -- 172.21.15.28:0/206940952 learned_addr learned my addr 172.21.15.28:0/206940952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f477fe640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 0x7f2f480698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.933+0000 7f2f46ffd640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48101ca0 0x7f2f4819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 -- 172.21.15.28:0/206940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48101ca0 msgr2=0x7f2f4819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48101ca0 0x7f2f4819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 -- 172.21.15.28:0/206940952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 msgr2=0x7f2f480698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 0x7f2f480698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 -- 172.21.15.28:0/206940952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4819fae0 con 0x7f2f48102660 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f477fe640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 0x7f2f480698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f47fff640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000e970 tx=0x7f2f4000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4000ccc0 con 0x7f2f48102660 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f4000ce60 con 0x7f2f48102660 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4819fdd0 con 0x7f2f48102660 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40010490 con 0x7f2f48102660 2024-03-22T02:13:17.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.934+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f481a0240 con 0x7f2f48102660 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.935+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f48107d70 con 0x7f2f48102660 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.938+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f2f40005000 con 0x7f2f48102660 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.938+0000 7f2f44ff9640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 0x7f2f300a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.938+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f2f400cff40 con 0x7f2f48102660 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.938+0000 7f2f477fe640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 0x7f2f300a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.938+0000 7f2f477fe640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 0x7f2f300a5470 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2f48101b00 tx=0x7f2f38009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:17.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:17.942+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f40053740 con 0x7f2f48102660 2024-03-22T02:13:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:17 smithi120 ceph-mon[36803]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:17 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/328613763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:17 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/317516165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:17 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/271399655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:18.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:17 smithi083 ceph-mon[36340]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:18.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:17 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/328613763' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:18.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:17 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/317516165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:18.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:17 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/271399655' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.125+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f4802d580 con 0x7f2f48102660 2024-03-22T02:13:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.127+0000 7f2f44ff9640 1 -- 172.21.15.28:0/206940952 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) ==== 74+0+24518 (secure 0 0 0) 0x7f2f4009c8d0 con 0x7f2f48102660 2024-03-22T02:13:18.129 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:18.129 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:12:28.476410+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 msgr2=0x7f2f300a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 0x7f2f300a5470 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2f48101b00 tx=0x7f2f38009500 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 msgr2=0x7f2f4819f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2f4000e970 tx=0x7f2f4000ed70 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 shutdown_connections 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2f300a2fb0 0x7f2f300a5470 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f48102660 0x7f2f4819f220 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2f48101ca0 0x7f2f4819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 --2- 172.21.15.28:0/206940952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2f48100ab0 0x7f2f480698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 >> 172.21.15.28:0/206940952 conn(0x7f2f480fc020 msgr2=0x7f2f48104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 shutdown_connections 2024-03-22T02:13:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.129+0000 7f2f4dc1c640 1 -- 172.21.15.28:0/206940952 wait complete. 2024-03-22T02:13:18.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 -- 172.21.15.28:0/472995470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500102650 msgr2=0x7fa500108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 --2- 172.21.15.28:0/472995470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500102650 0x7fa500108ee0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009be0 tx=0x7fa4f802bef0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 -- 172.21.15.28:0/472995470 shutdown_connections 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 --2- 172.21.15.28:0/472995470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500102650 0x7fa500108ee0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 --2- 172.21.15.28:0/472995470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500101c90 0x7fa500102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 --2- 172.21.15.28:0/472995470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500100aa0 0x7fa500100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 -- 172.21.15.28:0/472995470 >> 172.21.15.28:0/472995470 conn(0x7fa5000fc030 msgr2=0x7fa5000fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 -- 172.21.15.28:0/472995470 shutdown_connections 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.246+0000 7fa505993640 1 -- 172.21.15.28:0/472995470 wait complete. 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 Processor -- start 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 -- start start 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500101c90 0x7fa500197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500102650 0x7fa50019e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010c700 con 0x7fa500100aa0 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010c580 con 0x7fa500101c90 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa505993640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa50010c880 con 0x7fa500102650 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa4feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa4feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40964/0 (socket says 172.21.15.28:40964) 2024-03-22T02:13:18.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa4feffd640 1 -- 172.21.15.28:0/4052074707 learned_addr learned my addr 172.21.15.28:0/4052074707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa4ff7fe640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500102650 0x7fa50019e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.247+0000 7fa4fe7fc640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500101c90 0x7fa500197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 -- 172.21.15.28:0/4052074707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500102650 msgr2=0x7fa50019e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500102650 0x7fa50019e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 -- 172.21.15.28:0/4052074707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500101c90 msgr2=0x7fa500197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500101c90 0x7fa500197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 -- 172.21.15.28:0/4052074707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa50019e960 con 0x7fa500100aa0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa4feffd640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e9f0 tx=0x7fa4f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400ccc0 con 0x7fa500100aa0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa50019ec50 con 0x7fa500100aa0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4f40035f0 con 0x7fa500100aa0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f400ce60 con 0x7fa500100aa0 2024-03-22T02:13:18.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.248+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa50019f1a0 con 0x7fa500100aa0 2024-03-22T02:13:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.249+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa500107d60 con 0x7fa500100aa0 2024-03-22T02:13:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.250+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fa4f40037a0 con 0x7fa500100aa0 2024-03-22T02:13:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.251+0000 7fa504991640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 0x7fa4e80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.251+0000 7fa4fe7fc640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 0x7fa4e80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.251+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fa4f40cfbd0 con 0x7fa500100aa0 2024-03-22T02:13:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.251+0000 7fa4fe7fc640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 0x7fa4e80a5470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00095f0 tx=0x7fa4f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.255+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa4f409c560 con 0x7fa500100aa0 2024-03-22T02:13:18.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.481+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa50002d610 con 0x7fa500100aa0 2024-03-22T02:13:18.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.482+0000 7fa504991640 1 -- 172.21.15.28:0/4052074707 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7fa4f409be60 con 0x7fa500100aa0 2024-03-22T02:13:18.483 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:18.483 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:13:18.483 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:18.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 msgr2=0x7fa4e80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 0x7fa4e80a5470 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00095f0 tx=0x7fa4f0005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:18.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 msgr2=0x7fa500196d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400e9f0 tx=0x7fa4f400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 shutdown_connections 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4e80a2fb0 0x7fa4e80a5470 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa500102650 0x7fa50019e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa500101c90 0x7fa500197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 --2- 172.21.15.28:0/4052074707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa500100aa0 0x7fa500196d40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 >> 172.21.15.28:0/4052074707 conn(0x7fa5000fc030 msgr2=0x7fa500104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 shutdown_connections 2024-03-22T02:13:18.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.485+0000 7fa505993640 1 -- 172.21.15.28:0/4052074707 wait complete. 2024-03-22T02:13:18.497 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-22T02:11:33.259569+0000', 'modified': '2024-03-22T02:11:54.987732+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24508, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6835/3494340980', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3494340980}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 3494340980}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1917291995', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1917291995}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1917291995}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6837/3509701876', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 3509701876}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3509701876}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.120:6837/2149554336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 2149554336}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 2149554336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.120:6839/990751976', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 990751976}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 990751976}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6839/2818900206', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 2818900206}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 2818900206}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24478, 24508, 24509]} 2024-03-22T02:13:18.498 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-22T02:13:18.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-22T02:13:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 -- 172.21.15.28:0/2692628506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968100a90 msgr2=0x7f2968100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 --2- 172.21.15.28:0/2692628506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968100a90 0x7f2968100eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2964009ca0 tx=0x7f296402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 -- 172.21.15.28:0/2692628506 shutdown_connections 2024-03-22T02:13:18.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 --2- 172.21.15.28:0/2692628506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968102640 0x7f2968108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 --2- 172.21.15.28:0/2692628506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 --2- 172.21.15.28:0/2692628506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968100a90 0x7f2968100eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.605+0000 7f2970113640 1 -- 172.21.15.28:0/2692628506 >> 172.21.15.28:0/2692628506 conn(0x7f29680fc020 msgr2=0x7f29680fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- 172.21.15.28:0/2692628506 shutdown_connections 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- 172.21.15.28:0/2692628506 wait complete. 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 Processor -- start 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- start start 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 0x7f2968197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968102640 0x7f296819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f296810c710 con 0x7f2968101c80 2024-03-22T02:13:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f296810c590 con 0x7f2968100a90 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f2970113640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f296810c890 con 0x7f2968102640 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f296deaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 0x7f2968197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.606+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40978/0 (socket says 172.21.15.28:40978) 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 -- 172.21.15.28:0/1108106758 learned_addr learned my addr 172.21.15.28:0/1108106758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296e6b0640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968102640 0x7f296819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 -- 172.21.15.28:0/1108106758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968102640 msgr2=0x7f296819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968102640 0x7f296819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 -- 172.21.15.28:0/1108106758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 msgr2=0x7f2968197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 0x7f2968197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 -- 172.21.15.28:0/1108106758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296819faf0 con 0x7f2968101c80 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296deaf640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 0x7f2968197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.607+0000 7f296d6ae640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f295800e9f0 tx=0x7f295800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.609+0000 7f295effd640 1 -- 172.21.15.28:0/1108106758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295800cd80 con 0x7f2968101c80 2024-03-22T02:13:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.609+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296819fde0 con 0x7f2968101c80 2024-03-22T02:13:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.609+0000 7f295effd640 1 -- 172.21.15.28:0/1108106758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958010910 con 0x7f2968101c80 2024-03-22T02:13:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.609+0000 7f295effd640 1 -- 172.21.15.28:0/1108106758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29580188f0 con 0x7f2968101c80 2024-03-22T02:13:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.609+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29681a02f0 con 0x7f2968101c80 2024-03-22T02:13:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f295effd640 1 -- 172.21.15.28:0/1108106758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f2958018b70 con 0x7f2968101c80 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 Processor -- start 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 -- start start 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d670580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d670bf0 0x557f4d673080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f4d5cb210 con 0x557f4d670bf0 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f4d573c80 con 0x557f4d66b090 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f2972dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f4d5c71c0 con 0x557f4d66ba80 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296e6b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296deaf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d670580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296e6b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33966/0 (socket says 172.21.15.28:33966) 2024-03-22T02:13:18.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296deaf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d670580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34618/0 (socket says 172.21.15.28:34618) 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d670bf0 0x557f4d673080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.612+0000 7f296e6b0640 1 -- 172.21.15.28:0/2848005753 learned_addr learned my addr 172.21.15.28:0/2848005753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f295effd640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 0x7f29540a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 -- 172.21.15.28:0/2848005753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 msgr2=0x557f4d670580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d670580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 -- 172.21.15.28:0/2848005753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d670bf0 msgr2=0x557f4d673080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d670bf0 0x557f4d673080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 -- 172.21.15.28:0/2848005753 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f4d2df2d0 con 0x557f4d66b090 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 0x7f29540a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f295effd640 1 -- 172.21.15.28:0/1108106758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f2958026000 con 0x7f2968101c80 2024-03-22T02:13:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f296e6b0640 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2960009e20 tx=0x7f2960011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d209282ce6380bf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f2953fff640 1 -- 172.21.15.28:0/2848005753 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960017050 con 0x557f4d66b090 2024-03-22T02:13:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f2953fff640 1 -- 172.21.15.28:0/2848005753 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2960003c40 con 0x557f4d66b090 2024-03-22T02:13:18.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.613+0000 7f2953fff640 1 -- 172.21.15.28:0/2848005753 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296000e3b0 con 0x557f4d66b090 2024-03-22T02:13:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 -- 172.21.15.28:0/2848005753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 msgr2=0x557f4d66b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f296e6b0640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 0x7f29540a5470 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2960023000 tx=0x7f2960024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2960009e20 tx=0x7f2960011810 comp rx=0 tx=0).stop 2024-03-22T02:13:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 -- 172.21.15.28:0/2848005753 shutdown_connections 2024-03-22T02:13:18.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d670bf0 0x557f4d673080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d670580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2848005753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d66b090 0x557f4d66b4b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 -- 172.21.15.28:0/2848005753 >> 172.21.15.28:0/2848005753 conn(0x557f4d666600 msgr2=0x557f4d666a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 -- 172.21.15.28:0/2848005753 shutdown_connections 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.614+0000 7f2972dd9740 1 -- 172.21.15.28:0/2848005753 wait complete. 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 -- start start 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 Processor -- start 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f4d673ba0 con 0x557f4d66ba80 2024-03-22T02:13:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f4d586f80 con 0x557f4d670bf0 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f4d673d20 con 0x557f4d66b090 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f2972dd9740 10 client.0 _ll_register_callbacks cb 0x557f4d5c34f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40998/0 (socket says 172.21.15.28:40998) 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f296deaf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f296d6ae640 1 -- 172.21.15.28:0/361348951 learned_addr learned my addr 172.21.15.28:0/361348951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.615+0000 7f296e6b0640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 -- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 msgr2=0x557f4d675220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 -- 172.21.15.28:0/361348951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 msgr2=0x557f4d67c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 -- 172.21.15.28:0/361348951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f4d3a96b0 con 0x557f4d66ba80 2024-03-22T02:13:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296e6b0640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296deaf640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f296d6ae640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2958098400 tx=0x7f295809be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be97e1c4d071e2a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958018e60 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f29580cd520 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2960003c40 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f29580a4900 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f2972dd9740 10 client.15231 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557f4d67fb80 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2960003c40 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.616+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f29580173d0 con 0x557f4d66ba80 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 10 client.15231 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:18.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 10 client.15231 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 20 client.15231 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d682170 0x557f4d6835e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 4 client.15231 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f4d5d8750 con 0x557f4d682170 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f296deaf640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d682170 0x557f4d6835e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f296deaf640 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d682170 0x557f4d6835e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ee34278d8b549de0 server_cookie=3849aee479cc3896 in_seq=0 out_seq=0 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.617+0000 7f29517fa640 10 client.15231 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f29517fa640 1 -- 172.21.15.28:0/361348951 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f4d5d8750 con 0x557f4d682170 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f29517fa640 10 client.15231 handle_command_reply: tid=0 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 client.15231 shutdown 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 msgr2=0x557f4d675760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2958098400 tx=0x7f295809be30 comp rx=0 tx=0).stop 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 shutdown_connections 2024-03-22T02:13:18.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d682170 0x557f4d6835e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f29517fa640 0 client.15231 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 --2- 172.21.15.28:0/361348951 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 >> 172.21.15.28:0/361348951 conn(0x557f4d666600 msgr2=0x557f4d6614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 shutdown_connections 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 1 -- 172.21.15.28:0/361348951 wait complete. 2024-03-22T02:13:18.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.618+0000 7f2972dd9740 20 client.15231 trim_cache size 0 max 16384 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 Processor -- start 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 -- start start 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f4d681920 con 0x557f4d66b090 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f4d681ff0 con 0x557f4d670bf0 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f2972dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f4d6826c0 con 0x557f4d66ba80 2024-03-22T02:13:18.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41012/0 (socket says 172.21.15.28:41012) 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296d6ae640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 -- 172.21.15.28:0/2076010589 learned_addr learned my addr 172.21.15.28:0/2076010589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296e6b0640 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 -- 172.21.15.28:0/2076010589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 msgr2=0x557f4d675760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 -- 172.21.15.28:0/2076010589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 msgr2=0x557f4d67c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 -- 172.21.15.28:0/2076010589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f4d66be60 con 0x557f4d66b090 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296d6ae640 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.637+0000 7f296deaf640 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2964009ca0 tx=0x7f296402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d85af03f1340317 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f29537fe640 1 -- 172.21.15.28:0/2076010589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296404e070 con 0x557f4d66b090 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f29537fe640 1 -- 172.21.15.28:0/2076010589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2964032e50 con 0x557f4d66b090 2024-03-22T02:13:18.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f29537fe640 1 -- 172.21.15.28:0/2076010589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964039040 con 0x557f4d66b090 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 -- 172.21.15.28:0/2076010589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 msgr2=0x557f4d675220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2964009ca0 tx=0x7f296402e040 comp rx=0 tx=0).stop 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 -- 172.21.15.28:0/2076010589 shutdown_connections 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d675760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2076010589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d675220 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 -- 172.21.15.28:0/2076010589 >> 172.21.15.28:0/2076010589 conn(0x557f4d666600 msgr2=0x557f4d680ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 -- 172.21.15.28:0/2076010589 shutdown_connections 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.638+0000 7f2972dd9740 1 -- 172.21.15.28:0/2076010589 wait complete. 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 -- start start 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 Processor -- start 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d678ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f4d683460 con 0x557f4d66b090 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f4d682d90 con 0x557f4d670bf0 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f4d683b30 con 0x557f4d66ba80 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f2972dd9740 10 client.0 _ll_register_callbacks cb 0x7f29580982e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41016/0 (socket says 172.21.15.28:41016) 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 -- 172.21.15.28:0/2632901682 learned_addr learned my addr 172.21.15.28:0/2632901682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296d6ae640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d678ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296e6b0640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 -- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 msgr2=0x557f4d678ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d678ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 -- 172.21.15.28:0/2632901682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 msgr2=0x557f4d67b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.639+0000 7f296deaf640 1 -- 172.21.15.28:0/2632901682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f4d67bdb0 con 0x557f4d66b090 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f296deaf640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2964005e30 tx=0x7f296402c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460c63849682e040 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296404e070 con 0x557f4d66b090 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f2964032890 con 0x557f4d66b090 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2964032e50 con 0x557f4d66b090 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2964039040 con 0x557f4d66b090 2024-03-22T02:13:18.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 10 client.15237 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557f4d6e8200 con 0x557f4d66b090 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2964032e50 con 0x557f4d66b090 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2964054ac0 con 0x557f4d66b090 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 10 client.15237 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 10 client.15237 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 20 client.15237 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d6ec9f0 0x557f4d6eb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 4 client.15237 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.640+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "status"}) -- 0x557f4d5d2aa0 con 0x557f4d6ec9f0 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f296d6ae640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d6ec9f0 0x557f4d6eb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f296d6ae640 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d6ec9f0 0x557f4d6eb700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab2cd1ee6a647737 server_cookie=388dfb0f0083fca5 in_seq=0 out_seq=0 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f29517fa640 10 client.15237 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f29517fa640 1 -- 172.21.15.28:0/2632901682 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x557f4d5d2aa0 con 0x557f4d6ec9f0 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f29517fa640 10 client.15237 handle_command_reply: tid=0 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 client.15237 shutdown 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 msgr2=0x557f4d678990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2964005e30 tx=0x7f296402c8f0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 shutdown_connections 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557f4d6ec9f0 0x557f4d6eb700 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f4d670bf0 0x557f4d67b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4d66ba80 0x557f4d678ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 --2- 172.21.15.28:0/2632901682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4d66b090 0x557f4d678990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f29517fa640 0 client.15237 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 >> 172.21.15.28:0/2632901682 conn(0x557f4d666600 msgr2=0x557f4d66fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 shutdown_connections 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 1 -- 172.21.15.28:0/2632901682 wait complete. 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.641+0000 7f2972dd9740 20 client.15237 trim_cache size 0 max 16384 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24478, 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:18.650 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 84.661198224000003, 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 123.28336689699999 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 msgr2=0x7f29540a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 0x7f29540a5470 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2960023000 tx=0x7f2960024000 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 msgr2=0x7f2968198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f295800e9f0 tx=0x7f295800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 shutdown_connections 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29540a2fb0 0x7f29540a5470 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2968102640 0x7f296819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2968101c80 0x7f2968198410 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 --2- 172.21.15.28:0/1108106758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2968100a90 0x7f2968197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 >> 172.21.15.28:0/1108106758 conn(0x7f29680fc020 msgr2=0x7f29680fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 shutdown_connections 2024-03-22T02:13:18.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.642+0000 7f2970113640 1 -- 172.21.15.28:0/1108106758 wait complete. 2024-03-22T02:13:18.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-22T02:13:18.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3545956329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:18.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/206940952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:18.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:18 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4052074707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:18.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 -- 172.21.15.28:0/1686064083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518102e40 msgr2=0x7fb5181032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/1686064083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518102e40 0x7fb5181032c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009860 tx=0x7fb50c02bb70 comp rx=0 tx=0).stop 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 -- 172.21.15.28:0/1686064083 shutdown_connections 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/1686064083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb518103800 0x7fb51810a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/1686064083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518102e40 0x7fb5181032c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/1686064083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5180ffe70 0x7fb518100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 -- 172.21.15.28:0/1686064083 >> 172.21.15.28:0/1686064083 conn(0x7fb5180fb7a0 msgr2=0x7fb5180fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 -- 172.21.15.28:0/1686064083 shutdown_connections 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.758+0000 7fb51d9b5640 1 -- 172.21.15.28:0/1686064083 wait complete. 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 Processor -- start 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 -- start start 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb518102e40 0x7fb5181983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 0x7fb51819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb51810d850 con 0x7fb518102e40 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb51810d6d0 con 0x7fb518103800 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb51d9b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb51810d9d0 con 0x7fb5180ffe70 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 0x7fb51819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34666/0 (socket says 172.21.15.28:34666) 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb516ffd640 1 -- 172.21.15.28:0/2883229625 learned_addr learned my addr 172.21.15.28:0/2883229625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb518102e40 0x7fb5181983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 -- 172.21.15.28:0/2883229625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 msgr2=0x7fb518197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 -- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb518102e40 msgr2=0x7fb5181983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb518102e40 0x7fb5181983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb5177fe640 1 -- 172.21.15.28:0/2883229625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51819fa90 con 0x7fb518103800 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.759+0000 7fb516ffd640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb5177fe640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 0x7fb51819f420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb51000ef00 tx=0x7fb51000c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb51c9b3640 1 -- 172.21.15.28:0/2883229625 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb510004b20 con 0x7fb518103800 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb51c9b3640 1 -- 172.21.15.28:0/2883229625 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb510003e50 con 0x7fb518103800 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51819fd80 con 0x7fb518103800 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb51c9b3640 1 -- 172.21.15.28:0/2883229625 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51000b570 con 0x7fb518103800 2024-03-22T02:13:18.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.760+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5181a0240 con 0x7fb518103800 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.762+0000 7fb51c9b3640 1 -- 172.21.15.28:0/2883229625 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fb51001d020 con 0x7fb518103800 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.762+0000 7fb51c9b3640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 0x7fb5000a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.762+0000 7fb516ffd640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 0x7fb5000a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.762+0000 7fb51c9b3640 1 -- 172.21.15.28:0/2883229625 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb5100cf710 con 0x7fb518103800 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.763+0000 7fb516ffd640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 0x7fb5000a5470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5080058c0 tx=0x7fb508005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.763+0000 7fb52059f740 1 Processor -- start 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.763+0000 7fb52059f740 1 -- start start 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701bd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701bfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b27011c210 con 0x55b2701b8670 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b2700c4c80 con 0x55b2701bd7e0 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb52059f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2701181c0 con 0x55b2701b7c80 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701bd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701bd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41048/0 (socket says 172.21.15.28:41048) 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb5167fc640 1 -- 172.21.15.28:0/3906315672 learned_addr learned my addr 172.21.15.28:0/3906315672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb5177fe640 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701bfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 -- 172.21.15.28:0/3906315672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 msgr2=0x55b2701bfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701bfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 -- 172.21.15.28:0/3906315672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 msgr2=0x55b2701bd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701bd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 -- 172.21.15.28:0/3906315672 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b26fe302d0 con 0x55b2701b7c80 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb5177fe640 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.764+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701b80a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb508009fd0 tx=0x7fb508014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6868bab645d2cdf3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.765+0000 7fb4fdffb640 1 -- 172.21.15.28:0/3906315672 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb508005020 con 0x55b2701b7c80 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.765+0000 7fb4fdffb640 1 -- 172.21.15.28:0/3906315672 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5080047a0 con 0x55b2701b7c80 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.765+0000 7fb4fdffb640 1 -- 172.21.15.28:0/3906315672 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50801b700 con 0x55b2701b7c80 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.765+0000 7fb52059f740 1 -- 172.21.15.28:0/3906315672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 msgr2=0x55b2701b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.765+0000 7fb52059f740 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701b80a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb508009fd0 tx=0x7fb508014000 comp rx=0 tx=0).stop 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 -- 172.21.15.28:0/3906315672 shutdown_connections 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701bfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701bd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 --2- 172.21.15.28:0/3906315672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701b80a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 -- 172.21.15.28:0/3906315672 >> 172.21.15.28:0/3906315672 conn(0x55b2701b31f0 msgr2=0x55b2701b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 -- 172.21.15.28:0/3906315672 shutdown_connections 2024-03-22T02:13:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 -- 172.21.15.28:0/3906315672 wait complete. 2024-03-22T02:13:18.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 -- start start 2024-03-22T02:13:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.766+0000 7fb52059f740 1 Processor -- start 2024-03-22T02:13:18.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b8670 0x55b2701c23d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701bd7e0 0x55b2701c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2701c0790 con 0x55b2701b7c80 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b2700d7f80 con 0x55b2701b8670 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2701c0910 con 0x55b2701bd7e0 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41054/0 (socket says 172.21.15.28:41054) 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 -- 172.21.15.28:0/1294906663 learned_addr learned my addr 172.21.15.28:0/1294906663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 10 client.0 _ll_register_callbacks cb 0x55b2701144f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb5167fc640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b8670 0x55b2701c23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb5177fe640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701bd7e0 0x55b2701c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 -- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701bd7e0 msgr2=0x55b2701c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701bd7e0 0x55b2701c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 -- 172.21.15.28:0/1294906663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b8670 msgr2=0x55b2701c23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b8670 0x55b2701c23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 -- 172.21.15.28:0/1294906663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b26fefa6b0 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb516ffd640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb508027040 tx=0x7fb508004e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66027d57c2ddb2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb4f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb508032020 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb50801ba00 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.767+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5080047a0 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb50803d050 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 10 client.15249 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b2701cc710 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5080047a0 con 0x55b2701b7c80 2024-03-22T02:13:18.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb508024030 con 0x55b2701b7c80 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 10 client.15249 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 10 client.15249 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 20 client.15249 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b2701ceaf0 0x55b2701cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 4 client.15249 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b270129750 con 0x55b2701ceaf0 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.768+0000 7fb5167fc640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b2701ceaf0 0x55b2701cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.769+0000 7fb4f6ffd640 10 client.15249 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.769+0000 7fb5167fc640 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b2701ceaf0 0x55b2701cff60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1a96c881583096dc server_cookie=52dbabe7dc78c4a7 in_seq=0 out_seq=0 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/1294906663 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b270129750 con 0x55b2701ceaf0 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb4f6ffd640 10 client.15249 handle_command_reply: tid=0 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 client.15249 shutdown 2024-03-22T02:13:18.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 msgr2=0x55b2701c1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb508027040 tx=0x7fb508004e50 comp rx=0 tx=0).stop 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 shutdown_connections 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b2701ceaf0 0x55b2701cff60 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701bd7e0 0x55b2701c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb4f6ffd640 0 client.15249 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b8670 0x55b2701c23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 --2- 172.21.15.28:0/1294906663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b7c80 0x55b2701c1e90 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 >> 172.21.15.28:0/1294906663 conn(0x55b2701b31f0 msgr2=0x55b2701b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 shutdown_connections 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 1 -- 172.21.15.28:0/1294906663 wait complete. 2024-03-22T02:13:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.770+0000 7fb52059f740 20 client.15249 trim_cache size 0 max 16384 2024-03-22T02:13:18.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.807+0000 7fb52059f740 1 Processor -- start 2024-03-22T02:13:18.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.807+0000 7fb52059f740 1 -- start start 2024-03-22T02:13:18.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 0x55b2701c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 0x55b2701c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701bd7e0 0x55b2701c1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2701ce2a0 con 0x55b2701bd7e0 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b2701ce970 con 0x55b2701b7c80 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb52059f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2701cf040 con 0x55b2701b8670 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701bd7e0 0x55b2701c1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 0x55b2701c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 0x55b2701c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 0x55b2701c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34682/0 (socket says 172.21.15.28:34682) 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb5167fc640 1 -- 172.21.15.28:0/3859886395 learned_addr learned my addr 172.21.15.28:0/3859886395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 -- 172.21.15.28:0/3859886395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 msgr2=0x55b2701c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 0x55b2701c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 -- 172.21.15.28:0/3859886395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701bd7e0 msgr2=0x55b2701c1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701bd7e0 0x55b2701c1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 -- 172.21.15.28:0/3859886395 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2701ce0d0 con 0x55b2701b7c80 2024-03-22T02:13:18.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.808+0000 7fb516ffd640 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 0x55b2701c1aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb508011e50 tx=0x7fb50800f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef22942dd1d626ba server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.809+0000 7fb4fd7fa640 1 -- 172.21.15.28:0/3859886395 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb508032040 con 0x55b2701b7c80 2024-03-22T02:13:18.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.809+0000 7fb4fd7fa640 1 -- 172.21.15.28:0/3859886395 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb50800fd60 con 0x55b2701b7c80 2024-03-22T02:13:18.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.809+0000 7fb4fd7fa640 1 -- 172.21.15.28:0/3859886395 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb508004ec0 con 0x55b2701b7c80 2024-03-22T02:13:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- 172.21.15.28:0/3859886395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 msgr2=0x55b2701c1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 0x55b2701c1aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb508011e50 tx=0x7fb50800f610 comp rx=0 tx=0).stop 2024-03-22T02:13:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- 172.21.15.28:0/3859886395 shutdown_connections 2024-03-22T02:13:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701bd7e0 0x55b2701c1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b8670 0x55b2701c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- 172.21.15.28:0/3859886395 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701b7c80 0x55b2701c1aa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- 172.21.15.28:0/3859886395 >> 172.21.15.28:0/3859886395 conn(0x55b2701b31f0 msgr2=0x55b2701b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- 172.21.15.28:0/3859886395 shutdown_connections 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- 172.21.15.28:0/3859886395 wait complete. 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 -- start start 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 Processor -- start 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701c4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.810+0000 7fb52059f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701c7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b2701cfde0 con 0x55b2701b8670 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b2701cf710 con 0x55b2701bd7e0 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b2701d04b0 con 0x55b2701b7c80 2024-03-22T02:13:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 10 client.0 _ll_register_callbacks cb 0x7fb508005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41080/0 (socket says 172.21.15.28:41080) 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5177fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701c7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 -- 172.21.15.28:0/4001587367 learned_addr learned my addr 172.21.15.28:0/4001587367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb516ffd640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 -- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 msgr2=0x55b2701c4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701c4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 -- 172.21.15.28:0/4001587367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 msgr2=0x55b2701c7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701c7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 -- 172.21.15.28:0/4001587367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2701c80d0 con 0x55b2701b8670 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb5167fc640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0095f0 tx=0x7fb50c005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bc2acdd24c83f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb4f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c039040 con 0x55b2701b8670 2024-03-22T02:13:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fb50c004040 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb50800fd60 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb50c059050 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.811+0000 7fb52059f740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b2701c9440 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb50800fd60 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fb50c034ce0 con 0x55b2701b8670 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 10 client.15255 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 20 client.15255 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b270239020 0x55b270237d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 4 client.15255 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "status"}) -- 0x55b270123aa0 con 0x55b270239020 2024-03-22T02:13:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb516ffd640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b270239020 0x55b270237d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb4f6ffd640 10 client.15255 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.812+0000 7fb516ffd640 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b270239020 0x55b270237d30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d4c901fc5e22333e server_cookie=b6979837dab970ac in_seq=0 out_seq=0 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb4f6ffd640 1 -- 172.21.15.28:0/4001587367 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b270123aa0 con 0x55b270239020 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb4f6ffd640 10 client.15255 handle_command_reply: tid=0 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 client.15255 shutdown 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 msgr2=0x55b2701c4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb50c0095f0 tx=0x7fb50c005900 comp rx=0 tx=0).stop 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 shutdown_connections 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b270239020 0x55b270237d30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b2701bd7e0 0x55b2701c7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b2701b8670 0x55b2701c4ac0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 --2- 172.21.15.28:0/4001587367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b2701b7c80 0x55b2701c4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb4f6ffd640 0 client.15255 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 >> 172.21.15.28:0/4001587367 conn(0x55b2701b31f0 msgr2=0x55b2701b6450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 shutdown_connections 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 1 -- 172.21.15.28:0/4001587367 wait complete. 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.813+0000 7fb52059f740 20 client.15255 trim_cache size 0 max 16384 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24508, 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:18.820 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 85.837953583000001, 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 117.621476637 2024-03-22T02:13:18.821 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 msgr2=0x7fb5000a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 0x7fb5000a5470 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb5080058c0 tx=0x7fb508005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 msgr2=0x7fb51819f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 0x7fb51819f420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb51000ef00 tx=0x7fb51000c740 comp rx=0 tx=0).stop 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 shutdown_connections 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5000a2fb0 0x7fb5000a5470 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb518103800 0x7fb51819f420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb518102e40 0x7fb5181983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 --2- 172.21.15.28:0/2883229625 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5180ffe70 0x7fb518197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 >> 172.21.15.28:0/2883229625 conn(0x7fb5180fb7a0 msgr2=0x7fb518106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 shutdown_connections 2024-03-22T02:13:18.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.814+0000 7fb51d9b5640 1 -- 172.21.15.28:0/2883229625 wait complete. 2024-03-22T02:13:18.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.969+0000 7fe722271640 1 -- 172.21.15.28:0/337275483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 msgr2=0x7fe71c108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.969+0000 7fe722271640 1 --2- 172.21.15.28:0/337275483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c108ed0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe714009ca0 tx=0x7fe71402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 -- 172.21.15.28:0/337275483 shutdown_connections 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- 172.21.15.28:0/337275483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c108ed0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- 172.21.15.28:0/337275483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c101c80 0x7fe71c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- 172.21.15.28:0/337275483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c100a90 0x7fe71c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 -- 172.21.15.28:0/337275483 >> 172.21.15.28:0/337275483 conn(0x7fe71c0fc020 msgr2=0x7fe71c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 -- 172.21.15.28:0/337275483 shutdown_connections 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 -- 172.21.15.28:0/337275483 wait complete. 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 Processor -- start 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 -- start start 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 0x7fe71c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.970+0000 7fe722271640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe722271640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c10c6f0 con 0x7fe71c102640 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe722271640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c10c570 con 0x7fe71c100a90 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe722271640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe71c10c870 con 0x7fe71c101c80 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34060/0 (socket says 172.21.15.28:34060) 2024-03-22T02:13:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41092/0 (socket says 172.21.15.28:41092) 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71bfff640 1 -- 172.21.15.28:0/1560068132 learned_addr learned my addr 172.21.15.28:0/1560068132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 0x7fe71c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 -- 172.21.15.28:0/1560068132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 msgr2=0x7fe71c19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 0x7fe71c19cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 -- 172.21.15.28:0/1560068132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 msgr2=0x7fe71c0698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 -- 172.21.15.28:0/1560068132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71c19fa80 con 0x7fe71c102640 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 0x7fe71c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe72080e640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe71402c630 tx=0x7fe714032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe7197fa640 1 -- 172.21.15.28:0/1560068132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714041020 con 0x7fe71c102640 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71c19fd10 con 0x7fe71c102640 2024-03-22T02:13:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe7197fa640 1 -- 172.21.15.28:0/1560068132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe714034cd0 con 0x7fe71c102640 2024-03-22T02:13:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe7197fa640 1 -- 172.21.15.28:0/1560068132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714039040 con 0x7fe71c102640 2024-03-22T02:13:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.971+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe71c1a01d0 con 0x7fe71c102640 2024-03-22T02:13:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe7197fa640 1 -- 172.21.15.28:0/1560068132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7fe714044020 con 0x7fe71c102640 2024-03-22T02:13:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 Processor -- start 2024-03-22T02:13:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 -- start start 2024-03-22T02:13:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe7197fa640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 0x7fe7040a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c4670 0x560b908c9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c97e0 0x560b908cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b90828210 con 0x560b908c4670 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560b907d0c80 con 0x560b908c97e0 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe724e56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560b908241c0 con 0x560b908c3c80 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe7197fa640 1 -- 172.21.15.28:0/1560068132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe7140f1540 con 0x7fe71c102640 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 0x7fe7040a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34726/0 (socket says 172.21.15.28:34726) 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe71bfff640 1 -- 172.21.15.28:0/1836120374 learned_addr learned my addr 172.21.15.28:0/1836120374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c4670 0x560b908c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.974+0000 7fe72080e640 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c97e0 0x560b908cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 -- 172.21.15.28:0/1836120374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c97e0 msgr2=0x560b908cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c97e0 0x560b908cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 -- 172.21.15.28:0/1836120374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c4670 msgr2=0x560b908c9170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c4670 0x560b908c9170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 -- 172.21.15.28:0/1836120374 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b9053c2d0 con 0x560b908c3c80 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 0x7fe7040a5470 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710009ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe70c0115f0 tx=0x7fe70c0119f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdbe524cbd240114 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe7027fc640 1 -- 172.21.15.28:0/1836120374 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c01e070 con 0x560b908c3c80 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe7027fc640 1 -- 172.21.15.28:0/1836120374 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe70c00fac0 con 0x560b908c3c80 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.975+0000 7fe7027fc640 1 -- 172.21.15.28:0/1836120374 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c019b30 con 0x560b908c3c80 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- 172.21.15.28:0/1836120374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 msgr2=0x560b908c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe70c0115f0 tx=0x7fe70c0119f0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- 172.21.15.28:0/1836120374 shutdown_connections 2024-03-22T02:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c97e0 0x560b908cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c4670 0x560b908c9170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 --2- 172.21.15.28:0/1836120374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908c40a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- 172.21.15.28:0/1836120374 >> 172.21.15.28:0/1836120374 conn(0x560b908bf1f0 msgr2=0x560b908bf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- 172.21.15.28:0/1836120374 shutdown_connections 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- 172.21.15.28:0/1836120374 wait complete. 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 -- start start 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.976+0000 7fe724e56740 1 Processor -- start 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b908cc790 con 0x560b908c97e0 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560b907e3f80 con 0x560b908c4670 2024-03-22T02:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560b908cc910 con 0x560b908c3c80 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe724e56740 10 client.0 _ll_register_callbacks cb 0x560b908204f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34742/0 (socket says 172.21.15.28:34742) 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 -- 172.21.15.28:0/1113081462 learned_addr learned my addr 172.21.15.28:0/1113081462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe72080e640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 -- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 msgr2=0x560b908ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 -- 172.21.15.28:0/1113081462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 msgr2=0x560b908d5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 -- 172.21.15.28:0/1113081462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b906066b0 con 0x560b908c3c80 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01c040 tx=0x7fe70c00f660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f469d24ecd41fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.977+0000 7fe6fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c01e040 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe70c00f850 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe70c00fac0 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe70c019890 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 10 client.25093 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560b908d8790 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe70c00fac0 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe70c021070 con 0x560b908c3c80 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 10 client.25093 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 10 client.25093 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 20 client.25093 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b908dabe0 0x560b908dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 4 client.25093 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b90835750 con 0x560b908dabe0 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.978+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b908dabe0 0x560b908dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.979+0000 7fe6fb7fe640 10 client.25093 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.979+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b908dabe0 0x560b908dc050 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4041e41ccac7ce3f server_cookie=44adf44fde4e9a86 in_seq=0 out_seq=0 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.979+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1113081462 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b90835750 con 0x560b908dabe0 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.979+0000 7fe6fb7fe640 10 client.25093 handle_command_reply: tid=0 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.979+0000 7fe724e56740 1 client.25093 shutdown 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 msgr2=0x560b908cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe70c01c040 tx=0x7fe70c00f660 comp rx=0 tx=0).stop 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 shutdown_connections 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b908dabe0 0x560b908dc050 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe6fb7fe640 0 client.25093 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 --2- 172.21.15.28:0/1113081462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908cdea0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 >> 172.21.15.28:0/1113081462 conn(0x560b908bf1f0 msgr2=0x560b908be4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 shutdown_connections 2024-03-22T02:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 1 -- 172.21.15.28:0/1113081462 wait complete. 2024-03-22T02:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.980+0000 7fe724e56740 20 client.25093 trim_cache size 0 max 16384 2024-03-22T02:13:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:18 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3545956329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:18 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/206940952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:18 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4052074707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:19.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 Processor -- start 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 -- start start 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908bf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b908da390 con 0x560b908c97e0 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560b908daa60 con 0x560b908c4670 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe724e56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560b908db130 con 0x560b908c3c80 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41114/0 (socket says 172.21.15.28:41114) 2024-03-22T02:13:19.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 -- 172.21.15.28:0/933277888 learned_addr learned my addr 172.21.15.28:0/933277888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe71bfff640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908bf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 -- 172.21.15.28:0/933277888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 msgr2=0x560b908bf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908bf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 -- 172.21.15.28:0/933277888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 msgr2=0x560b908d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe72080e640 1 -- 172.21.15.28:0/933277888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b908ba380 con 0x560b908c97e0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.998+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe72080e640 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7140b9dd0 tx=0x7fe71402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9335472a50d3b454 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe701ffb640 1 -- 172.21.15.28:0/933277888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7140be070 con 0x560b908c97e0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe701ffb640 1 -- 172.21.15.28:0/933277888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7140098b0 con 0x560b908c97e0 2024-03-22T02:13:19.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe701ffb640 1 -- 172.21.15.28:0/933277888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe714003970 con 0x560b908c97e0 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 -- 172.21.15.28:0/933277888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 msgr2=0x560b908c5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe7140b9dd0 tx=0x7fe71402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 -- 172.21.15.28:0/933277888 shutdown_connections 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908c5110 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 --2- 172.21.15.28:0/933277888 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908bf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 -- 172.21.15.28:0/933277888 >> 172.21.15.28:0/933277888 conn(0x560b908bf1f0 msgr2=0x560b908c2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:18.999+0000 7fe724e56740 1 -- 172.21.15.28:0/933277888 shutdown_connections 2024-03-22T02:13:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 -- 172.21.15.28:0/933277888 wait complete. 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 -- start start 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 Processor -- start 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560b908dbed0 con 0x560b908c97e0 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560b908db800 con 0x560b908c4670 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560b908dc5a0 con 0x560b908c3c80 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe724e56740 10 client.0 _ll_register_callbacks cb 0x7fe70c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe72080e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41124/0 (socket says 172.21.15.28:41124) 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe72080e640 1 -- 172.21.15.28:0/1717882777 learned_addr learned my addr 172.21.15.28:0/1717882777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe71bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34770/0 (socket says 172.21.15.28:34770) 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.000+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 -- 172.21.15.28:0/1717882777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 msgr2=0x560b908d3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:19.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 -- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 msgr2=0x560b908d3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 -- 172.21.15.28:0/1717882777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b908d49c0 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe71b7fe640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe72080e640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe7140bd670 tx=0x7fe7140ef030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32727245a58f4fa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7140c62f0 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7fe7140c5e60 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7140098b0 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe7140be070 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe724e56740 10 client.15273 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560b90940cb0 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7140098b0 con 0x560b908c97e0 2024-03-22T02:13:19.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.001+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fe71403c030 con 0x560b908c97e0 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 10 client.15273 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 10 client.15273 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 20 client.15273 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b909455f0 0x560b90944300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 4 client.15273 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "status"}) -- 0x560b9082faa0 con 0x560b909455f0 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b909455f0 0x560b90944300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe6fb7fe640 10 client.15273 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe71bfff640 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b909455f0 0x560b90944300 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2062c3f266d21e78 server_cookie=c1c25fbdf9313e29 in_seq=0 out_seq=0 2024-03-22T02:13:19.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe6fb7fe640 1 -- 172.21.15.28:0/1717882777 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560b9082faa0 con 0x560b909455f0 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe6fb7fe640 10 client.15273 handle_command_reply: tid=0 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 client.15273 shutdown 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 msgr2=0x560b908d4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe7140bd670 tx=0x7fe7140ef030 comp rx=0 tx=0).stop 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 shutdown_connections 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560b909455f0 0x560b90944300 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b908c97e0 0x560b908d4190 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560b908c4670 0x560b908d3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe6fb7fe640 0 client.15273 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 --2- 172.21.15.28:0/1717882777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560b908c3c80 0x560b908d3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.002+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 >> 172.21.15.28:0/1717882777 conn(0x560b908bf1f0 msgr2=0x560b908c2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 shutdown_connections 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe724e56740 1 -- 172.21.15.28:0/1717882777 wait complete. 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe724e56740 20 client.15273 trim_cache size 0 max 16384 2024-03-22T02:13:19.017 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:19.018 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 105.10756548000001, 2024-03-22T02:13:19.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:19.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 92, 2024-03-22T02:13:19.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:19.019 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 109.32321389400001 2024-03-22T02:13:19.019 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 msgr2=0x7fe7040a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:19.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 0x7fe7040a5470 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe710009ee0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 msgr2=0x7fe71c19f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe71402c630 tx=0x7fe714032f20 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 shutdown_connections 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7040a2fb0 0x7fe7040a5470 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe71c102640 0x7fe71c19f1c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe71c101c80 0x7fe71c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 --2- 172.21.15.28:0/1560068132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe71c100a90 0x7fe71c0698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 >> 172.21.15.28:0/1560068132 conn(0x7fe71c0fc020 msgr2=0x7fe71c104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 shutdown_connections 2024-03-22T02:13:19.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:19.003+0000 7fe722271640 1 -- 172.21.15.28:0/1560068132 wait complete. 2024-03-22T02:13:19.022 INFO:tasks.cephadm:Setup complete, yielding 2024-03-22T02:13:19.022 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:13:19.033 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:13:19.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-22T02:13:19.033 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:18 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3545956329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:19.034 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:18 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/206940952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:19.034 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:18 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4052074707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:19.277 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:19.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:19 smithi028 ceph-mon[31916]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:19 smithi120 ceph-mon[36803]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:20.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:19 smithi083 ceph-mon[36340]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 -- 172.21.15.28:0/1794933187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88104280 msgr2=0x7f6b8810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 --2- 172.21.15.28:0/1794933187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88104280 0x7f6b8810ab30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b0d0 tx=0x7f6b8000b3e0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 -- 172.21.15.28:0/1794933187 shutdown_connections 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 --2- 172.21.15.28:0/1794933187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b88104280 0x7f6b8810ab30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 --2- 172.21.15.28:0/1794933187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b881026e0 0x7f6b88102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 --2- 172.21.15.28:0/1794933187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881038e0 0x7f6b88103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 -- 172.21.15.28:0/1794933187 >> 172.21.15.28:0/1794933187 conn(0x7f6b880fdc80 msgr2=0x7f6b881000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 -- 172.21.15.28:0/1794933187 shutdown_connections 2024-03-22T02:13:21.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.054+0000 7f6b90750700 1 -- 172.21.15.28:0/1794933187 wait complete. 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 Processor -- start 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 -- start start 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 0x7f6b8819fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810e2b0 con 0x7f6b881038e0 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810e130 con 0x7f6b881026e0 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b90750700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b8810e430 con 0x7f6b88104280 2024-03-22T02:13:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.055+0000 7f6b8dceb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34116/0 (socket says 172.21.15.28:34116) 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8dceb700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41136/0 (socket says 172.21.15.28:41136) 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 -- 172.21.15.28:0/1605486687 learned_addr learned my addr 172.21.15.28:0/1605486687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8eced700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 0x7f6b8819fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 -- 172.21.15.28:0/1605486687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 msgr2=0x7f6b8819fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 0x7f6b8819fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 -- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 msgr2=0x7f6b88198d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 -- 172.21.15.28:0/1605486687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b881a0470 con 0x7f6b881026e0 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8eced700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 0x7f6b8819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8dceb700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b8e4ec700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e970 tx=0x7f6b7c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c004220 con 0x7f6b881026e0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b881a0760 con 0x7f6b881026e0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b7c0043c0 con 0x7f6b881026e0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c003df0 con 0x7f6b881026e0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.056+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b881a0ca0 con 0x7f6b881026e0 2024-03-22T02:13:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.057+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b88069340 con 0x7f6b881026e0 2024-03-22T02:13:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.058+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f6b7c00c7e0 con 0x7f6b881026e0 2024-03-22T02:13:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.059+0000 7f6b7b7fe700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 0x7f6b740a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.059+0000 7f6b8dceb700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 0x7f6b740a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.059+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f6b7c0cfa90 con 0x7f6b881026e0 2024-03-22T02:13:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.059+0000 7f6b8dceb700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 0x7f6b740a5500 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6b84011490 tx=0x7f6b84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.063+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b7c09c3c0 con 0x7f6b881026e0 2024-03-22T02:13:21.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.286+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f6b88062ac0 con 0x7f6b740a3040 2024-03-22T02:13:21.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.287+0000 7f6b7b7fe700 1 -- 172.21.15.28:0/1605486687 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f6b88062ac0 con 0x7f6b740a3040 2024-03-22T02:13:21.288 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-03-22T02:13:21.288 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-03-22T02:13:21.288 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 msgr2=0x7f6b740a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 0x7f6b740a5500 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6b84011490 tx=0x7f6b84009260 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 msgr2=0x7f6b88198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00e970 tx=0x7f6b7c00ec80 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 shutdown_connections 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b88104280 0x7f6b8819fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b881026e0 0x7f6b88198810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b740a3040 0x7f6b740a5500 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 --2- 172.21.15.28:0/1605486687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b881038e0 0x7f6b88198d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:21.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 >> 172.21.15.28:0/1605486687 conn(0x7f6b880fdc80 msgr2=0x7f6b88106b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:21.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 shutdown_connections 2024-03-22T02:13:21.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:21.289+0000 7f6b90750700 1 -- 172.21.15.28:0/1605486687 wait complete. 2024-03-22T02:13:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:21 smithi120 ceph-mon[36803]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:21 smithi120 ceph-mon[36803]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:22.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:21 smithi083 ceph-mon[36340]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:22.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:21 smithi083 ceph-mon[36340]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:21 smithi028 ceph-mon[31916]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-22T02:13:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:21 smithi028 ceph-mon[31916]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:22.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-22T02:13:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 -- 172.21.15.28:0/2316292147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01038d0 msgr2=0x7f72b0103d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 --2- 172.21.15.28:0/2316292147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01038d0 0x7f72b0103d30 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f72a40099c0 tx=0x7f72a4009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 -- 172.21.15.28:0/2316292147 shutdown_connections 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 --2- 172.21.15.28:0/2316292147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b0104270 0x7f72b010ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 --2- 172.21.15.28:0/2316292147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b01026d0 0x7f72b0102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 --2- 172.21.15.28:0/2316292147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01038d0 0x7f72b0103d30 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 -- 172.21.15.28:0/2316292147 >> 172.21.15.28:0/2316292147 conn(0x7f72b00fdc90 msgr2=0x7f72b01000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 -- 172.21.15.28:0/2316292147 shutdown_connections 2024-03-22T02:13:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.287+0000 7f72b4c94700 1 -- 172.21.15.28:0/2316292147 wait complete. 2024-03-22T02:13:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:24 smithi028 ceph-mon[31916]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 Processor -- start 2024-03-22T02:13:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 -- start start 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b01038d0 0x7f72b019b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b0104270 0x7f72b01a2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b010e2c0 con 0x7f72b01026d0 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f72b010e140 con 0x7f72b01038d0 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72b4c94700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f72b010e440 con 0x7f72b0104270 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42704/0 (socket says 172.21.15.28:42704) 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 -- 172.21.15.28:0/3790189028 learned_addr learned my addr 172.21.15.28:0/3790189028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72aed9d700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b0104270 0x7f72b01a2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72add9b700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b01038d0 0x7f72b019b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 -- 172.21.15.28:0/3790189028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b0104270 msgr2=0x7f72b01a2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b0104270 0x7f72b01a2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 -- 172.21.15.28:0/3790189028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b01038d0 msgr2=0x7f72b019b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b01038d0 0x7f72b019b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.288+0000 7f72ae59c700 1 -- 172.21.15.28:0/3790189028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b01a2720 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f72ae59c700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f72a000e980 tx=0x7f72a000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0004230 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a00043d0 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b01a2a10 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0003dc0 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.289+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b01a2f00 con 0x7f72b01026d0 2024-03-22T02:13:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.290+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7290005330 con 0x7f72b01026d0 2024-03-22T02:13:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.291+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f72a000c810 con 0x7f72b01026d0 2024-03-22T02:13:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.292+0000 7f729f7fe700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 0x7f72980a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.292+0000 7f72add9b700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 0x7f72980a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.292+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f72a00cfa60 con 0x7f72b01026d0 2024-03-22T02:13:24.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.292+0000 7f72add9b700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 0x7f72980a5500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f72a4000c00 tx=0x7f72a4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:24.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.297+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72a009c3f0 con 0x7f72b01026d0 2024-03-22T02:13:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:24 smithi120 ceph-mon[36803]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.511+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f7290000bc0 con 0x7f72980a3040 2024-03-22T02:13:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:24 smithi083 ceph-mon[36340]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.525+0000 7f729f7fe700 1 -- 172.21.15.28:0/3790189028 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f7290000bc0 con 0x7f72980a3040 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (118s) 23s ago 117s 15.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed ff56731f8d2d 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi083 running (115s) 48s ago 114s 26.0M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 9e6641025827 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi120 running (2m) 48s ago 2m 18.6M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 78642f610968 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 23s ago 2m 13.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 86d0234f3b3a 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi083 running (2m) 48s ago 2m 16.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed bf52c6033e90 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi120 running (2m) 48s ago 2m 18.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 52a4e0b4566c 2024-03-22T02:13:24.526 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 23s ago 2m 15.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 13d3f9505bd6 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi083 running (2m) 48s ago 2m 19.7M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 174bfe21ccc3 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi120 running (2m) 48s ago 2m 17.8M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 102ba44f91bb 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 23s ago 2m 15.2M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 731ee3bdf871 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi083 running (2m) 48s ago 2m 15.4M - 19.0.0-2344-gc3a04c0e 9d92c22378ed e2cc96f1bf02 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi120 running (2m) 48s ago 2m 14.1M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 118c41b9f894 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (9m) 23s ago 9m 514M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 035fb819d0bd 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi083 *:8443,8765 running (7m) 48s ago 7m 454M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 9bff5d61bdce 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi120 *:8443,8765 running (7m) 48s ago 7m 452M - 19.0.0-2344-gc3a04c0e 9d92c22378ed 481400a94c66 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (9m) 23s ago 10m 50.7M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 9bbba678bd3f 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi083 running (8m) 48s ago 8m 40.9M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed a41eea566d9b 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi120 running (8m) 48s ago 8m 41.6M 2048M 19.0.0-2344-gc3a04c0e 9d92c22378ed 94c70714e4f0 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (7m) 23s ago 7m 57.8M 4273M 19.0.0-2344-gc3a04c0e 9d92c22378ed 06051385962d 2024-03-22T02:13:24.527 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi083 running (7m) 48s ago 7m 46.3M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed ec430ad646e2 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi120 running (6m) 48s ago 6m 49.8M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed a0b7d7f521e7 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 23s ago 6m 56.3M 4273M 19.0.0-2344-gc3a04c0e 9d92c22378ed 745ecd519fdf 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi083 running (5m) 48s ago 5m 51.0M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed f7e492e9f1a1 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi120 running (5m) 48s ago 5m 47.5M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 3759941b4ac8 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (4m) 23s ago 4m 41.7M 4273M 19.0.0-2344-gc3a04c0e 9d92c22378ed b9c672cd5d6c 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi083 running (4m) 48s ago 4m 47.2M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed b3d2882814f7 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi120 running (4m) 48s ago 4m 43.6M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed f82ae8dd55bb 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 23s ago 3m 42.0M 4273M 19.0.0-2344-gc3a04c0e 9d92c22378ed 7a5f252af47d 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi083 running (3m) 48s ago 3m 56.1M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed b86a0921af6a 2024-03-22T02:13:24.528 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi120 running (2m) 48s ago 2m 52.0M 4270M 19.0.0-2344-gc3a04c0e 9d92c22378ed 191406d335a9 2024-03-22T02:13:24.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 msgr2=0x7f72980a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:24.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 0x7f72980a5500 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f72a4000c00 tx=0x7f72a4005fb0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 msgr2=0x7f72b019aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f72a000e980 tx=0x7f72a000ec90 comp rx=0 tx=0).stop 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 shutdown_connections 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72980a3040 0x7f72980a5500 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b01026d0 0x7f72b019aac0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.527+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f72b0104270 0x7f72b01a2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.528+0000 7f72b4c94700 1 --2- 172.21.15.28:0/3790189028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f72b01038d0 0x7f72b019b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.528+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 >> 172.21.15.28:0/3790189028 conn(0x7f72b00fdc90 msgr2=0x7f72b00748b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.528+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 shutdown_connections 2024-03-22T02:13:24.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:24.528+0000 7f72b4c94700 1 -- 172.21.15.28:0/3790189028 wait complete. 2024-03-22T02:13:25.475 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:25 smithi028 ceph-mon[31916]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:25.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-22T02:13:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:25 smithi120 ceph-mon[36803]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:25.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:25 smithi083 ceph-mon[36340]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:26 smithi120 ceph-mon[36803]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:26 smithi083 ceph-mon[36340]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:26.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:26 smithi028 ceph-mon[31916]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:13:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 -- 172.21.15.28:0/3921162292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 msgr2=0x7f3938074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 --2- 172.21.15.28:0/3921162292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f3938074b10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3934009990 tx=0x7f3934009ca0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 -- 172.21.15.28:0/3921162292 shutdown_connections 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 --2- 172.21.15.28:0/3921162292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f3938074b10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 --2- 172.21.15.28:0/3921162292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 0x7f3938111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 --2- 172.21.15.28:0/3921162292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f3938075560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 -- 172.21.15.28:0/3921162292 >> 172.21.15.28:0/3921162292 conn(0x7f39380fdc60 msgr2=0x7f3938100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 -- 172.21.15.28:0/3921162292 shutdown_connections 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.472+0000 7f394066a700 1 -- 172.21.15.28:0/3921162292 wait complete. 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 Processor -- start 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 -- start start 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f393819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 0x7f39381a64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3938114860 con 0x7f39380766b0 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39381146e0 con 0x7f39380750e0 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f394066a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39381149e0 con 0x7f393810ec80 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393dc05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393dc05700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42734/0 (socket says 172.21.15.28:42734) 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393dc05700 1 -- 172.21.15.28:0/1581487942 learned_addr learned my addr 172.21.15.28:0/1581487942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f393819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393ec07700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 0x7f39381a64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 -- 172.21.15.28:0/1581487942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 msgr2=0x7f39381a64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 0x7f39381a64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 -- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 msgr2=0x7f393819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393e406700 1 -- 172.21.15.28:0/1581487942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39381a6b50 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.473+0000 7f393dc05700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f393e406700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f393819eef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3934000c00 tx=0x7f393400bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934026070 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f393400f460 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934021670 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39381a6de0 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.474+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39381a7180 con 0x7f39380750e0 2024-03-22T02:13:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.475+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f393802dd70 con 0x7f39380750e0 2024-03-22T02:13:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.476+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 17) ==== 137905+0+0 (secure 0 0 0) 0x7f3934029070 con 0x7f39380750e0 2024-03-22T02:13:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.476+0000 7f392f7fe700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 0x7f39240a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.476+0000 7f393dc05700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 0x7f39240a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.476+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9071+0+0 (secure 0 0 0) 0x7f39340d08e0 con 0x7f39380750e0 2024-03-22T02:13:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.477+0000 7f393dc05700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 0x7f39240a5500 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3930003ef0 tx=0x7f393000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.480+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3934013070 con 0x7f39380750e0 2024-03-22T02:13:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.695+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f3938062ac0 con 0x7f39240a3040 2024-03-22T02:13:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.701+0000 7f392f7fe700 1 -- 172.21.15.28:0/1581487942 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f3938062ac0 con 0x7f39240a3040 2024-03-22T02:13:27.701 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-22T02:13:27.701 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 52s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi083=b;smithi083=e;smithi083=h;smithi083=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-22T02:13:27.702 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 52s ago 7m smithi028=x;smithi083=y;smithi120=z;count:3 2024-03-22T02:13:27.702 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 52s ago 8m smithi028:172.21.15.28=a;smithi083:172.21.15.83=b;smithi120:172.21.15.120=c;count:3 2024-03-22T02:13:27.702 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 52s ago - 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 msgr2=0x7f39240a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 0x7f39240a5500 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3930003ef0 tx=0x7f393000b040 comp rx=0 tx=0).stop 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 msgr2=0x7f393819eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f393819eef0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3934000c00 tx=0x7f393400bbb0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 shutdown_connections 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f393810ec80 0x7f39381a64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f39380750e0 0x7f393819eef0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39240a3040 0x7f39240a5500 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 --2- 172.21.15.28:0/1581487942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39380766b0 0x7f393819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 >> 172.21.15.28:0/1581487942 conn(0x7f39380fdc60 msgr2=0x7f3938102020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 shutdown_connections 2024-03-22T02:13:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:27.703+0000 7f394066a700 1 -- 172.21.15.28:0/1581487942 wait complete. 2024-03-22T02:13:28.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-22T02:13:28.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-22T02:13:28.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-22T02:13:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-22T02:13:28.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-22T02:13:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: pgmap v352: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-22T02:13:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-03-22T02:13:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:29.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:29.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T02:13:29.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-22T02:13:29.488 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-22T02:13:29.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:29 smithi028 ceph-mon[31916]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T02:13:29.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:29.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T02:13:29.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-22T02:13:29.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-22T02:13:29.500 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:29 smithi120 ceph-mon[36803]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='client.25133 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-22T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-22T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-22T02:13:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-22T02:13:29.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-03-22T02:13:29.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-22T02:13:29.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-03-22T02:13:29.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-22T02:13:29.516 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:29 smithi083 ceph-mon[36340]: osdmap e93: 12 total, 12 up, 12 in 2024-03-22T02:13:30.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:30 smithi028 ceph-mon[31916]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-22T02:13:30.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:30 smithi028 ceph-mon[31916]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T02:13:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:30 smithi120 ceph-mon[36803]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-22T02:13:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:30 smithi120 ceph-mon[36803]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T02:13:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:30 smithi083 ceph-mon[36340]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-22T02:13:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:30 smithi083 ceph-mon[36340]: osdmap e94: 12 total, 12 up, 12 in 2024-03-22T02:13:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 -- 172.21.15.28:0/4187994831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b41027b0 msgr2=0x7f79b4102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 --2- 172.21.15.28:0/4187994831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b41027b0 0x7f79b4102c10 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79a80099c0 tx=0x7f79a8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 -- 172.21.15.28:0/4187994831 shutdown_connections 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 --2- 172.21.15.28:0/4187994831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b4103150 0x7f79b4109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 --2- 172.21.15.28:0/4187994831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41015b0 0x7f79b41019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 --2- 172.21.15.28:0/4187994831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b41027b0 0x7f79b4102c10 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 -- 172.21.15.28:0/4187994831 >> 172.21.15.28:0/4187994831 conn(0x7f79b40fcb50 msgr2=0x7f79b40fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 -- 172.21.15.28:0/4187994831 shutdown_connections 2024-03-22T02:13:30.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.566+0000 7f79bc1be700 1 -- 172.21.15.28:0/4187994831 wait complete. 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 Processor -- start 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 -- start start 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 0x7f79b4198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 0x7f79b4198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410d1f0 con 0x7f79b4103150 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410d070 con 0x7f79b41027b0 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79bc1be700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f79b410d370 con 0x7f79b41015b0 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79b9f5a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 0x7f79b4198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42748/0 (socket says 172.21.15.28:42748) 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79b9f5a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 0x7f79b4198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48604/0 (socket says 172.21.15.28:48604) 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 -- 172.21.15.28:0/681246645 learned_addr learned my addr 172.21.15.28:0/681246645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79b9759700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 0x7f79b4198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 -- 172.21.15.28:0/681246645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 msgr2=0x7f79b4198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 0x7f79b4198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 -- 172.21.15.28:0/681246645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 msgr2=0x7f79b4198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 0x7f79b4198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 -- 172.21.15.28:0/681246645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b41a0590 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79b9759700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 0x7f79b4198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.567+0000 7f79ba75b700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79a400e970 tx=0x7f79a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.568+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4004a30 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.568+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b41a0880 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.568+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79a4005500 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.568+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a400cdc0 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.568+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b41a0dc0 con 0x7f79b4103150 2024-03-22T02:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.569+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b4069340 con 0x7f79b4103150 2024-03-22T02:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.570+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f79a4005000 con 0x7f79b4103150 2024-03-22T02:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.570+0000 7f79b2ffd700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 0x7f799c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.571+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f79a40cfce0 con 0x7f79b4103150 2024-03-22T02:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.571+0000 7f79b9f5a700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 0x7f799c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.571+0000 7f79b9f5a700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 0x7f799c0a5470 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79b4102610 tx=0x7f79ac00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:30.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.574+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f79a4053740 con 0x7f79b4103150 2024-03-22T02:13:30.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.785+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f79b41a1080 con 0x7f799c0a2fb0 2024-03-22T02:13:30.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.787+0000 7f79b2ffd700 1 -- 172.21.15.28:0/681246645 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f79b41a1080 con 0x7f799c0a2fb0 2024-03-22T02:13:30.787 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-03-22T02:13:30.787 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-03-22T02:13:30.787 INFO:teuthology.orchestra.run.smithi028.stdout:smithi083 172.21.15.83 2024-03-22T02:13:30.788 INFO:teuthology.orchestra.run.smithi028.stdout:smithi120 172.21.15.120 2024-03-22T02:13:30.788 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-03-22T02:13:30.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 msgr2=0x7f799c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:30.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 0x7f799c0a5470 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f79b4102610 tx=0x7f79ac00b040 comp rx=0 tx=0).stop 2024-03-22T02:13:30.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 msgr2=0x7f79b419fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79a400e970 tx=0x7f79a400ec80 comp rx=0 tx=0).stop 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 shutdown_connections 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f799c0a2fb0 0x7f799c0a5470 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79b4103150 0x7f79b419fef0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f79b41015b0 0x7f79b4198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 --2- 172.21.15.28:0/681246645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f79b41027b0 0x7f79b4198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 >> 172.21.15.28:0/681246645 conn(0x7f79b40fcb50 msgr2=0x7f79b406a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 shutdown_connections 2024-03-22T02:13:30.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:30.789+0000 7f79bc1be700 1 -- 172.21.15.28:0/681246645 wait complete. 2024-03-22T02:13:31.566 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:31 smithi028 ceph-mon[31916]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-22T02:13:31.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-22T02:13:31.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:31 smithi120 ceph-mon[36803]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-22T02:13:31.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:31 smithi083 ceph-mon[36340]: Health check failed: Reduced data availability: 5 pgs peering (PG_AVAILABILITY) 2024-03-22T02:13:31.858 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:32 smithi028 ceph-mon[31916]: pgmap v356: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 5 op/s 2024-03-22T02:13:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:32 smithi028 ceph-mon[31916]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:32 smithi120 ceph-mon[36803]: pgmap v356: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 5 op/s 2024-03-22T02:13:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:32 smithi120 ceph-mon[36803]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:32.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:32 smithi083 ceph-mon[36340]: pgmap v356: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 5 op/s 2024-03-22T02:13:32.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:32 smithi083 ceph-mon[36340]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:33.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 -- 172.21.15.28:0/1884877718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 msgr2=0x7fe358101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 --2- 172.21.15.28:0/1884877718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358101190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe3540099c0 tx=0x7fe354009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 -- 172.21.15.28:0/1884877718 shutdown_connections 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 --2- 172.21.15.28:0/1884877718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358101190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 --2- 172.21.15.28:0/1884877718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 0x7fe3581091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 --2- 172.21.15.28:0/1884877718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 0x7fe3581023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 -- 172.21.15.28:0/1884877718 >> 172.21.15.28:0/1884877718 conn(0x7fe3580fc310 msgr2=0x7fe3580fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 -- 172.21.15.28:0/1884877718 shutdown_connections 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.675+0000 7fe360318700 1 -- 172.21.15.28:0/1884877718 wait complete. 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 Processor -- start 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 -- start start 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 0x7fe358197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 0x7fe35819ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c930 con 0x7fe358100d70 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810c7b0 con 0x7fe358101f70 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe360318700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe35810cab0 con 0x7fe358102910 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe35e0b4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe35e0b4700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35536/0 (socket says 172.21.15.28:35536) 2024-03-22T02:13:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe35e0b4700 1 -- 172.21.15.28:0/472172444 learned_addr learned my addr 172.21.15.28:0/472172444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe35e8b5700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 0x7fe35819ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.676+0000 7fe35d8b3700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 0x7fe358197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 -- 172.21.15.28:0/472172444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 msgr2=0x7fe35819ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 0x7fe35819ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 -- 172.21.15.28:0/472172444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 msgr2=0x7fe358197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 0x7fe358197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 -- 172.21.15.28:0/472172444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35819f410 con 0x7fe358100d70 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe35e0b4700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe35400bf90 tx=0x7fe35400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe354009e60 con 0x7fe358100d70 2024-03-22T02:13:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe354003e30 con 0x7fe358100d70 2024-03-22T02:13:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe35819f6a0 con 0x7fe358100d70 2024-03-22T02:13:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35400fd70 con 0x7fe358100d70 2024-03-22T02:13:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.677+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35819fbd0 con 0x7fe358100d70 2024-03-22T02:13:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.678+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe358069340 con 0x7fe358100d70 2024-03-22T02:13:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.679+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fe35400f460 con 0x7fe358100d70 2024-03-22T02:13:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.680+0000 7fe34b7fe700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 0x7fe3440a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.680+0000 7fe35d8b3700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 0x7fe3440a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.680+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe3540d05f0 con 0x7fe358100d70 2024-03-22T02:13:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.680+0000 7fe35d8b3700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 0x7fe3440a54f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009c60 tx=0x7fe34c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:33.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.683+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe354013070 con 0x7fe358100d70 2024-03-22T02:13:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.896+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe358062ac0 con 0x7fe3440a3030 2024-03-22T02:13:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.898+0000 7fe34b7fe700 1 -- 172.21.15.28:0/472172444 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe358062ac0 con 0x7fe3440a3030 2024-03-22T02:13:33.898 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-22T02:13:33.898 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-22T02:13:33.898 INFO:teuthology.orchestra.run.smithi028.stdout:smithi083 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040148400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-22T02:13:33.898 INFO:teuthology.orchestra.run.smithi028.stdout:smithi120 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 msgr2=0x7fe3440a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 0x7fe3440a54f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009c60 tx=0x7fe34c009260 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 msgr2=0x7fe358197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe35400bf90 tx=0x7fe35400bfc0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 shutdown_connections 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3440a3030 0x7fe3440a54f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe358100d70 0x7fe358197800 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe358102910 0x7fe35819ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 --2- 172.21.15.28:0/472172444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe358101f70 0x7fe358197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 >> 172.21.15.28:0/472172444 conn(0x7fe3580fc310 msgr2=0x7fe3580fc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:33.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 shutdown_connections 2024-03-22T02:13:33.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:33.900+0000 7fe360318700 1 -- 172.21.15.28:0/472172444 wait complete. 2024-03-22T02:13:34.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:34 smithi028 ceph-mon[31916]: pgmap v357: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-22T02:13:34.614 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T02:13:34.640 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:13:34.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-22T02:13:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:34 smithi120 ceph-mon[36803]: pgmap v357: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-22T02:13:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:34 smithi083 ceph-mon[36340]: pgmap v357: 129 pgs: 5 peering, 124 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 639 B/s wr, 7 op/s 2024-03-22T02:13:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:35.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:35 smithi028 ceph-mon[31916]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:35 smithi120 ceph-mon[36803]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:35 smithi083 ceph-mon[36340]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:36 smithi028 ceph-mon[31916]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 639 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:36 smithi028 ceph-mon[31916]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-22T02:13:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:36 smithi028 ceph-mon[31916]: Cluster is now healthy 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/496825163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41038b0 msgr2=0x7fdfa4103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/496825163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41038b0 0x7fdfa4103d10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdf940099c0 tx=0x7fdf94009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/496825163 shutdown_connections 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/496825163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 0x7fdfa410ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/496825163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41026b0 0x7fdfa4102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/496825163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41038b0 0x7fdfa4103d10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/496825163 >> 172.21.15.28:0/496825163 conn(0x7fdfa40fdc90 msgr2=0x7fdfa41000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:36.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/496825163 shutdown_connections 2024-03-22T02:13:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.606+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/496825163 wait complete. 2024-03-22T02:13:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 Processor -- start 2024-03-22T02:13:36.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 -- start start 2024-03-22T02:13:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 0x7fdfa4198e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 0x7fdfa419fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410e2a0 con 0x7fdfa41026b0 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410e120 con 0x7fdfa4104250 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfaa8ca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa410e420 con 0x7fdfa41038b0 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35540/0 (socket says 172.21.15.28:35540) 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 -- 172.21.15.28:0/2945826258 learned_addr learned my addr 172.21.15.28:0/2945826258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa37fe700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 0x7fdfa4198e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa8e67700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 0x7fdfa419fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 -- 172.21.15.28:0/2945826258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 msgr2=0x7fdfa4198e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 0x7fdfa4198e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.607+0000 7fdfa3fff700 1 -- 172.21.15.28:0/2945826258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 msgr2=0x7fdfa419fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa3fff700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 0x7fdfa419fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa3fff700 1 -- 172.21.15.28:0/2945826258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa41a0540 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa37fe700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 0x7fdfa4198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa3fff700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e980 tx=0x7fdf8c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c004230 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf8c0043d0 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfa41a0830 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c003dc0 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.608+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfa41a0d20 con 0x7fdfa41026b0 2024-03-22T02:13:36.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.609+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfa4069340 con 0x7fdfa41026b0 2024-03-22T02:13:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.610+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fdf8c00c810 con 0x7fdfa41026b0 2024-03-22T02:13:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.610+0000 7fdfa17fa700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 0x7fdf900a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.611+0000 7fdfa37fe700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 0x7fdf900a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.611+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdf8c0cfa70 con 0x7fdfa41026b0 2024-03-22T02:13:36.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.611+0000 7fdfa37fe700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 0x7fdf900a5470 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdf94000c00 tx=0x7fdf94005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:36.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.615+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf8c09c490 con 0x7fdfa41026b0 2024-03-22T02:13:36.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:36 smithi120 ceph-mon[36803]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 639 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:36.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:36 smithi120 ceph-mon[36803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-22T02:13:36.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:36 smithi120 ceph-mon[36803]: Cluster is now healthy 2024-03-22T02:13:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:36 smithi083 ceph-mon[36340]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 639 B/s wr, 6 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:36 smithi083 ceph-mon[36340]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-22T02:13:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:36 smithi083 ceph-mon[36340]: Cluster is now healthy 2024-03-22T02:13:36.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.868+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fdfa4063290 con 0x7fdfa41026b0 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.868+0000 7fdfa17fa700 1 -- 172.21.15.28:0/2945826258 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3047 (secure 0 0 0) 0x7fdf8c09bd90 con 0x7fdfa41026b0 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:e24 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:36.869 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 24 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-03-22T02:11:33.259569+0000 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-03-22T02:11:54.987732+0000 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-03-22T02:13:36.870 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 3 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24509,1=24508,2=24478} 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3,4] 2024-03-22T02:13:36.871 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24509 members: 24478,24508,24509 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{0:24509} state up:active seq 3 addr [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{0:24467} state up:standby-replay seq 1 addr [v2:172.21.15.83:6836/3509701876,v1:172.21.15.83:6837/3509701876] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{1:24508} state up:active seq 9 addr [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{1:24466} state up:standby-replay seq 1 addr [v2:172.21.15.120:6834/1917291995,v1:172.21.15.120:6835/1917291995] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{2:24478} state up:active seq 11 addr [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{2:24443} state up:standby-replay seq 1 addr [v2:172.21.15.83:6834/3494340980,v1:172.21.15.83:6835/3494340980] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.28:6834/4228376531,v1:172.21.15.28:6835/4228376531] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:14577} state up:standby seq 1 addr [v2:172.21.15.28:6836/1410938784,v1:172.21.15.28:6837/1410938784] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{-1:14637} state up:standby seq 1 addr [v2:172.21.15.28:6838/678476485,v1:172.21.15.28:6839/678476485] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.28:6840/2956844816,v1:172.21.15.28:6841/2956844816] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24410} state up:standby seq 1 addr [v2:172.21.15.83:6832/675405105,v1:172.21.15.83:6833/675405105] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.873 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24424} state up:standby seq 1 addr [v2:172.21.15.120:6832/2414441153,v1:172.21.15.120:6833/2414441153] compat {c=[1],r=[1],i=[1fff]}] 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.871+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 msgr2=0x7fdf900a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.871+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 0x7fdf900a5470 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdf94000c00 tx=0x7fdf94005fb0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.871+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 msgr2=0x7fdfa41988e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.871+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e980 tx=0x7fdf8c00ec90 comp rx=0 tx=0).stop 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 shutdown_connections 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf900a2fb0 0x7fdf900a5470 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfa41026b0 0x7fdfa41988e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfa4104250 0x7fdfa419fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 --2- 172.21.15.28:0/2945826258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfa41038b0 0x7fdfa4198e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 >> 172.21.15.28:0/2945826258 conn(0x7fdfa40fdc90 msgr2=0x7fdfa4106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 shutdown_connections 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:36.872+0000 7fdfaa8ca700 1 -- 172.21.15.28:0/2945826258 wait complete. 2024-03-22T02:13:36.875 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:37.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:37 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2945826258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:13:37.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-22T02:13:37.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:37 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2945826258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:13:37.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:37 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2945826258' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-22T02:13:37.818 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:38.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:38 smithi028 ceph-mon[31916]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 489 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:38.665 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:38 smithi083 ceph-mon[36340]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 489 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:38 smithi120 ceph-mon[36803]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 489 B/s wr, 5 op/s; 2 B/s, 1 keys/s, 0 objects/s recovering 2024-03-22T02:13:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- 172.21.15.28:0/3517251685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af81015a0 msgr2=0x7f1af81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:39.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 --2- 172.21.15.28:0/3517251685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af81015a0 0x7f1af81019c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1ae80099c0 tx=0x7f1ae8009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- 172.21.15.28:0/3517251685 shutdown_connections 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 --2- 172.21.15.28:0/3517251685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af81015a0 0x7f1af81019c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 --2- 172.21.15.28:0/3517251685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af8103140 0x7f1af81099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 --2- 172.21.15.28:0/3517251685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 0x7f1af8102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- 172.21.15.28:0/3517251685 >> 172.21.15.28:0/3517251685 conn(0x7f1af80fcb40 msgr2=0x7f1af80fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- 172.21.15.28:0/3517251685 shutdown_connections 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- 172.21.15.28:0/3517251685 wait complete. 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 Processor -- start 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.545+0000 7f1afd937700 1 -- start start 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 0x7f1af8198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 0x7f1af8198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810d180 con 0x7f1af8103140 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810d000 con 0x7f1af81015a0 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810d300 con 0x7f1af81027a0 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35556/0 (socket says 172.21.15.28:35556) 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af6ffd700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 0x7f1af8198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 -- 172.21.15.28:0/754558504 learned_addr learned my addr 172.21.15.28:0/754558504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af67fc700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 0x7f1af8198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 -- 172.21.15.28:0/754558504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 msgr2=0x7f1af8198e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 0x7f1af8198e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 -- 172.21.15.28:0/754558504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 msgr2=0x7f1af8198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 0x7f1af8198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 -- 172.21.15.28:0/754558504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af81a0520 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af6ffd700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 0x7f1af8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1af77fe700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1af000ef60 tx=0x7f1af000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af00092e0 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af81a07b0 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1af0004bb0 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.546+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0004280 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.547+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81a0ca0 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.547+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af8069340 con 0x7f1af8103140 2024-03-22T02:13:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.549+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f1af001b030 con 0x7f1af8103140 2024-03-22T02:13:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.549+0000 7f1afc935700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 0x7f1ae00a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.550+0000 7f1af6ffd700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 0x7f1ae00a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.550+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f1af00cf230 con 0x7f1af8103140 2024-03-22T02:13:39.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.550+0000 7f1af6ffd700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 0x7f1ae00a5470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800b580 tx=0x7f1ae8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.554+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1af0022020 con 0x7f1af8103140 2024-03-22T02:13:39.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.759+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f1af81a15c0 con 0x7f1af8103140 2024-03-22T02:13:39.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.760+0000 7f1afc935700 1 -- 172.21.15.28:0/754558504 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) ==== 56+0+5742 (secure 0 0 0) 0x7f1af0010040 con 0x7f1af8103140 2024-03-22T02:13:39.760 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 94 2024-03-22T02:13:39.760 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-03-22T02:03:23.538926+0000 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-03-22T02:13:29.222546+0000 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 27 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-22T02:13:39.761 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/2828048761,v1:172.21.15.28:6803/2828048761] [v2:172.21.15.28:6804/2828048761,v1:172.21.15.28:6805/2828048761] exists,up edd58ef4-5561-4043-85e2-d205cb7c9396 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6800/1302915006,v1:172.21.15.83:6801/1302915006] [v2:172.21.15.83:6802/1302915006,v1:172.21.15.83:6803/1302915006] exists,up dd32d180-5e18-4813-8e6d-74ed124ee68f 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/2212833348,v1:172.21.15.120:6801/2212833348] [v2:172.21.15.120:6802/2212833348,v1:172.21.15.120:6803/2212833348] exists,up e9244c70-b1b3-42f4-a4e9-285e6933cc9b 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/3156433849,v1:172.21.15.28:6811/3156433849] [v2:172.21.15.28:6812/3156433849,v1:172.21.15.28:6813/3156433849] exists,up 242938b2-bb3e-4a38-8c8e-7491101ee0bd 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 34 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6808/3880312837,v1:172.21.15.83:6809/3880312837] [v2:172.21.15.83:6810/3880312837,v1:172.21.15.83:6811/3880312837] exists,up 39ccc6c8-4269-4d9e-b30f-40311d5558b3 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 39 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/1653723326,v1:172.21.15.120:6809/1653723326] [v2:172.21.15.120:6810/1653723326,v1:172.21.15.120:6811/1653723326] exists,up 17fac3fc-aa99-48a3-8ce4-565522ae800d 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 47 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/1589283942,v1:172.21.15.28:6819/1589283942] [v2:172.21.15.28:6820/1589283942,v1:172.21.15.28:6821/1589283942] exists,up 1b4222ad-6096-499f-ba76-609404830d4d 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 52 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6816/1165824815,v1:172.21.15.83:6817/1165824815] [v2:172.21.15.83:6818/1165824815,v1:172.21.15.83:6819/1165824815] exists,up 5d99f5cf-caaa-4c34-96da-37dd87bc1491 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 60 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/739561916,v1:172.21.15.120:6817/739561916] [v2:172.21.15.120:6818/739561916,v1:172.21.15.120:6819/739561916] exists,up 574e783a-b2d6-439e-8bde-68e2d4b577b7 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/855775168,v1:172.21.15.28:6827/855775168] [v2:172.21.15.28:6828/855775168,v1:172.21.15.28:6829/855775168] exists,up f26642a2-c7eb-495a-979a-6002cf9f9438 2024-03-22T02:13:39.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 73 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.83:6824/3932179263,v1:172.21.15.83:6825/3932179263] [v2:172.21.15.83:6826/3932179263,v1:172.21.15.83:6827/3932179263] exists,up 9eafc799-33ff-4ef4-b4ec-235fcd7b9366 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 81 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/511858328,v1:172.21.15.120:6825/511858328] [v2:172.21.15.120:6826/511858328,v1:172.21.15.120:6827/511858328] exists,up ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.5 [8,11] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.a [2,11] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.16 [2,4] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.1d [2,4] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3 [4,9] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.6 [3,8] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.17 [4,6,5,7] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.22 [3,9] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.23 [4,8] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.28 [5,6] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.2a [1,9] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.3a [4,7] 2024-03-22T02:13:39.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.6 [5,8] 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.14 [9,8] 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.19 [9,8] 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1c [9,10] 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3429985942 expires 2024-03-23T02:04:26.467767+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2329372296 expires 2024-03-23T02:04:26.467767+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3429985942 expires 2024-03-23T02:04:26.467767+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/857886266 expires 2024-03-23T02:03:59.710695+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1957817657 expires 2024-03-23T02:04:26.467767+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/857886266 expires 2024-03-23T02:03:59.710695+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/548967036 expires 2024-03-23T02:03:59.710695+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/30661343 expires 2024-03-23T02:03:59.710695+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2074246765 expires 2024-03-23T02:04:26.467767+0000 2024-03-22T02:13:39.764 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4059471246 expires 2024-03-23T02:03:59.710695+0000 2024-03-22T02:13:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 msgr2=0x7f1ae00a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 0x7f1ae00a5470 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1ae800b580 tx=0x7f1ae8005fb0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 msgr2=0x7f1af819fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1af000ef60 tx=0x7f1af000ef90 comp rx=0 tx=0).stop 2024-03-22T02:13:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 shutdown_connections 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ae00a2fb0 0x7f1ae00a5470 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af8103140 0x7f1af819fe80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1af81015a0 0x7f1af8198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 --2- 172.21.15.28:0/754558504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1af81027a0 0x7f1af8198e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 >> 172.21.15.28:0/754558504 conn(0x7f1af80fcb40 msgr2=0x7f1af80fe610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 shutdown_connections 2024-03-22T02:13:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:39.763+0000 7f1afd937700 1 -- 172.21.15.28:0/754558504 wait complete. 2024-03-22T02:13:40.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:40 smithi028 ceph-mon[31916]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 409 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:40.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/754558504' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T02:13:40.416 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-22T02:13:40.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- 172.21.15.28:0/11751436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41031b0 msgr2=0x7ff0d41035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- 172.21.15.28:0/11751436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41031b0 0x7ff0d41035d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8009900 tx=0x7ff0c802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- 172.21.15.28:0/11751436 shutdown_connections 2024-03-22T02:13:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- 172.21.15.28:0/11751436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41005e0 0x7ff0d410a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- 172.21.15.28:0/11751436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d41000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- 172.21.15.28:0/11751436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41031b0 0x7ff0d41035d0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- 172.21.15.28:0/11751436 >> 172.21.15.28:0/11751436 conn(0x7ff0d40fb7c0 msgr2=0x7ff0d40fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- 172.21.15.28:0/11751436 shutdown_connections 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- 172.21.15.28:0/11751436 wait complete. 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 Processor -- start 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- start start 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41031b0 0x7ff0d419f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d410d830 con 0x7ff0d41005e0 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d410d6b0 con 0x7ff0d40ffc20 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.537+0000 7ff0db008640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d410d9b0 con 0x7ff0d41031b0 2024-03-22T02:13:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35584/0 (socket says 172.21.15.28:35584) 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 -- 172.21.15.28:0/14087080 learned_addr learned my addr 172.21.15.28:0/14087080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d8da4640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d95a5640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41031b0 0x7ff0d419f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 -- 172.21.15.28:0/14087080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41031b0 msgr2=0x7ff0d419f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41031b0 0x7ff0d419f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 -- 172.21.15.28:0/14087080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 msgr2=0x7ff0d4197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d4197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 -- 172.21.15.28:0/14087080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d419faf0 con 0x7ff0d41005e0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d8da4640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d3fff640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000e9f0 tx=0x7ff0c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c000cd80 con 0x7ff0d41005e0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0c0010910 con 0x7ff0d41005e0 2024-03-22T02:13:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d419fde0 con 0x7ff0d41005e0 2024-03-22T02:13:40.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c0018990 con 0x7ff0d41005e0 2024-03-22T02:13:40.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.538+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d41a02a0 con 0x7ff0d41005e0 2024-03-22T02:13:40.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.539+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d4109070 con 0x7ff0d41005e0 2024-03-22T02:13:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.540+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7ff0c0010ac0 con 0x7ff0d41005e0 2024-03-22T02:13:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.541+0000 7ff0d1ffb640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 0x7ff0c40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.541+0000 7ff0d8da4640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 0x7ff0c40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.541+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff0c00cfae0 con 0x7ff0d41005e0 2024-03-22T02:13:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.542+0000 7ff0d8da4640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 0x7ff0c40a53e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0c802efd0 tx=0x7ff0c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:40.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.546+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff0c009beb0 con 0x7ff0d41005e0 2024-03-22T02:13:40.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.732+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0d4062830 con 0x7ff0d41005e0 2024-03-22T02:13:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.733+0000 7ff0d1ffb640 1 -- 172.21.15.28:0/14087080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7ff0c009ff50 con 0x7ff0d41005e0 2024-03-22T02:13:40.734 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:40.734 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 msgr2=0x7ff0c40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 0x7ff0c40a53e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff0c802efd0 tx=0x7ff0c8000bc0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 msgr2=0x7ff0d4198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff0c000e9f0 tx=0x7ff0c000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 shutdown_connections 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff0c40a2f20 0x7ff0c40a53e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff0d41031b0 0x7ff0d419f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0d41005e0 0x7ff0d4198410 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 --2- 172.21.15.28:0/14087080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff0d40ffc20 0x7ff0d4197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 >> 172.21.15.28:0/14087080 conn(0x7ff0d40fb7c0 msgr2=0x7ff0d4106240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 shutdown_connections 2024-03-22T02:13:40.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.736+0000 7ff0db008640 1 -- 172.21.15.28:0/14087080 wait complete. 2024-03-22T02:13:40.748 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:40.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:40.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:40 smithi120 ceph-mon[36803]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 409 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:40.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/754558504' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T02:13:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:40 smithi083 ceph-mon[36340]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 409 B/s wr, 6 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/754558504' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-22T02:13:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3042949525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8101ba0 msgr2=0x7f7fd8101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3042949525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8101fc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009900 tx=0x7f7fc802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3042949525 shutdown_connections 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3042949525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8103750 0x7f7fd8109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3042949525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 0x7f7fd8103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3042949525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8101fc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3042949525 >> 172.21.15.28:0/3042949525 conn(0x7f7fd80fd130 msgr2=0x7f7fd80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3042949525 shutdown_connections 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3042949525 wait complete. 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.854+0000 7f7fdd82a640 1 Processor -- start 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 -- start start 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 0x7f7fd8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810d810 con 0x7f7fd8103750 2024-03-22T02:13:40.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810d690 con 0x7f7fd8101ba0 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdd82a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810d990 con 0x7f7fd8102d90 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35604/0 (socket says 172.21.15.28:35604) 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 -- 172.21.15.28:0/3661019598 learned_addr learned my addr 172.21.15.28:0/3661019598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd67fc640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 0x7f7fd8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd6ffd640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 -- 172.21.15.28:0/3661019598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 msgr2=0x7f7fd8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 0x7f7fd8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 -- 172.21.15.28:0/3661019598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8101ba0 msgr2=0x7f7fd8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 -- 172.21.15.28:0/3661019598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd819fb30 con 0x7f7fd8103750 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fd77fe640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000eb10 tx=0x7f7fd000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:40.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.855+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0004b20 con 0x7f7fd8103750 2024-03-22T02:13:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.856+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fd0004cc0 con 0x7f7fd8103750 2024-03-22T02:13:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.856+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd819fe20 con 0x7f7fd8103750 2024-03-22T02:13:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.856+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0010760 con 0x7f7fd8103750 2024-03-22T02:13:40.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.856+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd81a0290 con 0x7f7fd8103750 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.856+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd8108f30 con 0x7f7fd8103750 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.858+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f7fd0005020 con 0x7f7fd8103750 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.858+0000 7f7fdc828640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 0x7f7fc00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.858+0000 7f7fd6ffd640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 0x7f7fc00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.858+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7fd00cfd50 con 0x7f7fd8103750 2024-03-22T02:13:40.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.859+0000 7f7fd6ffd640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 0x7f7fc00a53e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8000c00 tx=0x7f7fc8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:40.863+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7fd00d3030 con 0x7f7fd8103750 2024-03-22T02:13:41.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.047+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7fd8062830 con 0x7f7fd8103750 2024-03-22T02:13:41.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.047+0000 7f7fdc828640 1 -- 172.21.15.28:0/3661019598 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f7fd009c770 con 0x7f7fd8103750 2024-03-22T02:13:41.048 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 msgr2=0x7f7fc00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 0x7f7fc00a53e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8000c00 tx=0x7f7fc8005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 msgr2=0x7f7fd819f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000eb10 tx=0x7f7fd000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 shutdown_connections 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.049+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7fc00a2f20 0x7f7fc00a53e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7fd8103750 0x7f7fd819f4c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fd8102d90 0x7f7fd8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 --2- 172.21.15.28:0/3661019598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7fd8101ba0 0x7f7fd8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 >> 172.21.15.28:0/3661019598 conn(0x7f7fd80fd130 msgr2=0x7f7fd8106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 shutdown_connections 2024-03-22T02:13:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.050+0000 7f7fdd82a640 1 -- 172.21.15.28:0/3661019598 wait complete. 2024-03-22T02:13:41.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:41.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 -- 172.21.15.28:0/2055821918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c103800 msgr2=0x7f929c10a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 --2- 172.21.15.28:0/2055821918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c103800 0x7f929c10a090 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9294009cd0 tx=0x7f929402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 -- 172.21.15.28:0/2055821918 shutdown_connections 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 --2- 172.21.15.28:0/2055821918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c103800 0x7f929c10a090 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 --2- 172.21.15.28:0/2055821918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c102e40 0x7f929c1032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 --2- 172.21.15.28:0/2055821918 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c0ffe70 0x7f929c100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 -- 172.21.15.28:0/2055821918 >> 172.21.15.28:0/2055821918 conn(0x7f929c0fb7c0 msgr2=0x7f929c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 -- 172.21.15.28:0/2055821918 shutdown_connections 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.167+0000 7f92a09bc640 1 -- 172.21.15.28:0/2055821918 wait complete. 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 Processor -- start 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 -- start start 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c102e40 0x7f929c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c103800 0x7f929c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10d820 con 0x7f929c0ffe70 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10d6a0 con 0x7f929c102e40 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92a09bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f929c10d9a0 con 0x7f929c103800 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35630/0 (socket says 172.21.15.28:35630) 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 -- 172.21.15.28:0/3037277696 learned_addr learned my addr 172.21.15.28:0/3037277696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929ad9d640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c103800 0x7f929c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f9293fff640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c102e40 0x7f929c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 -- 172.21.15.28:0/3037277696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c103800 msgr2=0x7f929c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c103800 0x7f929c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 -- 172.21.15.28:0/3037277696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c102e40 msgr2=0x7f929c198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c102e40 0x7f929c198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 -- 172.21.15.28:0/3037277696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c19fa50 con 0x7f929c0ffe70 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f929a59c640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f928c00e9c0 tx=0x7f928c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.168+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c00cd80 con 0x7f929c0ffe70 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.169+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c19fd40 con 0x7f929c0ffe70 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.169+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f928c0035f0 con 0x7f929c0ffe70 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.169+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c010690 con 0x7f929c0ffe70 2024-03-22T02:13:41.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.169+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c1a01d0 con 0x7f929c0ffe70 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.169+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c108f10 con 0x7f929c0ffe70 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.171+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f928c0037a0 con 0x7f929c0ffe70 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.171+0000 7f92937fe640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 0x7f92880a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.171+0000 7f9293fff640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 0x7f92880a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.171+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f928c0cfbd0 con 0x7f929c0ffe70 2024-03-22T02:13:41.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.172+0000 7f9293fff640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 0x7f92880a53e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f929c199340 tx=0x7f9284005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.176+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f928c09c5f0 con 0x7f929c0ffe70 2024-03-22T02:13:41.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:41 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/14087080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:41.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:41 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3661019598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.364+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f929c0628d0 con 0x7f929c0ffe70 2024-03-22T02:13:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.364+0000 7f92937fe640 1 -- 172.21.15.28:0/3037277696 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f928c09bef0 con 0x7f929c0ffe70 2024-03-22T02:13:41.364 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 msgr2=0x7f92880a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 0x7f92880a53e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f929c199340 tx=0x7f9284005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 msgr2=0x7f929c197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f928c00e9c0 tx=0x7f928c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 shutdown_connections 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f92880a2f20 0x7f92880a53e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f929c103800 0x7f929c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f929c102e40 0x7f929c198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 --2- 172.21.15.28:0/3037277696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f929c0ffe70 0x7f929c197e30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.366+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 >> 172.21.15.28:0/3037277696 conn(0x7f929c0fb7c0 msgr2=0x7f929c1060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.367+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 shutdown_connections 2024-03-22T02:13:41.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.367+0000 7f92a09bc640 1 -- 172.21.15.28:0/3037277696 wait complete. 2024-03-22T02:13:41.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- 172.21.15.28:0/3388407522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 msgr2=0x7fc238102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- 172.21.15.28:0/3388407522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238102100 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc228009860 tx=0x7fc22802bb70 comp rx=0 tx=0).stop 2024-03-22T02:13:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- 172.21.15.28:0/3388407522 shutdown_connections 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- 172.21.15.28:0/3388407522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238102640 0x7fc238108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- 172.21.15.28:0/3388407522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238102100 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- 172.21.15.28:0/3388407522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238100a90 0x7fc238100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- 172.21.15.28:0/3388407522 >> 172.21.15.28:0/3388407522 conn(0x7fc2380fc020 msgr2=0x7fc2380fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- 172.21.15.28:0/3388407522 shutdown_connections 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- 172.21.15.28:0/3388407522 wait complete. 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 Processor -- start 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 -- start start 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238100a90 0x7fc238197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.483+0000 7fc23fad4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23fad4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d830 con 0x7fc238102640 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23fad4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d6b0 con 0x7fc238100a90 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23fad4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc23810d9b0 con 0x7fc238101c80 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35646/0 (socket says 172.21.15.28:35646) 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 -- 172.21.15.28:0/2934456638 learned_addr learned my addr 172.21.15.28:0/2934456638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23d870640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238100a90 0x7fc238197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23d06f640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 -- 172.21.15.28:0/2934456638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 msgr2=0x7fc238198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 -- 172.21.15.28:0/2934456638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238100a90 msgr2=0x7fc238197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238100a90 0x7fc238197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 -- 172.21.15.28:0/2934456638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23819fae0 con 0x7fc238102640 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23e071640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc23000eb10 tx=0x7fc23000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23000ce20 con 0x7fc238102640 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23819fdd0 con 0x7fc238102640 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc230005020 con 0x7fc238102640 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc230010690 con 0x7fc238102640 2024-03-22T02:13:41.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.484+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2381a0290 con 0x7fc238102640 2024-03-22T02:13:41.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.485+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc238107e50 con 0x7fc238102640 2024-03-22T02:13:41.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.486+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fc230005540 con 0x7fc238102640 2024-03-22T02:13:41.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.487+0000 7fc22effd640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 0x7fc2240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.487+0000 7fc23d870640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 0x7fc2240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.487+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc2300cfdb0 con 0x7fc238102640 2024-03-22T02:13:41.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.488+0000 7fc23d870640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 0x7fc2240a53e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc238101ae0 tx=0x7fc234009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.492+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc2300d4070 con 0x7fc238102640 2024-03-22T02:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.677+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc238062790 con 0x7fc238102640 2024-03-22T02:13:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.677+0000 7fc22effd640 1 -- 172.21.15.28:0/2934456638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fc23009c770 con 0x7fc238102640 2024-03-22T02:13:41.678 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 msgr2=0x7fc2240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 0x7fc2240a53e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc238101ae0 tx=0x7fc234009260 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 msgr2=0x7fc23819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fc23000eb10 tx=0x7fc23000ef10 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 shutdown_connections 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc2240a2f20 0x7fc2240a53e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc238102640 0x7fc23819f470 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc238101c80 0x7fc238198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 --2- 172.21.15.28:0/2934456638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc238100a90 0x7fc238197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 >> 172.21.15.28:0/2934456638 conn(0x7fc2380fc020 msgr2=0x7fc238105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 shutdown_connections 2024-03-22T02:13:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.679+0000 7fc23fad4640 1 -- 172.21.15.28:0/2934456638 wait complete. 2024-03-22T02:13:41.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:41.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:41 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/14087080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:41.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:41 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3661019598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:41.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:41 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/14087080' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:41.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:41 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3661019598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 -- 172.21.15.28:0/3111297594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 msgr2=0x7fef90102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 --2- 172.21.15.28:0/3111297594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90102110 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fef84009900 tx=0x7fef8402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 -- 172.21.15.28:0/3111297594 shutdown_connections 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 --2- 172.21.15.28:0/3111297594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90102650 0x7fef90108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 --2- 172.21.15.28:0/3111297594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90102110 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 --2- 172.21.15.28:0/3111297594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90100aa0 0x7fef90100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 -- 172.21.15.28:0/3111297594 >> 172.21.15.28:0/3111297594 conn(0x7fef900fc030 msgr2=0x7fef900fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 -- 172.21.15.28:0/3111297594 shutdown_connections 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.797+0000 7fef98042640 1 -- 172.21.15.28:0/3111297594 wait complete. 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 Processor -- start 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 -- start start 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90100aa0 0x7fef90197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90102650 0x7fef9019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fef9010d870 con 0x7fef90101c90 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fef9010d6f0 con 0x7fef90100aa0 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef98042640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fef9010d9f0 con 0x7fef90102650 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49304/0 (socket says 172.21.15.28:49304) 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 -- 172.21.15.28:0/2790300580 learned_addr learned my addr 172.21.15.28:0/2790300580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef965df640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90102650 0x7fef9019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef95dde640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90100aa0 0x7fef90197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 -- 172.21.15.28:0/2790300580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90102650 msgr2=0x7fef9019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90102650 0x7fef9019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 -- 172.21.15.28:0/2790300580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90100aa0 msgr2=0x7fef90197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90100aa0 0x7fef90197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.798+0000 7fef955dd640 1 -- 172.21.15.28:0/2790300580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9019fb10 con 0x7fef90101c90 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef955dd640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef84000c00 tx=0x7fef8402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84043020 con 0x7fef90101c90 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef840054e0 con 0x7fef90101c90 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9019fda0 con 0x7fef90101c90 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef84003c40 con 0x7fef90101c90 2024-03-22T02:13:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.799+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef901a0270 con 0x7fef90101c90 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.800+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef90107dc0 con 0x7fef90101c90 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.801+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fef84046020 con 0x7fef90101c90 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.801+0000 7fef8effd640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 0x7fef780a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.801+0000 7fef95dde640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 0x7fef780a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.802+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fef840f1810 con 0x7fef90101c90 2024-03-22T02:13:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.802+0000 7fef95dde640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 0x7fef780a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef90101af0 tx=0x7fef88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:41.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.806+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fef8404d020 con 0x7fef90101c90 2024-03-22T02:13:41.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.991+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fef90062830 con 0x7fef90101c90 2024-03-22T02:13:41.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.992+0000 7fef8effd640 1 -- 172.21.15.28:0/2790300580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fef840be1d0 con 0x7fef90101c90 2024-03-22T02:13:41.992 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:41.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 msgr2=0x7fef780a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 0x7fef780a5460 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fef90101af0 tx=0x7fef88005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 msgr2=0x7fef90198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fef84000c00 tx=0x7fef8402c650 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 shutdown_connections 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fef780a2fa0 0x7fef780a5460 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef90102650 0x7fef9019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef90101c90 0x7fef90198430 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 --2- 172.21.15.28:0/2790300580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fef90100aa0 0x7fef90197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 >> 172.21.15.28:0/2790300580 conn(0x7fef900fc030 msgr2=0x7fef90104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 shutdown_connections 2024-03-22T02:13:41.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:41.994+0000 7fef98042640 1 -- 172.21.15.28:0/2790300580 wait complete. 2024-03-22T02:13:42.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- 172.21.15.28:0/3685467503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 msgr2=0x7f8404100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- 172.21.15.28:0/3685467503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f8404100ed0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83fc009ca0 tx=0x7f83fc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- 172.21.15.28:0/3685467503 shutdown_connections 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- 172.21.15.28:0/3685467503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f8404108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- 172.21.15.28:0/3685467503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 0x7f8404102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- 172.21.15.28:0/3685467503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f8404100ed0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- 172.21.15.28:0/3685467503 >> 172.21.15.28:0/3685467503 conn(0x7f84040fc040 msgr2=0x7f84040fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- 172.21.15.28:0/3685467503 shutdown_connections 2024-03-22T02:13:42.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- 172.21.15.28:0/3685467503 wait complete. 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 Processor -- start 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- start start 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 0x7f840419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f840419f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f840410c720 con 0x7f8404102660 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f840410c5a0 con 0x7f8404101ca0 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.108+0000 7f840b06d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f840410c8a0 con 0x7f8404100ab0 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f840419f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8408e09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8408e09640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52766/0 (socket says 172.21.15.28:52766) 2024-03-22T02:13:42.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8408e09640 1 -- 172.21.15.28:0/1523767978 learned_addr learned my addr 172.21.15.28:0/1523767978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8403fff640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 0x7f840419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 -- 172.21.15.28:0/1523767978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 msgr2=0x7f840406ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 -- 172.21.15.28:0/1523767978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 msgr2=0x7f840419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 0x7f840419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 -- 172.21.15.28:0/1523767978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840419fb10 con 0x7f8404102660 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8408e09640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840960a640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f840419f250 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00e970 tx=0x7f83ec00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec00ccc0 con 0x7f8404102660 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f840419fe00 con 0x7f8404102660 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83ec005020 con 0x7f8404102660 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ec004a00 con 0x7f8404102660 2024-03-22T02:13:42.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.109+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84041a0320 con 0x7f8404102660 2024-03-22T02:13:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.110+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8404107ea0 con 0x7f8404102660 2024-03-22T02:13:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.111+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f83ec0051d0 con 0x7f8404102660 2024-03-22T02:13:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.112+0000 7f8401ffb640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 0x7f83f00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.112+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f83ec0cf790 con 0x7f8404102660 2024-03-22T02:13:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.112+0000 7f8408e09640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 0x7f83f00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.112+0000 7f8408e09640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 0x7f83f00a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f83fc032420 tx=0x7f83fc032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.117+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83ec09c1b0 con 0x7f8404102660 2024-03-22T02:13:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.339+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f840402d3b0 con 0x7f8404102660 2024-03-22T02:13:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.340+0000 7f8401ffb640 1 -- 172.21.15.28:0/1523767978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7f83ec09bab0 con 0x7f8404102660 2024-03-22T02:13:42.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:42.341 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:13:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 msgr2=0x7f83f00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 0x7f83f00a53e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f83fc032420 tx=0x7f83fc032590 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 msgr2=0x7f840419f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f840419f250 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f83ec00e970 tx=0x7f83ec00ee30 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 shutdown_connections 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83f00a2f20 0x7f83f00a53e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8404102660 0x7f840419f250 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8404101ca0 0x7f840419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 --2- 172.21.15.28:0/1523767978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8404100ab0 0x7f840406ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 >> 172.21.15.28:0/1523767978 conn(0x7f84040fc040 msgr2=0x7f8404105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 shutdown_connections 2024-03-22T02:13:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.342+0000 7f840b06d640 1 -- 172.21.15.28:0/1523767978 wait complete. 2024-03-22T02:13:42.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:42.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.456+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1171824500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8100ab0 msgr2=0x7f65f8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.456+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1171824500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8100ab0 0x7f65f8100ed0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f65f0009860 tx=0x7f65f002bb70 comp rx=0 tx=0).stop 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1171824500 shutdown_connections 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1171824500 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f8108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1171824500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8101ca0 0x7f65f8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1171824500 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8100ab0 0x7f65f8100ed0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1171824500 >> 172.21.15.28:0/1171824500 conn(0x7f65f80fc020 msgr2=0x7f65f80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1171824500 shutdown_connections 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1171824500 wait complete. 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 Processor -- start 2024-03-22T02:13:42.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- start start 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8101ca0 0x7f65f8197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810c700 con 0x7f65f8100ab0 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.457+0000 7f65fd9d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810c580 con 0x7f65f8102660 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fd9d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810c880 con 0x7f65f8101ca0 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49340/0 (socket says 172.21.15.28:49340) 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f6ffd640 1 -- 172.21.15.28:0/1313653862 learned_addr learned my addr 172.21.15.28:0/1313653862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f67fc640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8101ca0 0x7f65f8197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 -- 172.21.15.28:0/1313653862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8101ca0 msgr2=0x7f65f8197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8101ca0 0x7f65f8197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 -- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 msgr2=0x7f65f8196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 -- 172.21.15.28:0/1313653862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f819e9f0 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f6ffd640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65f77fe640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f819e380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009fd0 tx=0x7f65e800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e800ce20 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f8068d50 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65e8005020 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8010720 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.458+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f80692a0 con 0x7f65f8102660 2024-03-22T02:13:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.459+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f8068980 con 0x7f65f8102660 2024-03-22T02:13:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.460+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f65e8005540 con 0x7f65f8102660 2024-03-22T02:13:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.461+0000 7f65fc9d5640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 0x7f65e40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.461+0000 7f65f6ffd640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 0x7f65e40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.461+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f65e80cfec0 con 0x7f65f8102660 2024-03-22T02:13:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.461+0000 7f65f6ffd640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 0x7f65e40a53e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f65f00095f0 tx=0x7f65f0034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.465+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f65e809c8e0 con 0x7f65f8102660 2024-03-22T02:13:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:42 smithi028 ceph-mon[31916]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 360 B/s wr, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:42 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3037277696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:42 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2934456638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:42 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2790300580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.650+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65f819ebc0 con 0x7f65f8102660 2024-03-22T02:13:42.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.652+0000 7f65fc9d5640 1 -- 172.21.15.28:0/1313653862 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7f65e809c1e0 con 0x7f65f8102660 2024-03-22T02:13:42.653 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:42.653 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:42.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 msgr2=0x7f65e40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 0x7f65e40a53e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f65f00095f0 tx=0x7f65f0034ad0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 msgr2=0x7f65f819e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f819e380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009fd0 tx=0x7f65e800ef10 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 shutdown_connections 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65e40a2f20 0x7f65e40a53e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f65f8102660 0x7f65f819e380 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65f8101ca0 0x7f65f8197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 --2- 172.21.15.28:0/1313653862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65f8100ab0 0x7f65f8196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 >> 172.21.15.28:0/1313653862 conn(0x7f65f80fc020 msgr2=0x7f65f8104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 shutdown_connections 2024-03-22T02:13:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.654+0000 7f65fd9d7640 1 -- 172.21.15.28:0/1313653862 wait complete. 2024-03-22T02:13:42.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:42.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:42 smithi120 ceph-mon[36803]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 360 B/s wr, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:42 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3037277696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:42 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2934456638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:42 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2790300580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:42 smithi083 ceph-mon[36340]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 360 B/s wr, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:42.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:42 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3037277696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:42 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2934456638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:42 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2790300580' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- 172.21.15.28:0/4291284408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484100cd0 msgr2=0x7fd484101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/4291284408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484100cd0 0x7fd484101150 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd474009900 tx=0x7fd47402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- 172.21.15.28:0/4291284408 shutdown_connections 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/4291284408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484101690 0x7fd484108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/4291284408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484100cd0 0x7fd484101150 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/4291284408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 0x7fd4841024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- 172.21.15.28:0/4291284408 >> 172.21.15.28:0/4291284408 conn(0x7fd4840fb7c0 msgr2=0x7fd4840fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- 172.21.15.28:0/4291284408 shutdown_connections 2024-03-22T02:13:42.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- 172.21.15.28:0/4291284408 wait complete. 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 Processor -- start 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.772+0000 7fd48c0d6640 1 -- start start 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 0x7fd484198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 0x7fd48419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd48410d890 con 0x7fd484100cd0 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd48410d710 con 0x7fd484102080 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48c0d6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd48410da10 con 0x7fd484101690 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49362/0 (socket says 172.21.15.28:49362) 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 -- 172.21.15.28:0/1439025702 learned_addr learned my addr 172.21.15.28:0/1439025702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489671640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 0x7fd484198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd48a673640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 0x7fd48419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 -- 172.21.15.28:0/1439025702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 msgr2=0x7fd484198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 0x7fd484198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 -- 172.21.15.28:0/1439025702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 msgr2=0x7fd48419f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 0x7fd48419f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 -- 172.21.15.28:0/1439025702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48419fae0 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489671640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 0x7fd484198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd489e72640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd48000e9c0 tx=0x7fd48000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.773+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd48000cd80 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.774+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48419fdd0 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.774+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4800035f0 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.774+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480010690 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.774+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4841a01e0 con 0x7fd484100cd0 2024-03-22T02:13:42.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.775+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd484107e90 con 0x7fd484100cd0 2024-03-22T02:13:42.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.776+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fd4800037a0 con 0x7fd484100cd0 2024-03-22T02:13:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.776+0000 7fd47affd640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 0x7fd4700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.776+0000 7fd489671640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 0x7fd4700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.776+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd4800cfb60 con 0x7fd484100cd0 2024-03-22T02:13:42.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.777+0000 7fd489671640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 0x7fd4700a53e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd47402c7b0 tx=0x7fd474005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:42.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.781+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd48009c580 con 0x7fd484100cd0 2024-03-22T02:13:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.964+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd484062860 con 0x7fd484100cd0 2024-03-22T02:13:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.965+0000 7fd47affd640 1 -- 172.21.15.28:0/1439025702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fd48009be80 con 0x7fd484100cd0 2024-03-22T02:13:42.965 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 msgr2=0x7fd4700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 0x7fd4700a53e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd47402c7b0 tx=0x7fd474005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 msgr2=0x7fd484197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd48000e9c0 tx=0x7fd48000ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 shutdown_connections 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd4700a2f20 0x7fd4700a53e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd484102080 0x7fd48419f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd484101690 0x7fd484198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 --2- 172.21.15.28:0/1439025702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd484100cd0 0x7fd484197ec0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.966+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 >> 172.21.15.28:0/1439025702 conn(0x7fd4840fb7c0 msgr2=0x7fd484105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.967+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 shutdown_connections 2024-03-22T02:13:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:42.967+0000 7fd48c0d6640 1 -- 172.21.15.28:0/1439025702 wait complete. 2024-03-22T02:13:42.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 -- 172.21.15.28:0/2761899222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc101c80 msgr2=0x7f7cdc102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/2761899222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc101c80 0x7f7cdc102100 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009860 tx=0x7f7cc402bb70 comp rx=0 tx=0).stop 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 -- 172.21.15.28:0/2761899222 shutdown_connections 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/2761899222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc102640 0x7f7cdc108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/2761899222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc101c80 0x7f7cdc102100 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/2761899222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc100a90 0x7f7cdc100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 -- 172.21.15.28:0/2761899222 >> 172.21.15.28:0/2761899222 conn(0x7f7cdc0fc020 msgr2=0x7f7cdc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 -- 172.21.15.28:0/2761899222 shutdown_connections 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.083+0000 7f7ce17c7640 1 -- 172.21.15.28:0/2761899222 wait complete. 2024-03-22T02:13:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 Processor -- start 2024-03-22T02:13:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 -- start start 2024-03-22T02:13:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 0x7f7cdc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc101c80 0x7f7cdc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10d830 con 0x7f7cdc102640 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10d6b0 con 0x7f7cdc100a90 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7ce17c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc10d9b0 con 0x7f7cdc101c80 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 0x7f7cdc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7cda7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc101c80 0x7f7cdc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49380/0 (socket says 172.21.15.28:49380) 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.084+0000 7f7cdb7fe640 1 -- 172.21.15.28:0/1999408411 learned_addr learned my addr 172.21.15.28:0/1999408411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 -- 172.21.15.28:0/1999408411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc101c80 msgr2=0x7f7cdc1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc101c80 0x7f7cdc1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 -- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 msgr2=0x7f7cdc19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 -- 172.21.15.28:0/1999408411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc19fad0 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdb7fe640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cdaffd640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 0x7f7cdc197eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc101ae0 tx=0x7f7cd400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4004440 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc19fdc0 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd400cda0 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40196e0 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.085+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc1a0230 con 0x7f7cdc100a90 2024-03-22T02:13:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.086+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc107db0 con 0x7f7cdc100a90 2024-03-22T02:13:43.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.087+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f7cd4003780 con 0x7f7cdc100a90 2024-03-22T02:13:43.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.088+0000 7f7cd3fff640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 0x7f7cc80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.088+0000 7f7cda7fc640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 0x7f7cc80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.088+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7cd40cfcd0 con 0x7f7cdc100a90 2024-03-22T02:13:43.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.088+0000 7f7cda7fc640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 0x7f7cc80a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7cc402c180 tx=0x7f7cc4034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.091+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cd4003e20 con 0x7f7cdc100a90 2024-03-22T02:13:43.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.277+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7cdc02d660 con 0x7f7cdc100a90 2024-03-22T02:13:43.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.278+0000 7f7cd3fff640 1 -- 172.21.15.28:0/1999408411 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f7cd4009510 con 0x7f7cdc100a90 2024-03-22T02:13:43.278 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.279+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 msgr2=0x7f7cc80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.279+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 0x7f7cc80a53e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7cc402c180 tx=0x7f7cc4034ad0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.279+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 msgr2=0x7f7cdc197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 0x7f7cdc197eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc101ae0 tx=0x7f7cd400ee50 comp rx=0 tx=0).stop 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 shutdown_connections 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7cc80a2f20 0x7f7cc80a53e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc102640 0x7f7cdc19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7cdc101c80 0x7f7cdc1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 --2- 172.21.15.28:0/1999408411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7cdc100a90 0x7f7cdc197eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 >> 172.21.15.28:0/1999408411 conn(0x7f7cdc0fc020 msgr2=0x7f7cdc104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 shutdown_connections 2024-03-22T02:13:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.280+0000 7f7ce17c7640 1 -- 172.21.15.28:0/1999408411 wait complete. 2024-03-22T02:13:43.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 -- 172.21.15.28:0/1186533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450100aa0 msgr2=0x7fa450100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1186533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450100aa0 0x7fa450100ec0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa448009900 tx=0x7fa44802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 -- 172.21.15.28:0/1186533602 shutdown_connections 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1186533602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa450108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1186533602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450101c90 0x7fa450102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1186533602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450100aa0 0x7fa450100ec0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 -- 172.21.15.28:0/1186533602 >> 172.21.15.28:0/1186533602 conn(0x7fa4500fc030 msgr2=0x7fa4500fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.397+0000 7fa4559bd640 1 -- 172.21.15.28:0/1186533602 shutdown_connections 2024-03-22T02:13:43.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 -- 172.21.15.28:0/1186533602 wait complete. 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 Processor -- start 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 -- start start 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450100aa0 0x7fa450197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa45019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d840 con 0x7fa450101c90 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d6c0 con 0x7fa450102650 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa4559bd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa45010d9c0 con 0x7fa450100aa0 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.398+0000 7fa44e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49400/0 (socket says 172.21.15.28:49400) 2024-03-22T02:13:43.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 -- 172.21.15.28:0/1790991540 learned_addr learned my addr 172.21.15.28:0/1790991540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44effd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450100aa0 0x7fa450197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa45019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 -- 172.21.15.28:0/1790991540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450100aa0 msgr2=0x7fa450197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450100aa0 0x7fa450197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 -- 172.21.15.28:0/1790991540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 msgr2=0x7fa45019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa45019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 -- 172.21.15.28:0/1790991540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45019fb40 con 0x7fa450101c90 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44f7fe640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa45019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa44e7fc640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa43800e9f0 tx=0x7fa43800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43800cd80 con 0x7fa450101c90 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa438005500 con 0x7fa450101c90 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45019fe30 con 0x7fa450101c90 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa438010690 con 0x7fa450101c90 2024-03-22T02:13:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.399+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4501a02a0 con 0x7fa450101c90 2024-03-22T02:13:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.400+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa450107e30 con 0x7fa450101c90 2024-03-22T02:13:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.401+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fa4380046c0 con 0x7fa450101c90 2024-03-22T02:13:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.402+0000 7fa4549bb640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 0x7fa43c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.402+0000 7fa44effd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 0x7fa43c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.402+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa4380cfd90 con 0x7fa450101c90 2024-03-22T02:13:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.402+0000 7fa44effd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 0x7fa43c0a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa448000c00 tx=0x7fa448005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.405+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa438053740 con 0x7fa450101c90 2024-03-22T02:13:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:43 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1523767978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:43 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1313653862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:43 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1439025702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:43 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1999408411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.590+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa45002d3b0 con 0x7fa450101c90 2024-03-22T02:13:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.591+0000 7fa4549bb640 1 -- 172.21.15.28:0/1790991540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fa4380bae20 con 0x7fa450101c90 2024-03-22T02:13:43.591 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 msgr2=0x7fa43c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 0x7fa43c0a53e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa448000c00 tx=0x7fa448005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 msgr2=0x7fa450198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa43800e9f0 tx=0x7fa43800edf0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 shutdown_connections 2024-03-22T02:13:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa43c0a2f20 0x7fa43c0a53e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa450102650 0x7fa45019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa450101c90 0x7fa450198460 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 --2- 172.21.15.28:0/1790991540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa450100aa0 0x7fa450197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 >> 172.21.15.28:0/1790991540 conn(0x7fa4500fc030 msgr2=0x7fa450105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 shutdown_connections 2024-03-22T02:13:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.593+0000 7fa4559bd640 1 -- 172.21.15.28:0/1790991540 wait complete. 2024-03-22T02:13:43.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- 172.21.15.28:0/524054633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101c80 msgr2=0x7fd098102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/524054633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101c80 0x7fd098102100 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009900 tx=0x7fd08c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- 172.21.15.28:0/524054633 shutdown_connections 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/524054633 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098102640 0x7fd098108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/524054633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098101c80 0x7fd098102100 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/524054633 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098100a90 0x7fd098100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- 172.21.15.28:0/524054633 >> 172.21.15.28:0/524054633 conn(0x7fd0980fc020 msgr2=0x7fd0980fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- 172.21.15.28:0/524054633 shutdown_connections 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- 172.21.15.28:0/524054633 wait complete. 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 Processor -- start 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- start start 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 0x7fd0981984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.710+0000 7fd09e6fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810c6f0 con 0x7fd098100a90 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09e6fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810c570 con 0x7fd098101c80 2024-03-22T02:13:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09e6fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd09810c870 con 0x7fd098102640 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09cc98640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49416/0 (socket says 172.21.15.28:49416) 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09cc98640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52838/0 (socket says 172.21.15.28:52838) 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 -- 172.21.15.28:0/2470645854 learned_addr learned my addr 172.21.15.28:0/2470645854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 0x7fd0981984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 -- 172.21.15.28:0/2470645854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 msgr2=0x7fd09819f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 -- 172.21.15.28:0/2470645854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 msgr2=0x7fd0981984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 0x7fd0981984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 -- 172.21.15.28:0/2470645854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09819fb50 con 0x7fd098100a90 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09cc98640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd0977fe640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 0x7fd0981984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd097fff640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd08800e9c0 tx=0x7fd08800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800cd80 con 0x7fd098100a90 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09819fe40 con 0x7fd098100a90 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd088010910 con 0x7fd098100a90 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088018980 con 0x7fd098100a90 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.711+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0981a02b0 con 0x7fd098100a90 2024-03-22T02:13:43.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.712+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0981a04e0 con 0x7fd098100a90 2024-03-22T02:13:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.714+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fd08801f020 con 0x7fd098100a90 2024-03-22T02:13:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.714+0000 7fd0957fa640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 0x7fd0800a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.714+0000 7fd0977fe640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 0x7fd0800a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.715+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0880cf8b0 con 0x7fd098100a90 2024-03-22T02:13:43.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.715+0000 7fd0977fe640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 0x7fd0800a53e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c7b0 tx=0x7fd08c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:43.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.719+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd08809c2d0 con 0x7fd098100a90 2024-03-22T02:13:43.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:43 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1523767978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:43 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1313653862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:43 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1439025702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:43 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1999408411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:43 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1523767978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:43 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1313653862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:43 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1439025702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:43 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1999408411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:43.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.903+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd0981a06b0 con 0x7fd098100a90 2024-03-22T02:13:43.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.904+0000 7fd0957fa640 1 -- 172.21.15.28:0/2470645854 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fd08809bbd0 con 0x7fd098100a90 2024-03-22T02:13:43.905 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:43.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 msgr2=0x7fd0800a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 0x7fd0800a53e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd08c02c7b0 tx=0x7fd08c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 msgr2=0x7fd098197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd08800e9c0 tx=0x7fd08800edc0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 shutdown_connections 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0800a2f20 0x7fd0800a53e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd098102640 0x7fd09819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd098101c80 0x7fd0981984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 --2- 172.21.15.28:0/2470645854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd098100a90 0x7fd098197f80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 >> 172.21.15.28:0/2470645854 conn(0x7fd0980fc020 msgr2=0x7fd0980fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 shutdown_connections 2024-03-22T02:13:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:43.906+0000 7fd09e6fb640 1 -- 172.21.15.28:0/2470645854 wait complete. 2024-03-22T02:13:43.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.020+0000 7f8d118b5640 1 -- 172.21.15.28:0/3647698252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c102da0 msgr2=0x7f8d0c103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.020+0000 7f8d118b5640 1 --2- 172.21.15.28:0/3647698252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c102da0 0x7f8d0c103220 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009900 tx=0x7f8d0002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- 172.21.15.28:0/3647698252 shutdown_connections 2024-03-22T02:13:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- 172.21.15.28:0/3647698252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c103760 0x7f8d0c109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- 172.21.15.28:0/3647698252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c102da0 0x7f8d0c103220 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- 172.21.15.28:0/3647698252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 0x7f8d0c101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- 172.21.15.28:0/3647698252 >> 172.21.15.28:0/3647698252 conn(0x7f8d0c0fd140 msgr2=0x7f8d0c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- 172.21.15.28:0/3647698252 shutdown_connections 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- 172.21.15.28:0/3647698252 wait complete. 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 Processor -- start 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- start start 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 0x7f8d0c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c102da0 0x7f8d0c198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d830 con 0x7f8d0c103760 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d6b0 con 0x7f8d0c101bb0 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.021+0000 7f8d118b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0c10d9b0 con 0x7f8d0c102da0 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0a7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c102da0 0x7f8d0c198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 0x7f8d0c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49432/0 (socket says 172.21.15.28:49432) 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 -- 172.21.15.28:0/1896301834 learned_addr learned my addr 172.21.15.28:0/1896301834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 -- 172.21.15.28:0/1896301834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c102da0 msgr2=0x7f8d0c198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c102da0 0x7f8d0c198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 -- 172.21.15.28:0/1896301834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 msgr2=0x7f8d0c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 0x7f8d0c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 -- 172.21.15.28:0/1896301834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0c19faf0 con 0x7f8d0c103760 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d0b7fe640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400eb10 tx=0x7f8d0400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04004b20 con 0x7f8d0c103760 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d04004cc0 con 0x7f8d0c103760 2024-03-22T02:13:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0c19fde0 con 0x7f8d0c103760 2024-03-22T02:13:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d04010760 con 0x7f8d0c103760 2024-03-22T02:13:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.022+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0c1a02f0 con 0x7f8d0c103760 2024-03-22T02:13:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.023+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d0c108ed0 con 0x7f8d0c103760 2024-03-22T02:13:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.024+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f8d04005000 con 0x7f8d0c103760 2024-03-22T02:13:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.025+0000 7f8d108b3640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 0x7f8cf40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.025+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8d040d0470 con 0x7f8d0c103760 2024-03-22T02:13:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.025+0000 7f8d0affd640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 0x7f8cf40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.025+0000 7f8d0affd640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 0x7f8cf40a5460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c102c00 tx=0x7f8cfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.029+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d0409ce30 con 0x7f8d0c103760 2024-03-22T02:13:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.250+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d0c02d3b0 con 0x7f8d0c103760 2024-03-22T02:13:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.251+0000 7f8d108b3640 1 -- 172.21.15.28:0/1896301834 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11866 (secure 0 0 0) 0x7f8d0409c730 con 0x7f8d0c103760 2024-03-22T02:13:44.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:44.252 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:11:54.987732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:13:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 msgr2=0x7f8cf40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 0x7f8cf40a5460 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c102c00 tx=0x7f8cfc005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 msgr2=0x7f8d0c19f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8d0400eb10 tx=0x7f8d0400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 shutdown_connections 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cf40a2fa0 0x7f8cf40a5460 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d0c103760 0x7f8d0c19f480 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d0c102da0 0x7f8d0c198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 --2- 172.21.15.28:0/1896301834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d0c101bb0 0x7f8d0c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 >> 172.21.15.28:0/1896301834 conn(0x7f8d0c0fd140 msgr2=0x7f8d0c1060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 shutdown_connections 2024-03-22T02:13:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.253+0000 7f8d118b5640 1 -- 172.21.15.28:0/1896301834 wait complete. 2024-03-22T02:13:44.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 -- 172.21.15.28:0/656947992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8103770 msgr2=0x7f93b810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 --2- 172.21.15.28:0/656947992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8103770 0x7f93b810a000 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009ca0 tx=0x7f93b002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 -- 172.21.15.28:0/656947992 shutdown_connections 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 --2- 172.21.15.28:0/656947992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8103770 0x7f93b810a000 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 --2- 172.21.15.28:0/656947992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 0x7f93b8103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 --2- 172.21.15.28:0/656947992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8101bc0 0x7f93b8101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 -- 172.21.15.28:0/656947992 >> 172.21.15.28:0/656947992 conn(0x7f93b80fd170 msgr2=0x7f93b80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 -- 172.21.15.28:0/656947992 shutdown_connections 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 -- 172.21.15.28:0/656947992 wait complete. 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.368+0000 7f93bd288640 1 Processor -- start 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 -- start start 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 0x7f93b8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8103770 0x7f93b819f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810d840 con 0x7f93b8101bc0 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810d6c0 con 0x7f93b8103770 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93bd288640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f93b810d9c0 con 0x7f93b8102db0 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49454/0 (socket says 172.21.15.28:49454) 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 -- 172.21.15.28:0/2872626438 learned_addr learned my addr 172.21.15.28:0/2872626438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b67fc640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 0x7f93b8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b77fe640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8103770 0x7f93b819f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 -- 172.21.15.28:0/2872626438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 msgr2=0x7f93b8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 0x7f93b8198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 -- 172.21.15.28:0/2872626438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8103770 msgr2=0x7f93b819f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8103770 0x7f93b819f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 -- 172.21.15.28:0/2872626438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b819fa50 con 0x7f93b8101bc0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.369+0000 7f93b6ffd640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f93a800e9c0 tx=0x7f93a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.370+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a800cd80 con 0x7f93b8101bc0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.370+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93a80035f0 con 0x7f93b8101bc0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.370+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93a8010690 con 0x7f93b8101bc0 2024-03-22T02:13:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.370+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b819fd40 con 0x7f93b8101bc0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.370+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b81a0180 con 0x7f93b8101bc0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.371+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93b8108e80 con 0x7f93b8101bc0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.372+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f93a80037a0 con 0x7f93b8101bc0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.373+0000 7f939ffff640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 0x7f93a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.373+0000 7f93b67fc640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 0x7f93a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.373+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f93a80cfb60 con 0x7f93b8101bc0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.373+0000 7f93b67fc640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 0x7f93a00a53e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0095f0 tx=0x7f93ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.377+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93a809c580 con 0x7f93b8101bc0 2024-03-22T02:13:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.563+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93b8062890 con 0x7f93b8101bc0 2024-03-22T02:13:44.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.565+0000 7f939ffff640 1 -- 172.21.15.28:0/2872626438 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7f93a809be80 con 0x7f93b8101bc0 2024-03-22T02:13:44.565 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:44.565 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 msgr2=0x7f93a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 0x7f93a00a53e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0095f0 tx=0x7f93ac005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 msgr2=0x7f93b8197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f93a800e9c0 tx=0x7f93a800ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 shutdown_connections 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f93a00a2f20 0x7f93a00a53e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f93b8103770 0x7f93b819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f93b8102db0 0x7f93b8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 --2- 172.21.15.28:0/2872626438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93b8101bc0 0x7f93b8197e30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.567+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 >> 172.21.15.28:0/2872626438 conn(0x7f93b80fd170 msgr2=0x7f93b8106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.568+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 shutdown_connections 2024-03-22T02:13:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.568+0000 7f93bd288640 1 -- 172.21.15.28:0/2872626438 wait complete. 2024-03-22T02:13:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:44 smithi028 ceph-mon[31916]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:44 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1790991540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:44 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2470645854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:44 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1896301834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:44.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-22T02:11:33.259569+0000', 'modified': '2024-03-22T02:11:54.987732+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24509, 'mds_1': 24508, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24443': {'gid': 24443, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6835/3494340980', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6834', 'nonce': 3494340980}, {'type': 'v1', 'addr': '172.21.15.83:6835', 'nonce': 3494340980}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24466': {'gid': 24466, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1917291995', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1917291995}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1917291995}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.83:6837/3509701876', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6836', 'nonce': 3509701876}, {'type': 'v1', 'addr': '172.21.15.83:6837', 'nonce': 3509701876}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.120:6837/2149554336', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 2149554336}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 2149554336}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.120:6839/990751976', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 990751976}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 990751976}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24509': {'gid': 24509, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.83:6839/2818900206', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.83:6838', 'nonce': 2818900206}, {'type': 'v1', 'addr': '172.21.15.83:6839', 'nonce': 2818900206}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24509, 'qdb_cluster': [24478, 24508, 24509]} 2024-03-22T02:13:44.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-22T02:13:44.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 -- 172.21.15.28:0/123051277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 msgr2=0x7fdb281073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/123051277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb281073e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009900 tx=0x7fdb1c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 -- 172.21.15.28:0/123051277 shutdown_connections 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/123051277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 0x7fdb28109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/123051277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb281073e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/123051277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 -- 172.21.15.28:0/123051277 >> 172.21.15.28:0/123051277 conn(0x7fdb280fc030 msgr2=0x7fdb280fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 -- 172.21.15.28:0/123051277 shutdown_connections 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.683+0000 7fdb2e42a640 1 -- 172.21.15.28:0/123051277 wait complete. 2024-03-22T02:13:44.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 Processor -- start 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 -- start start 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 0x7fdb2819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2810d7f0 con 0x7fdb28104ff0 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2810d670 con 0x7fdb281026c0 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2e42a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2810d970 con 0x7fdb28107920 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49460/0 (socket says 172.21.15.28:49460) 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb277fe640 1 -- 172.21.15.28:0/7832703 learned_addr learned my addr 172.21.15.28:0/7832703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 0x7fdb2819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 -- 172.21.15.28:0/7832703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 msgr2=0x7fdb2819f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 0x7fdb2819f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 -- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 msgr2=0x7fdb28198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb27fff640 1 -- 172.21.15.28:0/7832703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb2819fb10 con 0x7fdb281026c0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.684+0000 7fdb277fe640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb27fff640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28197ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e9f0 tx=0x7fdb1800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb257fa640 1 -- 172.21.15.28:0/7832703 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb1800cd80 con 0x7fdb281026c0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb257fa640 1 -- 172.21.15.28:0/7832703 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb180035f0 con 0x7fdb281026c0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb2819fe00 con 0x7fdb281026c0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb257fa640 1 -- 172.21.15.28:0/7832703 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18010690 con 0x7fdb281026c0 2024-03-22T02:13:44.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.685+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb281a0260 con 0x7fdb281026c0 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.687+0000 7fdb257fa640 1 -- 172.21.15.28:0/7832703 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fdb18003b30 con 0x7fdb281026c0 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 Processor -- start 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 -- start start 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb257fa640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 0x7fdb100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6a80 0x5576a8abb580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8abe080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a8a16210 con 0x5576a8ab6a80 2024-03-22T02:13:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5576a89bec80 con 0x5576a8abbbf0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb31011740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576a8a121c0 con 0x5576a8ab6090 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6a80 0x5576a8abb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52918/0 (socket says 172.21.15.28:52918) 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/3470205566 learned_addr learned my addr 172.21.15.28:0/3470205566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb27fff640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 0x7fdb100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8abe080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb257fa640 1 -- 172.21.15.28:0/7832703 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdb180cfb00 con 0x7fdb281026c0 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/3470205566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 msgr2=0x5576a8abe080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8abe080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/3470205566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6a80 msgr2=0x5576a8abb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6a80 0x5576a8abb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.688+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/3470205566 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a872a2d0 con 0x5576a8ab6090 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005f40 tx=0x7fdb20009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bc412253c262f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb27fff640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 0x7fdb100a53e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb18098220 tx=0x7fdb18003aa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb0e7fc640 1 -- 172.21.15.28:0/3470205566 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20011070 con 0x5576a8ab6090 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb0e7fc640 1 -- 172.21.15.28:0/3470205566 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb2000fd60 con 0x5576a8ab6090 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb0e7fc640 1 -- 172.21.15.28:0/3470205566 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20010c80 con 0x5576a8ab6090 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 -- 172.21.15.28:0/3470205566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 msgr2=0x5576a8ab64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005f40 tx=0x7fdb20009b80 comp rx=0 tx=0).stop 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 -- 172.21.15.28:0/3470205566 shutdown_connections 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8abe080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6a80 0x5576a8abb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 --2- 172.21.15.28:0/3470205566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ab64b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 -- 172.21.15.28:0/3470205566 >> 172.21.15.28:0/3470205566 conn(0x5576a8aad1f0 msgr2=0x5576a8aad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 -- 172.21.15.28:0/3470205566 shutdown_connections 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.689+0000 7fdb31011740 1 -- 172.21.15.28:0/3470205566 wait complete. 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 -- start start 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 Processor -- start 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6090 0x5576a8ac0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8ac07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a8abeba0 con 0x5576a8abbbf0 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5576a89d1f80 con 0x5576a8ab6090 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576a8abed20 con 0x5576a8ab6a80 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb31011740 10 client.0 _ll_register_callbacks cb 0x5576a8a0e4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49484/0 (socket says 172.21.15.28:49484) 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8ac07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1102972685 learned_addr learned my addr 172.21.15.28:0/1102972685 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.690+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6090 0x5576a8ac0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 msgr2=0x5576a8ac07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8ac07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1102972685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6090 msgr2=0x5576a8ac0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6090 0x5576a8ac0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1102972685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a87f46b0 con 0x5576a8abbbf0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb277fe640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8ac07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005b40 tx=0x7fdb20009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55449461bdc26864 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20004a40 con 0x5576a8abbbf0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdb20004be0 con 0x5576a8abbbf0 2024-03-22T02:13:44.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb2000fd60 con 0x5576a8abbbf0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb20034050 con 0x5576a8abbbf0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb31011740 10 client.15405 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5576a8acaaf0 con 0x5576a8abbbf0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb2000fd60 con 0x5576a8abbbf0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.691+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdb2002c020 con 0x5576a8abbbf0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 10 client.15405 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 10 client.15405 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 20 client.15405 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8acced0 0x5576a8ace340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 4 client.15405 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576a8a23750 con 0x5576a8acced0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb277fe640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8acced0 0x5576a8ace340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb077fe640 10 client.15405 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.692+0000 7fdb277fe640 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8acced0 0x5576a8ace340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f11ea1fddfc396e3 server_cookie=7834343475cfb838 in_seq=0 out_seq=0 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb077fe640 1 -- 172.21.15.28:0/1102972685 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576a8a23750 con 0x5576a8acced0 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb077fe640 10 client.15405 handle_command_reply: tid=0 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 client.15405 shutdown 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 msgr2=0x5576a8ac7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdb20005b40 tx=0x7fdb20009f40 comp rx=0 tx=0).stop 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 shutdown_connections 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8ac7780 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8ac07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8acced0 0x5576a8ace340 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 --2- 172.21.15.28:0/1102972685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6090 0x5576a8ac0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb077fe640 0 client.15405 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:44.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 >> 172.21.15.28:0/1102972685 conn(0x5576a8aad1f0 msgr2=0x5576a8aac4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 shutdown_connections 2024-03-22T02:13:44.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.693+0000 7fdb31011740 1 -- 172.21.15.28:0/1102972685 wait complete. 2024-03-22T02:13:44.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.694+0000 7fdb31011740 20 client.15405 trim_cache size 0 max 16384 2024-03-22T02:13:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.711+0000 7fdb31011740 1 Processor -- start 2024-03-22T02:13:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 -- start start 2024-03-22T02:13:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ac0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6a80 0x5576a8ac76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a8acc680 con 0x5576a8abbbf0 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5576a8accd50 con 0x5576a8ab6a80 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb31011740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576a8acd420 con 0x5576a8ab6090 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49486/0 (socket says 172.21.15.28:49486) 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1000067408 learned_addr learned my addr 172.21.15.28:0/1000067408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb277fe640 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6a80 0x5576a8ac76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ac0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1000067408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 msgr2=0x5576a8ac0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ac0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1000067408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6a80 msgr2=0x5576a8ac76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6a80 0x5576a8ac76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/1000067408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576a8abec60 con 0x5576a8abbbf0 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009f10 tx=0x7fdb20024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6618fbd84c43d22 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb0dffb640 1 -- 172.21.15.28:0/1000067408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb200106e0 con 0x5576a8abbbf0 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb0dffb640 1 -- 172.21.15.28:0/1000067408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb20003e50 con 0x5576a8abbbf0 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.712+0000 7fdb0dffb640 1 -- 172.21.15.28:0/1000067408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2000fbf0 con 0x5576a8abbbf0 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- 172.21.15.28:0/1000067408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 msgr2=0x5576a8aad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fdb20009f10 tx=0x7fdb20024040 comp rx=0 tx=0).stop 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- 172.21.15.28:0/1000067408 shutdown_connections 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8abbbf0 0x5576a8aad5d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8ab6a80 0x5576a8ac76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 --2- 172.21.15.28:0/1000067408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6090 0x5576a8ac0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- 172.21.15.28:0/1000067408 >> 172.21.15.28:0/1000067408 conn(0x5576a8aad1f0 msgr2=0x5576a8ab0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- 172.21.15.28:0/1000067408 shutdown_connections 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- 172.21.15.28:0/1000067408 wait complete. 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 -- start start 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.713+0000 7fdb31011740 1 Processor -- start 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8b2ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8b31170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576a8ace1c0 con 0x5576a8ab6090 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5576a8acdaf0 con 0x5576a8abbbf0 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5576a8ace890 con 0x5576a8ab6a80 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 10 client.0 _ll_register_callbacks cb 0x7fdb200093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49500/0 (socket says 172.21.15.28:49500) 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8b2ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/988094258 learned_addr learned my addr 172.21.15.28:0/988094258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8b31170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 msgr2=0x5576a8b2ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8b2ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/988094258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 msgr2=0x5576a8b31170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8b31170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 -- 172.21.15.28:0/988094258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576a8b316b0 con 0x5576a8ab6090 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb2c9c7640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdb2002d440 tx=0x7fdb2002d3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce5339783a9cc5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb077fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20009b80 con 0x5576a8ab6090 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb20003e50 con 0x5576a8ab6090 2024-03-22T02:13:44.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdb2000fbf0 con 0x5576a8ab6090 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.714+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdb2000fdf0 con 0x5576a8ab6090 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5576a8b32950 con 0x5576a8ab6090 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb20003e50 con 0x5576a8ab6090 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fdb2002c070 con 0x5576a8ab6090 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 10 client.15411 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 20 client.15411 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8b37210 0x5576a8b33d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 4 client.15411 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "status"}) -- 0x5576a8a1daa0 con 0x5576a8b37210 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb277fe640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8b37210 0x5576a8b33d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb077fe640 10 client.15411 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.715+0000 7fdb277fe640 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8b37210 0x5576a8b33d20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ac70a4317cbec52 server_cookie=8579dab8f02838ef in_seq=0 out_seq=0 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb077fe640 1 -- 172.21.15.28:0/988094258 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5576a8a1daa0 con 0x5576a8b37210 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb077fe640 10 client.15411 handle_command_reply: tid=0 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 client.15411 shutdown 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 msgr2=0x5576a8ac5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdb2002d440 tx=0x7fdb2002d3d0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 shutdown_connections 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5576a8b37210 0x5576a8b33d20 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5576a8abbbf0 0x5576a8b31170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5576a8ab6a80 0x5576a8b2ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb077fe640 0 client.15411 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 --2- 172.21.15.28:0/988094258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576a8ab6090 0x5576a8ac5690 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 >> 172.21.15.28:0/988094258 conn(0x5576a8aad1f0 msgr2=0x5576a8ab0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 shutdown_connections 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 1 -- 172.21.15.28:0/988094258 wait complete. 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb31011740 20 client.15411 trim_cache size 0 max 16384 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:44.724 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24478, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 110.735766142, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 149.35793551500001 2024-03-22T02:13:44.725 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 msgr2=0x7fdb100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 0x7fdb100a53e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb18098220 tx=0x7fdb18003aa0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 msgr2=0x7fdb28197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.716+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28197ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdb1800e9f0 tx=0x7fdb1800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 shutdown_connections 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb100a2f20 0x7fdb100a53e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb28107920 0x7fdb2819f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb28104ff0 0x7fdb28198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 --2- 172.21.15.28:0/7832703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb281026c0 0x7fdb28197ef0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 >> 172.21.15.28:0/7832703 conn(0x7fdb280fc030 msgr2=0x7fdb28103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 shutdown_connections 2024-03-22T02:13:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.717+0000 7fdb2e42a640 1 -- 172.21.15.28:0/7832703 wait complete. 2024-03-22T02:13:44.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-22T02:13:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:44 smithi120 ceph-mon[36803]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:44 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1790991540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:44 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2470645854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:44 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1896301834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:44 smithi083 ceph-mon[36340]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 341 B/s wr, 6 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:44 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1790991540' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:44 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2470645854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:44 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1896301834' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 -- 172.21.15.28:0/3988580519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164100a90 msgr2=0x7fa164100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 --2- 172.21.15.28:0/3988580519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164100a90 0x7fa164100eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009900 tx=0x7fa15c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 -- 172.21.15.28:0/3988580519 shutdown_connections 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 --2- 172.21.15.28:0/3988580519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa164108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 --2- 172.21.15.28:0/3988580519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164101c80 0x7fa164102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 --2- 172.21.15.28:0/3988580519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164100a90 0x7fa164100eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 -- 172.21.15.28:0/3988580519 >> 172.21.15.28:0/3988580519 conn(0x7fa1640fc020 msgr2=0x7fa1640fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 -- 172.21.15.28:0/3988580519 shutdown_connections 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.831+0000 7fa168e4e640 1 -- 172.21.15.28:0/3988580519 wait complete. 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 Processor -- start 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 -- start start 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 0x7fa16419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164101c80 0x7fa16419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa16410d830 con 0x7fa164102640 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa16410d6b0 con 0x7fa164100a90 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa168e4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa16410d9b0 con 0x7fa164101c80 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa162d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa162d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49530/0 (socket says 172.21.15.28:49530) 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa162d9d640 1 -- 172.21.15.28:0/1773280367 learned_addr learned my addr 172.21.15.28:0/1773280367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164101c80 0x7fa16419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.832+0000 7fa16259c640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 0x7fa16419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 -- 172.21.15.28:0/1773280367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164101c80 msgr2=0x7fa16419a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164101c80 0x7fa16419a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 -- 172.21.15.28:0/1773280367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 msgr2=0x7fa16419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 0x7fa16419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 -- 172.21.15.28:0/1773280367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1641a1d70 con 0x7fa164102640 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa16259c640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 0x7fa16419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa162d9d640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa15400eb10 tx=0x7fa15400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa1537fe640 1 -- 172.21.15.28:0/1773280367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa154004b20 con 0x7fa164102640 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1641a2060 con 0x7fa164102640 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa1537fe640 1 -- 172.21.15.28:0/1773280367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa154004cc0 con 0x7fa164102640 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1641a25b0 con 0x7fa164102640 2024-03-22T02:13:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.833+0000 7fa1537fe640 1 -- 172.21.15.28:0/1773280367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa154010700 con 0x7fa164102640 2024-03-22T02:13:44.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.835+0000 7fa1537fe640 1 -- 172.21.15.28:0/1773280367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fa154005000 con 0x7fa164102640 2024-03-22T02:13:44.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.835+0000 7fa16ba34740 1 Processor -- start 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 -- start start 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x5619314ea170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314ecc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561931449210 con 0x5619314e5670 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5619313f1c80 con 0x5619314ea7e0 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16ba34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5619314451c0 con 0x5619314e4c80 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa1537fe640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 0x7fa14c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa16259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x5619314ea170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52968/0 (socket says 172.21.15.28:52968) 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 -- 172.21.15.28:0/1378404578 learned_addr learned my addr 172.21.15.28:0/1378404578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314ecc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa162d9d640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 0x7fa14c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa1537fe640 1 -- 172.21.15.28:0/1773280367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa1540cfe80 con 0x7fa164102640 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 -- 172.21.15.28:0/1378404578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 msgr2=0x5619314ecc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314ecc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 -- 172.21.15.28:0/1378404578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 msgr2=0x5619314ea170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x5619314ea170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 -- 172.21.15.28:0/1378404578 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56193115d2d0 con 0x5619314e4c80 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa158011810 tx=0x7fa1580097d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9657b146554d2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.836+0000 7fa162d9d640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 0x7fa14c0a53e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa1540cd8e0 tx=0x7fa15409f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa150ff9640 1 -- 172.21.15.28:0/1378404578 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158004040 con 0x5619314e4c80 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa150ff9640 1 -- 172.21.15.28:0/1378404578 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa158011e50 con 0x5619314e4c80 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa150ff9640 1 -- 172.21.15.28:0/1378404578 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158003850 con 0x5619314e4c80 2024-03-22T02:13:44.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 -- 172.21.15.28:0/1378404578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 msgr2=0x5619314e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa158011810 tx=0x7fa1580097d0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 -- 172.21.15.28:0/1378404578 shutdown_connections 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314ecc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x5619314ea170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1378404578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314e50a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 -- 172.21.15.28:0/1378404578 >> 172.21.15.28:0/1378404578 conn(0x5619314e01f0 msgr2=0x5619314e05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 -- 172.21.15.28:0/1378404578 shutdown_connections 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.837+0000 7fa16ba34740 1 -- 172.21.15.28:0/1378404578 wait complete. 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 -- start start 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 Processor -- start 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619314ed790 con 0x5619314e4c80 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561931404f80 con 0x5619314ea7e0 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5619314ed910 con 0x5619314e5670 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16ba34740 10 client.0 _ll_register_callbacks cb 0x5619314414f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52976/0 (socket says 172.21.15.28:52976) 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa161d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 -- 172.21.15.28:0/3166144475 learned_addr learned my addr 172.21.15.28:0/3166144475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa161d9b640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 -- 172.21.15.28:0/3166144475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 msgr2=0x5619314f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 -- 172.21.15.28:0/3166144475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 msgr2=0x5619314eee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa16259c640 1 -- 172.21.15.28:0/3166144475 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5619312276b0 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.838+0000 7fa161d9b640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16259c640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009900 tx=0x7fa15c005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d46855fbca2c3936 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c005d50 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa158011e50 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa15c0048c0 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa15c005650 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16ba34740 10 client.25249 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5619314f97e0 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa158011e50 con 0x5619314e5670 2024-03-22T02:13:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa15c044070 con 0x5619314e5670 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16ba34740 10 client.25249 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.839+0000 7fa16ba34740 10 client.25249 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa16ba34740 20 client.25249 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619314fbde0 0x5619314fd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa16ba34740 4 client.25249 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561931456750 con 0x5619314fbde0 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa161d9b640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619314fbde0 0x5619314fd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa149ffb640 10 client.25249 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.840+0000 7fa161d9b640 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619314fbde0 0x5619314fd250 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fd302ebcd7e5093 server_cookie=d61fd86430f366ae in_seq=0 out_seq=0 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa149ffb640 1 -- 172.21.15.28:0/3166144475 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561931456750 con 0x5619314fbde0 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa149ffb640 10 client.25249 handle_command_reply: tid=0 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 client.25249 shutdown 2024-03-22T02:13:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 msgr2=0x5619314ef370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa15c009900 tx=0x7fa15c005250 comp rx=0 tx=0).stop 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 shutdown_connections 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619314fbde0 0x5619314fd250 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314ef370 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa149ffb640 0 client.25249 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 --2- 172.21.15.28:0/3166144475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 >> 172.21.15.28:0/3166144475 conn(0x5619314e01f0 msgr2=0x5619314df4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 shutdown_connections 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 1 -- 172.21.15.28:0/3166144475 wait complete. 2024-03-22T02:13:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.841+0000 7fa16ba34740 20 client.25249 trim_cache size 0 max 16384 2024-03-22T02:13:44.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 Processor -- start 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 -- start start 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314f6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314eef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619314fb590 con 0x5619314e4c80 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5619314fbc60 con 0x5619314ea7e0 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.859+0000 7fa16ba34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5619314fc330 con 0x5619314e5670 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 -- 172.21.15.28:0/1845036509 learned_addr learned my addr 172.21.15.28:0/1845036509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa16259c640 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 -- 172.21.15.28:0/1845036509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 msgr2=0x5619314f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 -- 172.21.15.28:0/1845036509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 msgr2=0x5619314eef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314eef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 -- 172.21.15.28:0/1845036509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5619314edee0 con 0x5619314e4c80 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa1580115b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aef955aa2af7d0f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa14bfff640 1 -- 172.21.15.28:0/1845036509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15800fcc0 con 0x5619314e4c80 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa14bfff640 1 -- 172.21.15.28:0/1845036509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15800fe60 con 0x5619314e4c80 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.860+0000 7fa14bfff640 1 -- 172.21.15.28:0/1845036509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580035f0 con 0x5619314e4c80 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- 172.21.15.28:0/1845036509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 msgr2=0x5619314eea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa158006010 tx=0x7fa1580115b0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- 172.21.15.28:0/1845036509 shutdown_connections 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x5619314eef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e5670 0x5619314f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1845036509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e4c80 0x5619314eea40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- 172.21.15.28:0/1845036509 >> 172.21.15.28:0/1845036509 conn(0x5619314e01f0 msgr2=0x5619314fac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- 172.21.15.28:0/1845036509 shutdown_connections 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- 172.21.15.28:0/1845036509 wait complete. 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- start start 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 Processor -- start 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314f4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x561931560080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5619314fd0d0 con 0x5619314e5670 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5619314fca00 con 0x5619314ea7e0 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5619314fd7a0 con 0x5619314e4c80 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16ba34740 10 client.0 _ll_register_callbacks cb 0x7fa15c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:44.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49570/0 (socket says 172.21.15.28:49570) 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa16259c640 1 -- 172.21.15.28:0/1096214147 learned_addr learned my addr 172.21.15.28:0/1096214147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.861+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x561931560080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 -- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 msgr2=0x5619314f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 -- 172.21.15.28:0/1096214147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 msgr2=0x561931560080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x561931560080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 -- 172.21.15.28:0/1096214147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561931560610 con 0x5619314e5670 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16259c640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa15c003dd0 tx=0x7fa15c009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78f18a648fd1d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa149ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c059060 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa15c046910 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa15800fe60 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa15c005d50 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16ba34740 10 client.15429 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x561931561a10 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.862+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa15800fe60 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa15c044040 con 0x5619314e5670 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 10 client.15429 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 20 client.15429 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619315665f0 0x561931565300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 4 client.15429 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "status"}) -- 0x561931450aa0 con 0x5619315665f0 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619315665f0 0x561931565300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa161d9b640 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619315665f0 0x561931565300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f7cec02c6e38ce2 server_cookie=cb9af864a7ea9be3 in_seq=0 out_seq=0 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa149ffb640 10 client.15429 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa149ffb640 1 -- 172.21.15.28:0/1096214147 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x561931450aa0 con 0x5619315665f0 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa149ffb640 10 client.15429 handle_command_reply: tid=0 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 1 client.15429 shutdown 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 msgr2=0x56193155db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.863+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa15c003dd0 tx=0x7fa15c009290 comp rx=0 tx=0).stop 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 shutdown_connections 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5619315665f0 0x561931565300 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5619314ea7e0 0x561931560080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa149ffb640 0 client.15429 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5619314e5670 0x56193155db30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 --2- 172.21.15.28:0/1096214147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5619314e4c80 0x5619314f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 >> 172.21.15.28:0/1096214147 conn(0x5619314e01f0 msgr2=0x5619314eea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 shutdown_connections 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 1 -- 172.21.15.28:0/1096214147 wait complete. 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa16ba34740 20 client.15429 trim_cache size 0 max 16384 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:44.872 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24508, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 111.88864108600001, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 143.672167793 2024-03-22T02:13:44.873 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:44.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 msgr2=0x7fa14c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 0x7fa14c0a53e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa1540cd8e0 tx=0x7fa15409f040 comp rx=0 tx=0).stop 2024-03-22T02:13:44.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 msgr2=0x7fa1641a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa15400eb10 tx=0x7fa15400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 shutdown_connections 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa14c0a2f20 0x7fa14c0a53e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa164102640 0x7fa1641a1700 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa164101c80 0x7fa16419a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 --2- 172.21.15.28:0/1773280367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa164100a90 0x7fa16419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 >> 172.21.15.28:0/1773280367 conn(0x7fa1640fc020 msgr2=0x7fa164109710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 shutdown_connections 2024-03-22T02:13:44.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.864+0000 7fa168e4e640 1 -- 172.21.15.28:0/1773280367 wait complete. 2024-03-22T02:13:44.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-22T02:13:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 -- 172.21.15.28:0/4273728956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14102530 msgr2=0x7f9c14108dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 --2- 172.21.15.28:0/4273728956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14102530 0x7f9c14108dc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009ca0 tx=0x7f9c0c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 -- 172.21.15.28:0/4273728956 shutdown_connections 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 --2- 172.21.15.28:0/4273728956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14102530 0x7f9c14108dc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 --2- 172.21.15.28:0/4273728956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14101b70 0x7f9c14101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 --2- 172.21.15.28:0/4273728956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14100980 0x7f9c14100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 -- 172.21.15.28:0/4273728956 >> 172.21.15.28:0/4273728956 conn(0x7f9c140fbf10 msgr2=0x7f9c140fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 -- 172.21.15.28:0/4273728956 shutdown_connections 2024-03-22T02:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.979+0000 7f9c18a01640 1 -- 172.21.15.28:0/4273728956 wait complete. 2024-03-22T02:13:44.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 Processor -- start 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 -- start start 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14101b70 0x7f9c14109940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14102530 0x7f9c14109e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1410d720 con 0x7f9c14100980 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1410d5a0 con 0x7f9c14101b70 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c18a01640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1410d8a0 con 0x7f9c14102530 2024-03-22T02:13:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49586/0 (socket says 172.21.15.28:49586) 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 -- 172.21.15.28:0/3732782703 learned_addr learned my addr 172.21.15.28:0/3732782703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14101b70 0x7f9c14109940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c1359e640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14102530 0x7f9c14109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 -- 172.21.15.28:0/3732782703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14102530 msgr2=0x7f9c14109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14102530 0x7f9c14109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 -- 172.21.15.28:0/3732782703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14101b70 msgr2=0x7f9c14109940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14101b70 0x7f9c14109940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.980+0000 7f9c12d9d640 1 -- 172.21.15.28:0/3732782703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c1410a710 con 0x7f9c14100980 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e9c0 tx=0x7f9c0400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9bfbfff640 1 -- 172.21.15.28:0/3732782703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0400cd80 con 0x7f9c14100980 2024-03-22T02:13:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9bfbfff640 1 -- 172.21.15.28:0/3732782703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c040035f0 con 0x7f9c14100980 2024-03-22T02:13:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9bfbfff640 1 -- 172.21.15.28:0/3732782703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c04010690 con 0x7f9c14100980 2024-03-22T02:13:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c141a1e40 con 0x7f9c14100980 2024-03-22T02:13:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.981+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c141a2250 con 0x7f9c14100980 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.983+0000 7f9bfbfff640 1 -- 172.21.15.28:0/3732782703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f9c040037a0 con 0x7f9c14100980 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.983+0000 7f9c1bdef740 1 Processor -- start 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.983+0000 7f9c1bdef740 1 -- start start 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034ee60 0x55d78034f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d780354350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d7802b3210 con 0x55d7803549c0 2024-03-22T02:13:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d78025bc80 con 0x55d78034ee60 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7802af1c0 con 0x55d78034f850 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9bfbfff640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 0x7f9bfc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d780354350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1359e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034ee60 0x55d78034f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49594/0 (socket says 172.21.15.28:49594) 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 -- 172.21.15.28:0/3931104695 learned_addr learned my addr 172.21.15.28:0/3931104695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1359e640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 0x7f9bfc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9bfbfff640 1 -- 172.21.15.28:0/3732782703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9c040cfba0 con 0x7f9c14100980 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 -- 172.21.15.28:0/3931104695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 msgr2=0x55d780354350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d780354350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 -- 172.21.15.28:0/3931104695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034ee60 msgr2=0x55d78034f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034ee60 0x55d78034f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 -- 172.21.15.28:0/3931104695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d77ffc72d0 con 0x55d7803549c0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1359e640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 0x7f9bfc0a53e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005e30 tx=0x7f9c0c009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.984+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009c90 tx=0x7f9c08011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f7a5626738f946 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9bf97fa640 1 -- 172.21.15.28:0/3931104695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0801e070 con 0x55d7803549c0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9bf97fa640 1 -- 172.21.15.28:0/3931104695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c08019b10 con 0x55d7803549c0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9bf97fa640 1 -- 172.21.15.28:0/3931104695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0800e9c0 con 0x55d7803549c0 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3931104695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 msgr2=0x55d780356e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9c08009c90 tx=0x7f9c08011810 comp rx=0 tx=0).stop 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3931104695 shutdown_connections 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d780356e50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d780354350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3931104695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034ee60 0x55d78034f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3931104695 >> 172.21.15.28:0/3931104695 conn(0x55d78034a3d0 msgr2=0x55d78034a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3931104695 shutdown_connections 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.985+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3931104695 wait complete. 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 -- start start 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 Processor -- start 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780359020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034f850 0x55d780359560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d780357970 con 0x55d7803549c0 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d78026ef80 con 0x55d78034f850 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d780357af0 con 0x55d78034ee60 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1bdef740 10 client.0 _ll_register_callbacks cb 0x55d7802ab4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780359020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49604/0 (socket says 172.21.15.28:49604) 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 -- 172.21.15.28:0/4166013546 learned_addr learned my addr 172.21.15.28:0/4166013546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1259c640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034f850 0x55d780359560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 -- 172.21.15.28:0/4166013546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 msgr2=0x55d780359020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780359020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 -- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034f850 msgr2=0x55d780359560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034f850 0x55d780359560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c1359e640 1 -- 172.21.15.28:0/4166013546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7800916b0 con 0x55d7803549c0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.986+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780359020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1359e640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009ca0 tx=0x7f9c0c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d78fe4c6f4308b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c051060 con 0x55d7803549c0 2024-03-22T02:13:44.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9c0c004040 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c08019b10 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9c0c058050 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 10 client.15441 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d780363980 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c08019b10 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9c0c03f040 con 0x55d7803549c0 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 10 client.15441 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 10 client.15441 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 20 client.15441 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 4 client.15441 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.987+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7802c0750 con 0x55d780366080 2024-03-22T02:13:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9bf6ff5640 10 client.15441 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803674f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecb4c0ce6aed2c21 server_cookie=3ce701a7509ff2b0 in_seq=0 out_seq=0 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/4166013546 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d7802c0750 con 0x55d780366080 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9bf6ff5640 10 client.15441 handle_command_reply: tid=0 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.988+0000 7f9c1bdef740 1 client.15441 shutdown 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 msgr2=0x55d7803605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009ca0 tx=0x7f9c0c02e040 comp rx=0 tx=0).stop 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 shutdown_connections 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803674f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d7803549c0 0x55d7803605a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d78034f850 0x55d780359560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9bf6ff5640 0 client.15441 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/4166013546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780359020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 >> 172.21.15.28:0/4166013546 conn(0x55d78034a3d0 msgr2=0x55d780349650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 shutdown_connections 2024-03-22T02:13:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 1 -- 172.21.15.28:0/4166013546 wait complete. 2024-03-22T02:13:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:44.989+0000 7f9c1bdef740 20 client.15441 trim_cache size 0 max 16384 2024-03-22T02:13:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 Processor -- start 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 -- start start 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780354070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78034a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d780365830 con 0x55d78034f850 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d780365f00 con 0x55d7803549c0 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d7803665d0 con 0x55d78034ee60 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49606/0 (socket says 172.21.15.28:49606) 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 -- 172.21.15.28:0/3445675575 learned_addr learned my addr 172.21.15.28:0/3445675575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780354070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1359e640 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78034a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 -- 172.21.15.28:0/3445675575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 msgr2=0x55d780354070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780354070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 -- 172.21.15.28:0/3445675575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 msgr2=0x55d78034a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78034a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 -- 172.21.15.28:0/3445675575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d780353cf0 con 0x55d78034f850 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.007+0000 7f9c1259c640 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55d78034b340 tx=0x7f9c08005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8b23f829a40105 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9bf8ff9640 1 -- 172.21.15.28:0/3445675575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08028070 con 0x55d78034f850 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9bf8ff9640 1 -- 172.21.15.28:0/3445675575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c0801f040 con 0x55d78034f850 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9bf8ff9640 1 -- 172.21.15.28:0/3445675575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0801e050 con 0x55d78034f850 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3445675575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 msgr2=0x55d780360560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x55d78034b340 tx=0x7f9c08005d70 comp rx=0 tx=0).stop 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3445675575 shutdown_connections 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78034a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034f850 0x55d780360560 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/3445675575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034ee60 0x55d780354070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3445675575 >> 172.21.15.28:0/3445675575 conn(0x55d78034a3d0 msgr2=0x55d78034dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3445675575 shutdown_connections 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- 172.21.15.28:0/3445675575 wait complete. 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.008+0000 7f9c1bdef740 1 -- start start 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 Processor -- start 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d78035c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78035f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d780367370 con 0x55d78034ee60 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d780366ca0 con 0x55d7803549c0 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d780367a40 con 0x55d78034f850 2024-03-22T02:13:45.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 10 client.0 _ll_register_callbacks cb 0x7f9c08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49614/0 (socket says 172.21.15.28:49614) 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d78035c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 -- 172.21.15.28:0/1633409163 learned_addr learned my addr 172.21.15.28:0/1633409163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1359e640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78035f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 -- 172.21.15.28:0/1633409163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 msgr2=0x55d78035c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d78035c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 -- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 msgr2=0x55d78035f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78035f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 -- 172.21.15.28:0/1633409163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d78035fc80 con 0x55d78034ee60 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c12d9d640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9c04098830 tx=0x7f9c0406b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc8b97d9ef26cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9bf6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:45.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0406a020 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9c04003b30 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.009+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9c0801f040 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f9c040babd0 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 10 client.15447 fetch_fsmap learned FSMap version 24 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7803cc360 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c0801f040 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f9c0409b2b0 con 0x55d78034ee60 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 10 client.15447 fetch_fsmap finished waiting for FSMap version 24 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 10 client.15447 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 20 client.15447 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803cd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 4 client.15447 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-22T02:13:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "status"}) -- 0x55d7802baaa0 con 0x55d780366080 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1259c640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9bf6ff5640 10 client.15447 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.010+0000 7f9c1259c640 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803cd370 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a81f734131144d6e server_cookie=a265c390047feb65 in_seq=0 out_seq=0 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9bf6ff5640 1 -- 172.21.15.28:0/1633409163 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7802baaa0 con 0x55d780366080 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9bf6ff5640 10 client.15447 handle_command_reply: tid=0 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 client.15447 shutdown 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 msgr2=0x55d78035c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9c04098830 tx=0x7f9c0406b980 comp rx=0 tx=0).stop 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 shutdown_connections 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d780366080 0x55d7803cd370 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d7803549c0 0x55d78035f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d78034f850 0x55d78035c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 --2- 172.21.15.28:0/1633409163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d78034ee60 0x55d78035c470 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9bf6ff5640 0 client.15447 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 >> 172.21.15.28:0/1633409163 conn(0x55d78034a3d0 msgr2=0x55d780348dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 shutdown_connections 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 1 -- 172.21.15.28:0/1633409163 wait complete. 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c1bdef740 20 client.15447 trim_cache size 0 max 16384 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "426c3602-e7f0-11ee-95cd-87774f69a715", 2024-03-22T02:13:45.019 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24509, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 131.11593016099999, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 24, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 94, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 90, 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 135.33157851999999 2024-03-22T02:13:45.020 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 msgr2=0x7f9bfc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 0x7f9bfc0a53e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c005e30 tx=0x7f9c0c009800 comp rx=0 tx=0).stop 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 msgr2=0x7f9c1410b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400e9c0 tx=0x7f9c0400ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 shutdown_connections 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9bfc0a2f20 0x7f9bfc0a53e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c14102530 0x7f9c14109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9c14101b70 0x7f9c14109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 --2- 172.21.15.28:0/3732782703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c14100980 0x7f9c1410b300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.011+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 >> 172.21.15.28:0/3732782703 conn(0x7f9c140fbf10 msgr2=0x7f9c14104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.012+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 shutdown_connections 2024-03-22T02:13:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:45.012+0000 7f9c18a01640 1 -- 172.21.15.28:0/3732782703 wait complete. 2024-03-22T02:13:45.023 INFO:teuthology.run_tasks:Running task sequential... 2024-03-22T02:13:45.033 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-22T02:13:45.033 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-22T02:13:45.033 INFO:teuthology.task.print:Enabling mgr modules 2024-03-22T02:13:45.033 INFO:teuthology.run_tasks:Running task kclient... 2024-03-22T02:13:45.049 INFO:tasks.kclient:Mounting kernel clients... 2024-03-22T02:13:45.049 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-22T02:13:45.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:45.050 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-22T02:13:45.064 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-22T02:13:45.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:45.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-22T02:13:45.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:45.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-03-22T02:13:45.244 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-22T02:13:45.244 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-22T02:13:45.244 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:45.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-22T02:13:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:13:45.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:45 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2872626438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:45.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:45 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2872626438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:45.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:45 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2872626438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:46 smithi028 ceph-mon[31916]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:46 smithi120 ceph-mon[36803]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:46 smithi083 ceph-mon[36340]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 -- 172.21.15.28:0/1345979428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 msgr2=0x7f9d4006a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 --2- 172.21.15.28:0/1345979428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d4006a790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0099c0 tx=0x7f9d3c009cd0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 -- 172.21.15.28:0/1345979428 shutdown_connections 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 --2- 172.21.15.28:0/1345979428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 0x7f9d4010f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 --2- 172.21.15.28:0/1345979428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d40108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 --2- 172.21.15.28:0/1345979428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d4006a790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 -- 172.21.15.28:0/1345979428 >> 172.21.15.28:0/1345979428 conn(0x7f9d400784b0 msgr2=0x7f9d400788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 -- 172.21.15.28:0/1345979428 shutdown_connections 2024-03-22T02:13:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.245+0000 7f9d48579700 1 -- 172.21.15.28:0/1345979428 wait complete. 2024-03-22T02:13:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 Processor -- start 2024-03-22T02:13:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 -- start start 2024-03-22T02:13:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 0x7f9d40199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:47.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d4019d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d401129c0 con 0x7f9d4006a330 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d40112840 con 0x7f9d40108640 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d48579700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d40112b40 con 0x7f9d4006acd0 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d46315700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d46315700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49634/0 (socket says 172.21.15.28:49634) 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d46315700 1 -- 172.21.15.28:0/22399135 learned_addr learned my addr 172.21.15.28:0/22399135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d46b16700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d4019d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.246+0000 7f9d45b14700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 0x7f9d40199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 -- 172.21.15.28:0/22399135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 msgr2=0x7f9d40199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 0x7f9d40199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 -- 172.21.15.28:0/22399135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 msgr2=0x7f9d4019d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d4019d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 -- 172.21.15.28:0/22399135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4019de00 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46b16700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d4019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d46315700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000e9a0 tx=0x7f9d3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000ca10 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4019e0f0 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d401a5970 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3000cbb0 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.247+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d30018680 con 0x7f9d4006a330 2024-03-22T02:13:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.248+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d40069340 con 0x7f9d4006a330 2024-03-22T02:13:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.249+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f9d30010430 con 0x7f9d4006a330 2024-03-22T02:13:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.250+0000 7f9d377fe700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 0x7f9d2c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.250+0000 7f9d45b14700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 0x7f9d2c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.250+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9d300cfde0 con 0x7f9d4006a330 2024-03-22T02:13:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.250+0000 7f9d45b14700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 0x7f9d2c0a54f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9d40078a20 tx=0x7f9d3c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:47.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.254+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d3009c800 con 0x7f9d4006a330 2024-03-22T02:13:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.503+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f9d4002dd70 con 0x7f9d4006a330 2024-03-22T02:13:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.504+0000 7f9d377fe700 1 -- 172.21.15.28:0/22399135 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f9d3009c100 con 0x7f9d4006a330 2024-03-22T02:13:47.505 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-22T02:13:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 msgr2=0x7f9d2c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 0x7f9d2c0a54f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9d40078a20 tx=0x7f9d3c005fb0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 msgr2=0x7f9d40198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000e9a0 tx=0x7f9d3000ecb0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 shutdown_connections 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d2c0a3030 0x7f9d2c0a54f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4006a330 0x7f9d40198df0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d40108640 0x7f9d4019d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 --2- 172.21.15.28:0/22399135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d4006acd0 0x7f9d40199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 >> 172.21.15.28:0/22399135 conn(0x7f9d400784b0 msgr2=0x7f9d400ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 shutdown_connections 2024-03-22T02:13:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:47.506+0000 7f9d48579700 1 -- 172.21.15.28:0/22399135 wait complete. 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-22T02:13:48.149 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-22T02:13:48.150 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-22T02:13:48.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:48.150 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-03-22T02:13:48.165 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-22T02:13:48.165 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-22T02:13:48.166 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42080sec preferred_lft 42080sec 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-03-22T02:13:48.167 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-03-22T02:13:48.167 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-22T02:13:48.167 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-22T02:13:48.168 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:13:48.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:48.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:48.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:48.320 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-22T02:13:48.346 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-03-22T02:13:48.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:48.347 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-03-22T02:13:48.402 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-03-22T02:13:48.403 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-22T02:13:48.403 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-22T02:13:48.403 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-22T02:13:48.403 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-22T02:13:48.403 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:13:48.403 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:13:48.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-22T02:13:48.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-22T02:13:48.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-22T02:13:48.404 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:13:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:48.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:48 smithi028 ceph-mon[31916]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:48.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/22399135' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:13:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:48.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:48.539 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-22T02:13:48.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:48.555 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-03-22T02:13:48.610 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:13:48.610 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:13:48.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:13:48.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-22T02:13:48.610 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:13:48.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:48.732 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-22T02:13:48.732 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:13:48.732 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:13:48.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-22T02:13:48.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-22T02:13:48.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-22T02:13:48.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-22T02:13:48.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-22T02:13:48.733 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:13:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:48 smithi120 ceph-mon[36803]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/22399135' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:13:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:48 smithi083 ceph-mon[36340]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-22T02:13:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/22399135' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-22T02:13:48.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:48.858 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-22T02:13:48.859 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-03-22T02:13:48.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-03-22T02:13:48.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-03-22T02:13:48.859 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-03-22T02:13:48.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:48.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:48.942 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:13:48.942 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:48.998 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-22T02:13:48.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-22T02:13:48.998 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:49.052 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-22T02:13:49.053 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-22T02:13:49.053 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 2024-03-22T02:13:49.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:49.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-22T02:13:49.137 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-22T02:13:49.138 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-22T02:13:49.272 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-22T02:13:49.272 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-22T02:13:49.272 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.83:3300,172.21.15.120:3300" 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@426c3602-e7f0-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.28:3300/172.21.15.83:3300/172.21.15.120:3300" will pass to kernel 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-03-22T02:13:49.273 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-22T02:13:49.275 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-22T02:13:49.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:49.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:49.304 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T02:13:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T02:13:49.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:49.400 INFO:teuthology.orchestra.run.smithi028.stdout:client25282 2024-03-22T02:13:49.401 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:13:49.401 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:13:49.416 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T02:13:49.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T02:13:49.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:49.517 INFO:teuthology.orchestra.run.smithi028.stdout:client25282 2024-03-22T02:13:49.517 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-22T02:13:49.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-22T02:13:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:49.575 INFO:teuthology.orchestra.run.smithi028.stdout:client25282 2024-03-22T02:13:49.575 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/426c3602-e7f0-11ee-95cd-87774f69a715.client25282/status') 2024-03-22T02:13:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25282 (3)172.21.15.28:0/1059673484 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:71 bytes copied, 2.1411e-05 s, 3.3 MB/s 2024-03-22T02:13:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-22T02:13:49.646 INFO:teuthology.run_tasks:Running task exec... 2024-03-22T02:13:49.983 INFO:teuthology.task.exec:Executing custom commands... 2024-03-22T02:13:49.984 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:13:49.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-22T02:13:50.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- 172.21.15.28:0/4278128815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 msgr2=0x7f40d0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:50.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 --2- 172.21.15.28:0/4278128815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0100ed0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009900 tx=0x7f40c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:50.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- 172.21.15.28:0/4278128815 shutdown_connections 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 --2- 172.21.15.28:0/4278128815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0102660 0x7f40d0108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 --2- 172.21.15.28:0/4278128815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0101ca0 0x7f40d0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 --2- 172.21.15.28:0/4278128815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0100ed0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- 172.21.15.28:0/4278128815 >> 172.21.15.28:0/4278128815 conn(0x7f40d00fc020 msgr2=0x7f40d00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- 172.21.15.28:0/4278128815 shutdown_connections 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- 172.21.15.28:0/4278128815 wait complete. 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 Processor -- start 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.088+0000 7f40d6478640 1 -- start start 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0101ca0 0x7f40d01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 0x7f40d019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:50.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010c720 con 0x7f40d0100ab0 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010c5a0 con 0x7f40d0101ca0 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d6478640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f40d010c8a0 con 0x7f40d0102660 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49656/0 (socket says 172.21.15.28:49656) 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40cffff640 1 -- 172.21.15.28:0/678284580 learned_addr learned my addr 172.21.15.28:0/678284580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 0x7f40d019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40cf7fe640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0101ca0 0x7f40d01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 -- 172.21.15.28:0/678284580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0101ca0 msgr2=0x7f40d01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0101ca0 0x7f40d01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 -- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 msgr2=0x7f40d0197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:50.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 -- 172.21.15.28:0/678284580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d019fa80 con 0x7f40d0102660 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40cffff640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.089+0000 7f40d4a15640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 0x7f40d019f410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009fd0 tx=0x7f40c800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.090+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c800ce20 con 0x7f40d0102660 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.090+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d019fd70 con 0x7f40d0102660 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.090+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c8005020 con 0x7f40d0102660 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.090+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d01a0260 con 0x7f40d0102660 2024-03-22T02:13:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.090+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8010720 con 0x7f40d0102660 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.091+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d01a0550 con 0x7f40d0102660 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.092+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f40c8005540 con 0x7f40d0102660 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.092+0000 7f40cd7fa640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 0x7f40b80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.092+0000 7f40cffff640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 0x7f40b80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.093+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f40c80cff20 con 0x7f40d0102660 2024-03-22T02:13:50.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.093+0000 7f40cffff640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 0x7f40b80a5480 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c8e0 tx=0x7f40c0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.101+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f40c809c8e0 con 0x7f40d0102660 2024-03-22T02:13:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:50.355+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f40d01a0720 con 0x7f40d0102660 2024-03-22T02:13:50.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:50 smithi028 ceph-mon[31916]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:50 smithi120 ceph-mon[36803]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:50 smithi083 ceph-mon[36340]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-22T02:13:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.327+0000 7f40cd7fa640 1 -- 172.21.15.28:0/678284580 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v25) ==== 118+0+0 (secure 0 0 0) 0x7f40c809c1e0 con 0x7f40d0102660 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 msgr2=0x7f40b80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 0x7f40b80a5480 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c8e0 tx=0x7f40c0005b50 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 msgr2=0x7f40d019f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 0x7f40d019f410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f40c8009fd0 tx=0x7f40c800ef10 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 shutdown_connections 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f40b80a2fc0 0x7f40b80a5480 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f40d0102660 0x7f40d019f410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f40d0101ca0 0x7f40d01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 --2- 172.21.15.28:0/678284580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40d0100ab0 0x7f40d0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 >> 172.21.15.28:0/678284580 conn(0x7f40d00fc020 msgr2=0x7f40d00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:51.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 shutdown_connections 2024-03-22T02:13:51.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.330+0000 7f40d6478640 1 -- 172.21.15.28:0/678284580 wait complete. 2024-03-22T02:13:51.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 -- 172.21.15.28:0/4145851806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 msgr2=0x7f958c1086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 --2- 172.21.15.28:0/4145851806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c1086a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9584009ca0 tx=0x7f958402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 -- 172.21.15.28:0/4145851806 shutdown_connections 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 --2- 172.21.15.28:0/4145851806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c1086a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 --2- 172.21.15.28:0/4145851806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 --2- 172.21.15.28:0/4145851806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c100680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 -- 172.21.15.28:0/4145851806 >> 172.21.15.28:0/4145851806 conn(0x7f958c0fb7f0 msgr2=0x7f958c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:51.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 -- 172.21.15.28:0/4145851806 shutdown_connections 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 -- 172.21.15.28:0/4145851806 wait complete. 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.457+0000 7f9592457640 1 Processor -- start 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 -- start start 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c10a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10d8b0 con 0x7f958c101e10 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10d730 con 0x7f958c100260 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f9592457640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10da30 con 0x7f958c101450 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f95909f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f95909f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49672/0 (socket says 172.21.15.28:49672) 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 -- 172.21.15.28:0/668749087 learned_addr learned my addr 172.21.15.28:0/668749087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958b7fe640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c10a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 -- 172.21.15.28:0/668749087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 msgr2=0x7f958c10a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c10a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 -- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 msgr2=0x7f958c10a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958bfff640 1 -- 172.21.15.28:0/668749087 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c10ae00 con 0x7f958c100260 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f958b7fe640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c10a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.458+0000 7f95909f4640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f958bfff640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e9c0 tx=0x7f957c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:51.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c00cd80 con 0x7f958c100260 2024-03-22T02:13:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f957c0035f0 con 0x7f958c100260 2024-03-22T02:13:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c1a1ec0 con 0x7f958c100260 2024-03-22T02:13:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c010690 con 0x7f958c100260 2024-03-22T02:13:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.459+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1a22d0 con 0x7f958c100260 2024-03-22T02:13:51.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.460+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c1074b0 con 0x7f958c100260 2024-03-22T02:13:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.461+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f957c0037a0 con 0x7f958c100260 2024-03-22T02:13:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.461+0000 7f95897fa640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.462+0000 7f958b7fe640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.462+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f957c0cfb50 con 0x7f958c100260 2024-03-22T02:13:51.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.462+0000 7f958b7fe640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f958c10b760 tx=0x7f9580005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:51.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.465+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f957c09c570 con 0x7f958c100260 2024-03-22T02:13:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/678284580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.677+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f958c062070 con 0x7f95740a2fc0 2024-03-22T02:13:51.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.691+0000 7f95897fa640 1 -- 172.21.15.28:0/668749087 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f958c062070 con 0x7f95740a2fc0 2024-03-22T02:13:51.692 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-03-22T02:13:51.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 msgr2=0x7f95740a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f958c10b760 tx=0x7f9580005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:51.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 msgr2=0x7f958c10b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f957c00e9c0 tx=0x7f957c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 shutdown_connections 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c101e10 0x7f958c10a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c101450 0x7f958c10a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 --2- 172.21.15.28:0/668749087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c100260 0x7f958c10b9f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 >> 172.21.15.28:0/668749087 conn(0x7f958c0fb7f0 msgr2=0x7f958c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 shutdown_connections 2024-03-22T02:13:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.693+0000 7f9592457640 1 -- 172.21.15.28:0/668749087 wait complete. 2024-03-22T02:13:51.705 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-22T02:13:51.717 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-22T02:13:51.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:51.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:51 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/678284580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:51 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:51 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/678284580' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:51 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-22T02:13:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 -- 172.21.15.28:0/1858169929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 msgr2=0x7f55ec10a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/1858169929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec10a020 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009900 tx=0x7f55dc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:51.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 -- 172.21.15.28:0/1858169929 shutdown_connections 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/1858169929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec10a020 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/1858169929 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec102dd0 0x7f55ec103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/1858169929 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec101be0 0x7f55ec102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 -- 172.21.15.28:0/1858169929 >> 172.21.15.28:0/1858169929 conn(0x7f55ec0fd130 msgr2=0x7f55ec0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 -- 172.21.15.28:0/1858169929 shutdown_connections 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.834+0000 7f55f2ea4640 1 -- 172.21.15.28:0/1858169929 wait complete. 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 Processor -- start 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 -- start start 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec101be0 0x7f55ec075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec102dd0 0x7f55ec073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10d860 con 0x7f55ec103790 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10d6e0 con 0x7f55ec101be0 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f2ea4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55ec10d9e0 con 0x7f55ec102dd0 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60112/0 (socket says 172.21.15.28:60112) 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f0c40640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec101be0 0x7f55ec075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 -- 172.21.15.28:0/3400558551 learned_addr learned my addr 172.21.15.28:0/3400558551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55ebfff640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec102dd0 0x7f55ec073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 -- 172.21.15.28:0/3400558551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec102dd0 msgr2=0x7f55ec073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec102dd0 0x7f55ec073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 -- 172.21.15.28:0/3400558551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec101be0 msgr2=0x7f55ec075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec101be0 0x7f55ec075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.835+0000 7f55f1441640 1 -- 172.21.15.28:0/3400558551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ec074d50 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55f1441640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005ab0 tx=0x7f55dc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc041070 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55dc004450 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ec1a1f30 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55dc003e60 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.836+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ec1a2390 con 0x7f55ec103790 2024-03-22T02:13:51.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.837+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ec108f00 con 0x7f55ec103790 2024-03-22T02:13:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.838+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7f55dc044070 con 0x7f55ec103790 2024-03-22T02:13:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.839+0000 7f55e9ffb640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 0x7f55d80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.839+0000 7f55f0c40640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 0x7f55d80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.839+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f55dc0f13c0 con 0x7f55ec103790 2024-03-22T02:13:51.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.840+0000 7f55f0c40640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 0x7f55d80a53e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f55ec102c30 tx=0x7f55e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:51.844+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55dc0bd730 con 0x7f55ec103790 2024-03-22T02:13:52.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.028+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55ec062830 con 0x7f55ec103790 2024-03-22T02:13:52.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.030+0000 7f55e9ffb640 1 -- 172.21.15.28:0/3400558551 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7f55dc0c17d0 con 0x7f55ec103790 2024-03-22T02:13:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:52.031 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:52.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 msgr2=0x7f55d80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 0x7f55d80a53e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f55ec102c30 tx=0x7f55e4009500 comp rx=0 tx=0).stop 2024-03-22T02:13:52.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 msgr2=0x7f55ec0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f55dc005ab0 tx=0x7f55dc02c6c0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 shutdown_connections 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d80a2f20 0x7f55d80a53e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55ec103790 0x7f55ec0744c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55ec102dd0 0x7f55ec073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 --2- 172.21.15.28:0/3400558551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55ec101be0 0x7f55ec075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 >> 172.21.15.28:0/3400558551 conn(0x7f55ec0fd130 msgr2=0x7f55ec1060d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 shutdown_connections 2024-03-22T02:13:52.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.032+0000 7f55f2ea4640 1 -- 172.21.15.28:0/3400558551 wait complete. 2024-03-22T02:13:52.043 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:13:51.681+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:13:52.043 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:13:51.681+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:13:52.043 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:13:51.681+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:13:52.043 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:13:51.681+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:13:52.043 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:13:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x[32146]: 2024-03-22T02:13:51.681+0000 7f0ce5ea8700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-22T02:13:52.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:52.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- 172.21.15.28:0/2325748199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 msgr2=0x7fb24c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- 172.21.15.28:0/2325748199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c102110 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb234009900 tx=0x7fb23402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- 172.21.15.28:0/2325748199 shutdown_connections 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- 172.21.15.28:0/2325748199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 0x7fb24c108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- 172.21.15.28:0/2325748199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c102110 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- 172.21.15.28:0/2325748199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 0x7fb24c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- 172.21.15.28:0/2325748199 >> 172.21.15.28:0/2325748199 conn(0x7fb24c0fc030 msgr2=0x7fb24c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- 172.21.15.28:0/2325748199 shutdown_connections 2024-03-22T02:13:52.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- 172.21.15.28:0/2325748199 wait complete. 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 Processor -- start 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 -- start start 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 0x7fb24c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.150+0000 7fb25177a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 0x7fb24c19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c10d840 con 0x7fb24c101c90 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c10d6c0 con 0x7fb24c102650 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb24c10d9c0 con 0x7fb24c100aa0 2024-03-22T02:13:52.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60130/0 (socket says 172.21.15.28:60130) 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb24affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 0x7fb24c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 -- 172.21.15.28:0/4114220956 learned_addr learned my addr 172.21.15.28:0/4114220956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb24b7fe640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 0x7fb24c19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 -- 172.21.15.28:0/4114220956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 msgr2=0x7fb24c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 0x7fb24c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 -- 172.21.15.28:0/4114220956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 msgr2=0x7fb24c19f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 0x7fb24c19f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 -- 172.21.15.28:0/4114220956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb24c19fb10 con 0x7fb24c101c90 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb243fff640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb234009230 tx=0x7fb23402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234041070 con 0x7fb24c101c90 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb234004450 con 0x7fb24c101c90 2024-03-22T02:13:52.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb24c19fda0 con 0x7fb24c101c90 2024-03-22T02:13:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb234003db0 con 0x7fb24c101c90 2024-03-22T02:13:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.151+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb24c1a0230 con 0x7fb24c101c90 2024-03-22T02:13:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.152+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb24c107e90 con 0x7fb24c101c90 2024-03-22T02:13:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.153+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 137854+0+0 (secure 0 0 0) 0x7fb234044070 con 0x7fb24c101c90 2024-03-22T02:13:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.154+0000 7fb248ff9640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 0x7fb2380a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.154+0000 7fb24affd640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 0x7fb2380a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.154+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb2340f1370 con 0x7fb24c101c90 2024-03-22T02:13:52.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.155+0000 7fb24affd640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 0x7fb2380a53e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb24c101af0 tx=0x7fb23c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.159+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb2340bdd90 con 0x7fb24c101c90 2024-03-22T02:13:52.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.335+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fb2340b9d00 con 0x7fb24c101c90 2024-03-22T02:13:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.345+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb24c062790 con 0x7fb24c101c90 2024-03-22T02:13:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.346+0000 7fb248ff9640 1 -- 172.21.15.28:0/4114220956 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7fb2340ae530 con 0x7fb24c101c90 2024-03-22T02:13:52.348 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 msgr2=0x7fb2380a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 0x7fb2380a53e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb24c101af0 tx=0x7fb23c005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 msgr2=0x7fb24c198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb234009230 tx=0x7fb23402c650 comp rx=0 tx=0).stop 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 shutdown_connections 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2380a2f20 0x7fb2380a53e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb24c102650 0x7fb24c19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb24c101c90 0x7fb24c198430 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 --2- 172.21.15.28:0/4114220956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb24c100aa0 0x7fb24c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 >> 172.21.15.28:0/4114220956 conn(0x7fb24c0fc030 msgr2=0x7fb24c105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 shutdown_connections 2024-03-22T02:13:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.348+0000 7fb25177a640 1 -- 172.21.15.28:0/4114220956 wait complete. 2024-03-22T02:13:52.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:52.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 -- 172.21.15.28:0/2101281224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 msgr2=0x7f919c1035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 --2- 172.21.15.28:0/2101281224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c1035b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f918c009900 tx=0x7f918c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 -- 172.21.15.28:0/2101281224 shutdown_connections 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 --2- 172.21.15.28:0/2101281224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c1005c0 0x7f919c10a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 --2- 172.21.15.28:0/2101281224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c0ffc00 0x7f919c100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 --2- 172.21.15.28:0/2101281224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c1035b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 -- 172.21.15.28:0/2101281224 >> 172.21.15.28:0/2101281224 conn(0x7f919c0fb7c0 msgr2=0x7f919c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 -- 172.21.15.28:0/2101281224 shutdown_connections 2024-03-22T02:13:52.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.464+0000 7f91a0837640 1 -- 172.21.15.28:0/2101281224 wait complete. 2024-03-22T02:13:52.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 Processor -- start 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 -- start start 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 0x7f919c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c1005c0 0x7f919c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10d830 con 0x7f919c103190 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10d6b0 con 0x7f919c0ffc00 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f91a0837640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10d9b0 con 0x7f919c1005c0 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f9193fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c1005c0 0x7f919c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60144/0 (socket says 172.21.15.28:60144) 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 -- 172.21.15.28:0/3708361876 learned_addr learned my addr 172.21.15.28:0/3708361876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919a59c640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 0x7f919c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 -- 172.21.15.28:0/3708361876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c1005c0 msgr2=0x7f919c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c1005c0 0x7f919c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 -- 172.21.15.28:0/3708361876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 msgr2=0x7f919c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 0x7f919c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919ad9d640 1 -- 172.21.15.28:0/3708361876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c19fb20 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.465+0000 7f919a59c640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 0x7f919c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f919ad9d640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f919400eb10 tx=0x7f919400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194004b20 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9194004cc0 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c19fe10 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194010760 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.466+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c1a02d0 con 0x7f919c103190 2024-03-22T02:13:52.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.467+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f919c1090a0 con 0x7f919c103190 2024-03-22T02:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.468+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f9194005020 con 0x7f919c103190 2024-03-22T02:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.468+0000 7f91937fe640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 0x7f91880a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.469+0000 7f919a59c640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 0x7f91880a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.469+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f91940cfe90 con 0x7f919c103190 2024-03-22T02:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.469+0000 7f919a59c640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 0x7f91880a54f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f918c02efd0 tx=0x7f918c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.473+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f919409c750 con 0x7f919c103190 2024-03-22T02:13:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.655+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f919c02d600 con 0x7f919c103190 2024-03-22T02:13:52.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.656+0000 7f91937fe640 1 -- 172.21.15.28:0/3708361876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f919409c050 con 0x7f919c103190 2024-03-22T02:13:52.657 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 msgr2=0x7f91880a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 0x7f91880a54f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f918c02efd0 tx=0x7f918c000bc0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 msgr2=0x7f919c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f919400eb10 tx=0x7f919400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 shutdown_connections 2024-03-22T02:13:52.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f91880a3030 0x7f91880a54f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f919c103190 0x7f919c19f4b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f919c1005c0 0x7f919c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 --2- 172.21.15.28:0/3708361876 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f919c0ffc00 0x7f919c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.657+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 >> 172.21.15.28:0/3708361876 conn(0x7f919c0fb7c0 msgr2=0x7f919c106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.658+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 shutdown_connections 2024-03-22T02:13:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.658+0000 7f91a0837640 1 -- 172.21.15.28:0/3708361876 wait complete. 2024-03-22T02:13:52.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:52 smithi028 ceph-mon[31916]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-22T02:13:52.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:52 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-22T02:13:52.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:52 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:13:52.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:52 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3400558551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:52.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:52 smithi120 ceph-mon[36803]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-22T02:13:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:52 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-22T02:13:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:52 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:13:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:52 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3400558551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:52 smithi083 ceph-mon[36340]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-22T02:13:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:52 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-22T02:13:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:52 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:13:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:52 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3400558551' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.777+0000 7fe30e7c9640 1 -- 172.21.15.28:0/1143541213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 msgr2=0x7fe308104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.777+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/1143541213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308104a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009900 tx=0x7fe2f802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- 172.21.15.28:0/1143541213 shutdown_connections 2024-03-22T02:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/1143541213 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 0x7fe308109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/1143541213 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 0x7fe308107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/1143541213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308104a20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- 172.21.15.28:0/1143541213 >> 172.21.15.28:0/1143541213 conn(0x7fe3080fc040 msgr2=0x7fe3080fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- 172.21.15.28:0/1143541213 shutdown_connections 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- 172.21.15.28:0/1143541213 wait complete. 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 Processor -- start 2024-03-22T02:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- start start 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 0x7fe308198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 0x7fe30819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe30810d860 con 0x7fe308102630 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe30810d6e0 con 0x7fe308107890 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.778+0000 7fe30e7c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe30810d9e0 con 0x7fe308104f60 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60166/0 (socket says 172.21.15.28:60166) 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 -- 172.21.15.28:0/3975929603 learned_addr learned my addr 172.21.15.28:0/3975929603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe30cd66640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 0x7fe30819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe3077fe640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 0x7fe308198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 -- 172.21.15.28:0/3975929603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 msgr2=0x7fe308198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 0x7fe308198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 -- 172.21.15.28:0/3975929603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 msgr2=0x7fe30819f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 0x7fe30819f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 -- 172.21.15.28:0/3975929603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30819fb60 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe307fff640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009230 tx=0x7fe2f802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8041070 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2f8004450 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8003db0 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30819fdf0 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.779+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3081a01d0 con 0x7fe308102630 2024-03-22T02:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.780+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3080fdb30 con 0x7fe308102630 2024-03-22T02:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.781+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fe2f8044070 con 0x7fe308102630 2024-03-22T02:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.782+0000 7fe3057fa640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 0x7fe2f40a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.782+0000 7fe3077fe640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 0x7fe2f40a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.782+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe2f80f13d0 con 0x7fe308102630 2024-03-22T02:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.783+0000 7fe3077fe640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 0x7fe2f40a54f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe308199450 tx=0x7fe2f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.787+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe2f80bdd40 con 0x7fe308102630 2024-03-22T02:13:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.971+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe308062860 con 0x7fe308102630 2024-03-22T02:13:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.971+0000 7fe3057fa640 1 -- 172.21.15.28:0/3975929603 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fe2f80b9020 con 0x7fe308102630 2024-03-22T02:13:52.973 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 msgr2=0x7fe2f40a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 0x7fe2f40a54f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fe308199450 tx=0x7fe2f0005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 msgr2=0x7fe308197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8009230 tx=0x7fe2f802c650 comp rx=0 tx=0).stop 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 shutdown_connections 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe2f40a3030 0x7fe2f40a54f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe308107890 0x7fe30819f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe308104f60 0x7fe308198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 --2- 172.21.15.28:0/3975929603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe308102630 0x7fe308197f40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 >> 172.21.15.28:0/3975929603 conn(0x7fe3080fc040 msgr2=0x7fe308102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 shutdown_connections 2024-03-22T02:13:52.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:52.973+0000 7fe30e7c9640 1 -- 172.21.15.28:0/3975929603 wait complete. 2024-03-22T02:13:52.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.089+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2619374150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4102610 msgr2=0x7fc0a4104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.089+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2619374150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4102610 0x7fc0a4104a00 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009900 tx=0x7fc09c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2619374150 shutdown_connections 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2619374150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 0x7fc0a4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2619374150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4104f40 0x7fc0a4107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2619374150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4102610 0x7fc0a4104a00 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2619374150 >> 172.21.15.28:0/2619374150 conn(0x7fc0a40fc020 msgr2=0x7fc0a40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2619374150 shutdown_connections 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2619374150 wait complete. 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 Processor -- start 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- start start 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 0x7fc0a41005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 0x7fc0a4100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a410d810 con 0x7fc0a4104f40 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a410d690 con 0x7fc0a4107870 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.090+0000 7fc0a8dac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a410d990 con 0x7fc0a4102610 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 0x7fc0a41005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58656/0 (socket says 172.21.15.28:58656) 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 -- 172.21.15.28:0/2204304701 learned_addr learned my addr 172.21.15.28:0/2204304701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a2d9d640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 0x7fc0a4100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 -- 172.21.15.28:0/2204304701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 msgr2=0x7fc0a4100b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 0x7fc0a4100b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 -- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 msgr2=0x7fc0a41005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 0x7fc0a41005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 -- 172.21.15.28:0/2204304701 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a4101390 con 0x7fc0a4102610 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a1d9b640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 0x7fc0a41005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a259c640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009230 tx=0x7fc09c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c041070 con 0x7fc0a4102610 2024-03-22T02:13:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc09c004450 con 0x7fc0a4102610 2024-03-22T02:13:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a41a1fd0 con 0x7fc0a4102610 2024-03-22T02:13:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c003e20 con 0x7fc0a4102610 2024-03-22T02:13:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.091+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a41a23e0 con 0x7fc0a4102610 2024-03-22T02:13:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.092+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a40fdb10 con 0x7fc0a4102610 2024-03-22T02:13:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.093+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fc09c044070 con 0x7fc0a4102610 2024-03-22T02:13:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.094+0000 7fc09b7fe640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 0x7fc0900a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.094+0000 7fc0a1d9b640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 0x7fc0900a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.094+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc09c0f1370 con 0x7fc0a4102610 2024-03-22T02:13:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.094+0000 7fc0a1d9b640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 0x7fc0900a5470 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0095f0 tx=0x7fc08c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.099+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc09c0bd6f0 con 0x7fc0a4102610 2024-03-22T02:13:53.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.282+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc0a402d3b0 con 0x7fc0a4102610 2024-03-22T02:13:53.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.282+0000 7fc09b7fe640 1 -- 172.21.15.28:0/2204304701 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fc09c0c1790 con 0x7fc0a4102610 2024-03-22T02:13:53.284 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 msgr2=0x7fc0900a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 0x7fc0900a5470 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0095f0 tx=0x7fc08c005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 msgr2=0x7fc0a4101f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc09c009230 tx=0x7fc09c02c650 comp rx=0 tx=0).stop 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 shutdown_connections 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0900a2fb0 0x7fc0900a5470 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0a4107870 0x7fc0a4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0a4104f40 0x7fc0a41005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 --2- 172.21.15.28:0/2204304701 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0a4102610 0x7fc0a4101f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 >> 172.21.15.28:0/2204304701 conn(0x7fc0a40fc020 msgr2=0x7fc0a41030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 shutdown_connections 2024-03-22T02:13:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.284+0000 7fc0a8dac640 1 -- 172.21.15.28:0/2204304701 wait complete. 2024-03-22T02:13:53.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 -- 172.21.15.28:0/1372674463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44101ca0 msgr2=0x7fdb44102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 --2- 172.21.15.28:0/1372674463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44101ca0 0x7fdb44102120 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009900 tx=0x7fdb3402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 -- 172.21.15.28:0/1372674463 shutdown_connections 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 --2- 172.21.15.28:0/1372674463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44102660 0x7fdb44108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 --2- 172.21.15.28:0/1372674463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44101ca0 0x7fdb44102120 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 --2- 172.21.15.28:0/1372674463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 0x7fdb44100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 -- 172.21.15.28:0/1372674463 >> 172.21.15.28:0/1372674463 conn(0x7fdb440fc020 msgr2=0x7fdb440fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 -- 172.21.15.28:0/1372674463 shutdown_connections 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.399+0000 7fdb4a881640 1 -- 172.21.15.28:0/1372674463 wait complete. 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 Processor -- start 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 -- start start 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 0x7fdb44069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 0x7fdb4419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4410c710 con 0x7fdb44102660 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4410c590 con 0x7fdb44101ca0 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb4a881640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4410c890 con 0x7fdb44100ab0 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60192/0 (socket says 172.21.15.28:60192) 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 -- 172.21.15.28:0/227595922 learned_addr learned my addr 172.21.15.28:0/227595922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb437fe640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 0x7fdb4419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb43fff640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 0x7fdb44069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 -- 172.21.15.28:0/227595922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 msgr2=0x7fdb44069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 0x7fdb44069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 -- 172.21.15.28:0/227595922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 msgr2=0x7fdb4419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.400+0000 7fdb48e1e640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 0x7fdb4419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb48e1e640 1 -- 172.21.15.28:0/227595922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4419fb60 con 0x7fdb44102660 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb437fe640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 0x7fdb4419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb48e1e640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00efd0 tx=0x7fdb3c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c003b70 con 0x7fdb44102660 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c003d10 con 0x7fdb44102660 2024-03-22T02:13:53.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4419fdf0 con 0x7fdb44102660 2024-03-22T02:13:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0109f0 con 0x7fdb44102660 2024-03-22T02:13:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.401+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb441a0260 con 0x7fdb44102660 2024-03-22T02:13:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.402+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb44107dd0 con 0x7fdb44102660 2024-03-22T02:13:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.403+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fdb3c004020 con 0x7fdb44102660 2024-03-22T02:13:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.403+0000 7fdb417fa640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 0x7fdb2c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.404+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdb3c0cfe00 con 0x7fdb44102660 2024-03-22T02:13:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.404+0000 7fdb43fff640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 0x7fdb2c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.404+0000 7fdb43fff640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 0x7fdb2c0a5470 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb44101b00 tx=0x7fdb38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.409+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdb3c09c660 con 0x7fdb44102660 2024-03-22T02:13:53.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:53.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T02:13:53.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4114220956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3708361876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3975929603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2204304701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.592+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb4402d5d0 con 0x7fdb44102660 2024-03-22T02:13:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.593+0000 7fdb417fa640 1 -- 172.21.15.28:0/227595922 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7fdb3c09bf60 con 0x7fdb44102660 2024-03-22T02:13:53.595 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:53.595 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 msgr2=0x7fdb2c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 0x7fdb2c0a5470 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdb44101b00 tx=0x7fdb38005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 msgr2=0x7fdb4419f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00efd0 tx=0x7fdb3c00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 shutdown_connections 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb2c0a2fb0 0x7fdb2c0a5470 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb44102660 0x7fdb4419f250 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb44101ca0 0x7fdb4419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 --2- 172.21.15.28:0/227595922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb44100ab0 0x7fdb44069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 >> 172.21.15.28:0/227595922 conn(0x7fdb440fc020 msgr2=0x7fdb44104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 shutdown_connections 2024-03-22T02:13:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.596+0000 7fdb4a881640 1 -- 172.21.15.28:0/227595922 wait complete. 2024-03-22T02:13:53.608 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:53.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- 172.21.15.28:0/143282069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 msgr2=0x7f94dc101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- 172.21.15.28:0/143282069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc101fe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f94d0009860 tx=0x7f94d002bb70 comp rx=0 tx=0).stop 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- 172.21.15.28:0/143282069 shutdown_connections 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- 172.21.15.28:0/143282069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- 172.21.15.28:0/143282069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc101fe0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- 172.21.15.28:0/143282069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 0x7f94dc100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- 172.21.15.28:0/143282069 >> 172.21.15.28:0/143282069 conn(0x7f94dc0fbf00 msgr2=0x7f94dc0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- 172.21.15.28:0/143282069 shutdown_connections 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- 172.21.15.28:0/143282069 wait complete. 2024-03-22T02:13:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 Processor -- start 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- start start 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 0x7f94dc199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc19a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc10c5d0 con 0x7f94dc102520 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc10c450 con 0x7f94dc101b60 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e31a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc10c750 con 0x7f94dc100970 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e299f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e219e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 0x7f94dc199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e299f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60208/0 (socket says 172.21.15.28:60208) 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e299f640 1 -- 172.21.15.28:0/2742174861 learned_addr learned my addr 172.21.15.28:0/2742174861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.714+0000 7f94e199d640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc19a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 -- 172.21.15.28:0/2742174861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 msgr2=0x7f94dc199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 0x7f94dc199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 -- 172.21.15.28:0/2742174861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 msgr2=0x7f94dc19a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc19a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 -- 172.21.15.28:0/2742174861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc1a1b90 con 0x7f94dc102520 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e299f640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e970 tx=0x7f94d400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ccc0 con 0x7f94dc102520 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc1a1e80 con 0x7f94dc102520 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94d4005500 con 0x7f94dc102520 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4010690 con 0x7f94dc102520 2024-03-22T02:13:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.715+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc1a22f0 con 0x7f94dc102520 2024-03-22T02:13:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.716+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94dc1a2580 con 0x7f94dc102520 2024-03-22T02:13:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.717+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f94d4004e50 con 0x7f94dc102520 2024-03-22T02:13:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.717+0000 7f94db7fe640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 0x7f94c40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.718+0000 7f94e219e640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 0x7f94c40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.718+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94d40cfc30 con 0x7f94dc102520 2024-03-22T02:13:53.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.718+0000 7f94e219e640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 0x7f94c40a5470 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0099a0 tx=0x7f94cc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:53.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.723+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f94d409c4f0 con 0x7f94dc102520 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4114220956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3708361876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3975929603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2204304701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T02:13:53.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-22T02:13:53.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4114220956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3708361876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3975929603' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2204304701' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.905+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94dc1a2750 con 0x7f94dc102520 2024-03-22T02:13:53.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.906+0000 7f94db7fe640 1 -- 172.21.15.28:0/2742174861 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f94d409bdf0 con 0x7f94dc102520 2024-03-22T02:13:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 msgr2=0x7f94c40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 0x7f94c40a5470 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0099a0 tx=0x7f94cc009500 comp rx=0 tx=0).stop 2024-03-22T02:13:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 msgr2=0x7f94dc1a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:53.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e970 tx=0x7f94d400ee30 comp rx=0 tx=0).stop 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 shutdown_connections 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94c40a2fb0 0x7f94c40a5470 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94dc102520 0x7f94dc1a1520 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94dc101b60 0x7f94dc19a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 --2- 172.21.15.28:0/2742174861 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94dc100970 0x7f94dc199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 >> 172.21.15.28:0/2742174861 conn(0x7f94dc0fbf00 msgr2=0x7f94dc0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 shutdown_connections 2024-03-22T02:13:53.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:53.908+0000 7f94e31a0640 1 -- 172.21.15.28:0/2742174861 wait complete. 2024-03-22T02:13:53.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1914638313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 msgr2=0x7f67ec10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1914638313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec10a000 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f67e4009ca0 tx=0x7f67e402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1914638313 shutdown_connections 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1914638313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec10a000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1914638313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 0x7f67ec103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1914638313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1914638313 >> 172.21.15.28:0/1914638313 conn(0x7f67ec0fd130 msgr2=0x7f67ec0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1914638313 shutdown_connections 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1914638313 wait complete. 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 Processor -- start 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.028+0000 7f67f1b3a640 1 -- start start 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 0x7f67ec073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec10d840 con 0x7f67ec101bc0 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec10d6c0 con 0x7f67ec103770 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67ec10d9c0 con 0x7f67ec102db0 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60220/0 (socket says 172.21.15.28:60220) 2024-03-22T02:13:54.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 -- 172.21.15.28:0/1199263731 learned_addr learned my addr 172.21.15.28:0/1199263731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eaffd640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 0x7f67ec073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67ebfff640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 -- 172.21.15.28:0/1199263731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 msgr2=0x7f67ec073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 0x7f67ec073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 -- 172.21.15.28:0/1199263731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 msgr2=0x7f67ec0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 -- 172.21.15.28:0/1199263731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67ec074d50 con 0x7f67ec101bc0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67eb7fe640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9c0 tx=0x7f67dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00cd80 con 0x7f67ec101bc0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67dc0035f0 con 0x7f67ec101bc0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67ec1a1fa0 con 0x7f67ec101bc0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67dc010690 con 0x7f67ec101bc0 2024-03-22T02:13:54.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.029+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67ec1a23b0 con 0x7f67ec101bc0 2024-03-22T02:13:54.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.030+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67ec108ee0 con 0x7f67ec101bc0 2024-03-22T02:13:54.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.032+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f67dc0037a0 con 0x7f67ec101bc0 2024-03-22T02:13:54.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.032+0000 7f67e8ff9640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 0x7f67d40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.032+0000 7f67eaffd640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 0x7f67d40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.032+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f67dc0cfcc0 con 0x7f67ec101bc0 2024-03-22T02:13:54.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.033+0000 7f67eaffd640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 0x7f67d40a5470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0756b0 tx=0x7f67e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.037+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f67dc09c580 con 0x7f67ec101bc0 2024-03-22T02:13:54.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.220+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f67ec02d3b0 con 0x7f67ec101bc0 2024-03-22T02:13:54.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.220+0000 7f67e8ff9640 1 -- 172.21.15.28:0/1199263731 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f67ec02d3b0 con 0x7f67ec101bc0 2024-03-22T02:13:54.221 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 msgr2=0x7f67d40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 0x7f67d40a5470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f67ec0756b0 tx=0x7f67e0005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 msgr2=0x7f67ec075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00e9c0 tx=0x7f67dc00ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 shutdown_connections 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67d40a2fb0 0x7f67d40a5470 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67ec103770 0x7f67ec0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67ec102db0 0x7f67ec073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 --2- 172.21.15.28:0/1199263731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67ec101bc0 0x7f67ec075940 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 >> 172.21.15.28:0/1199263731 conn(0x7f67ec0fd130 msgr2=0x7f67ec1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.222+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 shutdown_connections 2024-03-22T02:13:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.223+0000 7f67f1b3a640 1 -- 172.21.15.28:0/1199263731 wait complete. 2024-03-22T02:13:54.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:54.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 -- 172.21.15.28:0/3844646427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 msgr2=0x7f7658102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 --2- 172.21.15.28:0/3844646427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658102100 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7640009900 tx=0x7f764002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 -- 172.21.15.28:0/3844646427 shutdown_connections 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 --2- 172.21.15.28:0/3844646427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f7658108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 --2- 172.21.15.28:0/3844646427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658102100 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 --2- 172.21.15.28:0/3844646427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 -- 172.21.15.28:0/3844646427 >> 172.21.15.28:0/3844646427 conn(0x7f76580fc020 msgr2=0x7f76580fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 -- 172.21.15.28:0/3844646427 shutdown_connections 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.339+0000 7f765d69d640 1 -- 172.21.15.28:0/3844646427 wait complete. 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 Processor -- start 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 -- start start 2024-03-22T02:13:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f765819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f765810d830 con 0x7f7658101c80 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f765810d6b0 con 0x7f7658102640 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f765d69d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f765810d9b0 con 0x7f7658100a90 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f76567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f7656ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f76567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60244/0 (socket says 172.21.15.28:60244) 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f76567fc640 1 -- 172.21.15.28:0/3528884880 learned_addr learned my addr 172.21.15.28:0/3528884880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.340+0000 7f76577fe640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f765819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 -- 172.21.15.28:0/3528884880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 msgr2=0x7f7658197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 -- 172.21.15.28:0/3528884880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 msgr2=0x7f765819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f765819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 -- 172.21.15.28:0/3528884880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765819fb00 con 0x7f7658101c80 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f7656ffd640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76577fe640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f765819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f76567fc640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7640009230 tx=0x7f764002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640041070 con 0x7f7658101c80 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7640004450 con 0x7f7658101c80 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765819fd90 con 0x7f7658101c80 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7640003db0 con 0x7f7658101c80 2024-03-22T02:13:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.341+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76581a0270 con 0x7f7658101c80 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.342+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7658107ed0 con 0x7f7658101c80 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.343+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f7640044070 con 0x7f7658101c80 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.344+0000 7f764ffff640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 0x7f76440a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.344+0000 7f7656ffd640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 0x7f76440a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.344+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f76400f1380 con 0x7f7658101c80 2024-03-22T02:13:54.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.344+0000 7f7656ffd640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 0x7f76440a54f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7658101ae0 tx=0x7f7650005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.350+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76400bdd50 con 0x7f7658101c80 2024-03-22T02:13:54.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.533+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f765802d560 con 0x7f7658101c80 2024-03-22T02:13:54.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.534+0000 7f764ffff640 1 -- 172.21.15.28:0/3528884880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f76400bd650 con 0x7f7658101c80 2024-03-22T02:13:54.534 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 msgr2=0x7f76440a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 0x7f76440a54f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7658101ae0 tx=0x7f7650005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 msgr2=0x7f7658198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7640009230 tx=0x7f764002c650 comp rx=0 tx=0).stop 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 shutdown_connections 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76440a3030 0x7f76440a54f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7658102640 0x7f765819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7658101c80 0x7f7658198420 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 --2- 172.21.15.28:0/3528884880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7658100a90 0x7f7658197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 >> 172.21.15.28:0/3528884880 conn(0x7f76580fc020 msgr2=0x7f76581050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 shutdown_connections 2024-03-22T02:13:54.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.536+0000 7f765d69d640 1 -- 172.21.15.28:0/3528884880 wait complete. 2024-03-22T02:13:54.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:54 smithi028 ceph-mon[31916]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-22T02:13:54.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/227595922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:54.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2742174861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.546 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1199263731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:54.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 -- 172.21.15.28:0/2311172653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118103190 msgr2=0x7fb1181035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 --2- 172.21.15.28:0/2311172653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118103190 0x7fb1181035b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb110009900 tx=0x7fb11002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 -- 172.21.15.28:0/2311172653 shutdown_connections 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 --2- 172.21.15.28:0/2311172653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 0x7fb11810a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 --2- 172.21.15.28:0/2311172653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1180ffc00 0x7fb118100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 --2- 172.21.15.28:0/2311172653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118103190 0x7fb1181035b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 -- 172.21.15.28:0/2311172653 >> 172.21.15.28:0/2311172653 conn(0x7fb1180fb7c0 msgr2=0x7fb1180fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 -- 172.21.15.28:0/2311172653 shutdown_connections 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 -- 172.21.15.28:0/2311172653 wait complete. 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.653+0000 7fb11d17a640 1 Processor -- start 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 -- start start 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 0x7fb118198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 0x7fb11819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb11810d830 con 0x7fb1180ffc00 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb11810d6b0 con 0x7fb118103190 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11d17a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb11810d9b0 con 0x7fb1181005c0 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60258/0 (socket says 172.21.15.28:60258) 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 -- 172.21.15.28:0/1982261162 learned_addr learned my addr 172.21.15.28:0/1982261162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11759e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 0x7fb11819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11659c640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 0x7fb118198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 -- 172.21.15.28:0/1982261162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 msgr2=0x7fb118198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 0x7fb118198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 -- 172.21.15.28:0/1982261162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 msgr2=0x7fb11819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 0x7fb11819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb116d9d640 1 -- 172.21.15.28:0/1982261162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11819faf0 con 0x7fb1180ffc00 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.654+0000 7fb11759e640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 0x7fb11819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb116d9d640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb110000c00 tx=0x7fb11002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110041070 con 0x7fb1180ffc00 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb110004450 con 0x7fb1180ffc00 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11819fd80 con 0x7fb1180ffc00 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110034440 con 0x7fb1180ffc00 2024-03-22T02:13:54.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.655+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1181a01c0 con 0x7fb1180ffc00 2024-03-22T02:13:54.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.656+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1181090d0 con 0x7fb1180ffc00 2024-03-22T02:13:54.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.657+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fb110044070 con 0x7fb1180ffc00 2024-03-22T02:13:54.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.657+0000 7fb10ffff640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 0x7fb1040a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.658+0000 7fb11659c640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 0x7fb1040a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.658+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb1100f13a0 con 0x7fb1180ffc00 2024-03-22T02:13:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.658+0000 7fb11659c640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 0x7fb1040a5470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb1000095f0 tx=0x7fb100005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.662+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1100bd720 con 0x7fb1180ffc00 2024-03-22T02:13:54.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:54 smithi120 ceph-mon[36803]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-22T02:13:54.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/227595922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:54.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2742174861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1199263731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:54 smithi083 ceph-mon[36340]: pgmap v367: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2024-03-22T02:13:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/227595922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2742174861' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1199263731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.845+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb118062790 con 0x7fb1180ffc00 2024-03-22T02:13:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.846+0000 7fb10ffff640 1 -- 172.21.15.28:0/1982261162 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fb1100c17c0 con 0x7fb1180ffc00 2024-03-22T02:13:54.847 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 msgr2=0x7fb1040a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 0x7fb1040a5470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb1000095f0 tx=0x7fb100005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 msgr2=0x7fb118197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb110000c00 tx=0x7fb11002c650 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 shutdown_connections 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1040a2fb0 0x7fb1040a5470 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb118103190 0x7fb11819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1181005c0 0x7fb118198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 --2- 172.21.15.28:0/1982261162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1180ffc00 0x7fb118197ed0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 >> 172.21.15.28:0/1982261162 conn(0x7fb1180fb7c0 msgr2=0x7fb1181062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 shutdown_connections 2024-03-22T02:13:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.848+0000 7fb11d17a640 1 -- 172.21.15.28:0/1982261162 wait complete. 2024-03-22T02:13:54.859 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-22T02:13:54.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 -- 172.21.15.28:0/3613719355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374100ab0 msgr2=0x7f8374100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 --2- 172.21.15.28:0/3613719355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374100ab0 0x7f8374100ed0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8364009900 tx=0x7f836402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 -- 172.21.15.28:0/3613719355 shutdown_connections 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 --2- 172.21.15.28:0/3613719355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 0x7f8374108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 --2- 172.21.15.28:0/3613719355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374101ca0 0x7f8374102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 --2- 172.21.15.28:0/3613719355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374100ab0 0x7f8374100ed0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 -- 172.21.15.28:0/3613719355 >> 172.21.15.28:0/3613719355 conn(0x7f83740fc040 msgr2=0x7f83740fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 -- 172.21.15.28:0/3613719355 shutdown_connections 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.962+0000 7f837a6eb640 1 -- 172.21.15.28:0/3613719355 wait complete. 2024-03-22T02:13:54.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 Processor -- start 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 -- start start 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374101ca0 0x7f837419cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 0x7f837419f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f837410c730 con 0x7f8374100ab0 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f837410c5b0 con 0x7f8374102660 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f837a6eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f837410c8b0 con 0x7f8374101ca0 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f8373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f8373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60274/0 (socket says 172.21.15.28:60274) 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f8373fff640 1 -- 172.21.15.28:0/2101802693 learned_addr learned my addr 172.21.15.28:0/2101802693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f8378c88640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 0x7f837419f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.963+0000 7f83737fe640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374101ca0 0x7f837419cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 -- 172.21.15.28:0/2101802693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374101ca0 msgr2=0x7f837419cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374101ca0 0x7f837419cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 -- 172.21.15.28:0/2101802693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 msgr2=0x7f837419f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 0x7f837419f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 -- 172.21.15.28:0/2101802693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837419fb30 con 0x7f8374100ab0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f8373fff640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8364000c00 tx=0x7f836402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8364041070 con 0x7f8374100ab0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837419fd60 con 0x7f8374100ab0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8364004450 con 0x7f8374100ab0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8364034440 con 0x7f8374100ab0 2024-03-22T02:13:54.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.964+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83741a01e0 con 0x7f8374100ab0 2024-03-22T02:13:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.965+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8374107dd0 con 0x7f8374100ab0 2024-03-22T02:13:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.966+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f8364044070 con 0x7f8374100ab0 2024-03-22T02:13:54.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.967+0000 7f83717fa640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 0x7f835c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.967+0000 7f83737fe640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 0x7f835c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.967+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f83640f14f0 con 0x7f8374100ab0 2024-03-22T02:13:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.967+0000 7f83737fe640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 0x7f835c0a5470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f837406aac0 tx=0x7f8368005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:54.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:54.970+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83640bd8a0 con 0x7f8374100ab0 2024-03-22T02:13:55.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.193+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f837402d3b0 con 0x7f8374100ab0 2024-03-22T02:13:55.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.194+0000 7f83717fa640 1 -- 172.21.15.28:0/2101802693 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11866 (secure 0 0 0) 0x7f83640c1940 con 0x7f8374100ab0 2024-03-22T02:13:55.194 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:55.195 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:13:50.358406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:13:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-03-22T02:13:55.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 msgr2=0x7f835c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 0x7f835c0a5470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f837406aac0 tx=0x7f8368005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 msgr2=0x7f837406ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8364000c00 tx=0x7f836402c640 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 shutdown_connections 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f835c0a2fb0 0x7f835c0a5470 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8374102660 0x7f837419f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8374101ca0 0x7f837419cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 --2- 172.21.15.28:0/2101802693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8374100ab0 0x7f837406ad50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 >> 172.21.15.28:0/2101802693 conn(0x7f83740fc040 msgr2=0x7f8374104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 shutdown_connections 2024-03-22T02:13:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.196+0000 7f837a6eb640 1 -- 172.21.15.28:0/2101802693 wait complete. 2024-03-22T02:13:55.207 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-22T02:13:55.207 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-22T02:13:55.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T02:13:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.310+0000 7f4b0ab32640 1 -- 172.21.15.28:0/3813807268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04101bb0 msgr2=0x7f4b04101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.310+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/3813807268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04101bb0 0x7f4b04101fd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009900 tx=0x7f4af402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.310+0000 7f4b0ab32640 1 -- 172.21.15.28:0/3813807268 shutdown_connections 2024-03-22T02:13:55.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.310+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/3813807268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04103760 0x7f4b04109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.310+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/3813807268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 0x7f4b04103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/3813807268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04101bb0 0x7f4b04101fd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- 172.21.15.28:0/3813807268 >> 172.21.15.28:0/3813807268 conn(0x7f4b040fd140 msgr2=0x7f4b040ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- 172.21.15.28:0/3813807268 shutdown_connections 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- 172.21.15.28:0/3813807268 wait complete. 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 Processor -- start 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- start start 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04101bb0 0x7f4b04197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 0x7f4b04198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410d800 con 0x7f4b04103760 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410d680 con 0x7f4b04102da0 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b0ab32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0410d980 con 0x7f4b04101bb0 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b090cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b090cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60300/0 (socket says 172.21.15.28:60300) 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b090cf640 1 -- 172.21.15.28:0/2561560593 learned_addr learned my addr 172.21.15.28:0/2561560593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b088ce640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04101bb0 0x7f4b04197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.311+0000 7f4b03fff640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 0x7f4b04198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 -- 172.21.15.28:0/2561560593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04101bb0 msgr2=0x7f4b04197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04101bb0 0x7f4b04197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 -- 172.21.15.28:0/2561560593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 msgr2=0x7f4b04198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 0x7f4b04198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 -- 172.21.15.28:0/2561560593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b0419fb00 con 0x7f4b04103760 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b090cf640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00efd0 tx=0x7f4afc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc003b70 con 0x7f4b04103760 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4afc003d10 con 0x7f4b04103760 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b0419fdf0 con 0x7f4b04103760 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc0109f0 con 0x7f4b04103760 2024-03-22T02:13:55.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.312+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b041a02b0 con 0x7f4b04103760 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.313+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b04108f50 con 0x7f4b04103760 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.314+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f4afc004020 con 0x7f4b04103760 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.315+0000 7f4b01ffb640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 0x7f4aec0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.315+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4afc0cfd30 con 0x7f4b04103760 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.315+0000 7f4b088ce640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 0x7f4aec0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.315+0000 7f4b088ce640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 0x7f4aec0a54f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4af4000c00 tx=0x7f4af4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.319+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4afc0d4070 con 0x7f4b04103760 2024-03-22T02:13:55.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.499+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4b04061fd0 con 0x7f4aec0a3030 2024-03-22T02:13:55.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.508+0000 7f4b01ffb640 1 -- 172.21.15.28:0/2561560593 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340801 (secure 0 0 0) 0x7f4b04061fd0 con 0x7f4aec0a3030 2024-03-22T02:13:55.513 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:13:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 msgr2=0x7f4aec0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 0x7f4aec0a54f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4af4000c00 tx=0x7f4af4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 msgr2=0x7f4b0419f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4afc00efd0 tx=0x7f4afc00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 shutdown_connections 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4aec0a3030 0x7f4aec0a54f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b04103760 0x7f4b0419f490 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4b04102da0 0x7f4b04198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 --2- 172.21.15.28:0/2561560593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b04101bb0 0x7f4b04197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 >> 172.21.15.28:0/2561560593 conn(0x7f4b040fd140 msgr2=0x7f4b04106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 shutdown_connections 2024-03-22T02:13:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.512+0000 7f4b0ab32640 1 -- 172.21.15.28:0/2561560593 wait complete. 2024-03-22T02:13:55.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:55 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3528884880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:55 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1982261162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:55 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2101802693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:55.523 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-22T02:13:54.587702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664729,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":859,"num_read_kb":552,"num_write":358,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365888,"kb_used_data":12544,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575936,"statfs":{"total":1151940427776,"available":1151565758464,"internally_reserved":0,"allocated":12845056,"data_stored":8240576,"data_compressed":432609,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6380,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232507+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:13:29.232507+0000","last_peered":"2024-03-22T02:13:29.232507+0000","last_clean":"2024-03-22T02:13:29.232507+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:13:29.232507+0000","last_undegraded":"2024-03-22T02:13:29.232507+0000","last_fullsized":"2024-03-22T02:13:29.232507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234634+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:13:29.234634+0000","last_peered":"2024-03-22T02:13:29.234634+0000","last_clean":"2024-03-22T02:13:29.234634+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:13:29.234634+0000","last_undegraded":"2024-03-22T02:13:29.234634+0000","last_fullsized":"2024-03-22T02:13:29.234634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234266+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:13:29.234266+0000","last_peered":"2024-03-22T02:13:29.234266+0000","last_clean":"2024-03-22T02:13:29.234266+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:13:29.234266+0000","last_undegraded":"2024-03-22T02:13:29.234266+0000","last_fullsized":"2024-03-22T02:13:29.234266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229827+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:13:29.229827+0000","last_peered":"2024-03-22T02:13:29.229827+0000","last_clean":"2024-03-22T02:13:29.229827+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:13:29.229827+0000","last_undegraded":"2024-03-22T02:13:29.229827+0000","last_fullsized":"2024-03-22T02:13:29.229827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235150+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:13:29.235150+0000","last_peered":"2024-03-22T02:13:29.235150+0000","last_clean":"2024-03-22T02:13:29.235150+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:13:29.235150+0000","last_undegraded":"2024-03-22T02:13:29.235150+0000","last_fullsized":"2024-03-22T02:13:29.235150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666425+0000","last_change":"2024-03-22T02:12:28.521805+0000","last_active":"2024-03-22T02:13:29.666425+0000","last_peered":"2024-03-22T02:13:29.666425+0000","last_clean":"2024-03-22T02:13:29.666425+0000","last_became_active":"2024-03-22T02:12:28.521419+0000","last_became_peered":"2024-03-22T02:12:28.521419+0000","last_unstale":"2024-03-22T02:13:29.666425+0000","last_undegraded":"2024-03-22T02:13:29.666425+0000","last_fullsized":"2024-03-22T02:13:29.666425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:01:11.831486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232631+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:13:29.232631+0000","last_peered":"2024-03-22T02:13:29.232631+0000","last_clean":"2024-03-22T02:13:29.232631+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:13:29.232631+0000","last_undegraded":"2024-03-22T02:13:29.232631+0000","last_fullsized":"2024-03-22T02:13:29.232631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229177+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:13:29.229177+0000","last_peered":"2024-03-22T02:13:29.229177+0000","last_clean":"2024-03-22T02:13:29.229177+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:13:29.229177+0000","last_undegraded":"2024-03-22T02:13:29.229177+0000","last_fullsized":"2024-03-22T02:13:29.229177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235354+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:13:29.235354+0000","last_peered":"2024-03-22T02:13:29.235354+0000","last_clean":"2024-03-22T02:13:29.235354+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:13:29.235354+0000","last_undegraded":"2024-03-22T02:13:29.235354+0000","last_fullsized":"2024-03-22T02:13:29.235354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666800+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:13:29.666800+0000","last_peered":"2024-03-22T02:13:29.666800+0000","last_clean":"2024-03-22T02:13:29.666800+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:13:29.666800+0000","last_undegraded":"2024-03-22T02:13:29.666800+0000","last_fullsized":"2024-03-22T02:13:29.666800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233246+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:13:29.233246+0000","last_peered":"2024-03-22T02:13:29.233246+0000","last_clean":"2024-03-22T02:13:29.233246+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:13:29.233246+0000","last_undegraded":"2024-03-22T02:13:29.233246+0000","last_fullsized":"2024-03-22T02:13:29.233246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229832+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:13:29.229832+0000","last_peered":"2024-03-22T02:13:29.229832+0000","last_clean":"2024-03-22T02:13:29.229832+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:13:29.229832+0000","last_undegraded":"2024-03-22T02:13:29.229832+0000","last_fullsized":"2024-03-22T02:13:29.229832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247995+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:13:29.247995+0000","last_peered":"2024-03-22T02:13:29.247995+0000","last_clean":"2024-03-22T02:13:29.247995+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:13:29.247995+0000","last_undegraded":"2024-03-22T02:13:29.247995+0000","last_fullsized":"2024-03-22T02:13:29.247995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234497+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:13:29.234497+0000","last_peered":"2024-03-22T02:13:29.234497+0000","last_clean":"2024-03-22T02:13:29.234497+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:13:29.234497+0000","last_undegraded":"2024-03-22T02:13:29.234497+0000","last_fullsized":"2024-03-22T02:13:29.234497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234649+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:13:29.234649+0000","last_peered":"2024-03-22T02:13:29.234649+0000","last_clean":"2024-03-22T02:13:29.234649+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:13:29.234649+0000","last_undegraded":"2024-03-22T02:13:29.234649+0000","last_fullsized":"2024-03-22T02:13:29.234649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234959+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:13:29.234959+0000","last_peered":"2024-03-22T02:13:29.234959+0000","last_clean":"2024-03-22T02:13:29.234959+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:13:29.234959+0000","last_undegraded":"2024-03-22T02:13:29.234959+0000","last_fullsized":"2024-03-22T02:13:29.234959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231082+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:13:29.231082+0000","last_peered":"2024-03-22T02:13:29.231082+0000","last_clean":"2024-03-22T02:13:29.231082+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:13:29.231082+0000","last_undegraded":"2024-03-22T02:13:29.231082+0000","last_fullsized":"2024-03-22T02:13:29.231082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235450+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:13:29.235450+0000","last_peered":"2024-03-22T02:13:29.235450+0000","last_clean":"2024-03-22T02:13:29.235450+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:13:29.235450+0000","last_undegraded":"2024-03-22T02:13:29.235450+0000","last_fullsized":"2024-03-22T02:13:29.235450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232547+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:13:29.232547+0000","last_peered":"2024-03-22T02:13:29.232547+0000","last_clean":"2024-03-22T02:13:29.232547+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:13:29.232547+0000","last_undegraded":"2024-03-22T02:13:29.232547+0000","last_fullsized":"2024-03-22T02:13:29.232547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.236492+0000","last_change":"2024-03-22T02:13:29.236492+0000","last_active":"2024-03-22T02:13:29.236492+0000","last_peered":"2024-03-22T02:13:29.236492+0000","last_clean":"2024-03-22T02:13:29.236492+0000","last_became_active":"2024-03-22T02:13:29.236092+0000","last_became_peered":"2024-03-22T02:13:29.236092+0000","last_unstale":"2024-03-22T02:13:29.236492+0000","last_undegraded":"2024-03-22T02:13:29.236492+0000","last_fullsized":"2024-03-22T02:13:29.236492+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:42.511417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235774+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:13:29.235774+0000","last_peered":"2024-03-22T02:13:29.235774+0000","last_clean":"2024-03-22T02:13:29.235774+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:13:29.235774+0000","last_undegraded":"2024-03-22T02:13:29.235774+0000","last_fullsized":"2024-03-22T02:13:29.235774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229678+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:13:29.229678+0000","last_peered":"2024-03-22T02:13:29.229678+0000","last_clean":"2024-03-22T02:13:29.229678+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:13:29.229678+0000","last_undegraded":"2024-03-22T02:13:29.229678+0000","last_fullsized":"2024-03-22T02:13:29.229678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229358+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:13:29.229358+0000","last_peered":"2024-03-22T02:13:29.229358+0000","last_clean":"2024-03-22T02:13:29.229358+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:13:29.229358+0000","last_undegraded":"2024-03-22T02:13:29.229358+0000","last_fullsized":"2024-03-22T02:13:29.229358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229936+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:13:29.229936+0000","last_peered":"2024-03-22T02:13:29.229936+0000","last_clean":"2024-03-22T02:13:29.229936+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:13:29.229936+0000","last_undegraded":"2024-03-22T02:13:29.229936+0000","last_fullsized":"2024-03-22T02:13:29.229936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234882+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:13:29.234882+0000","last_peered":"2024-03-22T02:13:29.234882+0000","last_clean":"2024-03-22T02:13:29.234882+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:13:29.234882+0000","last_undegraded":"2024-03-22T02:13:29.234882+0000","last_fullsized":"2024-03-22T02:13:29.234882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229464+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:13:29.229464+0000","last_peered":"2024-03-22T02:13:29.229464+0000","last_clean":"2024-03-22T02:13:29.229464+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:13:29.229464+0000","last_undegraded":"2024-03-22T02:13:29.229464+0000","last_fullsized":"2024-03-22T02:13:29.229464+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242108+0000","last_change":"2024-03-22T02:13:29.242108+0000","last_active":"2024-03-22T02:13:29.242108+0000","last_peered":"2024-03-22T02:13:29.242108+0000","last_clean":"2024-03-22T02:13:29.242108+0000","last_became_active":"2024-03-22T02:13:29.236686+0000","last_became_peered":"2024-03-22T02:13:29.236686+0000","last_unstale":"2024-03-22T02:13:29.242108+0000","last_undegraded":"2024-03-22T02:13:29.242108+0000","last_fullsized":"2024-03-22T02:13:29.242108+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:42:01.218922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235315+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:13:29.235315+0000","last_peered":"2024-03-22T02:13:29.235315+0000","last_clean":"2024-03-22T02:13:29.235315+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:13:29.235315+0000","last_undegraded":"2024-03-22T02:13:29.235315+0000","last_fullsized":"2024-03-22T02:13:29.235315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229375+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:13:29.229375+0000","last_peered":"2024-03-22T02:13:29.229375+0000","last_clean":"2024-03-22T02:13:29.229375+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:13:29.229375+0000","last_undegraded":"2024-03-22T02:13:29.229375+0000","last_fullsized":"2024-03-22T02:13:29.229375+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235311+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:13:29.235311+0000","last_peered":"2024-03-22T02:13:29.235311+0000","last_clean":"2024-03-22T02:13:29.235311+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:13:29.235311+0000","last_undegraded":"2024-03-22T02:13:29.235311+0000","last_fullsized":"2024-03-22T02:13:29.235311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536646+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:13:29.536646+0000","last_peered":"2024-03-22T02:13:29.536646+0000","last_clean":"2024-03-22T02:13:29.536646+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:13:29.536646+0000","last_undegraded":"2024-03-22T02:13:29.536646+0000","last_fullsized":"2024-03-22T02:13:29.536646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231729+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:13:29.231729+0000","last_peered":"2024-03-22T02:13:29.231729+0000","last_clean":"2024-03-22T02:13:29.231729+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:13:29.231729+0000","last_undegraded":"2024-03-22T02:13:29.231729+0000","last_fullsized":"2024-03-22T02:13:29.231729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:51.111514+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:13:51.111514+0000","last_peered":"2024-03-22T02:13:51.111514+0000","last_clean":"2024-03-22T02:13:51.111514+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:13:51.111514+0000","last_undegraded":"2024-03-22T02:13:51.111514+0000","last_fullsized":"2024-03-22T02:13:51.111514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234972+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:13:29.234972+0000","last_peered":"2024-03-22T02:13:29.234972+0000","last_clean":"2024-03-22T02:13:29.234972+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:13:29.234972+0000","last_undegraded":"2024-03-22T02:13:29.234972+0000","last_fullsized":"2024-03-22T02:13:29.234972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667818+0000","last_change":"2024-03-22T02:13:29.667818+0000","last_active":"2024-03-22T02:13:29.667818+0000","last_peered":"2024-03-22T02:13:29.667818+0000","last_clean":"2024-03-22T02:13:29.667818+0000","last_became_active":"2024-03-22T02:13:29.667344+0000","last_became_peered":"2024-03-22T02:13:29.667344+0000","last_unstale":"2024-03-22T02:13:29.667818+0000","last_undegraded":"2024-03-22T02:13:29.667818+0000","last_fullsized":"2024-03-22T02:13:29.667818+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:08.802171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666359+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:13:29.666359+0000","last_peered":"2024-03-22T02:13:29.666359+0000","last_clean":"2024-03-22T02:13:29.666359+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:13:29.666359+0000","last_undegraded":"2024-03-22T02:13:29.666359+0000","last_fullsized":"2024-03-22T02:13:29.666359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536554+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:13:29.536554+0000","last_peered":"2024-03-22T02:13:29.536554+0000","last_clean":"2024-03-22T02:13:29.536554+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:13:29.536554+0000","last_undegraded":"2024-03-22T02:13:29.536554+0000","last_fullsized":"2024-03-22T02:13:29.536554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231009+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:13:29.231009+0000","last_peered":"2024-03-22T02:13:29.231009+0000","last_clean":"2024-03-22T02:13:29.231009+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:13:29.231009+0000","last_undegraded":"2024-03-22T02:13:29.231009+0000","last_fullsized":"2024-03-22T02:13:29.231009+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247642+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:13:29.247642+0000","last_peered":"2024-03-22T02:13:29.247642+0000","last_clean":"2024-03-22T02:13:29.247642+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:13:29.247642+0000","last_undegraded":"2024-03-22T02:13:29.247642+0000","last_fullsized":"2024-03-22T02:13:29.247642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229323+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:13:29.229323+0000","last_peered":"2024-03-22T02:13:29.229323+0000","last_clean":"2024-03-22T02:13:29.229323+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:13:29.229323+0000","last_undegraded":"2024-03-22T02:13:29.229323+0000","last_fullsized":"2024-03-22T02:13:29.229323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234585+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:13:29.234585+0000","last_peered":"2024-03-22T02:13:29.234585+0000","last_clean":"2024-03-22T02:13:29.234585+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:13:29.234585+0000","last_undegraded":"2024-03-22T02:13:29.234585+0000","last_fullsized":"2024-03-22T02:13:29.234585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536228+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:13:29.536228+0000","last_peered":"2024-03-22T02:13:29.536228+0000","last_clean":"2024-03-22T02:13:29.536228+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:13:29.536228+0000","last_undegraded":"2024-03-22T02:13:29.536228+0000","last_fullsized":"2024-03-22T02:13:29.536228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234870+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:13:29.234870+0000","last_peered":"2024-03-22T02:13:29.234870+0000","last_clean":"2024-03-22T02:13:29.234870+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:13:29.234870+0000","last_undegraded":"2024-03-22T02:13:29.234870+0000","last_fullsized":"2024-03-22T02:13:29.234870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229276+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.229276+0000","last_peered":"2024-03-22T02:13:29.229276+0000","last_clean":"2024-03-22T02:13:29.229276+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:13:29.229276+0000","last_undegraded":"2024-03-22T02:13:29.229276+0000","last_fullsized":"2024-03-22T02:13:29.229276+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:49.108917+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:13:49.108917+0000","last_peered":"2024-03-22T02:13:49.108917+0000","last_clean":"2024-03-22T02:13:49.108917+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:13:49.108917+0000","last_undegraded":"2024-03-22T02:13:49.108917+0000","last_fullsized":"2024-03-22T02:13:49.108917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235071+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:13:29.235071+0000","last_peered":"2024-03-22T02:13:29.235071+0000","last_clean":"2024-03-22T02:13:29.235071+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:13:29.235071+0000","last_undegraded":"2024-03-22T02:13:29.235071+0000","last_fullsized":"2024-03-22T02:13:29.235071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234467+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:13:29.234467+0000","last_peered":"2024-03-22T02:13:29.234467+0000","last_clean":"2024-03-22T02:13:29.234467+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:13:29.234467+0000","last_undegraded":"2024-03-22T02:13:29.234467+0000","last_fullsized":"2024-03-22T02:13:29.234467+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.238453+0000","last_change":"2024-03-22T02:13:29.238453+0000","last_active":"2024-03-22T02:13:29.238453+0000","last_peered":"2024-03-22T02:13:29.238453+0000","last_clean":"2024-03-22T02:13:29.238453+0000","last_became_active":"2024-03-22T02:13:29.235861+0000","last_became_peered":"2024-03-22T02:13:29.235861+0000","last_unstale":"2024-03-22T02:13:29.238453+0000","last_undegraded":"2024-03-22T02:13:29.238453+0000","last_fullsized":"2024-03-22T02:13:29.238453+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:50:35.945707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234160+0000","last_change":"2024-03-22T02:12:28.521865+0000","last_active":"2024-03-22T02:13:29.234160+0000","last_peered":"2024-03-22T02:13:29.234160+0000","last_clean":"2024-03-22T02:13:29.234160+0000","last_became_active":"2024-03-22T02:12:28.521413+0000","last_became_peered":"2024-03-22T02:12:28.521413+0000","last_unstale":"2024-03-22T02:13:29.234160+0000","last_undegraded":"2024-03-22T02:13:29.234160+0000","last_fullsized":"2024-03-22T02:13:29.234160+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:54:26.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229759+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:13:29.229759+0000","last_peered":"2024-03-22T02:13:29.229759+0000","last_clean":"2024-03-22T02:13:29.229759+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:13:29.229759+0000","last_undegraded":"2024-03-22T02:13:29.229759+0000","last_fullsized":"2024-03-22T02:13:29.229759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536583+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:13:29.536583+0000","last_peered":"2024-03-22T02:13:29.536583+0000","last_clean":"2024-03-22T02:13:29.536583+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:13:29.536583+0000","last_undegraded":"2024-03-22T02:13:29.536583+0000","last_fullsized":"2024-03-22T02:13:29.536583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235431+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:13:29.235431+0000","last_peered":"2024-03-22T02:13:29.235431+0000","last_clean":"2024-03-22T02:13:29.235431+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:13:29.235431+0000","last_undegraded":"2024-03-22T02:13:29.235431+0000","last_fullsized":"2024-03-22T02:13:29.235431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235070+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.235070+0000","last_peered":"2024-03-22T02:13:29.235070+0000","last_clean":"2024-03-22T02:13:29.235070+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:13:29.235070+0000","last_undegraded":"2024-03-22T02:13:29.235070+0000","last_fullsized":"2024-03-22T02:13:29.235070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":311,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.109991+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:13:50.109991+0000","last_peered":"2024-03-22T02:13:50.109991+0000","last_clean":"2024-03-22T02:13:50.109991+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:13:50.109991+0000","last_undegraded":"2024-03-22T02:13:50.109991+0000","last_fullsized":"2024-03-22T02:13:50.109991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":242,"num_read_kb":189,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235127+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:13:29.235127+0000","last_peered":"2024-03-22T02:13:29.235127+0000","last_clean":"2024-03-22T02:13:29.235127+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:13:29.235127+0000","last_undegraded":"2024-03-22T02:13:29.235127+0000","last_fullsized":"2024-03-22T02:13:29.235127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536841+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:13:29.536841+0000","last_peered":"2024-03-22T02:13:29.536841+0000","last_clean":"2024-03-22T02:13:29.536841+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:13:29.536841+0000","last_undegraded":"2024-03-22T02:13:29.536841+0000","last_fullsized":"2024-03-22T02:13:29.536841+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231083+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:13:29.231083+0000","last_peered":"2024-03-22T02:13:29.231083+0000","last_clean":"2024-03-22T02:13:29.231083+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:13:29.231083+0000","last_undegraded":"2024-03-22T02:13:29.231083+0000","last_fullsized":"2024-03-22T02:13:29.231083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247705+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:13:29.247705+0000","last_peered":"2024-03-22T02:13:29.247705+0000","last_clean":"2024-03-22T02:13:29.247705+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:13:29.247705+0000","last_undegraded":"2024-03-22T02:13:29.247705+0000","last_fullsized":"2024-03-22T02:13:29.247705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231568+0000","last_change":"2024-03-22T02:12:28.520817+0000","last_active":"2024-03-22T02:13:29.231568+0000","last_peered":"2024-03-22T02:13:29.231568+0000","last_clean":"2024-03-22T02:13:29.231568+0000","last_became_active":"2024-03-22T02:12:28.520360+0000","last_became_peered":"2024-03-22T02:12:28.520360+0000","last_unstale":"2024-03-22T02:13:29.231568+0000","last_undegraded":"2024-03-22T02:13:29.231568+0000","last_fullsized":"2024-03-22T02:13:29.231568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:34:21.654588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536568+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:13:29.536568+0000","last_peered":"2024-03-22T02:13:29.536568+0000","last_clean":"2024-03-22T02:13:29.536568+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:13:29.536568+0000","last_undegraded":"2024-03-22T02:13:29.536568+0000","last_fullsized":"2024-03-22T02:13:29.536568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233232+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:13:29.233232+0000","last_peered":"2024-03-22T02:13:29.233232+0000","last_clean":"2024-03-22T02:13:29.233232+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:13:29.233232+0000","last_undegraded":"2024-03-22T02:13:29.233232+0000","last_fullsized":"2024-03-22T02:13:29.233232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235648+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:13:29.235648+0000","last_peered":"2024-03-22T02:13:29.235648+0000","last_clean":"2024-03-22T02:13:29.235648+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:13:29.235648+0000","last_undegraded":"2024-03-22T02:13:29.235648+0000","last_fullsized":"2024-03-22T02:13:29.235648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'5","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:53.143913+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:13:53.143913+0000","last_peered":"2024-03-22T02:13:53.143913+0000","last_clean":"2024-03-22T02:13:53.143913+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:13:53.143913+0000","last_undegraded":"2024-03-22T02:13:53.143913+0000","last_fullsized":"2024-03-22T02:13:53.143913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232769+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:13:29.232769+0000","last_peered":"2024-03-22T02:13:29.232769+0000","last_clean":"2024-03-22T02:13:29.232769+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:13:29.232769+0000","last_undegraded":"2024-03-22T02:13:29.232769+0000","last_fullsized":"2024-03-22T02:13:29.232769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248019+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:13:29.248019+0000","last_peered":"2024-03-22T02:13:29.248019+0000","last_clean":"2024-03-22T02:13:29.248019+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:13:29.248019+0000","last_undegraded":"2024-03-22T02:13:29.248019+0000","last_fullsized":"2024-03-22T02:13:29.248019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234158+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:13:29.234158+0000","last_peered":"2024-03-22T02:13:29.234158+0000","last_clean":"2024-03-22T02:13:29.234158+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:13:29.234158+0000","last_undegraded":"2024-03-22T02:13:29.234158+0000","last_fullsized":"2024-03-22T02:13:29.234158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536334+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:13:29.536334+0000","last_peered":"2024-03-22T02:13:29.536334+0000","last_clean":"2024-03-22T02:13:29.536334+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:13:29.536334+0000","last_undegraded":"2024-03-22T02:13:29.536334+0000","last_fullsized":"2024-03-22T02:13:29.536334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536312+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:13:29.536312+0000","last_peered":"2024-03-22T02:13:29.536312+0000","last_clean":"2024-03-22T02:13:29.536312+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:13:29.536312+0000","last_undegraded":"2024-03-22T02:13:29.536312+0000","last_fullsized":"2024-03-22T02:13:29.536312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235355+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:13:29.235355+0000","last_peered":"2024-03-22T02:13:29.235355+0000","last_clean":"2024-03-22T02:13:29.235355+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:13:29.235355+0000","last_undegraded":"2024-03-22T02:13:29.235355+0000","last_fullsized":"2024-03-22T02:13:29.235355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231288+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:13:29.231288+0000","last_peered":"2024-03-22T02:13:29.231288+0000","last_clean":"2024-03-22T02:13:29.231288+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:13:29.231288+0000","last_undegraded":"2024-03-22T02:13:29.231288+0000","last_fullsized":"2024-03-22T02:13:29.231288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234912+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:13:29.234912+0000","last_peered":"2024-03-22T02:13:29.234912+0000","last_clean":"2024-03-22T02:13:29.234912+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:13:29.234912+0000","last_undegraded":"2024-03-22T02:13:29.234912+0000","last_fullsized":"2024-03-22T02:13:29.234912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235141+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:13:29.235141+0000","last_peered":"2024-03-22T02:13:29.235141+0000","last_clean":"2024-03-22T02:13:29.235141+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:13:29.235141+0000","last_undegraded":"2024-03-22T02:13:29.235141+0000","last_fullsized":"2024-03-22T02:13:29.235141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666733+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:13:29.666733+0000","last_peered":"2024-03-22T02:13:29.666733+0000","last_clean":"2024-03-22T02:13:29.666733+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:13:29.666733+0000","last_undegraded":"2024-03-22T02:13:29.666733+0000","last_fullsized":"2024-03-22T02:13:29.666733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248014+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:13:29.248014+0000","last_peered":"2024-03-22T02:13:29.248014+0000","last_clean":"2024-03-22T02:13:29.248014+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:13:29.248014+0000","last_undegraded":"2024-03-22T02:13:29.248014+0000","last_fullsized":"2024-03-22T02:13:29.248014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232955+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:13:29.232955+0000","last_peered":"2024-03-22T02:13:29.232955+0000","last_clean":"2024-03-22T02:13:29.232955+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:13:29.232955+0000","last_undegraded":"2024-03-22T02:13:29.232955+0000","last_fullsized":"2024-03-22T02:13:29.232955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666435+0000","last_change":"2024-03-22T02:12:28.488917+0000","last_active":"2024-03-22T02:13:29.666435+0000","last_peered":"2024-03-22T02:13:29.666435+0000","last_clean":"2024-03-22T02:13:29.666435+0000","last_became_active":"2024-03-22T02:12:28.488542+0000","last_became_peered":"2024-03-22T02:12:28.488542+0000","last_unstale":"2024-03-22T02:13:29.666435+0000","last_undegraded":"2024-03-22T02:13:29.666435+0000","last_fullsized":"2024-03-22T02:13:29.666435+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:41.575475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248137+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:13:29.248137+0000","last_peered":"2024-03-22T02:13:29.248137+0000","last_clean":"2024-03-22T02:13:29.248137+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:13:29.248137+0000","last_undegraded":"2024-03-22T02:13:29.248137+0000","last_fullsized":"2024-03-22T02:13:29.248137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536336+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:13:29.536336+0000","last_peered":"2024-03-22T02:13:29.536336+0000","last_clean":"2024-03-22T02:13:29.536336+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:13:29.536336+0000","last_undegraded":"2024-03-22T02:13:29.536336+0000","last_fullsized":"2024-03-22T02:13:29.536336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.138873+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:13:50.138873+0000","last_peered":"2024-03-22T02:13:50.138873+0000","last_clean":"2024-03-22T02:13:50.138873+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:13:50.138873+0000","last_undegraded":"2024-03-22T02:13:50.138873+0000","last_fullsized":"2024-03-22T02:13:50.138873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232977+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:13:29.232977+0000","last_peered":"2024-03-22T02:13:29.232977+0000","last_clean":"2024-03-22T02:13:29.232977+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:13:29.232977+0000","last_undegraded":"2024-03-22T02:13:29.232977+0000","last_fullsized":"2024-03-22T02:13:29.232977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229511+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:13:29.229511+0000","last_peered":"2024-03-22T02:13:29.229511+0000","last_clean":"2024-03-22T02:13:29.229511+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:13:29.229511+0000","last_undegraded":"2024-03-22T02:13:29.229511+0000","last_fullsized":"2024-03-22T02:13:29.229511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229686+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:13:29.229686+0000","last_peered":"2024-03-22T02:13:29.229686+0000","last_clean":"2024-03-22T02:13:29.229686+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:13:29.229686+0000","last_undegraded":"2024-03-22T02:13:29.229686+0000","last_fullsized":"2024-03-22T02:13:29.229686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232583+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:13:29.232583+0000","last_peered":"2024-03-22T02:13:29.232583+0000","last_clean":"2024-03-22T02:13:29.232583+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:13:29.232583+0000","last_undegraded":"2024-03-22T02:13:29.232583+0000","last_fullsized":"2024-03-22T02:13:29.232583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667520+0000","last_change":"2024-03-22T02:13:29.667520+0000","last_active":"2024-03-22T02:13:29.667520+0000","last_peered":"2024-03-22T02:13:29.667520+0000","last_clean":"2024-03-22T02:13:29.667520+0000","last_became_active":"2024-03-22T02:13:29.666948+0000","last_became_peered":"2024-03-22T02:13:29.666948+0000","last_unstale":"2024-03-22T02:13:29.667520+0000","last_undegraded":"2024-03-22T02:13:29.667520+0000","last_fullsized":"2024-03-22T02:13:29.667520+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:00:56.014210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247578+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:13:29.247578+0000","last_peered":"2024-03-22T02:13:29.247578+0000","last_clean":"2024-03-22T02:13:29.247578+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:13:29.247578+0000","last_undegraded":"2024-03-22T02:13:29.247578+0000","last_fullsized":"2024-03-22T02:13:29.247578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232655+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:13:29.232655+0000","last_peered":"2024-03-22T02:13:29.232655+0000","last_clean":"2024-03-22T02:13:29.232655+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:13:29.232655+0000","last_undegraded":"2024-03-22T02:13:29.232655+0000","last_fullsized":"2024-03-22T02:13:29.232655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231152+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:13:29.231152+0000","last_peered":"2024-03-22T02:13:29.231152+0000","last_clean":"2024-03-22T02:13:29.231152+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:13:29.231152+0000","last_undegraded":"2024-03-22T02:13:29.231152+0000","last_fullsized":"2024-03-22T02:13:29.231152+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242062+0000","last_change":"2024-03-22T02:13:29.241667+0000","last_active":"2024-03-22T02:13:29.242062+0000","last_peered":"2024-03-22T02:13:29.242062+0000","last_clean":"2024-03-22T02:13:29.242062+0000","last_became_active":"2024-03-22T02:13:29.236378+0000","last_became_peered":"2024-03-22T02:13:29.236378+0000","last_unstale":"2024-03-22T02:13:29.242062+0000","last_undegraded":"2024-03-22T02:13:29.242062+0000","last_fullsized":"2024-03-22T02:13:29.242062+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:41:25.050034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248131+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:13:29.248131+0000","last_peered":"2024-03-22T02:13:29.248131+0000","last_clean":"2024-03-22T02:13:29.248131+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:13:29.248131+0000","last_undegraded":"2024-03-22T02:13:29.248131+0000","last_fullsized":"2024-03-22T02:13:29.248131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229304+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:13:29.229304+0000","last_peered":"2024-03-22T02:13:29.229304+0000","last_clean":"2024-03-22T02:13:29.229304+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:13:29.229304+0000","last_undegraded":"2024-03-22T02:13:29.229304+0000","last_fullsized":"2024-03-22T02:13:29.229304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:44.917232+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:13:44.917232+0000","last_peered":"2024-03-22T02:13:44.917232+0000","last_clean":"2024-03-22T02:13:44.917232+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:13:44.917232+0000","last_undegraded":"2024-03-22T02:13:44.917232+0000","last_fullsized":"2024-03-22T02:13:44.917232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229947+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:13:29.229947+0000","last_peered":"2024-03-22T02:13:29.229947+0000","last_clean":"2024-03-22T02:13:29.229947+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:13:29.229947+0000","last_undegraded":"2024-03-22T02:13:29.229947+0000","last_fullsized":"2024-03-22T02:13:29.229947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231155+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:13:29.231155+0000","last_peered":"2024-03-22T02:13:29.231155+0000","last_clean":"2024-03-22T02:13:29.231155+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:13:29.231155+0000","last_undegraded":"2024-03-22T02:13:29.231155+0000","last_fullsized":"2024-03-22T02:13:29.231155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229873+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:13:29.229873+0000","last_peered":"2024-03-22T02:13:29.229873+0000","last_clean":"2024-03-22T02:13:29.229873+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:13:29.229873+0000","last_undegraded":"2024-03-22T02:13:29.229873+0000","last_fullsized":"2024-03-22T02:13:29.229873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234699+0000","last_change":"2024-03-22T02:12:28.489608+0000","last_active":"2024-03-22T02:13:29.234699+0000","last_peered":"2024-03-22T02:13:29.234699+0000","last_clean":"2024-03-22T02:13:29.234699+0000","last_became_active":"2024-03-22T02:12:28.489005+0000","last_became_peered":"2024-03-22T02:12:28.489005+0000","last_unstale":"2024-03-22T02:13:29.234699+0000","last_undegraded":"2024-03-22T02:13:29.234699+0000","last_fullsized":"2024-03-22T02:13:29.234699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:01.427654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229642+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:13:29.229642+0000","last_peered":"2024-03-22T02:13:29.229642+0000","last_clean":"2024-03-22T02:13:29.229642+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:13:29.229642+0000","last_undegraded":"2024-03-22T02:13:29.229642+0000","last_fullsized":"2024-03-22T02:13:29.229642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234767+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:13:29.234767+0000","last_peered":"2024-03-22T02:13:29.234767+0000","last_clean":"2024-03-22T02:13:29.234767+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:13:29.234767+0000","last_undegraded":"2024-03-22T02:13:29.234767+0000","last_fullsized":"2024-03-22T02:13:29.234767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234695+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:13:29.234695+0000","last_peered":"2024-03-22T02:13:29.234695+0000","last_clean":"2024-03-22T02:13:29.234695+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:13:29.234695+0000","last_undegraded":"2024-03-22T02:13:29.234695+0000","last_fullsized":"2024-03-22T02:13:29.234695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235725+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:13:29.235725+0000","last_peered":"2024-03-22T02:13:29.235725+0000","last_clean":"2024-03-22T02:13:29.235725+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:13:29.235725+0000","last_undegraded":"2024-03-22T02:13:29.235725+0000","last_fullsized":"2024-03-22T02:13:29.235725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229456+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:13:29.229456+0000","last_peered":"2024-03-22T02:13:29.229456+0000","last_clean":"2024-03-22T02:13:29.229456+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:13:29.229456+0000","last_undegraded":"2024-03-22T02:13:29.229456+0000","last_fullsized":"2024-03-22T02:13:29.229456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230885+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:13:29.230885+0000","last_peered":"2024-03-22T02:13:29.230885+0000","last_clean":"2024-03-22T02:13:29.230885+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:13:29.230885+0000","last_undegraded":"2024-03-22T02:13:29.230885+0000","last_fullsized":"2024-03-22T02:13:29.230885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229707+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:13:29.229707+0000","last_peered":"2024-03-22T02:13:29.229707+0000","last_clean":"2024-03-22T02:13:29.229707+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:13:29.229707+0000","last_undegraded":"2024-03-22T02:13:29.229707+0000","last_fullsized":"2024-03-22T02:13:29.229707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.239556+0000","last_change":"2024-03-22T02:13:29.239080+0000","last_active":"2024-03-22T02:13:29.239556+0000","last_peered":"2024-03-22T02:13:29.239556+0000","last_clean":"2024-03-22T02:13:29.239556+0000","last_became_active":"2024-03-22T02:13:29.234626+0000","last_became_peered":"2024-03-22T02:13:29.234626+0000","last_unstale":"2024-03-22T02:13:29.239556+0000","last_undegraded":"2024-03-22T02:13:29.239556+0000","last_fullsized":"2024-03-22T02:13:29.239556+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:21:19.317485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229714+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:13:29.229714+0000","last_peered":"2024-03-22T02:13:29.229714+0000","last_clean":"2024-03-22T02:13:29.229714+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:13:29.229714+0000","last_undegraded":"2024-03-22T02:13:29.229714+0000","last_fullsized":"2024-03-22T02:13:29.229714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229605+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:13:29.229605+0000","last_peered":"2024-03-22T02:13:29.229605+0000","last_clean":"2024-03-22T02:13:29.229605+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:13:29.229605+0000","last_undegraded":"2024-03-22T02:13:29.229605+0000","last_fullsized":"2024-03-22T02:13:29.229605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235113+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:13:29.235113+0000","last_peered":"2024-03-22T02:13:29.235113+0000","last_clean":"2024-03-22T02:13:29.235113+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:13:29.235113+0000","last_undegraded":"2024-03-22T02:13:29.235113+0000","last_fullsized":"2024-03-22T02:13:29.235113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233084+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:13:29.233084+0000","last_peered":"2024-03-22T02:13:29.233084+0000","last_clean":"2024-03-22T02:13:29.233084+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:13:29.233084+0000","last_undegraded":"2024-03-22T02:13:29.233084+0000","last_fullsized":"2024-03-22T02:13:29.233084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234895+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:13:29.234895+0000","last_peered":"2024-03-22T02:13:29.234895+0000","last_clean":"2024-03-22T02:13:29.234895+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:13:29.234895+0000","last_undegraded":"2024-03-22T02:13:29.234895+0000","last_fullsized":"2024-03-22T02:13:29.234895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231835+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:13:29.231835+0000","last_peered":"2024-03-22T02:13:29.231835+0000","last_clean":"2024-03-22T02:13:29.231835+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:13:29.231835+0000","last_undegraded":"2024-03-22T02:13:29.231835+0000","last_fullsized":"2024-03-22T02:13:29.231835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233399+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:13:29.233399+0000","last_peered":"2024-03-22T02:13:29.233399+0000","last_clean":"2024-03-22T02:13:29.233399+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:13:29.233399+0000","last_undegraded":"2024-03-22T02:13:29.233399+0000","last_fullsized":"2024-03-22T02:13:29.233399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234112+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:13:29.234112+0000","last_peered":"2024-03-22T02:13:29.234112+0000","last_clean":"2024-03-22T02:13:29.234112+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:13:29.234112+0000","last_undegraded":"2024-03-22T02:13:29.234112+0000","last_fullsized":"2024-03-22T02:13:29.234112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536753+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:13:29.536753+0000","last_peered":"2024-03-22T02:13:29.536753+0000","last_clean":"2024-03-22T02:13:29.536753+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:13:29.536753+0000","last_undegraded":"2024-03-22T02:13:29.536753+0000","last_fullsized":"2024-03-22T02:13:29.536753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232917+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:13:29.232917+0000","last_peered":"2024-03-22T02:13:29.232917+0000","last_clean":"2024-03-22T02:13:29.232917+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:13:29.232917+0000","last_undegraded":"2024-03-22T02:13:29.232917+0000","last_fullsized":"2024-03-22T02:13:29.232917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536144+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:13:29.536144+0000","last_peered":"2024-03-22T02:13:29.536144+0000","last_clean":"2024-03-22T02:13:29.536144+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:13:29.536144+0000","last_undegraded":"2024-03-22T02:13:29.536144+0000","last_fullsized":"2024-03-22T02:13:29.536144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248047+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:13:29.248047+0000","last_peered":"2024-03-22T02:13:29.248047+0000","last_clean":"2024-03-22T02:13:29.248047+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:13:29.248047+0000","last_undegraded":"2024-03-22T02:13:29.248047+0000","last_fullsized":"2024-03-22T02:13:29.248047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233098+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:13:29.233098+0000","last_peered":"2024-03-22T02:13:29.233098+0000","last_clean":"2024-03-22T02:13:29.233098+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:13:29.233098+0000","last_undegraded":"2024-03-22T02:13:29.233098+0000","last_fullsized":"2024-03-22T02:13:29.233098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231002+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:13:29.231002+0000","last_peered":"2024-03-22T02:13:29.231002+0000","last_clean":"2024-03-22T02:13:29.231002+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:13:29.231002+0000","last_undegraded":"2024-03-22T02:13:29.231002+0000","last_fullsized":"2024-03-22T02:13:29.231002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231480+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:13:29.231480+0000","last_peered":"2024-03-22T02:13:29.231480+0000","last_clean":"2024-03-22T02:13:29.231480+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:13:29.231480+0000","last_undegraded":"2024-03-22T02:13:29.231480+0000","last_fullsized":"2024-03-22T02:13:29.231480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231359+0000","last_change":"2024-03-22T02:12:28.489359+0000","last_active":"2024-03-22T02:13:29.231359+0000","last_peered":"2024-03-22T02:13:29.231359+0000","last_clean":"2024-03-22T02:13:29.231359+0000","last_became_active":"2024-03-22T02:12:28.489038+0000","last_became_peered":"2024-03-22T02:12:28.489038+0000","last_unstale":"2024-03-22T02:13:29.231359+0000","last_undegraded":"2024-03-22T02:13:29.231359+0000","last_fullsized":"2024-03-22T02:13:29.231359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:15.886990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234341+0000","last_change":"2024-03-22T02:12:28.489164+0000","last_active":"2024-03-22T02:13:29.234341+0000","last_peered":"2024-03-22T02:13:29.234341+0000","last_clean":"2024-03-22T02:13:29.234341+0000","last_became_active":"2024-03-22T02:12:28.488707+0000","last_became_peered":"2024-03-22T02:12:28.488707+0000","last_unstale":"2024-03-22T02:13:29.234341+0000","last_undegraded":"2024-03-22T02:13:29.234341+0000","last_fullsized":"2024-03-22T02:13:29.234341+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:40.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248083+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:13:29.248083+0000","last_peered":"2024-03-22T02:13:29.248083+0000","last_clean":"2024-03-22T02:13:29.248083+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:13:29.248083+0000","last_undegraded":"2024-03-22T02:13:29.248083+0000","last_fullsized":"2024-03-22T02:13:29.248083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232320+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:13:29.232320+0000","last_peered":"2024-03-22T02:13:29.232320+0000","last_clean":"2024-03-22T02:13:29.232320+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:13:29.232320+0000","last_undegraded":"2024-03-22T02:13:29.232320+0000","last_fullsized":"2024-03-22T02:13:29.232320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232088+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:13:29.232088+0000","last_peered":"2024-03-22T02:13:29.232088+0000","last_clean":"2024-03-22T02:13:29.232088+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:13:29.232088+0000","last_undegraded":"2024-03-22T02:13:29.232088+0000","last_fullsized":"2024-03-22T02:13:29.232088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230024+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:13:29.230024+0000","last_peered":"2024-03-22T02:13:29.230024+0000","last_clean":"2024-03-22T02:13:29.230024+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:13:29.230024+0000","last_undegraded":"2024-03-22T02:13:29.230024+0000","last_fullsized":"2024-03-22T02:13:29.230024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247487+0000","last_change":"2024-03-22T02:12:28.520917+0000","last_active":"2024-03-22T02:13:29.247487+0000","last_peered":"2024-03-22T02:13:29.247487+0000","last_clean":"2024-03-22T02:13:29.247487+0000","last_became_active":"2024-03-22T02:12:28.520533+0000","last_became_peered":"2024-03-22T02:12:28.520533+0000","last_unstale":"2024-03-22T02:13:29.247487+0000","last_undegraded":"2024-03-22T02:13:29.247487+0000","last_fullsized":"2024-03-22T02:13:29.247487+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:44.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229453+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:13:29.229453+0000","last_peered":"2024-03-22T02:13:29.229453+0000","last_clean":"2024-03-22T02:13:29.229453+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:13:29.229453+0000","last_undegraded":"2024-03-22T02:13:29.229453+0000","last_fullsized":"2024-03-22T02:13:29.229453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234939+0000","last_change":"2024-03-22T02:12:28.521533+0000","last_active":"2024-03-22T02:13:29.234939+0000","last_peered":"2024-03-22T02:13:29.234939+0000","last_clean":"2024-03-22T02:13:29.234939+0000","last_became_active":"2024-03-22T02:12:28.521176+0000","last_became_peered":"2024-03-22T02:12:28.521176+0000","last_unstale":"2024-03-22T02:13:29.234939+0000","last_undegraded":"2024-03-22T02:13:29.234939+0000","last_fullsized":"2024-03-22T02:13:29.234939+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:10:11.909377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248177+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:13:29.248177+0000","last_peered":"2024-03-22T02:13:29.248177+0000","last_clean":"2024-03-22T02:13:29.248177+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:13:29.248177+0000","last_undegraded":"2024-03-22T02:13:29.248177+0000","last_fullsized":"2024-03-22T02:13:29.248177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231467+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:13:29.231467+0000","last_peered":"2024-03-22T02:13:29.231467+0000","last_clean":"2024-03-22T02:13:29.231467+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:13:29.231467+0000","last_undegraded":"2024-03-22T02:13:29.231467+0000","last_fullsized":"2024-03-22T02:13:29.231467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74127,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":733,"num_read_kb":443,"num_write":175,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":222381,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":523008,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1179848,"data_compressed":64340,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527318,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523120,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":1119891,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518398,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":582221,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":519786,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":589867,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":518592,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":523747,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1114780,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69286,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71559,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":71172,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:13:55.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 -- 172.21.15.28:0/3466877738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102660 msgr2=0x7f1448108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 --2- 172.21.15.28:0/3466877738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102660 0x7f1448108ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1440009ca0 tx=0x7f144002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 -- 172.21.15.28:0/3466877738 shutdown_connections 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 --2- 172.21.15.28:0/3466877738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102660 0x7f1448108ef0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 --2- 172.21.15.28:0/3466877738 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448101ca0 0x7f1448102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 --2- 172.21.15.28:0/3466877738 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448100ab0 0x7f1448100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 -- 172.21.15.28:0/3466877738 >> 172.21.15.28:0/3466877738 conn(0x7f14480fc040 msgr2=0x7f14480fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 -- 172.21.15.28:0/3466877738 shutdown_connections 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.667+0000 7f144ec0e640 1 -- 172.21.15.28:0/3466877738 wait complete. 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 Processor -- start 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 -- start start 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101ca0 0x7f14481983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448102660 0x7f144819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c730 con 0x7f1448100ab0 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c5b0 con 0x7f1448102660 2024-03-22T02:13:55.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144ec0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c8b0 con 0x7f1448101ca0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60318/0 (socket says 172.21.15.28:60318) 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 -- 172.21.15.28:0/4220797568 learned_addr learned my addr 172.21.15.28:0/4220797568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f1447fff640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101ca0 0x7f14481983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144d1ab640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448102660 0x7f144819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 -- 172.21.15.28:0/4220797568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101ca0 msgr2=0x7f14481983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101ca0 0x7f14481983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 -- 172.21.15.28:0/4220797568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448102660 msgr2=0x7f144819f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448102660 0x7f144819f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 -- 172.21.15.28:0/4220797568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144819fa90 con 0x7f1448100ab0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.668+0000 7f144c9aa640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e9c0 tx=0x7f143c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:55.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.669+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c00cd80 con 0x7f1448100ab0 2024-03-22T02:13:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.669+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f143c0035f0 con 0x7f1448100ab0 2024-03-22T02:13:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.669+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144819fd80 con 0x7f1448100ab0 2024-03-22T02:13:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.669+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c010690 con 0x7f1448100ab0 2024-03-22T02:13:55.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.669+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14481a0210 con 0x7f1448100ab0 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.670+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14481a04a0 con 0x7f1448100ab0 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.671+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f143c0037a0 con 0x7f1448100ab0 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.671+0000 7f1445ffb640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 0x7f14380a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.671+0000 7f1447fff640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 0x7f14380a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.672+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f143c0cfce0 con 0x7f1448100ab0 2024-03-22T02:13:55.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.672+0000 7f1447fff640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 0x7f14380a54f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1448199380 tx=0x7f1434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:55.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.676+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f143c09c540 con 0x7f1448100ab0 2024-03-22T02:13:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:55 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3528884880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:55 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1982261162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:55 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2101802693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:55.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:55 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3528884880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:55 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1982261162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:55.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:55 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2101802693' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:55.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.856+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14481a0670 con 0x7f14380a3030 2024-03-22T02:13:55.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.865+0000 7f1445ffb640 1 -- 172.21.15.28:0/4220797568 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340801 (secure 0 0 0) 0x7f14481a0670 con 0x7f14380a3030 2024-03-22T02:13:55.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:55.871 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:13:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 msgr2=0x7f14380a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 0x7f14380a54f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1448199380 tx=0x7f1434005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:55.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 msgr2=0x7f1448197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f143c00e9c0 tx=0x7f143c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 shutdown_connections 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14380a3030 0x7f14380a54f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448102660 0x7f144819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101ca0 0x7f14481983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 --2- 172.21.15.28:0/4220797568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448100ab0 0x7f1448197e70 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 >> 172.21.15.28:0/4220797568 conn(0x7f14480fc040 msgr2=0x7f144806a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 shutdown_connections 2024-03-22T02:13:55.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:55.869+0000 7f144ec0e640 1 -- 172.21.15.28:0/4220797568 wait complete. 2024-03-22T02:13:55.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-22T02:13:54.587702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664729,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":859,"num_read_kb":552,"num_write":358,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365888,"kb_used_data":12544,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575936,"statfs":{"total":1151940427776,"available":1151565758464,"internally_reserved":0,"allocated":12845056,"data_stored":8240576,"data_compressed":432609,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6380,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232507+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:13:29.232507+0000","last_peered":"2024-03-22T02:13:29.232507+0000","last_clean":"2024-03-22T02:13:29.232507+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:13:29.232507+0000","last_undegraded":"2024-03-22T02:13:29.232507+0000","last_fullsized":"2024-03-22T02:13:29.232507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234634+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:13:29.234634+0000","last_peered":"2024-03-22T02:13:29.234634+0000","last_clean":"2024-03-22T02:13:29.234634+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:13:29.234634+0000","last_undegraded":"2024-03-22T02:13:29.234634+0000","last_fullsized":"2024-03-22T02:13:29.234634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234266+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:13:29.234266+0000","last_peered":"2024-03-22T02:13:29.234266+0000","last_clean":"2024-03-22T02:13:29.234266+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:13:29.234266+0000","last_undegraded":"2024-03-22T02:13:29.234266+0000","last_fullsized":"2024-03-22T02:13:29.234266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229827+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:13:29.229827+0000","last_peered":"2024-03-22T02:13:29.229827+0000","last_clean":"2024-03-22T02:13:29.229827+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:13:29.229827+0000","last_undegraded":"2024-03-22T02:13:29.229827+0000","last_fullsized":"2024-03-22T02:13:29.229827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235150+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:13:29.235150+0000","last_peered":"2024-03-22T02:13:29.235150+0000","last_clean":"2024-03-22T02:13:29.235150+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:13:29.235150+0000","last_undegraded":"2024-03-22T02:13:29.235150+0000","last_fullsized":"2024-03-22T02:13:29.235150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666425+0000","last_change":"2024-03-22T02:12:28.521805+0000","last_active":"2024-03-22T02:13:29.666425+0000","last_peered":"2024-03-22T02:13:29.666425+0000","last_clean":"2024-03-22T02:13:29.666425+0000","last_became_active":"2024-03-22T02:12:28.521419+0000","last_became_peered":"2024-03-22T02:12:28.521419+0000","last_unstale":"2024-03-22T02:13:29.666425+0000","last_undegraded":"2024-03-22T02:13:29.666425+0000","last_fullsized":"2024-03-22T02:13:29.666425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:01:11.831486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232631+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:13:29.232631+0000","last_peered":"2024-03-22T02:13:29.232631+0000","last_clean":"2024-03-22T02:13:29.232631+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:13:29.232631+0000","last_undegraded":"2024-03-22T02:13:29.232631+0000","last_fullsized":"2024-03-22T02:13:29.232631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229177+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:13:29.229177+0000","last_peered":"2024-03-22T02:13:29.229177+0000","last_clean":"2024-03-22T02:13:29.229177+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:13:29.229177+0000","last_undegraded":"2024-03-22T02:13:29.229177+0000","last_fullsized":"2024-03-22T02:13:29.229177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235354+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:13:29.235354+0000","last_peered":"2024-03-22T02:13:29.235354+0000","last_clean":"2024-03-22T02:13:29.235354+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:13:29.235354+0000","last_undegraded":"2024-03-22T02:13:29.235354+0000","last_fullsized":"2024-03-22T02:13:29.235354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666800+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:13:29.666800+0000","last_peered":"2024-03-22T02:13:29.666800+0000","last_clean":"2024-03-22T02:13:29.666800+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:13:29.666800+0000","last_undegraded":"2024-03-22T02:13:29.666800+0000","last_fullsized":"2024-03-22T02:13:29.666800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233246+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:13:29.233246+0000","last_peered":"2024-03-22T02:13:29.233246+0000","last_clean":"2024-03-22T02:13:29.233246+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:13:29.233246+0000","last_undegraded":"2024-03-22T02:13:29.233246+0000","last_fullsized":"2024-03-22T02:13:29.233246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229832+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:13:29.229832+0000","last_peered":"2024-03-22T02:13:29.229832+0000","last_clean":"2024-03-22T02:13:29.229832+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:13:29.229832+0000","last_undegraded":"2024-03-22T02:13:29.229832+0000","last_fullsized":"2024-03-22T02:13:29.229832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247995+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:13:29.247995+0000","last_peered":"2024-03-22T02:13:29.247995+0000","last_clean":"2024-03-22T02:13:29.247995+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:13:29.247995+0000","last_undegraded":"2024-03-22T02:13:29.247995+0000","last_fullsized":"2024-03-22T02:13:29.247995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234497+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:13:29.234497+0000","last_peered":"2024-03-22T02:13:29.234497+0000","last_clean":"2024-03-22T02:13:29.234497+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:13:29.234497+0000","last_undegraded":"2024-03-22T02:13:29.234497+0000","last_fullsized":"2024-03-22T02:13:29.234497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234649+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:13:29.234649+0000","last_peered":"2024-03-22T02:13:29.234649+0000","last_clean":"2024-03-22T02:13:29.234649+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:13:29.234649+0000","last_undegraded":"2024-03-22T02:13:29.234649+0000","last_fullsized":"2024-03-22T02:13:29.234649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234959+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:13:29.234959+0000","last_peered":"2024-03-22T02:13:29.234959+0000","last_clean":"2024-03-22T02:13:29.234959+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:13:29.234959+0000","last_undegraded":"2024-03-22T02:13:29.234959+0000","last_fullsized":"2024-03-22T02:13:29.234959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231082+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:13:29.231082+0000","last_peered":"2024-03-22T02:13:29.231082+0000","last_clean":"2024-03-22T02:13:29.231082+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:13:29.231082+0000","last_undegraded":"2024-03-22T02:13:29.231082+0000","last_fullsized":"2024-03-22T02:13:29.231082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235450+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:13:29.235450+0000","last_peered":"2024-03-22T02:13:29.235450+0000","last_clean":"2024-03-22T02:13:29.235450+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:13:29.235450+0000","last_undegraded":"2024-03-22T02:13:29.235450+0000","last_fullsized":"2024-03-22T02:13:29.235450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232547+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:13:29.232547+0000","last_peered":"2024-03-22T02:13:29.232547+0000","last_clean":"2024-03-22T02:13:29.232547+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:13:29.232547+0000","last_undegraded":"2024-03-22T02:13:29.232547+0000","last_fullsized":"2024-03-22T02:13:29.232547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.236492+0000","last_change":"2024-03-22T02:13:29.236492+0000","last_active":"2024-03-22T02:13:29.236492+0000","last_peered":"2024-03-22T02:13:29.236492+0000","last_clean":"2024-03-22T02:13:29.236492+0000","last_became_active":"2024-03-22T02:13:29.236092+0000","last_became_peered":"2024-03-22T02:13:29.236092+0000","last_unstale":"2024-03-22T02:13:29.236492+0000","last_undegraded":"2024-03-22T02:13:29.236492+0000","last_fullsized":"2024-03-22T02:13:29.236492+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:42.511417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235774+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:13:29.235774+0000","last_peered":"2024-03-22T02:13:29.235774+0000","last_clean":"2024-03-22T02:13:29.235774+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:13:29.235774+0000","last_undegraded":"2024-03-22T02:13:29.235774+0000","last_fullsized":"2024-03-22T02:13:29.235774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229678+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:13:29.229678+0000","last_peered":"2024-03-22T02:13:29.229678+0000","last_clean":"2024-03-22T02:13:29.229678+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:13:29.229678+0000","last_undegraded":"2024-03-22T02:13:29.229678+0000","last_fullsized":"2024-03-22T02:13:29.229678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229358+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:13:29.229358+0000","last_peered":"2024-03-22T02:13:29.229358+0000","last_clean":"2024-03-22T02:13:29.229358+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:13:29.229358+0000","last_undegraded":"2024-03-22T02:13:29.229358+0000","last_fullsized":"2024-03-22T02:13:29.229358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229936+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:13:29.229936+0000","last_peered":"2024-03-22T02:13:29.229936+0000","last_clean":"2024-03-22T02:13:29.229936+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:13:29.229936+0000","last_undegraded":"2024-03-22T02:13:29.229936+0000","last_fullsized":"2024-03-22T02:13:29.229936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234882+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:13:29.234882+0000","last_peered":"2024-03-22T02:13:29.234882+0000","last_clean":"2024-03-22T02:13:29.234882+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:13:29.234882+0000","last_undegraded":"2024-03-22T02:13:29.234882+0000","last_fullsized":"2024-03-22T02:13:29.234882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229464+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:13:29.229464+0000","last_peered":"2024-03-22T02:13:29.229464+0000","last_clean":"2024-03-22T02:13:29.229464+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:13:29.229464+0000","last_undegraded":"2024-03-22T02:13:29.229464+0000","last_fullsized":"2024-03-22T02:13:29.229464+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242108+0000","last_change":"2024-03-22T02:13:29.242108+0000","last_active":"2024-03-22T02:13:29.242108+0000","last_peered":"2024-03-22T02:13:29.242108+0000","last_clean":"2024-03-22T02:13:29.242108+0000","last_became_active":"2024-03-22T02:13:29.236686+0000","last_became_peered":"2024-03-22T02:13:29.236686+0000","last_unstale":"2024-03-22T02:13:29.242108+0000","last_undegraded":"2024-03-22T02:13:29.242108+0000","last_fullsized":"2024-03-22T02:13:29.242108+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:42:01.218922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235315+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:13:29.235315+0000","last_peered":"2024-03-22T02:13:29.235315+0000","last_clean":"2024-03-22T02:13:29.235315+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:13:29.235315+0000","last_undegraded":"2024-03-22T02:13:29.235315+0000","last_fullsized":"2024-03-22T02:13:29.235315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229375+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:13:29.229375+0000","last_peered":"2024-03-22T02:13:29.229375+0000","last_clean":"2024-03-22T02:13:29.229375+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:13:29.229375+0000","last_undegraded":"2024-03-22T02:13:29.229375+0000","last_fullsized":"2024-03-22T02:13:29.229375+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235311+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:13:29.235311+0000","last_peered":"2024-03-22T02:13:29.235311+0000","last_clean":"2024-03-22T02:13:29.235311+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:13:29.235311+0000","last_undegraded":"2024-03-22T02:13:29.235311+0000","last_fullsized":"2024-03-22T02:13:29.235311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536646+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:13:29.536646+0000","last_peered":"2024-03-22T02:13:29.536646+0000","last_clean":"2024-03-22T02:13:29.536646+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:13:29.536646+0000","last_undegraded":"2024-03-22T02:13:29.536646+0000","last_fullsized":"2024-03-22T02:13:29.536646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231729+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:13:29.231729+0000","last_peered":"2024-03-22T02:13:29.231729+0000","last_clean":"2024-03-22T02:13:29.231729+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:13:29.231729+0000","last_undegraded":"2024-03-22T02:13:29.231729+0000","last_fullsized":"2024-03-22T02:13:29.231729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:51.111514+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:13:51.111514+0000","last_peered":"2024-03-22T02:13:51.111514+0000","last_clean":"2024-03-22T02:13:51.111514+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:13:51.111514+0000","last_undegraded":"2024-03-22T02:13:51.111514+0000","last_fullsized":"2024-03-22T02:13:51.111514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234972+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:13:29.234972+0000","last_peered":"2024-03-22T02:13:29.234972+0000","last_clean":"2024-03-22T02:13:29.234972+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:13:29.234972+0000","last_undegraded":"2024-03-22T02:13:29.234972+0000","last_fullsized":"2024-03-22T02:13:29.234972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667818+0000","last_change":"2024-03-22T02:13:29.667818+0000","last_active":"2024-03-22T02:13:29.667818+0000","last_peered":"2024-03-22T02:13:29.667818+0000","last_clean":"2024-03-22T02:13:29.667818+0000","last_became_active":"2024-03-22T02:13:29.667344+0000","last_became_peered":"2024-03-22T02:13:29.667344+0000","last_unstale":"2024-03-22T02:13:29.667818+0000","last_undegraded":"2024-03-22T02:13:29.667818+0000","last_fullsized":"2024-03-22T02:13:29.667818+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:08.802171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666359+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:13:29.666359+0000","last_peered":"2024-03-22T02:13:29.666359+0000","last_clean":"2024-03-22T02:13:29.666359+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:13:29.666359+0000","last_undegraded":"2024-03-22T02:13:29.666359+0000","last_fullsized":"2024-03-22T02:13:29.666359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536554+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:13:29.536554+0000","last_peered":"2024-03-22T02:13:29.536554+0000","last_clean":"2024-03-22T02:13:29.536554+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:13:29.536554+0000","last_undegraded":"2024-03-22T02:13:29.536554+0000","last_fullsized":"2024-03-22T02:13:29.536554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231009+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:13:29.231009+0000","last_peered":"2024-03-22T02:13:29.231009+0000","last_clean":"2024-03-22T02:13:29.231009+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:13:29.231009+0000","last_undegraded":"2024-03-22T02:13:29.231009+0000","last_fullsized":"2024-03-22T02:13:29.231009+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247642+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:13:29.247642+0000","last_peered":"2024-03-22T02:13:29.247642+0000","last_clean":"2024-03-22T02:13:29.247642+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:13:29.247642+0000","last_undegraded":"2024-03-22T02:13:29.247642+0000","last_fullsized":"2024-03-22T02:13:29.247642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229323+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:13:29.229323+0000","last_peered":"2024-03-22T02:13:29.229323+0000","last_clean":"2024-03-22T02:13:29.229323+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:13:29.229323+0000","last_undegraded":"2024-03-22T02:13:29.229323+0000","last_fullsized":"2024-03-22T02:13:29.229323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234585+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:13:29.234585+0000","last_peered":"2024-03-22T02:13:29.234585+0000","last_clean":"2024-03-22T02:13:29.234585+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:13:29.234585+0000","last_undegraded":"2024-03-22T02:13:29.234585+0000","last_fullsized":"2024-03-22T02:13:29.234585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536228+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:13:29.536228+0000","last_peered":"2024-03-22T02:13:29.536228+0000","last_clean":"2024-03-22T02:13:29.536228+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:13:29.536228+0000","last_undegraded":"2024-03-22T02:13:29.536228+0000","last_fullsized":"2024-03-22T02:13:29.536228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234870+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:13:29.234870+0000","last_peered":"2024-03-22T02:13:29.234870+0000","last_clean":"2024-03-22T02:13:29.234870+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:13:29.234870+0000","last_undegraded":"2024-03-22T02:13:29.234870+0000","last_fullsized":"2024-03-22T02:13:29.234870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229276+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.229276+0000","last_peered":"2024-03-22T02:13:29.229276+0000","last_clean":"2024-03-22T02:13:29.229276+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:13:29.229276+0000","last_undegraded":"2024-03-22T02:13:29.229276+0000","last_fullsized":"2024-03-22T02:13:29.229276+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:49.108917+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:13:49.108917+0000","last_peered":"2024-03-22T02:13:49.108917+0000","last_clean":"2024-03-22T02:13:49.108917+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:13:49.108917+0000","last_undegraded":"2024-03-22T02:13:49.108917+0000","last_fullsized":"2024-03-22T02:13:49.108917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235071+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:13:29.235071+0000","last_peered":"2024-03-22T02:13:29.235071+0000","last_clean":"2024-03-22T02:13:29.235071+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:13:29.235071+0000","last_undegraded":"2024-03-22T02:13:29.235071+0000","last_fullsized":"2024-03-22T02:13:29.235071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234467+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:13:29.234467+0000","last_peered":"2024-03-22T02:13:29.234467+0000","last_clean":"2024-03-22T02:13:29.234467+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:13:29.234467+0000","last_undegraded":"2024-03-22T02:13:29.234467+0000","last_fullsized":"2024-03-22T02:13:29.234467+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.238453+0000","last_change":"2024-03-22T02:13:29.238453+0000","last_active":"2024-03-22T02:13:29.238453+0000","last_peered":"2024-03-22T02:13:29.238453+0000","last_clean":"2024-03-22T02:13:29.238453+0000","last_became_active":"2024-03-22T02:13:29.235861+0000","last_became_peered":"2024-03-22T02:13:29.235861+0000","last_unstale":"2024-03-22T02:13:29.238453+0000","last_undegraded":"2024-03-22T02:13:29.238453+0000","last_fullsized":"2024-03-22T02:13:29.238453+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:50:35.945707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234160+0000","last_change":"2024-03-22T02:12:28.521865+0000","last_active":"2024-03-22T02:13:29.234160+0000","last_peered":"2024-03-22T02:13:29.234160+0000","last_clean":"2024-03-22T02:13:29.234160+0000","last_became_active":"2024-03-22T02:12:28.521413+0000","last_became_peered":"2024-03-22T02:12:28.521413+0000","last_unstale":"2024-03-22T02:13:29.234160+0000","last_undegraded":"2024-03-22T02:13:29.234160+0000","last_fullsized":"2024-03-22T02:13:29.234160+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:54:26.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229759+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:13:29.229759+0000","last_peered":"2024-03-22T02:13:29.229759+0000","last_clean":"2024-03-22T02:13:29.229759+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:13:29.229759+0000","last_undegraded":"2024-03-22T02:13:29.229759+0000","last_fullsized":"2024-03-22T02:13:29.229759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536583+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:13:29.536583+0000","last_peered":"2024-03-22T02:13:29.536583+0000","last_clean":"2024-03-22T02:13:29.536583+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:13:29.536583+0000","last_undegraded":"2024-03-22T02:13:29.536583+0000","last_fullsized":"2024-03-22T02:13:29.536583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235431+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:13:29.235431+0000","last_peered":"2024-03-22T02:13:29.235431+0000","last_clean":"2024-03-22T02:13:29.235431+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:13:29.235431+0000","last_undegraded":"2024-03-22T02:13:29.235431+0000","last_fullsized":"2024-03-22T02:13:29.235431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235070+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.235070+0000","last_peered":"2024-03-22T02:13:29.235070+0000","last_clean":"2024-03-22T02:13:29.235070+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:13:29.235070+0000","last_undegraded":"2024-03-22T02:13:29.235070+0000","last_fullsized":"2024-03-22T02:13:29.235070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":311,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.109991+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:13:50.109991+0000","last_peered":"2024-03-22T02:13:50.109991+0000","last_clean":"2024-03-22T02:13:50.109991+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:13:50.109991+0000","last_undegraded":"2024-03-22T02:13:50.109991+0000","last_fullsized":"2024-03-22T02:13:50.109991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":242,"num_read_kb":189,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235127+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:13:29.235127+0000","last_peered":"2024-03-22T02:13:29.235127+0000","last_clean":"2024-03-22T02:13:29.235127+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:13:29.235127+0000","last_undegraded":"2024-03-22T02:13:29.235127+0000","last_fullsized":"2024-03-22T02:13:29.235127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536841+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:13:29.536841+0000","last_peered":"2024-03-22T02:13:29.536841+0000","last_clean":"2024-03-22T02:13:29.536841+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:13:29.536841+0000","last_undegraded":"2024-03-22T02:13:29.536841+0000","last_fullsized":"2024-03-22T02:13:29.536841+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231083+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:13:29.231083+0000","last_peered":"2024-03-22T02:13:29.231083+0000","last_clean":"2024-03-22T02:13:29.231083+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:13:29.231083+0000","last_undegraded":"2024-03-22T02:13:29.231083+0000","last_fullsized":"2024-03-22T02:13:29.231083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247705+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:13:29.247705+0000","last_peered":"2024-03-22T02:13:29.247705+0000","last_clean":"2024-03-22T02:13:29.247705+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:13:29.247705+0000","last_undegraded":"2024-03-22T02:13:29.247705+0000","last_fullsized":"2024-03-22T02:13:29.247705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231568+0000","last_change":"2024-03-22T02:12:28.520817+0000","last_active":"2024-03-22T02:13:29.231568+0000","last_peered":"2024-03-22T02:13:29.231568+0000","last_clean":"2024-03-22T02:13:29.231568+0000","last_became_active":"2024-03-22T02:12:28.520360+0000","last_became_peered":"2024-03-22T02:12:28.520360+0000","last_unstale":"2024-03-22T02:13:29.231568+0000","last_undegraded":"2024-03-22T02:13:29.231568+0000","last_fullsized":"2024-03-22T02:13:29.231568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:34:21.654588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536568+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:13:29.536568+0000","last_peered":"2024-03-22T02:13:29.536568+0000","last_clean":"2024-03-22T02:13:29.536568+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:13:29.536568+0000","last_undegraded":"2024-03-22T02:13:29.536568+0000","last_fullsized":"2024-03-22T02:13:29.536568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233232+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:13:29.233232+0000","last_peered":"2024-03-22T02:13:29.233232+0000","last_clean":"2024-03-22T02:13:29.233232+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:13:29.233232+0000","last_undegraded":"2024-03-22T02:13:29.233232+0000","last_fullsized":"2024-03-22T02:13:29.233232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235648+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:13:29.235648+0000","last_peered":"2024-03-22T02:13:29.235648+0000","last_clean":"2024-03-22T02:13:29.235648+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:13:29.235648+0000","last_undegraded":"2024-03-22T02:13:29.235648+0000","last_fullsized":"2024-03-22T02:13:29.235648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'5","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:53.143913+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:13:53.143913+0000","last_peered":"2024-03-22T02:13:53.143913+0000","last_clean":"2024-03-22T02:13:53.143913+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:13:53.143913+0000","last_undegraded":"2024-03-22T02:13:53.143913+0000","last_fullsized":"2024-03-22T02:13:53.143913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232769+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:13:29.232769+0000","last_peered":"2024-03-22T02:13:29.232769+0000","last_clean":"2024-03-22T02:13:29.232769+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:13:29.232769+0000","last_undegraded":"2024-03-22T02:13:29.232769+0000","last_fullsized":"2024-03-22T02:13:29.232769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248019+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:13:29.248019+0000","last_peered":"2024-03-22T02:13:29.248019+0000","last_clean":"2024-03-22T02:13:29.248019+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:13:29.248019+0000","last_undegraded":"2024-03-22T02:13:29.248019+0000","last_fullsized":"2024-03-22T02:13:29.248019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234158+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:13:29.234158+0000","last_peered":"2024-03-22T02:13:29.234158+0000","last_clean":"2024-03-22T02:13:29.234158+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:13:29.234158+0000","last_undegraded":"2024-03-22T02:13:29.234158+0000","last_fullsized":"2024-03-22T02:13:29.234158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536334+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:13:29.536334+0000","last_peered":"2024-03-22T02:13:29.536334+0000","last_clean":"2024-03-22T02:13:29.536334+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:13:29.536334+0000","last_undegraded":"2024-03-22T02:13:29.536334+0000","last_fullsized":"2024-03-22T02:13:29.536334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536312+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:13:29.536312+0000","last_peered":"2024-03-22T02:13:29.536312+0000","last_clean":"2024-03-22T02:13:29.536312+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:13:29.536312+0000","last_undegraded":"2024-03-22T02:13:29.536312+0000","last_fullsized":"2024-03-22T02:13:29.536312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235355+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:13:29.235355+0000","last_peered":"2024-03-22T02:13:29.235355+0000","last_clean":"2024-03-22T02:13:29.235355+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:13:29.235355+0000","last_undegraded":"2024-03-22T02:13:29.235355+0000","last_fullsized":"2024-03-22T02:13:29.235355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231288+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:13:29.231288+0000","last_peered":"2024-03-22T02:13:29.231288+0000","last_clean":"2024-03-22T02:13:29.231288+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:13:29.231288+0000","last_undegraded":"2024-03-22T02:13:29.231288+0000","last_fullsized":"2024-03-22T02:13:29.231288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234912+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:13:29.234912+0000","last_peered":"2024-03-22T02:13:29.234912+0000","last_clean":"2024-03-22T02:13:29.234912+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:13:29.234912+0000","last_undegraded":"2024-03-22T02:13:29.234912+0000","last_fullsized":"2024-03-22T02:13:29.234912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235141+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:13:29.235141+0000","last_peered":"2024-03-22T02:13:29.235141+0000","last_clean":"2024-03-22T02:13:29.235141+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:13:29.235141+0000","last_undegraded":"2024-03-22T02:13:29.235141+0000","last_fullsized":"2024-03-22T02:13:29.235141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666733+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:13:29.666733+0000","last_peered":"2024-03-22T02:13:29.666733+0000","last_clean":"2024-03-22T02:13:29.666733+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:13:29.666733+0000","last_undegraded":"2024-03-22T02:13:29.666733+0000","last_fullsized":"2024-03-22T02:13:29.666733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248014+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:13:29.248014+0000","last_peered":"2024-03-22T02:13:29.248014+0000","last_clean":"2024-03-22T02:13:29.248014+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:13:29.248014+0000","last_undegraded":"2024-03-22T02:13:29.248014+0000","last_fullsized":"2024-03-22T02:13:29.248014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232955+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:13:29.232955+0000","last_peered":"2024-03-22T02:13:29.232955+0000","last_clean":"2024-03-22T02:13:29.232955+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:13:29.232955+0000","last_undegraded":"2024-03-22T02:13:29.232955+0000","last_fullsized":"2024-03-22T02:13:29.232955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666435+0000","last_change":"2024-03-22T02:12:28.488917+0000","last_active":"2024-03-22T02:13:29.666435+0000","last_peered":"2024-03-22T02:13:29.666435+0000","last_clean":"2024-03-22T02:13:29.666435+0000","last_became_active":"2024-03-22T02:12:28.488542+0000","last_became_peered":"2024-03-22T02:12:28.488542+0000","last_unstale":"2024-03-22T02:13:29.666435+0000","last_undegraded":"2024-03-22T02:13:29.666435+0000","last_fullsized":"2024-03-22T02:13:29.666435+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:41.575475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248137+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:13:29.248137+0000","last_peered":"2024-03-22T02:13:29.248137+0000","last_clean":"2024-03-22T02:13:29.248137+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:13:29.248137+0000","last_undegraded":"2024-03-22T02:13:29.248137+0000","last_fullsized":"2024-03-22T02:13:29.248137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536336+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:13:29.536336+0000","last_peered":"2024-03-22T02:13:29.536336+0000","last_clean":"2024-03-22T02:13:29.536336+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:13:29.536336+0000","last_undegraded":"2024-03-22T02:13:29.536336+0000","last_fullsized":"2024-03-22T02:13:29.536336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.138873+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:13:50.138873+0000","last_peered":"2024-03-22T02:13:50.138873+0000","last_clean":"2024-03-22T02:13:50.138873+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:13:50.138873+0000","last_undegraded":"2024-03-22T02:13:50.138873+0000","last_fullsized":"2024-03-22T02:13:50.138873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232977+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:13:29.232977+0000","last_peered":"2024-03-22T02:13:29.232977+0000","last_clean":"2024-03-22T02:13:29.232977+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:13:29.232977+0000","last_undegraded":"2024-03-22T02:13:29.232977+0000","last_fullsized":"2024-03-22T02:13:29.232977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229511+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:13:29.229511+0000","last_peered":"2024-03-22T02:13:29.229511+0000","last_clean":"2024-03-22T02:13:29.229511+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:13:29.229511+0000","last_undegraded":"2024-03-22T02:13:29.229511+0000","last_fullsized":"2024-03-22T02:13:29.229511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229686+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:13:29.229686+0000","last_peered":"2024-03-22T02:13:29.229686+0000","last_clean":"2024-03-22T02:13:29.229686+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:13:29.229686+0000","last_undegraded":"2024-03-22T02:13:29.229686+0000","last_fullsized":"2024-03-22T02:13:29.229686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232583+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:13:29.232583+0000","last_peered":"2024-03-22T02:13:29.232583+0000","last_clean":"2024-03-22T02:13:29.232583+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:13:29.232583+0000","last_undegraded":"2024-03-22T02:13:29.232583+0000","last_fullsized":"2024-03-22T02:13:29.232583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667520+0000","last_change":"2024-03-22T02:13:29.667520+0000","last_active":"2024-03-22T02:13:29.667520+0000","last_peered":"2024-03-22T02:13:29.667520+0000","last_clean":"2024-03-22T02:13:29.667520+0000","last_became_active":"2024-03-22T02:13:29.666948+0000","last_became_peered":"2024-03-22T02:13:29.666948+0000","last_unstale":"2024-03-22T02:13:29.667520+0000","last_undegraded":"2024-03-22T02:13:29.667520+0000","last_fullsized":"2024-03-22T02:13:29.667520+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:00:56.014210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247578+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:13:29.247578+0000","last_peered":"2024-03-22T02:13:29.247578+0000","last_clean":"2024-03-22T02:13:29.247578+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:13:29.247578+0000","last_undegraded":"2024-03-22T02:13:29.247578+0000","last_fullsized":"2024-03-22T02:13:29.247578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232655+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:13:29.232655+0000","last_peered":"2024-03-22T02:13:29.232655+0000","last_clean":"2024-03-22T02:13:29.232655+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:13:29.232655+0000","last_undegraded":"2024-03-22T02:13:29.232655+0000","last_fullsized":"2024-03-22T02:13:29.232655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231152+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:13:29.231152+0000","last_peered":"2024-03-22T02:13:29.231152+0000","last_clean":"2024-03-22T02:13:29.231152+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:13:29.231152+0000","last_undegraded":"2024-03-22T02:13:29.231152+0000","last_fullsized":"2024-03-22T02:13:29.231152+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242062+0000","last_change":"2024-03-22T02:13:29.241667+0000","last_active":"2024-03-22T02:13:29.242062+0000","last_peered":"2024-03-22T02:13:29.242062+0000","last_clean":"2024-03-22T02:13:29.242062+0000","last_became_active":"2024-03-22T02:13:29.236378+0000","last_became_peered":"2024-03-22T02:13:29.236378+0000","last_unstale":"2024-03-22T02:13:29.242062+0000","last_undegraded":"2024-03-22T02:13:29.242062+0000","last_fullsized":"2024-03-22T02:13:29.242062+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:41:25.050034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248131+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:13:29.248131+0000","last_peered":"2024-03-22T02:13:29.248131+0000","last_clean":"2024-03-22T02:13:29.248131+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:13:29.248131+0000","last_undegraded":"2024-03-22T02:13:29.248131+0000","last_fullsized":"2024-03-22T02:13:29.248131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229304+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:13:29.229304+0000","last_peered":"2024-03-22T02:13:29.229304+0000","last_clean":"2024-03-22T02:13:29.229304+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:13:29.229304+0000","last_undegraded":"2024-03-22T02:13:29.229304+0000","last_fullsized":"2024-03-22T02:13:29.229304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:44.917232+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:13:44.917232+0000","last_peered":"2024-03-22T02:13:44.917232+0000","last_clean":"2024-03-22T02:13:44.917232+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:13:44.917232+0000","last_undegraded":"2024-03-22T02:13:44.917232+0000","last_fullsized":"2024-03-22T02:13:44.917232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229947+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:13:29.229947+0000","last_peered":"2024-03-22T02:13:29.229947+0000","last_clean":"2024-03-22T02:13:29.229947+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:13:29.229947+0000","last_undegraded":"2024-03-22T02:13:29.229947+0000","last_fullsized":"2024-03-22T02:13:29.229947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231155+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:13:29.231155+0000","last_peered":"2024-03-22T02:13:29.231155+0000","last_clean":"2024-03-22T02:13:29.231155+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:13:29.231155+0000","last_undegraded":"2024-03-22T02:13:29.231155+0000","last_fullsized":"2024-03-22T02:13:29.231155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229873+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:13:29.229873+0000","last_peered":"2024-03-22T02:13:29.229873+0000","last_clean":"2024-03-22T02:13:29.229873+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:13:29.229873+0000","last_undegraded":"2024-03-22T02:13:29.229873+0000","last_fullsized":"2024-03-22T02:13:29.229873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234699+0000","last_change":"2024-03-22T02:12:28.489608+0000","last_active":"2024-03-22T02:13:29.234699+0000","last_peered":"2024-03-22T02:13:29.234699+0000","last_clean":"2024-03-22T02:13:29.234699+0000","last_became_active":"2024-03-22T02:12:28.489005+0000","last_became_peered":"2024-03-22T02:12:28.489005+0000","last_unstale":"2024-03-22T02:13:29.234699+0000","last_undegraded":"2024-03-22T02:13:29.234699+0000","last_fullsized":"2024-03-22T02:13:29.234699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:01.427654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229642+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:13:29.229642+0000","last_peered":"2024-03-22T02:13:29.229642+0000","last_clean":"2024-03-22T02:13:29.229642+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:13:29.229642+0000","last_undegraded":"2024-03-22T02:13:29.229642+0000","last_fullsized":"2024-03-22T02:13:29.229642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234767+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:13:29.234767+0000","last_peered":"2024-03-22T02:13:29.234767+0000","last_clean":"2024-03-22T02:13:29.234767+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:13:29.234767+0000","last_undegraded":"2024-03-22T02:13:29.234767+0000","last_fullsized":"2024-03-22T02:13:29.234767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234695+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:13:29.234695+0000","last_peered":"2024-03-22T02:13:29.234695+0000","last_clean":"2024-03-22T02:13:29.234695+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:13:29.234695+0000","last_undegraded":"2024-03-22T02:13:29.234695+0000","last_fullsized":"2024-03-22T02:13:29.234695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235725+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:13:29.235725+0000","last_peered":"2024-03-22T02:13:29.235725+0000","last_clean":"2024-03-22T02:13:29.235725+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:13:29.235725+0000","last_undegraded":"2024-03-22T02:13:29.235725+0000","last_fullsized":"2024-03-22T02:13:29.235725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229456+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:13:29.229456+0000","last_peered":"2024-03-22T02:13:29.229456+0000","last_clean":"2024-03-22T02:13:29.229456+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:13:29.229456+0000","last_undegraded":"2024-03-22T02:13:29.229456+0000","last_fullsized":"2024-03-22T02:13:29.229456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230885+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:13:29.230885+0000","last_peered":"2024-03-22T02:13:29.230885+0000","last_clean":"2024-03-22T02:13:29.230885+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:13:29.230885+0000","last_undegraded":"2024-03-22T02:13:29.230885+0000","last_fullsized":"2024-03-22T02:13:29.230885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229707+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:13:29.229707+0000","last_peered":"2024-03-22T02:13:29.229707+0000","last_clean":"2024-03-22T02:13:29.229707+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:13:29.229707+0000","last_undegraded":"2024-03-22T02:13:29.229707+0000","last_fullsized":"2024-03-22T02:13:29.229707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.239556+0000","last_change":"2024-03-22T02:13:29.239080+0000","last_active":"2024-03-22T02:13:29.239556+0000","last_peered":"2024-03-22T02:13:29.239556+0000","last_clean":"2024-03-22T02:13:29.239556+0000","last_became_active":"2024-03-22T02:13:29.234626+0000","last_became_peered":"2024-03-22T02:13:29.234626+0000","last_unstale":"2024-03-22T02:13:29.239556+0000","last_undegraded":"2024-03-22T02:13:29.239556+0000","last_fullsized":"2024-03-22T02:13:29.239556+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:21:19.317485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229714+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:13:29.229714+0000","last_peered":"2024-03-22T02:13:29.229714+0000","last_clean":"2024-03-22T02:13:29.229714+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:13:29.229714+0000","last_undegraded":"2024-03-22T02:13:29.229714+0000","last_fullsized":"2024-03-22T02:13:29.229714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229605+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:13:29.229605+0000","last_peered":"2024-03-22T02:13:29.229605+0000","last_clean":"2024-03-22T02:13:29.229605+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:13:29.229605+0000","last_undegraded":"2024-03-22T02:13:29.229605+0000","last_fullsized":"2024-03-22T02:13:29.229605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235113+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:13:29.235113+0000","last_peered":"2024-03-22T02:13:29.235113+0000","last_clean":"2024-03-22T02:13:29.235113+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:13:29.235113+0000","last_undegraded":"2024-03-22T02:13:29.235113+0000","last_fullsized":"2024-03-22T02:13:29.235113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233084+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:13:29.233084+0000","last_peered":"2024-03-22T02:13:29.233084+0000","last_clean":"2024-03-22T02:13:29.233084+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:13:29.233084+0000","last_undegraded":"2024-03-22T02:13:29.233084+0000","last_fullsized":"2024-03-22T02:13:29.233084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234895+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:13:29.234895+0000","last_peered":"2024-03-22T02:13:29.234895+0000","last_clean":"2024-03-22T02:13:29.234895+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:13:29.234895+0000","last_undegraded":"2024-03-22T02:13:29.234895+0000","last_fullsized":"2024-03-22T02:13:29.234895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231835+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:13:29.231835+0000","last_peered":"2024-03-22T02:13:29.231835+0000","last_clean":"2024-03-22T02:13:29.231835+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:13:29.231835+0000","last_undegraded":"2024-03-22T02:13:29.231835+0000","last_fullsized":"2024-03-22T02:13:29.231835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233399+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:13:29.233399+0000","last_peered":"2024-03-22T02:13:29.233399+0000","last_clean":"2024-03-22T02:13:29.233399+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:13:29.233399+0000","last_undegraded":"2024-03-22T02:13:29.233399+0000","last_fullsized":"2024-03-22T02:13:29.233399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234112+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:13:29.234112+0000","last_peered":"2024-03-22T02:13:29.234112+0000","last_clean":"2024-03-22T02:13:29.234112+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:13:29.234112+0000","last_undegraded":"2024-03-22T02:13:29.234112+0000","last_fullsized":"2024-03-22T02:13:29.234112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536753+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:13:29.536753+0000","last_peered":"2024-03-22T02:13:29.536753+0000","last_clean":"2024-03-22T02:13:29.536753+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:13:29.536753+0000","last_undegraded":"2024-03-22T02:13:29.536753+0000","last_fullsized":"2024-03-22T02:13:29.536753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232917+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:13:29.232917+0000","last_peered":"2024-03-22T02:13:29.232917+0000","last_clean":"2024-03-22T02:13:29.232917+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:13:29.232917+0000","last_undegraded":"2024-03-22T02:13:29.232917+0000","last_fullsized":"2024-03-22T02:13:29.232917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536144+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:13:29.536144+0000","last_peered":"2024-03-22T02:13:29.536144+0000","last_clean":"2024-03-22T02:13:29.536144+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:13:29.536144+0000","last_undegraded":"2024-03-22T02:13:29.536144+0000","last_fullsized":"2024-03-22T02:13:29.536144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248047+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:13:29.248047+0000","last_peered":"2024-03-22T02:13:29.248047+0000","last_clean":"2024-03-22T02:13:29.248047+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:13:29.248047+0000","last_undegraded":"2024-03-22T02:13:29.248047+0000","last_fullsized":"2024-03-22T02:13:29.248047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233098+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:13:29.233098+0000","last_peered":"2024-03-22T02:13:29.233098+0000","last_clean":"2024-03-22T02:13:29.233098+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:13:29.233098+0000","last_undegraded":"2024-03-22T02:13:29.233098+0000","last_fullsized":"2024-03-22T02:13:29.233098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231002+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:13:29.231002+0000","last_peered":"2024-03-22T02:13:29.231002+0000","last_clean":"2024-03-22T02:13:29.231002+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:13:29.231002+0000","last_undegraded":"2024-03-22T02:13:29.231002+0000","last_fullsized":"2024-03-22T02:13:29.231002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231480+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:13:29.231480+0000","last_peered":"2024-03-22T02:13:29.231480+0000","last_clean":"2024-03-22T02:13:29.231480+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:13:29.231480+0000","last_undegraded":"2024-03-22T02:13:29.231480+0000","last_fullsized":"2024-03-22T02:13:29.231480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231359+0000","last_change":"2024-03-22T02:12:28.489359+0000","last_active":"2024-03-22T02:13:29.231359+0000","last_peered":"2024-03-22T02:13:29.231359+0000","last_clean":"2024-03-22T02:13:29.231359+0000","last_became_active":"2024-03-22T02:12:28.489038+0000","last_became_peered":"2024-03-22T02:12:28.489038+0000","last_unstale":"2024-03-22T02:13:29.231359+0000","last_undegraded":"2024-03-22T02:13:29.231359+0000","last_fullsized":"2024-03-22T02:13:29.231359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:15.886990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234341+0000","last_change":"2024-03-22T02:12:28.489164+0000","last_active":"2024-03-22T02:13:29.234341+0000","last_peered":"2024-03-22T02:13:29.234341+0000","last_clean":"2024-03-22T02:13:29.234341+0000","last_became_active":"2024-03-22T02:12:28.488707+0000","last_became_peered":"2024-03-22T02:12:28.488707+0000","last_unstale":"2024-03-22T02:13:29.234341+0000","last_undegraded":"2024-03-22T02:13:29.234341+0000","last_fullsized":"2024-03-22T02:13:29.234341+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:40.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248083+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:13:29.248083+0000","last_peered":"2024-03-22T02:13:29.248083+0000","last_clean":"2024-03-22T02:13:29.248083+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:13:29.248083+0000","last_undegraded":"2024-03-22T02:13:29.248083+0000","last_fullsized":"2024-03-22T02:13:29.248083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232320+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:13:29.232320+0000","last_peered":"2024-03-22T02:13:29.232320+0000","last_clean":"2024-03-22T02:13:29.232320+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:13:29.232320+0000","last_undegraded":"2024-03-22T02:13:29.232320+0000","last_fullsized":"2024-03-22T02:13:29.232320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232088+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:13:29.232088+0000","last_peered":"2024-03-22T02:13:29.232088+0000","last_clean":"2024-03-22T02:13:29.232088+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:13:29.232088+0000","last_undegraded":"2024-03-22T02:13:29.232088+0000","last_fullsized":"2024-03-22T02:13:29.232088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230024+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:13:29.230024+0000","last_peered":"2024-03-22T02:13:29.230024+0000","last_clean":"2024-03-22T02:13:29.230024+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:13:29.230024+0000","last_undegraded":"2024-03-22T02:13:29.230024+0000","last_fullsized":"2024-03-22T02:13:29.230024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247487+0000","last_change":"2024-03-22T02:12:28.520917+0000","last_active":"2024-03-22T02:13:29.247487+0000","last_peered":"2024-03-22T02:13:29.247487+0000","last_clean":"2024-03-22T02:13:29.247487+0000","last_became_active":"2024-03-22T02:12:28.520533+0000","last_became_peered":"2024-03-22T02:12:28.520533+0000","last_unstale":"2024-03-22T02:13:29.247487+0000","last_undegraded":"2024-03-22T02:13:29.247487+0000","last_fullsized":"2024-03-22T02:13:29.247487+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:44.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229453+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:13:29.229453+0000","last_peered":"2024-03-22T02:13:29.229453+0000","last_clean":"2024-03-22T02:13:29.229453+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:13:29.229453+0000","last_undegraded":"2024-03-22T02:13:29.229453+0000","last_fullsized":"2024-03-22T02:13:29.229453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234939+0000","last_change":"2024-03-22T02:12:28.521533+0000","last_active":"2024-03-22T02:13:29.234939+0000","last_peered":"2024-03-22T02:13:29.234939+0000","last_clean":"2024-03-22T02:13:29.234939+0000","last_became_active":"2024-03-22T02:12:28.521176+0000","last_became_peered":"2024-03-22T02:12:28.521176+0000","last_unstale":"2024-03-22T02:13:29.234939+0000","last_undegraded":"2024-03-22T02:13:29.234939+0000","last_fullsized":"2024-03-22T02:13:29.234939+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:10:11.909377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248177+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:13:29.248177+0000","last_peered":"2024-03-22T02:13:29.248177+0000","last_clean":"2024-03-22T02:13:29.248177+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:13:29.248177+0000","last_undegraded":"2024-03-22T02:13:29.248177+0000","last_fullsized":"2024-03-22T02:13:29.248177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231467+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:13:29.231467+0000","last_peered":"2024-03-22T02:13:29.231467+0000","last_clean":"2024-03-22T02:13:29.231467+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:13:29.231467+0000","last_undegraded":"2024-03-22T02:13:29.231467+0000","last_fullsized":"2024-03-22T02:13:29.231467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74127,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":733,"num_read_kb":443,"num_write":175,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":222381,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":523008,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1179848,"data_compressed":64340,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527318,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523120,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":1119891,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518398,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":582221,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":519786,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":589867,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":518592,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":523747,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1114780,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69286,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71559,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":71172,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:13:55.886 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-22T02:13:55.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-22T02:13:56.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 -- 172.21.15.28:0/1909427978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c102660 msgr2=0x7f4e8c108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1909427978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c102660 0x7f4e8c108ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009be0 tx=0x7f4e8402bef0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 -- 172.21.15.28:0/1909427978 shutdown_connections 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1909427978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c102660 0x7f4e8c108ef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1909427978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c101ca0 0x7f4e8c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1909427978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c100ab0 0x7f4e8c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 -- 172.21.15.28:0/1909427978 >> 172.21.15.28:0/1909427978 conn(0x7f4e8c0fc040 msgr2=0x7f4e8c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 -- 172.21.15.28:0/1909427978 shutdown_connections 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.024+0000 7f4e9340e640 1 -- 172.21.15.28:0/1909427978 wait complete. 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 Processor -- start 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 -- start start 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c100ab0 0x7f4e8c06ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c101ca0 0x7f4e8c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c10c730 con 0x7f4e8c102660 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c10c5b0 con 0x7f4e8c100ab0 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e9340e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c10c8b0 con 0x7f4e8c101ca0 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e909a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c101ca0 0x7f4e8c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60322/0 (socket says 172.21.15.28:60322) 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 -- 172.21.15.28:0/1900642584 learned_addr learned my addr 172.21.15.28:0/1900642584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e911aa640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c100ab0 0x7f4e8c06ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 -- 172.21.15.28:0/1900642584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c101ca0 msgr2=0x7f4e8c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c101ca0 0x7f4e8c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 -- 172.21.15.28:0/1900642584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c100ab0 msgr2=0x7f4e8c06ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c100ab0 0x7f4e8c06ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 -- 172.21.15.28:0/1900642584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c19fb20 con 0x7f4e8c102660 2024-03-22T02:13:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.025+0000 7f4e919ab640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036040 tx=0x7f4e840337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.026+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84043020 con 0x7f4e8c102660 2024-03-22T02:13:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.026+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e8402ca60 con 0x7f4e8c102660 2024-03-22T02:13:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.026+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c19fd50 con 0x7f4e8c102660 2024-03-22T02:13:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.026+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84034690 con 0x7f4e8c102660 2024-03-22T02:13:56.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.026+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c1a0240 con 0x7f4e8c102660 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.027+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c107dd0 con 0x7f4e8c102660 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.028+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f4e84046020 con 0x7f4e8c102660 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.028+0000 7f4e827fc640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 0x7f4e780a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.028+0000 7f4e911aa640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 0x7f4e780a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.028+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4e840f1e20 con 0x7f4e8c102660 2024-03-22T02:13:56.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.029+0000 7f4e911aa640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 0x7f4e780a5470 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c101b00 tx=0x7f4e88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.033+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e8404d020 con 0x7f4e8c102660 2024-03-22T02:13:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.213+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 --> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4e8c061fd0 con 0x7f4e780a2fb0 2024-03-22T02:13:56.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.222+0000 7f4e827fc640 1 -- 172.21.15.28:0/1900642584 <== mgr.14152 v2:172.21.15.28:6800/4205492780 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340801 (secure 0 0 0) 0x7f4e8c061fd0 con 0x7f4e780a2fb0 2024-03-22T02:13:56.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:56.228 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 msgr2=0x7f4e780a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 0x7f4e780a5470 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c101b00 tx=0x7f4e88009260 comp rx=0 tx=0).stop 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 msgr2=0x7f4e8c19f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4e84036040 tx=0x7f4e840337b0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 shutdown_connections 2024-03-22T02:13:56.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4e780a2fb0 0x7f4e780a5470 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c102660 0x7f4e8c19f210 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4e8c101ca0 0x7f4e8c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 --2- 172.21.15.28:0/1900642584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4e8c100ab0 0x7f4e8c06ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 >> 172.21.15.28:0/1900642584 conn(0x7f4e8c0fc040 msgr2=0x7f4e8c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 shutdown_connections 2024-03-22T02:13:56.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.225+0000 7f4e9340e640 1 -- 172.21.15.28:0/1900642584 wait complete. 2024-03-22T02:13:56.237 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-03-22T02:13:54.587702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664729,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":859,"num_read_kb":552,"num_write":358,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180758,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365888,"kb_used_data":12544,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124575936,"statfs":{"total":1151940427776,"available":1151565758464,"internally_reserved":0,"allocated":12845056,"data_stored":8240576,"data_compressed":432609,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19142,"internal_metadata":361674042},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6380,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":38,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003025"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232507+0000","last_change":"2024-03-22T02:11:33.640969+0000","last_active":"2024-03-22T02:13:29.232507+0000","last_peered":"2024-03-22T02:13:29.232507+0000","last_clean":"2024-03-22T02:13:29.232507+0000","last_became_active":"2024-03-22T02:11:33.640549+0000","last_became_peered":"2024-03-22T02:11:33.640549+0000","last_unstale":"2024-03-22T02:13:29.232507+0000","last_undegraded":"2024-03-22T02:13:29.232507+0000","last_fullsized":"2024-03-22T02:13:29.232507+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:19.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234634+0000","last_change":"2024-03-22T02:11:33.647215+0000","last_active":"2024-03-22T02:13:29.234634+0000","last_peered":"2024-03-22T02:13:29.234634+0000","last_clean":"2024-03-22T02:13:29.234634+0000","last_became_active":"2024-03-22T02:11:33.645173+0000","last_became_peered":"2024-03-22T02:11:33.645173+0000","last_unstale":"2024-03-22T02:13:29.234634+0000","last_undegraded":"2024-03-22T02:13:29.234634+0000","last_fullsized":"2024-03-22T02:13:29.234634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:24:41.525859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234266+0000","last_change":"2024-03-22T02:11:33.641010+0000","last_active":"2024-03-22T02:13:29.234266+0000","last_peered":"2024-03-22T02:13:29.234266+0000","last_clean":"2024-03-22T02:13:29.234266+0000","last_became_active":"2024-03-22T02:11:33.640681+0000","last_became_peered":"2024-03-22T02:11:33.640681+0000","last_unstale":"2024-03-22T02:13:29.234266+0000","last_undegraded":"2024-03-22T02:13:29.234266+0000","last_fullsized":"2024-03-22T02:13:29.234266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:55:58.765768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229827+0000","last_change":"2024-03-22T02:11:33.638099+0000","last_active":"2024-03-22T02:13:29.229827+0000","last_peered":"2024-03-22T02:13:29.229827+0000","last_clean":"2024-03-22T02:13:29.229827+0000","last_became_active":"2024-03-22T02:11:33.637682+0000","last_became_peered":"2024-03-22T02:11:33.637682+0000","last_unstale":"2024-03-22T02:13:29.229827+0000","last_undegraded":"2024-03-22T02:13:29.229827+0000","last_fullsized":"2024-03-22T02:13:29.229827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:23:45.267023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235150+0000","last_change":"2024-03-22T02:11:33.630104+0000","last_active":"2024-03-22T02:13:29.235150+0000","last_peered":"2024-03-22T02:13:29.235150+0000","last_clean":"2024-03-22T02:13:29.235150+0000","last_became_active":"2024-03-22T02:11:33.629624+0000","last_became_peered":"2024-03-22T02:11:33.629624+0000","last_unstale":"2024-03-22T02:13:29.235150+0000","last_undegraded":"2024-03-22T02:13:29.235150+0000","last_fullsized":"2024-03-22T02:13:29.235150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:43.571682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666425+0000","last_change":"2024-03-22T02:12:28.521805+0000","last_active":"2024-03-22T02:13:29.666425+0000","last_peered":"2024-03-22T02:13:29.666425+0000","last_clean":"2024-03-22T02:13:29.666425+0000","last_became_active":"2024-03-22T02:12:28.521419+0000","last_became_peered":"2024-03-22T02:12:28.521419+0000","last_unstale":"2024-03-22T02:13:29.666425+0000","last_undegraded":"2024-03-22T02:13:29.666425+0000","last_fullsized":"2024-03-22T02:13:29.666425+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:01:11.831486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232631+0000","last_change":"2024-03-22T02:11:33.636731+0000","last_active":"2024-03-22T02:13:29.232631+0000","last_peered":"2024-03-22T02:13:29.232631+0000","last_clean":"2024-03-22T02:13:29.232631+0000","last_became_active":"2024-03-22T02:11:33.636307+0000","last_became_peered":"2024-03-22T02:11:33.636307+0000","last_unstale":"2024-03-22T02:13:29.232631+0000","last_undegraded":"2024-03-22T02:13:29.232631+0000","last_fullsized":"2024-03-22T02:13:29.232631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:53:35.131261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229177+0000","last_change":"2024-03-22T02:11:33.639529+0000","last_active":"2024-03-22T02:13:29.229177+0000","last_peered":"2024-03-22T02:13:29.229177+0000","last_clean":"2024-03-22T02:13:29.229177+0000","last_became_active":"2024-03-22T02:11:33.639225+0000","last_became_peered":"2024-03-22T02:11:33.639225+0000","last_unstale":"2024-03-22T02:13:29.229177+0000","last_undegraded":"2024-03-22T02:13:29.229177+0000","last_fullsized":"2024-03-22T02:13:29.229177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:54.643205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235354+0000","last_change":"2024-03-22T02:11:33.636964+0000","last_active":"2024-03-22T02:13:29.235354+0000","last_peered":"2024-03-22T02:13:29.235354+0000","last_clean":"2024-03-22T02:13:29.235354+0000","last_became_active":"2024-03-22T02:11:33.635351+0000","last_became_peered":"2024-03-22T02:11:33.635351+0000","last_unstale":"2024-03-22T02:13:29.235354+0000","last_undegraded":"2024-03-22T02:13:29.235354+0000","last_fullsized":"2024-03-22T02:13:29.235354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:17:38.060329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666800+0000","last_change":"2024-03-22T02:11:33.630282+0000","last_active":"2024-03-22T02:13:29.666800+0000","last_peered":"2024-03-22T02:13:29.666800+0000","last_clean":"2024-03-22T02:13:29.666800+0000","last_became_active":"2024-03-22T02:11:33.629906+0000","last_became_peered":"2024-03-22T02:11:33.629906+0000","last_unstale":"2024-03-22T02:13:29.666800+0000","last_undegraded":"2024-03-22T02:13:29.666800+0000","last_fullsized":"2024-03-22T02:13:29.666800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:12:00.166114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233246+0000","last_change":"2024-03-22T02:11:33.635152+0000","last_active":"2024-03-22T02:13:29.233246+0000","last_peered":"2024-03-22T02:13:29.233246+0000","last_clean":"2024-03-22T02:13:29.233246+0000","last_became_active":"2024-03-22T02:11:33.634836+0000","last_became_peered":"2024-03-22T02:11:33.634836+0000","last_unstale":"2024-03-22T02:13:29.233246+0000","last_undegraded":"2024-03-22T02:13:29.233246+0000","last_fullsized":"2024-03-22T02:13:29.233246+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:03:37.979347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229832+0000","last_change":"2024-03-22T02:11:33.642758+0000","last_active":"2024-03-22T02:13:29.229832+0000","last_peered":"2024-03-22T02:13:29.229832+0000","last_clean":"2024-03-22T02:13:29.229832+0000","last_became_active":"2024-03-22T02:11:33.642414+0000","last_became_peered":"2024-03-22T02:11:33.642414+0000","last_unstale":"2024-03-22T02:13:29.229832+0000","last_undegraded":"2024-03-22T02:13:29.229832+0000","last_fullsized":"2024-03-22T02:13:29.229832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:20:17.337290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247995+0000","last_change":"2024-03-22T02:11:33.634630+0000","last_active":"2024-03-22T02:13:29.247995+0000","last_peered":"2024-03-22T02:13:29.247995+0000","last_clean":"2024-03-22T02:13:29.247995+0000","last_became_active":"2024-03-22T02:11:33.632605+0000","last_became_peered":"2024-03-22T02:11:33.632605+0000","last_unstale":"2024-03-22T02:13:29.247995+0000","last_undegraded":"2024-03-22T02:13:29.247995+0000","last_fullsized":"2024-03-22T02:13:29.247995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:46:07.416821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234497+0000","last_change":"2024-03-22T02:11:33.646903+0000","last_active":"2024-03-22T02:13:29.234497+0000","last_peered":"2024-03-22T02:13:29.234497+0000","last_clean":"2024-03-22T02:13:29.234497+0000","last_became_active":"2024-03-22T02:11:33.644850+0000","last_became_peered":"2024-03-22T02:11:33.644850+0000","last_unstale":"2024-03-22T02:13:29.234497+0000","last_undegraded":"2024-03-22T02:13:29.234497+0000","last_fullsized":"2024-03-22T02:13:29.234497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:07:35.389129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234649+0000","last_change":"2024-03-22T02:11:33.648921+0000","last_active":"2024-03-22T02:13:29.234649+0000","last_peered":"2024-03-22T02:13:29.234649+0000","last_clean":"2024-03-22T02:13:29.234649+0000","last_became_active":"2024-03-22T02:11:33.648454+0000","last_became_peered":"2024-03-22T02:11:33.648454+0000","last_unstale":"2024-03-22T02:13:29.234649+0000","last_undegraded":"2024-03-22T02:13:29.234649+0000","last_fullsized":"2024-03-22T02:13:29.234649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:02:13.643942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234959+0000","last_change":"2024-03-22T02:11:33.640285+0000","last_active":"2024-03-22T02:13:29.234959+0000","last_peered":"2024-03-22T02:13:29.234959+0000","last_clean":"2024-03-22T02:13:29.234959+0000","last_became_active":"2024-03-22T02:11:33.639856+0000","last_became_peered":"2024-03-22T02:11:33.639856+0000","last_unstale":"2024-03-22T02:13:29.234959+0000","last_undegraded":"2024-03-22T02:13:29.234959+0000","last_fullsized":"2024-03-22T02:13:29.234959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:08:39.575708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231082+0000","last_change":"2024-03-22T02:11:33.643692+0000","last_active":"2024-03-22T02:13:29.231082+0000","last_peered":"2024-03-22T02:13:29.231082+0000","last_clean":"2024-03-22T02:13:29.231082+0000","last_became_active":"2024-03-22T02:11:33.643377+0000","last_became_peered":"2024-03-22T02:11:33.643377+0000","last_unstale":"2024-03-22T02:13:29.231082+0000","last_undegraded":"2024-03-22T02:13:29.231082+0000","last_fullsized":"2024-03-22T02:13:29.231082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:11:19.411616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235450+0000","last_change":"2024-03-22T02:11:33.635795+0000","last_active":"2024-03-22T02:13:29.235450+0000","last_peered":"2024-03-22T02:13:29.235450+0000","last_clean":"2024-03-22T02:13:29.235450+0000","last_became_active":"2024-03-22T02:11:33.635479+0000","last_became_peered":"2024-03-22T02:11:33.635479+0000","last_unstale":"2024-03-22T02:13:29.235450+0000","last_undegraded":"2024-03-22T02:13:29.235450+0000","last_fullsized":"2024-03-22T02:13:29.235450+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:56:06.931364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232547+0000","last_change":"2024-03-22T02:11:33.633325+0000","last_active":"2024-03-22T02:13:29.232547+0000","last_peered":"2024-03-22T02:13:29.232547+0000","last_clean":"2024-03-22T02:13:29.232547+0000","last_became_active":"2024-03-22T02:11:33.633007+0000","last_became_peered":"2024-03-22T02:11:33.633007+0000","last_unstale":"2024-03-22T02:13:29.232547+0000","last_undegraded":"2024-03-22T02:13:29.232547+0000","last_fullsized":"2024-03-22T02:13:29.232547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:45:30.411463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.236492+0000","last_change":"2024-03-22T02:13:29.236492+0000","last_active":"2024-03-22T02:13:29.236492+0000","last_peered":"2024-03-22T02:13:29.236492+0000","last_clean":"2024-03-22T02:13:29.236492+0000","last_became_active":"2024-03-22T02:13:29.236092+0000","last_became_peered":"2024-03-22T02:13:29.236092+0000","last_unstale":"2024-03-22T02:13:29.236492+0000","last_undegraded":"2024-03-22T02:13:29.236492+0000","last_fullsized":"2024-03-22T02:13:29.236492+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:42.511417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'9","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235774+0000","last_change":"2024-03-22T02:11:32.914157+0000","last_active":"2024-03-22T02:13:29.235774+0000","last_peered":"2024-03-22T02:13:29.235774+0000","last_clean":"2024-03-22T02:13:29.235774+0000","last_became_active":"2024-03-22T02:11:32.913858+0000","last_became_peered":"2024-03-22T02:11:32.913858+0000","last_unstale":"2024-03-22T02:13:29.235774+0000","last_undegraded":"2024-03-22T02:13:29.235774+0000","last_fullsized":"2024-03-22T02:13:29.235774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:01:09.170401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229678+0000","last_change":"2024-03-22T02:11:33.641380+0000","last_active":"2024-03-22T02:13:29.229678+0000","last_peered":"2024-03-22T02:13:29.229678+0000","last_clean":"2024-03-22T02:13:29.229678+0000","last_became_active":"2024-03-22T02:11:33.641079+0000","last_became_peered":"2024-03-22T02:11:33.641079+0000","last_unstale":"2024-03-22T02:13:29.229678+0000","last_undegraded":"2024-03-22T02:13:29.229678+0000","last_fullsized":"2024-03-22T02:13:29.229678+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:26.244351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229358+0000","last_change":"2024-03-22T02:11:37.937254+0000","last_active":"2024-03-22T02:13:29.229358+0000","last_peered":"2024-03-22T02:13:29.229358+0000","last_clean":"2024-03-22T02:13:29.229358+0000","last_became_active":"2024-03-22T02:11:37.936791+0000","last_became_peered":"2024-03-22T02:11:37.936791+0000","last_unstale":"2024-03-22T02:13:29.229358+0000","last_undegraded":"2024-03-22T02:13:29.229358+0000","last_fullsized":"2024-03-22T02:13:29.229358+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:48.970305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229936+0000","last_change":"2024-03-22T02:11:32.908490+0000","last_active":"2024-03-22T02:13:29.229936+0000","last_peered":"2024-03-22T02:13:29.229936+0000","last_clean":"2024-03-22T02:13:29.229936+0000","last_became_active":"2024-03-22T02:11:32.908162+0000","last_became_peered":"2024-03-22T02:11:32.908162+0000","last_unstale":"2024-03-22T02:13:29.229936+0000","last_undegraded":"2024-03-22T02:13:29.229936+0000","last_fullsized":"2024-03-22T02:13:29.229936+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:24.122646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234882+0000","last_change":"2024-03-22T02:11:33.647675+0000","last_active":"2024-03-22T02:13:29.234882+0000","last_peered":"2024-03-22T02:13:29.234882+0000","last_clean":"2024-03-22T02:13:29.234882+0000","last_became_active":"2024-03-22T02:11:33.647362+0000","last_became_peered":"2024-03-22T02:11:33.647362+0000","last_unstale":"2024-03-22T02:13:29.234882+0000","last_undegraded":"2024-03-22T02:13:29.234882+0000","last_fullsized":"2024-03-22T02:13:29.234882+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:32:15.829742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229464+0000","last_change":"2024-03-22T02:11:37.941253+0000","last_active":"2024-03-22T02:13:29.229464+0000","last_peered":"2024-03-22T02:13:29.229464+0000","last_clean":"2024-03-22T02:13:29.229464+0000","last_became_active":"2024-03-22T02:11:37.940868+0000","last_became_peered":"2024-03-22T02:11:37.940868+0000","last_unstale":"2024-03-22T02:13:29.229464+0000","last_undegraded":"2024-03-22T02:13:29.229464+0000","last_fullsized":"2024-03-22T02:13:29.229464+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:02:28.760339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'4","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242108+0000","last_change":"2024-03-22T02:13:29.242108+0000","last_active":"2024-03-22T02:13:29.242108+0000","last_peered":"2024-03-22T02:13:29.242108+0000","last_clean":"2024-03-22T02:13:29.242108+0000","last_became_active":"2024-03-22T02:13:29.236686+0000","last_became_peered":"2024-03-22T02:13:29.236686+0000","last_unstale":"2024-03-22T02:13:29.242108+0000","last_undegraded":"2024-03-22T02:13:29.242108+0000","last_fullsized":"2024-03-22T02:13:29.242108+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:42:01.218922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235315+0000","last_change":"2024-03-22T02:11:33.635659+0000","last_active":"2024-03-22T02:13:29.235315+0000","last_peered":"2024-03-22T02:13:29.235315+0000","last_clean":"2024-03-22T02:13:29.235315+0000","last_became_active":"2024-03-22T02:11:33.634797+0000","last_became_peered":"2024-03-22T02:11:33.634797+0000","last_unstale":"2024-03-22T02:13:29.235315+0000","last_undegraded":"2024-03-22T02:13:29.235315+0000","last_fullsized":"2024-03-22T02:13:29.235315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:01.272838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229375+0000","last_change":"2024-03-22T02:11:37.937217+0000","last_active":"2024-03-22T02:13:29.229375+0000","last_peered":"2024-03-22T02:13:29.229375+0000","last_clean":"2024-03-22T02:13:29.229375+0000","last_became_active":"2024-03-22T02:11:37.936782+0000","last_became_peered":"2024-03-22T02:11:37.936782+0000","last_unstale":"2024-03-22T02:13:29.229375+0000","last_undegraded":"2024-03-22T02:13:29.229375+0000","last_fullsized":"2024-03-22T02:13:29.229375+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:58:23.862889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235311+0000","last_change":"2024-03-22T02:11:32.909349+0000","last_active":"2024-03-22T02:13:29.235311+0000","last_peered":"2024-03-22T02:13:29.235311+0000","last_clean":"2024-03-22T02:13:29.235311+0000","last_became_active":"2024-03-22T02:11:32.908402+0000","last_became_peered":"2024-03-22T02:11:32.908402+0000","last_unstale":"2024-03-22T02:13:29.235311+0000","last_undegraded":"2024-03-22T02:13:29.235311+0000","last_fullsized":"2024-03-22T02:13:29.235311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:19:05.587134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536646+0000","last_change":"2024-03-22T02:11:33.637247+0000","last_active":"2024-03-22T02:13:29.536646+0000","last_peered":"2024-03-22T02:13:29.536646+0000","last_clean":"2024-03-22T02:13:29.536646+0000","last_became_active":"2024-03-22T02:11:33.636947+0000","last_became_peered":"2024-03-22T02:11:33.636947+0000","last_unstale":"2024-03-22T02:13:29.536646+0000","last_undegraded":"2024-03-22T02:13:29.536646+0000","last_fullsized":"2024-03-22T02:13:29.536646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:59.078910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231729+0000","last_change":"2024-03-22T02:11:37.932612+0000","last_active":"2024-03-22T02:13:29.231729+0000","last_peered":"2024-03-22T02:13:29.231729+0000","last_clean":"2024-03-22T02:13:29.231729+0000","last_became_active":"2024-03-22T02:11:37.932037+0000","last_became_peered":"2024-03-22T02:11:37.932037+0000","last_unstale":"2024-03-22T02:13:29.231729+0000","last_undegraded":"2024-03-22T02:13:29.231729+0000","last_fullsized":"2024-03-22T02:13:29.231729+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:25:48.063910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:51.111514+0000","last_change":"2024-03-22T02:11:32.916577+0000","last_active":"2024-03-22T02:13:51.111514+0000","last_peered":"2024-03-22T02:13:51.111514+0000","last_clean":"2024-03-22T02:13:51.111514+0000","last_became_active":"2024-03-22T02:11:32.916281+0000","last_became_peered":"2024-03-22T02:11:32.916281+0000","last_unstale":"2024-03-22T02:13:51.111514+0000","last_undegraded":"2024-03-22T02:13:51.111514+0000","last_fullsized":"2024-03-22T02:13:51.111514+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:31:06.154873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234972+0000","last_change":"2024-03-22T02:11:33.639590+0000","last_active":"2024-03-22T02:13:29.234972+0000","last_peered":"2024-03-22T02:13:29.234972+0000","last_clean":"2024-03-22T02:13:29.234972+0000","last_became_active":"2024-03-22T02:11:33.639289+0000","last_became_peered":"2024-03-22T02:11:33.639289+0000","last_unstale":"2024-03-22T02:13:29.234972+0000","last_undegraded":"2024-03-22T02:13:29.234972+0000","last_fullsized":"2024-03-22T02:13:29.234972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:57:22.322228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667818+0000","last_change":"2024-03-22T02:13:29.667818+0000","last_active":"2024-03-22T02:13:29.667818+0000","last_peered":"2024-03-22T02:13:29.667818+0000","last_clean":"2024-03-22T02:13:29.667818+0000","last_became_active":"2024-03-22T02:13:29.667344+0000","last_became_peered":"2024-03-22T02:13:29.667344+0000","last_unstale":"2024-03-22T02:13:29.667818+0000","last_undegraded":"2024-03-22T02:13:29.667818+0000","last_fullsized":"2024-03-22T02:13:29.667818+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:45:08.802171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666359+0000","last_change":"2024-03-22T02:11:32.914779+0000","last_active":"2024-03-22T02:13:29.666359+0000","last_peered":"2024-03-22T02:13:29.666359+0000","last_clean":"2024-03-22T02:13:29.666359+0000","last_became_active":"2024-03-22T02:11:32.914394+0000","last_became_peered":"2024-03-22T02:11:32.914394+0000","last_unstale":"2024-03-22T02:13:29.666359+0000","last_undegraded":"2024-03-22T02:13:29.666359+0000","last_fullsized":"2024-03-22T02:13:29.666359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:20:49.562118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536554+0000","last_change":"2024-03-22T02:11:33.636818+0000","last_active":"2024-03-22T02:13:29.536554+0000","last_peered":"2024-03-22T02:13:29.536554+0000","last_clean":"2024-03-22T02:13:29.536554+0000","last_became_active":"2024-03-22T02:11:33.636377+0000","last_became_peered":"2024-03-22T02:11:33.636377+0000","last_unstale":"2024-03-22T02:13:29.536554+0000","last_undegraded":"2024-03-22T02:13:29.536554+0000","last_fullsized":"2024-03-22T02:13:29.536554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:17.010089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231009+0000","last_change":"2024-03-22T02:11:37.944942+0000","last_active":"2024-03-22T02:13:29.231009+0000","last_peered":"2024-03-22T02:13:29.231009+0000","last_clean":"2024-03-22T02:13:29.231009+0000","last_became_active":"2024-03-22T02:11:37.944514+0000","last_became_peered":"2024-03-22T02:11:37.944514+0000","last_unstale":"2024-03-22T02:13:29.231009+0000","last_undegraded":"2024-03-22T02:13:29.231009+0000","last_fullsized":"2024-03-22T02:13:29.231009+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:36:54.144207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247642+0000","last_change":"2024-03-22T02:11:32.900678+0000","last_active":"2024-03-22T02:13:29.247642+0000","last_peered":"2024-03-22T02:13:29.247642+0000","last_clean":"2024-03-22T02:13:29.247642+0000","last_became_active":"2024-03-22T02:11:32.900322+0000","last_became_peered":"2024-03-22T02:11:32.900322+0000","last_unstale":"2024-03-22T02:13:29.247642+0000","last_undegraded":"2024-03-22T02:13:29.247642+0000","last_fullsized":"2024-03-22T02:13:29.247642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:14:40.587548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229323+0000","last_change":"2024-03-22T02:11:33.633486+0000","last_active":"2024-03-22T02:13:29.229323+0000","last_peered":"2024-03-22T02:13:29.229323+0000","last_clean":"2024-03-22T02:13:29.229323+0000","last_became_active":"2024-03-22T02:11:33.633029+0000","last_became_peered":"2024-03-22T02:11:33.633029+0000","last_unstale":"2024-03-22T02:13:29.229323+0000","last_undegraded":"2024-03-22T02:13:29.229323+0000","last_fullsized":"2024-03-22T02:13:29.229323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:13:30.565651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234585+0000","last_change":"2024-03-22T02:11:37.943585+0000","last_active":"2024-03-22T02:13:29.234585+0000","last_peered":"2024-03-22T02:13:29.234585+0000","last_clean":"2024-03-22T02:13:29.234585+0000","last_became_active":"2024-03-22T02:11:37.943024+0000","last_became_peered":"2024-03-22T02:11:37.943024+0000","last_unstale":"2024-03-22T02:13:29.234585+0000","last_undegraded":"2024-03-22T02:13:29.234585+0000","last_fullsized":"2024-03-22T02:13:29.234585+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:13:40.645998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536228+0000","last_change":"2024-03-22T02:11:32.915688+0000","last_active":"2024-03-22T02:13:29.536228+0000","last_peered":"2024-03-22T02:13:29.536228+0000","last_clean":"2024-03-22T02:13:29.536228+0000","last_became_active":"2024-03-22T02:11:32.915392+0000","last_became_peered":"2024-03-22T02:11:32.915392+0000","last_unstale":"2024-03-22T02:13:29.536228+0000","last_undegraded":"2024-03-22T02:13:29.536228+0000","last_fullsized":"2024-03-22T02:13:29.536228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:55:39.011504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234870+0000","last_change":"2024-03-22T02:11:33.635151+0000","last_active":"2024-03-22T02:13:29.234870+0000","last_peered":"2024-03-22T02:13:29.234870+0000","last_clean":"2024-03-22T02:13:29.234870+0000","last_became_active":"2024-03-22T02:11:33.634813+0000","last_became_peered":"2024-03-22T02:11:33.634813+0000","last_unstale":"2024-03-22T02:13:29.234870+0000","last_undegraded":"2024-03-22T02:13:29.234870+0000","last_fullsized":"2024-03-22T02:13:29.234870+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:20:34.497878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229276+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.229276+0000","last_peered":"2024-03-22T02:13:29.229276+0000","last_clean":"2024-03-22T02:13:29.229276+0000","last_became_active":"2024-03-22T02:11:37.938287+0000","last_became_peered":"2024-03-22T02:11:37.938287+0000","last_unstale":"2024-03-22T02:13:29.229276+0000","last_undegraded":"2024-03-22T02:13:29.229276+0000","last_fullsized":"2024-03-22T02:13:29.229276+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:13:48.542566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'4","reported_seq":151,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:49.108917+0000","last_change":"2024-03-22T02:11:32.916337+0000","last_active":"2024-03-22T02:13:49.108917+0000","last_peered":"2024-03-22T02:13:49.108917+0000","last_clean":"2024-03-22T02:13:49.108917+0000","last_became_active":"2024-03-22T02:11:32.915997+0000","last_became_peered":"2024-03-22T02:11:32.915997+0000","last_unstale":"2024-03-22T02:13:49.108917+0000","last_undegraded":"2024-03-22T02:13:49.108917+0000","last_fullsized":"2024-03-22T02:13:49.108917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:45:05.697255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":116,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235071+0000","last_change":"2024-03-22T02:11:33.636079+0000","last_active":"2024-03-22T02:13:29.235071+0000","last_peered":"2024-03-22T02:13:29.235071+0000","last_clean":"2024-03-22T02:13:29.235071+0000","last_became_active":"2024-03-22T02:11:33.635155+0000","last_became_peered":"2024-03-22T02:11:33.635155+0000","last_unstale":"2024-03-22T02:13:29.235071+0000","last_undegraded":"2024-03-22T02:13:29.235071+0000","last_fullsized":"2024-03-22T02:13:29.235071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:47:38.002632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234467+0000","last_change":"2024-03-22T02:11:37.951089+0000","last_active":"2024-03-22T02:13:29.234467+0000","last_peered":"2024-03-22T02:13:29.234467+0000","last_clean":"2024-03-22T02:13:29.234467+0000","last_became_active":"2024-03-22T02:11:37.949847+0000","last_became_peered":"2024-03-22T02:11:37.949847+0000","last_unstale":"2024-03-22T02:13:29.234467+0000","last_undegraded":"2024-03-22T02:13:29.234467+0000","last_fullsized":"2024-03-22T02:13:29.234467+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:54:01.309935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.238453+0000","last_change":"2024-03-22T02:13:29.238453+0000","last_active":"2024-03-22T02:13:29.238453+0000","last_peered":"2024-03-22T02:13:29.238453+0000","last_clean":"2024-03-22T02:13:29.238453+0000","last_became_active":"2024-03-22T02:13:29.235861+0000","last_became_peered":"2024-03-22T02:13:29.235861+0000","last_unstale":"2024-03-22T02:13:29.238453+0000","last_undegraded":"2024-03-22T02:13:29.238453+0000","last_fullsized":"2024-03-22T02:13:29.238453+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:50:35.945707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234160+0000","last_change":"2024-03-22T02:12:28.521865+0000","last_active":"2024-03-22T02:13:29.234160+0000","last_peered":"2024-03-22T02:13:29.234160+0000","last_clean":"2024-03-22T02:13:29.234160+0000","last_became_active":"2024-03-22T02:12:28.521413+0000","last_became_peered":"2024-03-22T02:12:28.521413+0000","last_unstale":"2024-03-22T02:13:29.234160+0000","last_undegraded":"2024-03-22T02:13:29.234160+0000","last_fullsized":"2024-03-22T02:13:29.234160+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:54:26.175128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229759+0000","last_change":"2024-03-22T02:11:37.935073+0000","last_active":"2024-03-22T02:13:29.229759+0000","last_peered":"2024-03-22T02:13:29.229759+0000","last_clean":"2024-03-22T02:13:29.229759+0000","last_became_active":"2024-03-22T02:11:37.933513+0000","last_became_peered":"2024-03-22T02:11:37.933513+0000","last_unstale":"2024-03-22T02:13:29.229759+0000","last_undegraded":"2024-03-22T02:13:29.229759+0000","last_fullsized":"2024-03-22T02:13:29.229759+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:28:29.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536583+0000","last_change":"2024-03-22T02:11:32.913239+0000","last_active":"2024-03-22T02:13:29.536583+0000","last_peered":"2024-03-22T02:13:29.536583+0000","last_clean":"2024-03-22T02:13:29.536583+0000","last_became_active":"2024-03-22T02:11:32.912260+0000","last_became_peered":"2024-03-22T02:11:32.912260+0000","last_unstale":"2024-03-22T02:13:29.536583+0000","last_undegraded":"2024-03-22T02:13:29.536583+0000","last_fullsized":"2024-03-22T02:13:29.536583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:03:53.782155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235431+0000","last_change":"2024-03-22T02:11:33.636143+0000","last_active":"2024-03-22T02:13:29.235431+0000","last_peered":"2024-03-22T02:13:29.235431+0000","last_clean":"2024-03-22T02:13:29.235431+0000","last_became_active":"2024-03-22T02:11:33.635401+0000","last_became_peered":"2024-03-22T02:11:33.635401+0000","last_unstale":"2024-03-22T02:13:29.235431+0000","last_undegraded":"2024-03-22T02:13:29.235431+0000","last_fullsized":"2024-03-22T02:13:29.235431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:10:20.286598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235070+0000","last_change":"2024-03-22T02:11:37.938792+0000","last_active":"2024-03-22T02:13:29.235070+0000","last_peered":"2024-03-22T02:13:29.235070+0000","last_clean":"2024-03-22T02:13:29.235070+0000","last_became_active":"2024-03-22T02:11:37.938377+0000","last_became_peered":"2024-03-22T02:11:37.938377+0000","last_unstale":"2024-03-22T02:13:29.235070+0000","last_undegraded":"2024-03-22T02:13:29.235070+0000","last_fullsized":"2024-03-22T02:13:29.235070+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:25:49.215456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'38","reported_seq":311,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.109991+0000","last_change":"2024-03-22T02:11:32.912105+0000","last_active":"2024-03-22T02:13:50.109991+0000","last_peered":"2024-03-22T02:13:50.109991+0000","last_clean":"2024-03-22T02:13:50.109991+0000","last_became_active":"2024-03-22T02:11:32.911782+0000","last_became_peered":"2024-03-22T02:11:32.911782+0000","last_unstale":"2024-03-22T02:13:50.109991+0000","last_undegraded":"2024-03-22T02:13:50.109991+0000","last_fullsized":"2024-03-22T02:13:50.109991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:00:37.170569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":242,"num_read_kb":189,"num_write":39,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235127+0000","last_change":"2024-03-22T02:11:33.632732+0000","last_active":"2024-03-22T02:13:29.235127+0000","last_peered":"2024-03-22T02:13:29.235127+0000","last_clean":"2024-03-22T02:13:29.235127+0000","last_became_active":"2024-03-22T02:11:33.632435+0000","last_became_peered":"2024-03-22T02:11:33.632435+0000","last_unstale":"2024-03-22T02:13:29.235127+0000","last_undegraded":"2024-03-22T02:13:29.235127+0000","last_fullsized":"2024-03-22T02:13:29.235127+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:51:02.755855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536841+0000","last_change":"2024-03-22T02:11:37.937393+0000","last_active":"2024-03-22T02:13:29.536841+0000","last_peered":"2024-03-22T02:13:29.536841+0000","last_clean":"2024-03-22T02:13:29.536841+0000","last_became_active":"2024-03-22T02:11:37.936994+0000","last_became_peered":"2024-03-22T02:11:37.936994+0000","last_unstale":"2024-03-22T02:13:29.536841+0000","last_undegraded":"2024-03-22T02:13:29.536841+0000","last_fullsized":"2024-03-22T02:13:29.536841+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:52:33.233991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231083+0000","last_change":"2024-03-22T02:11:32.911638+0000","last_active":"2024-03-22T02:13:29.231083+0000","last_peered":"2024-03-22T02:13:29.231083+0000","last_clean":"2024-03-22T02:13:29.231083+0000","last_became_active":"2024-03-22T02:11:32.910500+0000","last_became_peered":"2024-03-22T02:11:32.910500+0000","last_unstale":"2024-03-22T02:13:29.231083+0000","last_undegraded":"2024-03-22T02:13:29.231083+0000","last_fullsized":"2024-03-22T02:13:29.231083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:18:16.583441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247705+0000","last_change":"2024-03-22T02:11:33.634944+0000","last_active":"2024-03-22T02:13:29.247705+0000","last_peered":"2024-03-22T02:13:29.247705+0000","last_clean":"2024-03-22T02:13:29.247705+0000","last_became_active":"2024-03-22T02:11:33.634563+0000","last_became_peered":"2024-03-22T02:11:33.634563+0000","last_unstale":"2024-03-22T02:13:29.247705+0000","last_undegraded":"2024-03-22T02:13:29.247705+0000","last_fullsized":"2024-03-22T02:13:29.247705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:10:48.703936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231568+0000","last_change":"2024-03-22T02:12:28.520817+0000","last_active":"2024-03-22T02:13:29.231568+0000","last_peered":"2024-03-22T02:13:29.231568+0000","last_clean":"2024-03-22T02:13:29.231568+0000","last_became_active":"2024-03-22T02:12:28.520360+0000","last_became_peered":"2024-03-22T02:12:28.520360+0000","last_unstale":"2024-03-22T02:13:29.231568+0000","last_undegraded":"2024-03-22T02:13:29.231568+0000","last_fullsized":"2024-03-22T02:13:29.231568+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:34:21.654588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536568+0000","last_change":"2024-03-22T02:11:32.914999+0000","last_active":"2024-03-22T02:13:29.536568+0000","last_peered":"2024-03-22T02:13:29.536568+0000","last_clean":"2024-03-22T02:13:29.536568+0000","last_became_active":"2024-03-22T02:11:32.914669+0000","last_became_peered":"2024-03-22T02:11:32.914669+0000","last_unstale":"2024-03-22T02:13:29.536568+0000","last_undegraded":"2024-03-22T02:13:29.536568+0000","last_fullsized":"2024-03-22T02:13:29.536568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:30:50.546703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233232+0000","last_change":"2024-03-22T02:11:33.632961+0000","last_active":"2024-03-22T02:13:29.233232+0000","last_peered":"2024-03-22T02:13:29.233232+0000","last_clean":"2024-03-22T02:13:29.233232+0000","last_became_active":"2024-03-22T02:11:33.632642+0000","last_became_peered":"2024-03-22T02:11:33.632642+0000","last_unstale":"2024-03-22T02:13:29.233232+0000","last_undegraded":"2024-03-22T02:13:29.233232+0000","last_fullsized":"2024-03-22T02:13:29.233232+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:40:15.267842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235648+0000","last_change":"2024-03-22T02:11:37.950426+0000","last_active":"2024-03-22T02:13:29.235648+0000","last_peered":"2024-03-22T02:13:29.235648+0000","last_clean":"2024-03-22T02:13:29.235648+0000","last_became_active":"2024-03-22T02:11:37.950024+0000","last_became_peered":"2024-03-22T02:11:37.950024+0000","last_unstale":"2024-03-22T02:13:29.235648+0000","last_undegraded":"2024-03-22T02:13:29.235648+0000","last_fullsized":"2024-03-22T02:13:29.235648+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:12:16.325692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'5","reported_seq":160,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:53.143913+0000","last_change":"2024-03-22T02:11:32.910565+0000","last_active":"2024-03-22T02:13:53.143913+0000","last_peered":"2024-03-22T02:13:53.143913+0000","last_clean":"2024-03-22T02:13:53.143913+0000","last_became_active":"2024-03-22T02:11:32.910276+0000","last_became_peered":"2024-03-22T02:11:32.910276+0000","last_unstale":"2024-03-22T02:13:53.143913+0000","last_undegraded":"2024-03-22T02:13:53.143913+0000","last_fullsized":"2024-03-22T02:13:53.143913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:22:16.173292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1287,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":124,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232769+0000","last_change":"2024-03-22T02:11:33.636642+0000","last_active":"2024-03-22T02:13:29.232769+0000","last_peered":"2024-03-22T02:13:29.232769+0000","last_clean":"2024-03-22T02:13:29.232769+0000","last_became_active":"2024-03-22T02:11:33.636269+0000","last_became_peered":"2024-03-22T02:11:33.636269+0000","last_unstale":"2024-03-22T02:13:29.232769+0000","last_undegraded":"2024-03-22T02:13:29.232769+0000","last_fullsized":"2024-03-22T02:13:29.232769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:50:21.668795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248019+0000","last_change":"2024-03-22T02:11:37.946550+0000","last_active":"2024-03-22T02:13:29.248019+0000","last_peered":"2024-03-22T02:13:29.248019+0000","last_clean":"2024-03-22T02:13:29.248019+0000","last_became_active":"2024-03-22T02:11:37.946161+0000","last_became_peered":"2024-03-22T02:11:37.946161+0000","last_unstale":"2024-03-22T02:13:29.248019+0000","last_undegraded":"2024-03-22T02:13:29.248019+0000","last_fullsized":"2024-03-22T02:13:29.248019+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:41.689233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'15","reported_seq":48,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234158+0000","last_change":"2024-03-22T02:11:32.913480+0000","last_active":"2024-03-22T02:13:29.234158+0000","last_peered":"2024-03-22T02:13:29.234158+0000","last_clean":"2024-03-22T02:13:29.234158+0000","last_became_active":"2024-03-22T02:11:32.913198+0000","last_became_peered":"2024-03-22T02:11:32.913198+0000","last_unstale":"2024-03-22T02:13:29.234158+0000","last_undegraded":"2024-03-22T02:13:29.234158+0000","last_fullsized":"2024-03-22T02:13:29.234158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:16:41.279928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536334+0000","last_change":"2024-03-22T02:11:33.631765+0000","last_active":"2024-03-22T02:13:29.536334+0000","last_peered":"2024-03-22T02:13:29.536334+0000","last_clean":"2024-03-22T02:13:29.536334+0000","last_became_active":"2024-03-22T02:11:33.631436+0000","last_became_peered":"2024-03-22T02:11:33.631436+0000","last_unstale":"2024-03-22T02:13:29.536334+0000","last_undegraded":"2024-03-22T02:13:29.536334+0000","last_fullsized":"2024-03-22T02:13:29.536334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:43:56.027416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536312+0000","last_change":"2024-03-22T02:11:37.944107+0000","last_active":"2024-03-22T02:13:29.536312+0000","last_peered":"2024-03-22T02:13:29.536312+0000","last_clean":"2024-03-22T02:13:29.536312+0000","last_became_active":"2024-03-22T02:11:37.943624+0000","last_became_peered":"2024-03-22T02:11:37.943624+0000","last_unstale":"2024-03-22T02:13:29.536312+0000","last_undegraded":"2024-03-22T02:13:29.536312+0000","last_fullsized":"2024-03-22T02:13:29.536312+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:17:47.521899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235355+0000","last_change":"2024-03-22T02:11:32.910195+0000","last_active":"2024-03-22T02:13:29.235355+0000","last_peered":"2024-03-22T02:13:29.235355+0000","last_clean":"2024-03-22T02:13:29.235355+0000","last_became_active":"2024-03-22T02:11:32.909794+0000","last_became_peered":"2024-03-22T02:11:32.909794+0000","last_unstale":"2024-03-22T02:13:29.235355+0000","last_undegraded":"2024-03-22T02:13:29.235355+0000","last_fullsized":"2024-03-22T02:13:29.235355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:26:14.915242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231288+0000","last_change":"2024-03-22T02:11:33.637043+0000","last_active":"2024-03-22T02:13:29.231288+0000","last_peered":"2024-03-22T02:13:29.231288+0000","last_clean":"2024-03-22T02:13:29.231288+0000","last_became_active":"2024-03-22T02:11:33.636745+0000","last_became_peered":"2024-03-22T02:11:33.636745+0000","last_unstale":"2024-03-22T02:13:29.231288+0000","last_undegraded":"2024-03-22T02:13:29.231288+0000","last_fullsized":"2024-03-22T02:13:29.231288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:13:41.177242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234912+0000","last_change":"2024-03-22T02:11:37.934401+0000","last_active":"2024-03-22T02:13:29.234912+0000","last_peered":"2024-03-22T02:13:29.234912+0000","last_clean":"2024-03-22T02:13:29.234912+0000","last_became_active":"2024-03-22T02:11:37.933930+0000","last_became_peered":"2024-03-22T02:11:37.933930+0000","last_unstale":"2024-03-22T02:13:29.234912+0000","last_undegraded":"2024-03-22T02:13:29.234912+0000","last_fullsized":"2024-03-22T02:13:29.234912+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:38:15.310702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235141+0000","last_change":"2024-03-22T02:11:32.911728+0000","last_active":"2024-03-22T02:13:29.235141+0000","last_peered":"2024-03-22T02:13:29.235141+0000","last_clean":"2024-03-22T02:13:29.235141+0000","last_became_active":"2024-03-22T02:11:32.911313+0000","last_became_peered":"2024-03-22T02:11:32.911313+0000","last_unstale":"2024-03-22T02:13:29.235141+0000","last_undegraded":"2024-03-22T02:13:29.235141+0000","last_fullsized":"2024-03-22T02:13:29.235141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.694765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666733+0000","last_change":"2024-03-22T02:11:33.633296+0000","last_active":"2024-03-22T02:13:29.666733+0000","last_peered":"2024-03-22T02:13:29.666733+0000","last_clean":"2024-03-22T02:13:29.666733+0000","last_became_active":"2024-03-22T02:11:33.633011+0000","last_became_peered":"2024-03-22T02:11:33.633011+0000","last_unstale":"2024-03-22T02:13:29.666733+0000","last_undegraded":"2024-03-22T02:13:29.666733+0000","last_fullsized":"2024-03-22T02:13:29.666733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:32:31.070994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248014+0000","last_change":"2024-03-22T02:11:37.945094+0000","last_active":"2024-03-22T02:13:29.248014+0000","last_peered":"2024-03-22T02:13:29.248014+0000","last_clean":"2024-03-22T02:13:29.248014+0000","last_became_active":"2024-03-22T02:11:37.944691+0000","last_became_peered":"2024-03-22T02:11:37.944691+0000","last_unstale":"2024-03-22T02:13:29.248014+0000","last_undegraded":"2024-03-22T02:13:29.248014+0000","last_fullsized":"2024-03-22T02:13:29.248014+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:08:08.253026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'1","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232955+0000","last_change":"2024-03-22T02:11:32.910644+0000","last_active":"2024-03-22T02:13:29.232955+0000","last_peered":"2024-03-22T02:13:29.232955+0000","last_clean":"2024-03-22T02:13:29.232955+0000","last_became_active":"2024-03-22T02:11:32.909816+0000","last_became_peered":"2024-03-22T02:11:32.909816+0000","last_unstale":"2024-03-22T02:13:29.232955+0000","last_undegraded":"2024-03-22T02:13:29.232955+0000","last_fullsized":"2024-03-22T02:13:29.232955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:47:03.394970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.666435+0000","last_change":"2024-03-22T02:12:28.488917+0000","last_active":"2024-03-22T02:13:29.666435+0000","last_peered":"2024-03-22T02:13:29.666435+0000","last_clean":"2024-03-22T02:13:29.666435+0000","last_became_active":"2024-03-22T02:12:28.488542+0000","last_became_peered":"2024-03-22T02:12:28.488542+0000","last_unstale":"2024-03-22T02:13:29.666435+0000","last_undegraded":"2024-03-22T02:13:29.666435+0000","last_fullsized":"2024-03-22T02:13:29.666435+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:57:41.575475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248137+0000","last_change":"2024-03-22T02:11:37.947287+0000","last_active":"2024-03-22T02:13:29.248137+0000","last_peered":"2024-03-22T02:13:29.248137+0000","last_clean":"2024-03-22T02:13:29.248137+0000","last_became_active":"2024-03-22T02:11:37.946867+0000","last_became_peered":"2024-03-22T02:11:37.946867+0000","last_unstale":"2024-03-22T02:13:29.248137+0000","last_undegraded":"2024-03-22T02:13:29.248137+0000","last_fullsized":"2024-03-22T02:13:29.248137+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:59:06.391882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536336+0000","last_change":"2024-03-22T02:11:33.643014+0000","last_active":"2024-03-22T02:13:29.536336+0000","last_peered":"2024-03-22T02:13:29.536336+0000","last_clean":"2024-03-22T02:13:29.536336+0000","last_became_active":"2024-03-22T02:11:33.642708+0000","last_became_peered":"2024-03-22T02:11:33.642708+0000","last_unstale":"2024-03-22T02:13:29.536336+0000","last_undegraded":"2024-03-22T02:13:29.536336+0000","last_fullsized":"2024-03-22T02:13:29.536336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:48:24.540742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'5","reported_seq":154,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:50.138873+0000","last_change":"2024-03-22T02:11:32.902799+0000","last_active":"2024-03-22T02:13:50.138873+0000","last_peered":"2024-03-22T02:13:50.138873+0000","last_clean":"2024-03-22T02:13:50.138873+0000","last_became_active":"2024-03-22T02:11:32.902432+0000","last_became_peered":"2024-03-22T02:11:32.902432+0000","last_unstale":"2024-03-22T02:13:50.138873+0000","last_undegraded":"2024-03-22T02:13:50.138873+0000","last_fullsized":"2024-03-22T02:13:50.138873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:14.751324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232977+0000","last_change":"2024-03-22T02:11:37.941356+0000","last_active":"2024-03-22T02:13:29.232977+0000","last_peered":"2024-03-22T02:13:29.232977+0000","last_clean":"2024-03-22T02:13:29.232977+0000","last_became_active":"2024-03-22T02:11:37.941004+0000","last_became_peered":"2024-03-22T02:11:37.941004+0000","last_unstale":"2024-03-22T02:13:29.232977+0000","last_undegraded":"2024-03-22T02:13:29.232977+0000","last_fullsized":"2024-03-22T02:13:29.232977+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:29:10.545278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229511+0000","last_change":"2024-03-22T02:11:33.649267+0000","last_active":"2024-03-22T02:13:29.229511+0000","last_peered":"2024-03-22T02:13:29.229511+0000","last_clean":"2024-03-22T02:13:29.229511+0000","last_became_active":"2024-03-22T02:11:33.648935+0000","last_became_peered":"2024-03-22T02:11:33.648935+0000","last_unstale":"2024-03-22T02:13:29.229511+0000","last_undegraded":"2024-03-22T02:13:29.229511+0000","last_fullsized":"2024-03-22T02:13:29.229511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:42:29.881360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229686+0000","last_change":"2024-03-22T02:11:32.903394+0000","last_active":"2024-03-22T02:13:29.229686+0000","last_peered":"2024-03-22T02:13:29.229686+0000","last_clean":"2024-03-22T02:13:29.229686+0000","last_became_active":"2024-03-22T02:11:32.903052+0000","last_became_peered":"2024-03-22T02:11:32.903052+0000","last_unstale":"2024-03-22T02:13:29.229686+0000","last_undegraded":"2024-03-22T02:13:29.229686+0000","last_fullsized":"2024-03-22T02:13:29.229686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:04:58.920948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232583+0000","last_change":"2024-03-22T02:10:19.179259+0000","last_active":"2024-03-22T02:13:29.232583+0000","last_peered":"2024-03-22T02:13:29.232583+0000","last_clean":"2024-03-22T02:13:29.232583+0000","last_became_active":"2024-03-22T02:10:19.178725+0000","last_became_peered":"2024-03-22T02:10:19.178725+0000","last_unstale":"2024-03-22T02:13:29.232583+0000","last_undegraded":"2024-03-22T02:13:29.232583+0000","last_fullsized":"2024-03-22T02:13:29.232583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:06:54.600680+0000","last_clean_scrub_stamp":"2024-03-22T02:06:54.600680+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:00:53.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.667520+0000","last_change":"2024-03-22T02:13:29.667520+0000","last_active":"2024-03-22T02:13:29.667520+0000","last_peered":"2024-03-22T02:13:29.667520+0000","last_clean":"2024-03-22T02:13:29.667520+0000","last_became_active":"2024-03-22T02:13:29.666948+0000","last_became_peered":"2024-03-22T02:13:29.666948+0000","last_unstale":"2024-03-22T02:13:29.667520+0000","last_undegraded":"2024-03-22T02:13:29.667520+0000","last_fullsized":"2024-03-22T02:13:29.667520+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:00:56.014210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6,10],"acting":[8,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247578+0000","last_change":"2024-03-22T02:11:33.630070+0000","last_active":"2024-03-22T02:13:29.247578+0000","last_peered":"2024-03-22T02:13:29.247578+0000","last_clean":"2024-03-22T02:13:29.247578+0000","last_became_active":"2024-03-22T02:11:33.629707+0000","last_became_peered":"2024-03-22T02:11:33.629707+0000","last_unstale":"2024-03-22T02:13:29.247578+0000","last_undegraded":"2024-03-22T02:13:29.247578+0000","last_fullsized":"2024-03-22T02:13:29.247578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:47:46.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232655+0000","last_change":"2024-03-22T02:11:32.916184+0000","last_active":"2024-03-22T02:13:29.232655+0000","last_peered":"2024-03-22T02:13:29.232655+0000","last_clean":"2024-03-22T02:13:29.232655+0000","last_became_active":"2024-03-22T02:11:32.915872+0000","last_became_peered":"2024-03-22T02:11:32.915872+0000","last_unstale":"2024-03-22T02:13:29.232655+0000","last_undegraded":"2024-03-22T02:13:29.232655+0000","last_fullsized":"2024-03-22T02:13:29.232655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:57:06.456293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231152+0000","last_change":"2024-03-22T02:11:37.936336+0000","last_active":"2024-03-22T02:13:29.231152+0000","last_peered":"2024-03-22T02:13:29.231152+0000","last_clean":"2024-03-22T02:13:29.231152+0000","last_became_active":"2024-03-22T02:11:37.935899+0000","last_became_peered":"2024-03-22T02:11:37.935899+0000","last_unstale":"2024-03-22T02:13:29.231152+0000","last_undegraded":"2024-03-22T02:13:29.231152+0000","last_fullsized":"2024-03-22T02:13:29.231152+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:24:10.412736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.242062+0000","last_change":"2024-03-22T02:13:29.241667+0000","last_active":"2024-03-22T02:13:29.242062+0000","last_peered":"2024-03-22T02:13:29.242062+0000","last_clean":"2024-03-22T02:13:29.242062+0000","last_became_active":"2024-03-22T02:13:29.236378+0000","last_became_peered":"2024-03-22T02:13:29.236378+0000","last_unstale":"2024-03-22T02:13:29.242062+0000","last_undegraded":"2024-03-22T02:13:29.242062+0000","last_fullsized":"2024-03-22T02:13:29.242062+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:41:25.050034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248131+0000","last_change":"2024-03-22T02:11:33.628538+0000","last_active":"2024-03-22T02:13:29.248131+0000","last_peered":"2024-03-22T02:13:29.248131+0000","last_clean":"2024-03-22T02:13:29.248131+0000","last_became_active":"2024-03-22T02:11:33.628225+0000","last_became_peered":"2024-03-22T02:11:33.628225+0000","last_unstale":"2024-03-22T02:13:29.248131+0000","last_undegraded":"2024-03-22T02:13:29.248131+0000","last_fullsized":"2024-03-22T02:13:29.248131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:56:36.248921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229304+0000","last_change":"2024-03-22T02:11:37.945993+0000","last_active":"2024-03-22T02:13:29.229304+0000","last_peered":"2024-03-22T02:13:29.229304+0000","last_clean":"2024-03-22T02:13:29.229304+0000","last_became_active":"2024-03-22T02:11:37.945566+0000","last_became_peered":"2024-03-22T02:11:37.945566+0000","last_unstale":"2024-03-22T02:13:29.229304+0000","last_undegraded":"2024-03-22T02:13:29.229304+0000","last_fullsized":"2024-03-22T02:13:29.229304+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:46:22.943749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'10","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:44.917232+0000","last_change":"2024-03-22T02:11:32.909171+0000","last_active":"2024-03-22T02:13:44.917232+0000","last_peered":"2024-03-22T02:13:44.917232+0000","last_clean":"2024-03-22T02:13:44.917232+0000","last_became_active":"2024-03-22T02:11:32.908850+0000","last_became_peered":"2024-03-22T02:11:32.908850+0000","last_unstale":"2024-03-22T02:13:44.917232+0000","last_undegraded":"2024-03-22T02:13:44.917232+0000","last_fullsized":"2024-03-22T02:13:44.917232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:09:12.948953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229947+0000","last_change":"2024-03-22T02:11:33.633819+0000","last_active":"2024-03-22T02:13:29.229947+0000","last_peered":"2024-03-22T02:13:29.229947+0000","last_clean":"2024-03-22T02:13:29.229947+0000","last_became_active":"2024-03-22T02:11:33.633443+0000","last_became_peered":"2024-03-22T02:11:33.633443+0000","last_unstale":"2024-03-22T02:13:29.229947+0000","last_undegraded":"2024-03-22T02:13:29.229947+0000","last_fullsized":"2024-03-22T02:13:29.229947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:27:46.370278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231155+0000","last_change":"2024-03-22T02:11:37.930992+0000","last_active":"2024-03-22T02:13:29.231155+0000","last_peered":"2024-03-22T02:13:29.231155+0000","last_clean":"2024-03-22T02:13:29.231155+0000","last_became_active":"2024-03-22T02:11:37.930592+0000","last_became_peered":"2024-03-22T02:11:37.930592+0000","last_unstale":"2024-03-22T02:13:29.231155+0000","last_undegraded":"2024-03-22T02:13:29.231155+0000","last_fullsized":"2024-03-22T02:13:29.231155+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:16:12.171531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229873+0000","last_change":"2024-03-22T02:11:32.910370+0000","last_active":"2024-03-22T02:13:29.229873+0000","last_peered":"2024-03-22T02:13:29.229873+0000","last_clean":"2024-03-22T02:13:29.229873+0000","last_became_active":"2024-03-22T02:11:32.910062+0000","last_became_peered":"2024-03-22T02:11:32.910062+0000","last_unstale":"2024-03-22T02:13:29.229873+0000","last_undegraded":"2024-03-22T02:13:29.229873+0000","last_fullsized":"2024-03-22T02:13:29.229873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:42:47.376197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234699+0000","last_change":"2024-03-22T02:12:28.489608+0000","last_active":"2024-03-22T02:13:29.234699+0000","last_peered":"2024-03-22T02:13:29.234699+0000","last_clean":"2024-03-22T02:13:29.234699+0000","last_became_active":"2024-03-22T02:12:28.489005+0000","last_became_peered":"2024-03-22T02:12:28.489005+0000","last_unstale":"2024-03-22T02:13:29.234699+0000","last_undegraded":"2024-03-22T02:13:29.234699+0000","last_fullsized":"2024-03-22T02:13:29.234699+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:13:01.427654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,4],"acting":[0,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229642+0000","last_change":"2024-03-22T02:11:37.934956+0000","last_active":"2024-03-22T02:13:29.229642+0000","last_peered":"2024-03-22T02:13:29.229642+0000","last_clean":"2024-03-22T02:13:29.229642+0000","last_became_active":"2024-03-22T02:11:37.933489+0000","last_became_peered":"2024-03-22T02:11:37.933489+0000","last_unstale":"2024-03-22T02:13:29.229642+0000","last_undegraded":"2024-03-22T02:13:29.229642+0000","last_fullsized":"2024-03-22T02:13:29.229642+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:45:51.293559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234767+0000","last_change":"2024-03-22T02:11:32.908375+0000","last_active":"2024-03-22T02:13:29.234767+0000","last_peered":"2024-03-22T02:13:29.234767+0000","last_clean":"2024-03-22T02:13:29.234767+0000","last_became_active":"2024-03-22T02:11:32.908070+0000","last_became_peered":"2024-03-22T02:11:32.908070+0000","last_unstale":"2024-03-22T02:13:29.234767+0000","last_undegraded":"2024-03-22T02:13:29.234767+0000","last_fullsized":"2024-03-22T02:13:29.234767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:17:53.322631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234695+0000","last_change":"2024-03-22T02:11:33.630478+0000","last_active":"2024-03-22T02:13:29.234695+0000","last_peered":"2024-03-22T02:13:29.234695+0000","last_clean":"2024-03-22T02:13:29.234695+0000","last_became_active":"2024-03-22T02:11:33.629558+0000","last_became_peered":"2024-03-22T02:11:33.629558+0000","last_unstale":"2024-03-22T02:13:29.234695+0000","last_undegraded":"2024-03-22T02:13:29.234695+0000","last_fullsized":"2024-03-22T02:13:29.234695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:18:41.897066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235725+0000","last_change":"2024-03-22T02:11:37.940961+0000","last_active":"2024-03-22T02:13:29.235725+0000","last_peered":"2024-03-22T02:13:29.235725+0000","last_clean":"2024-03-22T02:13:29.235725+0000","last_became_active":"2024-03-22T02:11:37.940331+0000","last_became_peered":"2024-03-22T02:11:37.940331+0000","last_unstale":"2024-03-22T02:13:29.235725+0000","last_undegraded":"2024-03-22T02:13:29.235725+0000","last_fullsized":"2024-03-22T02:13:29.235725+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:54:10.444925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229456+0000","last_change":"2024-03-22T02:11:32.903153+0000","last_active":"2024-03-22T02:13:29.229456+0000","last_peered":"2024-03-22T02:13:29.229456+0000","last_clean":"2024-03-22T02:13:29.229456+0000","last_became_active":"2024-03-22T02:11:32.902716+0000","last_became_peered":"2024-03-22T02:11:32.902716+0000","last_unstale":"2024-03-22T02:13:29.229456+0000","last_undegraded":"2024-03-22T02:13:29.229456+0000","last_fullsized":"2024-03-22T02:13:29.229456+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:56:12.579228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230885+0000","last_change":"2024-03-22T02:11:33.640853+0000","last_active":"2024-03-22T02:13:29.230885+0000","last_peered":"2024-03-22T02:13:29.230885+0000","last_clean":"2024-03-22T02:13:29.230885+0000","last_became_active":"2024-03-22T02:11:33.640529+0000","last_became_peered":"2024-03-22T02:11:33.640529+0000","last_unstale":"2024-03-22T02:13:29.230885+0000","last_undegraded":"2024-03-22T02:13:29.230885+0000","last_fullsized":"2024-03-22T02:13:29.230885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:07:25.614671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":24,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229707+0000","last_change":"2024-03-22T02:11:37.935685+0000","last_active":"2024-03-22T02:13:29.229707+0000","last_peered":"2024-03-22T02:13:29.229707+0000","last_clean":"2024-03-22T02:13:29.229707+0000","last_became_active":"2024-03-22T02:11:37.935270+0000","last_became_peered":"2024-03-22T02:11:37.935270+0000","last_unstale":"2024-03-22T02:13:29.229707+0000","last_undegraded":"2024-03-22T02:13:29.229707+0000","last_fullsized":"2024-03-22T02:13:29.229707+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:27:20.513318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":15,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.239556+0000","last_change":"2024-03-22T02:13:29.239080+0000","last_active":"2024-03-22T02:13:29.239556+0000","last_peered":"2024-03-22T02:13:29.239556+0000","last_clean":"2024-03-22T02:13:29.239556+0000","last_became_active":"2024-03-22T02:13:29.234626+0000","last_became_peered":"2024-03-22T02:13:29.234626+0000","last_unstale":"2024-03-22T02:13:29.239556+0000","last_undegraded":"2024-03-22T02:13:29.239556+0000","last_fullsized":"2024-03-22T02:13:29.239556+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:21:19.317485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229714+0000","last_change":"2024-03-22T02:11:33.648198+0000","last_active":"2024-03-22T02:13:29.229714+0000","last_peered":"2024-03-22T02:13:29.229714+0000","last_clean":"2024-03-22T02:13:29.229714+0000","last_became_active":"2024-03-22T02:11:33.645276+0000","last_became_peered":"2024-03-22T02:11:33.645276+0000","last_unstale":"2024-03-22T02:13:29.229714+0000","last_undegraded":"2024-03-22T02:13:29.229714+0000","last_fullsized":"2024-03-22T02:13:29.229714+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:12:38.954223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229605+0000","last_change":"2024-03-22T02:11:37.937688+0000","last_active":"2024-03-22T02:13:29.229605+0000","last_peered":"2024-03-22T02:13:29.229605+0000","last_clean":"2024-03-22T02:13:29.229605+0000","last_became_active":"2024-03-22T02:11:37.937049+0000","last_became_peered":"2024-03-22T02:11:37.937049+0000","last_unstale":"2024-03-22T02:13:29.229605+0000","last_undegraded":"2024-03-22T02:13:29.229605+0000","last_fullsized":"2024-03-22T02:13:29.229605+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:58:55.312223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.235113+0000","last_change":"2024-03-22T02:11:32.917474+0000","last_active":"2024-03-22T02:13:29.235113+0000","last_peered":"2024-03-22T02:13:29.235113+0000","last_clean":"2024-03-22T02:13:29.235113+0000","last_became_active":"2024-03-22T02:11:32.917183+0000","last_became_peered":"2024-03-22T02:11:32.917183+0000","last_unstale":"2024-03-22T02:13:29.235113+0000","last_undegraded":"2024-03-22T02:13:29.235113+0000","last_fullsized":"2024-03-22T02:13:29.235113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:22:45.241417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233084+0000","last_change":"2024-03-22T02:11:33.635321+0000","last_active":"2024-03-22T02:13:29.233084+0000","last_peered":"2024-03-22T02:13:29.233084+0000","last_clean":"2024-03-22T02:13:29.233084+0000","last_became_active":"2024-03-22T02:11:33.634954+0000","last_became_peered":"2024-03-22T02:11:33.634954+0000","last_unstale":"2024-03-22T02:13:29.233084+0000","last_undegraded":"2024-03-22T02:13:29.233084+0000","last_fullsized":"2024-03-22T02:13:29.233084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:20:30.185180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234895+0000","last_change":"2024-03-22T02:11:37.933166+0000","last_active":"2024-03-22T02:13:29.234895+0000","last_peered":"2024-03-22T02:13:29.234895+0000","last_clean":"2024-03-22T02:13:29.234895+0000","last_became_active":"2024-03-22T02:11:37.932770+0000","last_became_peered":"2024-03-22T02:11:37.932770+0000","last_unstale":"2024-03-22T02:13:29.234895+0000","last_undegraded":"2024-03-22T02:13:29.234895+0000","last_fullsized":"2024-03-22T02:13:29.234895+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:21:24.941679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231835+0000","last_change":"2024-03-22T02:11:32.912871+0000","last_active":"2024-03-22T02:13:29.231835+0000","last_peered":"2024-03-22T02:13:29.231835+0000","last_clean":"2024-03-22T02:13:29.231835+0000","last_became_active":"2024-03-22T02:11:32.911758+0000","last_became_peered":"2024-03-22T02:11:32.911758+0000","last_unstale":"2024-03-22T02:13:29.231835+0000","last_undegraded":"2024-03-22T02:13:29.231835+0000","last_fullsized":"2024-03-22T02:13:29.231835+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:34:49.700242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233399+0000","last_change":"2024-03-22T02:11:33.637129+0000","last_active":"2024-03-22T02:13:29.233399+0000","last_peered":"2024-03-22T02:13:29.233399+0000","last_clean":"2024-03-22T02:13:29.233399+0000","last_became_active":"2024-03-22T02:11:33.636094+0000","last_became_peered":"2024-03-22T02:11:33.636094+0000","last_unstale":"2024-03-22T02:13:29.233399+0000","last_undegraded":"2024-03-22T02:13:29.233399+0000","last_fullsized":"2024-03-22T02:13:29.233399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T10:19:33.174294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234112+0000","last_change":"2024-03-22T02:11:37.944522+0000","last_active":"2024-03-22T02:13:29.234112+0000","last_peered":"2024-03-22T02:13:29.234112+0000","last_clean":"2024-03-22T02:13:29.234112+0000","last_became_active":"2024-03-22T02:11:37.944058+0000","last_became_peered":"2024-03-22T02:11:37.944058+0000","last_unstale":"2024-03-22T02:13:29.234112+0000","last_undegraded":"2024-03-22T02:13:29.234112+0000","last_fullsized":"2024-03-22T02:13:29.234112+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T04:15:32.484342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536753+0000","last_change":"2024-03-22T02:11:32.913926+0000","last_active":"2024-03-22T02:13:29.536753+0000","last_peered":"2024-03-22T02:13:29.536753+0000","last_clean":"2024-03-22T02:13:29.536753+0000","last_became_active":"2024-03-22T02:11:32.913634+0000","last_became_peered":"2024-03-22T02:11:32.913634+0000","last_unstale":"2024-03-22T02:13:29.536753+0000","last_undegraded":"2024-03-22T02:13:29.536753+0000","last_fullsized":"2024-03-22T02:13:29.536753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:28:03.631775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232917+0000","last_change":"2024-03-22T02:11:33.641862+0000","last_active":"2024-03-22T02:13:29.232917+0000","last_peered":"2024-03-22T02:13:29.232917+0000","last_clean":"2024-03-22T02:13:29.232917+0000","last_became_active":"2024-03-22T02:11:33.641511+0000","last_became_peered":"2024-03-22T02:11:33.641511+0000","last_unstale":"2024-03-22T02:13:29.232917+0000","last_undegraded":"2024-03-22T02:13:29.232917+0000","last_fullsized":"2024-03-22T02:13:29.232917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:59:21.976575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.536144+0000","last_change":"2024-03-22T02:11:37.933609+0000","last_active":"2024-03-22T02:13:29.536144+0000","last_peered":"2024-03-22T02:13:29.536144+0000","last_clean":"2024-03-22T02:13:29.536144+0000","last_became_active":"2024-03-22T02:11:37.933226+0000","last_became_peered":"2024-03-22T02:11:37.933226+0000","last_unstale":"2024-03-22T02:13:29.536144+0000","last_undegraded":"2024-03-22T02:13:29.536144+0000","last_fullsized":"2024-03-22T02:13:29.536144+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:36.910307+0000","last_clean_scrub_stamp":"2024-03-22T02:11:36.910307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:35:49.176877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248047+0000","last_change":"2024-03-22T02:11:32.907858+0000","last_active":"2024-03-22T02:13:29.248047+0000","last_peered":"2024-03-22T02:13:29.248047+0000","last_clean":"2024-03-22T02:13:29.248047+0000","last_became_active":"2024-03-22T02:11:32.907523+0000","last_became_peered":"2024-03-22T02:11:32.907523+0000","last_unstale":"2024-03-22T02:13:29.248047+0000","last_undegraded":"2024-03-22T02:13:29.248047+0000","last_fullsized":"2024-03-22T02:13:29.248047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:31.883160+0000","last_clean_scrub_stamp":"2024-03-22T02:11:31.883160+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T08:37:12.696841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.233098+0000","last_change":"2024-03-22T02:11:33.630128+0000","last_active":"2024-03-22T02:13:29.233098+0000","last_peered":"2024-03-22T02:13:29.233098+0000","last_clean":"2024-03-22T02:13:29.233098+0000","last_became_active":"2024-03-22T02:11:33.629797+0000","last_became_peered":"2024-03-22T02:11:33.629797+0000","last_unstale":"2024-03-22T02:13:29.233098+0000","last_undegraded":"2024-03-22T02:13:29.233098+0000","last_fullsized":"2024-03-22T02:13:29.233098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:45:55.215806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231002+0000","last_change":"2024-03-22T02:11:33.636272+0000","last_active":"2024-03-22T02:13:29.231002+0000","last_peered":"2024-03-22T02:13:29.231002+0000","last_clean":"2024-03-22T02:13:29.231002+0000","last_became_active":"2024-03-22T02:11:33.635947+0000","last_became_peered":"2024-03-22T02:11:33.635947+0000","last_unstale":"2024-03-22T02:13:29.231002+0000","last_undegraded":"2024-03-22T02:13:29.231002+0000","last_fullsized":"2024-03-22T02:13:29.231002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T12:14:23.332921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231480+0000","last_change":"2024-03-22T02:11:33.638531+0000","last_active":"2024-03-22T02:13:29.231480+0000","last_peered":"2024-03-22T02:13:29.231480+0000","last_clean":"2024-03-22T02:13:29.231480+0000","last_became_active":"2024-03-22T02:11:33.638208+0000","last_became_peered":"2024-03-22T02:11:33.638208+0000","last_unstale":"2024-03-22T02:13:29.231480+0000","last_undegraded":"2024-03-22T02:13:29.231480+0000","last_fullsized":"2024-03-22T02:13:29.231480+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T05:14:02.429536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231359+0000","last_change":"2024-03-22T02:12:28.489359+0000","last_active":"2024-03-22T02:13:29.231359+0000","last_peered":"2024-03-22T02:13:29.231359+0000","last_clean":"2024-03-22T02:13:29.231359+0000","last_became_active":"2024-03-22T02:12:28.489038+0000","last_became_peered":"2024-03-22T02:12:28.489038+0000","last_unstale":"2024-03-22T02:13:29.231359+0000","last_undegraded":"2024-03-22T02:13:29.231359+0000","last_fullsized":"2024-03-22T02:13:29.231359+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:24:15.886990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234341+0000","last_change":"2024-03-22T02:12:28.489164+0000","last_active":"2024-03-22T02:13:29.234341+0000","last_peered":"2024-03-22T02:13:29.234341+0000","last_clean":"2024-03-22T02:13:29.234341+0000","last_became_active":"2024-03-22T02:12:28.488707+0000","last_became_peered":"2024-03-22T02:12:28.488707+0000","last_unstale":"2024-03-22T02:13:29.234341+0000","last_undegraded":"2024-03-22T02:13:29.234341+0000","last_fullsized":"2024-03-22T02:13:29.234341+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:42:40.807687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248083+0000","last_change":"2024-03-22T02:11:33.643417+0000","last_active":"2024-03-22T02:13:29.248083+0000","last_peered":"2024-03-22T02:13:29.248083+0000","last_clean":"2024-03-22T02:13:29.248083+0000","last_became_active":"2024-03-22T02:11:33.642951+0000","last_became_peered":"2024-03-22T02:11:33.642951+0000","last_unstale":"2024-03-22T02:13:29.248083+0000","last_undegraded":"2024-03-22T02:13:29.248083+0000","last_fullsized":"2024-03-22T02:13:29.248083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T02:31:49.338933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232320+0000","last_change":"2024-03-22T02:11:33.637814+0000","last_active":"2024-03-22T02:13:29.232320+0000","last_peered":"2024-03-22T02:13:29.232320+0000","last_clean":"2024-03-22T02:13:29.232320+0000","last_became_active":"2024-03-22T02:11:33.637487+0000","last_became_peered":"2024-03-22T02:11:33.637487+0000","last_unstale":"2024-03-22T02:13:29.232320+0000","last_undegraded":"2024-03-22T02:13:29.232320+0000","last_fullsized":"2024-03-22T02:13:29.232320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T07:23:50.904535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.232088+0000","last_change":"2024-03-22T02:11:33.630648+0000","last_active":"2024-03-22T02:13:29.232088+0000","last_peered":"2024-03-22T02:13:29.232088+0000","last_clean":"2024-03-22T02:13:29.232088+0000","last_became_active":"2024-03-22T02:11:33.630346+0000","last_became_peered":"2024-03-22T02:11:33.630346+0000","last_unstale":"2024-03-22T02:13:29.232088+0000","last_undegraded":"2024-03-22T02:13:29.232088+0000","last_fullsized":"2024-03-22T02:13:29.232088+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:35:54.862477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.230024+0000","last_change":"2024-03-22T02:11:33.639334+0000","last_active":"2024-03-22T02:13:29.230024+0000","last_peered":"2024-03-22T02:13:29.230024+0000","last_clean":"2024-03-22T02:13:29.230024+0000","last_became_active":"2024-03-22T02:11:33.638980+0000","last_became_peered":"2024-03-22T02:11:33.638980+0000","last_unstale":"2024-03-22T02:13:29.230024+0000","last_undegraded":"2024-03-22T02:13:29.230024+0000","last_fullsized":"2024-03-22T02:13:29.230024+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:44:47.030361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.247487+0000","last_change":"2024-03-22T02:12:28.520917+0000","last_active":"2024-03-22T02:13:29.247487+0000","last_peered":"2024-03-22T02:13:29.247487+0000","last_clean":"2024-03-22T02:13:29.247487+0000","last_became_active":"2024-03-22T02:12:28.520533+0000","last_became_peered":"2024-03-22T02:12:28.520533+0000","last_unstale":"2024-03-22T02:13:29.247487+0000","last_undegraded":"2024-03-22T02:13:29.247487+0000","last_fullsized":"2024-03-22T02:13:29.247487+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T13:03:44.988896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,3],"acting":[6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.229453+0000","last_change":"2024-03-22T02:11:33.633779+0000","last_active":"2024-03-22T02:13:29.229453+0000","last_peered":"2024-03-22T02:13:29.229453+0000","last_clean":"2024-03-22T02:13:29.229453+0000","last_became_active":"2024-03-22T02:11:33.633175+0000","last_became_peered":"2024-03-22T02:11:33.633175+0000","last_unstale":"2024-03-22T02:13:29.229453+0000","last_undegraded":"2024-03-22T02:13:29.229453+0000","last_fullsized":"2024-03-22T02:13:29.229453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T03:41:12.078496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":25,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.234939+0000","last_change":"2024-03-22T02:12:28.521533+0000","last_active":"2024-03-22T02:13:29.234939+0000","last_peered":"2024-03-22T02:13:29.234939+0000","last_clean":"2024-03-22T02:13:29.234939+0000","last_became_active":"2024-03-22T02:12:28.521176+0000","last_became_peered":"2024-03-22T02:12:28.521176+0000","last_unstale":"2024-03-22T02:13:29.234939+0000","last_undegraded":"2024-03-22T02:13:29.234939+0000","last_fullsized":"2024-03-22T02:13:29.234939+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T14:10:11.909377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,3],"acting":[9,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":41,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.248177+0000","last_change":"2024-03-22T02:11:33.646112+0000","last_active":"2024-03-22T02:13:29.248177+0000","last_peered":"2024-03-22T02:13:29.248177+0000","last_clean":"2024-03-22T02:13:29.248177+0000","last_became_active":"2024-03-22T02:11:33.645787+0000","last_became_peered":"2024-03-22T02:11:33.645787+0000","last_unstale":"2024-03-22T02:13:29.248177+0000","last_undegraded":"2024-03-22T02:13:29.248177+0000","last_fullsized":"2024-03-22T02:13:29.248177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T06:39:48.213517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-22T02:13:29.231467+0000","last_change":"2024-03-22T02:11:33.634081+0000","last_active":"2024-03-22T02:13:29.231467+0000","last_peered":"2024-03-22T02:13:29.231467+0000","last_clean":"2024-03-22T02:13:29.231467+0000","last_became_active":"2024-03-22T02:11:33.633748+0000","last_became_peered":"2024-03-22T02:11:33.633748+0000","last_unstale":"2024-03-22T02:13:29.231467+0000","last_undegraded":"2024-03-22T02:13:29.231467+0000","last_fullsized":"2024-03-22T02:13:29.231467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T02:11:32.888137+0000","last_clean_scrub_stamp":"2024-03-22T02:11:32.888137+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T09:21:23.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":283816,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":74127,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":733,"num_read_kb":443,"num_write":175,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":22,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":222381,"data_compressed":4881,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351017,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":523008,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612654,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32324,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712828,"statfs":{"total":95995035648,"available":95961935872,"internally_reserved":0,"allocated":1175552,"data_stored":1179848,"data_compressed":64340,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776180,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":527318,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037818,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523120,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":1119891,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462979,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518398,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":582221,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1048576,"data_stored":519786,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":589867,"data_compressed":28248,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1036288,"data_stored":518592,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":523747,"data_compressed":26621,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1101824,"data_stored":1114780,"data_compressed":62713,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34452,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69286,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":71559,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":71172,"data_compressed":1627,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T02:13:56.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 -- 172.21.15.28:0/1718130048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40100ab0 msgr2=0x7fbb40100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 --2- 172.21.15.28:0/1718130048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40100ab0 0x7fbb40100ed0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbb30009900 tx=0x7fbb3002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 -- 172.21.15.28:0/1718130048 shutdown_connections 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 --2- 172.21.15.28:0/1718130048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40102660 0x7fbb40108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 --2- 172.21.15.28:0/1718130048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40101ca0 0x7fbb40102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 --2- 172.21.15.28:0/1718130048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40100ab0 0x7fbb40100ed0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 -- 172.21.15.28:0/1718130048 >> 172.21.15.28:0/1718130048 conn(0x7fbb400fc020 msgr2=0x7fbb400fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 -- 172.21.15.28:0/1718130048 shutdown_connections 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.380+0000 7fbb45855640 1 -- 172.21.15.28:0/1718130048 wait complete. 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 Processor -- start 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 -- start start 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40100ab0 0x7fbb40197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40102660 0x7fbb4019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4010c700 con 0x7fbb40101ca0 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4010c580 con 0x7fbb40100ab0 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb45855640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4010c880 con 0x7fbb40102660 2024-03-22T02:13:56.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60344/0 (socket says 172.21.15.28:60344) 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 -- 172.21.15.28:0/1961798038 learned_addr learned my addr 172.21.15.28:0/1961798038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb3f7fe640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40102660 0x7fbb4019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb3effd640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40100ab0 0x7fbb40197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 -- 172.21.15.28:0/1961798038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40102660 msgr2=0x7fbb4019f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40102660 0x7fbb4019f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 -- 172.21.15.28:0/1961798038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40100ab0 msgr2=0x7fbb40197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40100ab0 0x7fbb40197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.381+0000 7fbb37fff640 1 -- 172.21.15.28:0/1961798038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4019fb40 con 0x7fbb40101ca0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb37fff640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800cd80 con 0x7fbb40101ca0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb28010910 con 0x7fbb40101ca0 2024-03-22T02:13:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4019fe30 con 0x7fbb40101ca0 2024-03-22T02:13:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28018990 con 0x7fbb40101ca0 2024-03-22T02:13:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.382+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb401a0340 con 0x7fbb40101ca0 2024-03-22T02:13:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.383+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb401a0570 con 0x7fbb40101ca0 2024-03-22T02:13:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.384+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fbb28010ac0 con 0x7fbb40101ca0 2024-03-22T02:13:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.384+0000 7fbb44853640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 0x7fbb2c0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.384+0000 7fbb3effd640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 0x7fbb2c0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.385+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fbb280cfc30 con 0x7fbb40101ca0 2024-03-22T02:13:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.385+0000 7fbb3effd640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 0x7fbb2c0a54f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbb30000c00 tx=0x7fbb30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.389+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb2809be40 con 0x7fbb40101ca0 2024-03-22T02:13:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:56 smithi028 ceph-mon[31916]: pgmap v368: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-22T02:13:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:56 smithi028 ceph-mon[31916]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:56.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.575+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb401a0c70 con 0x7fbb40101ca0 2024-03-22T02:13:56.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.576+0000 7fbb44853640 1 -- 172.21.15.28:0/1961798038 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7fbb2809fee0 con 0x7fbb40101ca0 2024-03-22T02:13:56.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:56.577 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.578+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 msgr2=0x7fbb2c0a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.578+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 0x7fbb2c0a54f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbb30000c00 tx=0x7fbb30005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.578+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 msgr2=0x7fbb40198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.578+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800e9f0 tx=0x7fbb2800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 shutdown_connections 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbb2c0a3030 0x7fbb2c0a54f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb40102660 0x7fbb4019f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb40101ca0 0x7fbb40198460 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 --2- 172.21.15.28:0/1961798038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbb40100ab0 0x7fbb40197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 >> 172.21.15.28:0/1961798038 conn(0x7fbb400fc020 msgr2=0x7fbb400fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 shutdown_connections 2024-03-22T02:13:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.579+0000 7fbb45855640 1 -- 172.21.15.28:0/1961798038 wait complete. 2024-03-22T02:13:56.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:13:56.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:13:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 -- 172.21.15.28:0/2478629193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d241020c0 msgr2=0x7f9d241024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 --2- 172.21.15.28:0/2478629193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d241020c0 0x7f9d241024e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009900 tx=0x7f9d1c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 -- 172.21.15.28:0/2478629193 shutdown_connections 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 --2- 172.21.15.28:0/2478629193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d24100600 0x7f9d24100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 --2- 172.21.15.28:0/2478629193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d240ffc40 0x7f9d241000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 --2- 172.21.15.28:0/2478629193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d241020c0 0x7f9d241024e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 -- 172.21.15.28:0/2478629193 >> 172.21.15.28:0/2478629193 conn(0x7f9d240fb7c0 msgr2=0x7f9d240fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.696+0000 7f9d2b437640 1 -- 172.21.15.28:0/2478629193 shutdown_connections 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 -- 172.21.15.28:0/2478629193 wait complete. 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 Processor -- start 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 -- start start 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d240ffc40 0x7f9d2419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d241020c0 0x7f9d241a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410fa50 con 0x7f9d24100600 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410f8d0 con 0x7f9d240ffc40 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d2b437640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410fbd0 con 0x7f9d241020c0 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d289d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d289d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60364/0 (socket says 172.21.15.28:60364) 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d289d2640 1 -- 172.21.15.28:0/4104950076 learned_addr learned my addr 172.21.15.28:0/4104950076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d291d3640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d240ffc40 0x7f9d2419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.697+0000 7f9d299d4640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d241020c0 0x7f9d241a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 -- 172.21.15.28:0/4104950076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d241020c0 msgr2=0x7f9d241a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d241020c0 0x7f9d241a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 -- 172.21.15.28:0/4104950076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d240ffc40 msgr2=0x7f9d2419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d240ffc40 0x7f9d2419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 -- 172.21.15.28:0/4104950076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d241a1cf0 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d289d2640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800e9f0 tx=0x7f9d1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800cd80 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d18005500 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d241a1fe0 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18010690 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.698+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d241a2450 con 0x7f9d24100600 2024-03-22T02:13:56.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.699+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d2410bca0 con 0x7f9d24100600 2024-03-22T02:13:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.700+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f9d180046c0 con 0x7f9d24100600 2024-03-22T02:13:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.701+0000 7f9d227fc640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 0x7f9d0c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.701+0000 7f9d291d3640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 0x7f9d0c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.701+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9d180cfd60 con 0x7f9d24100600 2024-03-22T02:13:56.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.701+0000 7f9d291d3640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 0x7f9d0c0a5470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c000c00 tx=0x7f9d1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:56.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.705+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d1809c620 con 0x7f9d24100600 2024-03-22T02:13:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:56 smithi120 ceph-mon[36803]: pgmap v368: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-22T02:13:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:56 smithi120 ceph-mon[36803]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:56 smithi083 ceph-mon[36340]: pgmap v368: 129 pgs: 129 active+clean; 649 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 6 op/s 2024-03-22T02:13:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:56 smithi083 ceph-mon[36340]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:56.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.890+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d2402d3b0 con 0x7f9d24100600 2024-03-22T02:13:56.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.891+0000 7f9d227fc640 1 -- 172.21.15.28:0/4104950076 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f9d1809bf20 con 0x7f9d24100600 2024-03-22T02:13:56.891 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:13:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 msgr2=0x7f9d0c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 0x7f9d0c0a5470 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c000c00 tx=0x7f9d1c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 msgr2=0x7f9d2419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800e9f0 tx=0x7f9d1800edf0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 shutdown_connections 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d0c0a2fb0 0x7f9d0c0a5470 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d241020c0 0x7f9d241a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d24100600 0x7f9d2419a610 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 --2- 172.21.15.28:0/4104950076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d240ffc40 0x7f9d2419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 >> 172.21.15.28:0/4104950076 conn(0x7f9d240fb7c0 msgr2=0x7f9d240fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 shutdown_connections 2024-03-22T02:13:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:56.893+0000 7f9d2b437640 1 -- 172.21.15.28:0/4104950076 wait complete. 2024-03-22T02:13:56.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:13:57.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- 172.21.15.28:0/1738636437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354101bb0 msgr2=0x7fc354101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/1738636437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354101bb0 0x7fc354101fd0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc350009900 tx=0x7fc35002bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- 172.21.15.28:0/1738636437 shutdown_connections 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/1738636437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354103760 0x7fc354109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/1738636437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 0x7fc354103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/1738636437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354101bb0 0x7fc354101fd0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- 172.21.15.28:0/1738636437 >> 172.21.15.28:0/1738636437 conn(0x7fc3540fd120 msgr2=0x7fc3540ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- 172.21.15.28:0/1738636437 shutdown_connections 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- 172.21.15.28:0/1738636437 wait complete. 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 Processor -- start 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.010+0000 7fc35b4fb640 1 -- start start 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354101bb0 0x7fc354197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 0x7fc3541983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc35410d830 con 0x7fc354103760 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc35410d6b0 con 0x7fc354102da0 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc35410d9b0 con 0x7fc354101bb0 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60380/0 (socket says 172.21.15.28:60380) 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 -- 172.21.15.28:0/887165081 learned_addr learned my addr 172.21.15.28:0/887165081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359297640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354101bb0 0x7fc354197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc358a96640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 0x7fc3541983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 -- 172.21.15.28:0/887165081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354101bb0 msgr2=0x7fc354197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354101bb0 0x7fc354197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 -- 172.21.15.28:0/887165081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 msgr2=0x7fc3541983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 0x7fc3541983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 -- 172.21.15.28:0/887165081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc35419fac0 con 0x7fc354103760 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc359a98640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00eb10 tx=0x7fc34c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c004b20 con 0x7fc354103760 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc34c004cc0 con 0x7fc354103760 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc35419fdb0 con 0x7fc354103760 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc34c010760 con 0x7fc354103760 2024-03-22T02:13:57.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.011+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3541a0270 con 0x7fc354103760 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.013+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc354108f50 con 0x7fc354103760 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.014+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fc34c005020 con 0x7fc354103760 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.014+0000 7fc34a7fc640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 0x7fc3400a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.014+0000 7fc359297640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 0x7fc3400a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.014+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc34c0d0040 con 0x7fc354103760 2024-03-22T02:13:57.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.015+0000 7fc359297640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 0x7fc3400a5470 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc350005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.019+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc34c09c900 con 0x7fc354103760 2024-03-22T02:13:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.203+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc354062830 con 0x7fc354103760 2024-03-22T02:13:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.203+0000 7fc34a7fc640 1 -- 172.21.15.28:0/887165081 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc34c09c200 con 0x7fc354103760 2024-03-22T02:13:57.204 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:57.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 msgr2=0x7fc3400a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 0x7fc3400a5470 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc350000c00 tx=0x7fc350005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 msgr2=0x7fc35419f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00eb10 tx=0x7fc34c00c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 shutdown_connections 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc3400a2fb0 0x7fc3400a5470 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc354103760 0x7fc35419f450 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc354102da0 0x7fc3541983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 --2- 172.21.15.28:0/887165081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc354101bb0 0x7fc354197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 >> 172.21.15.28:0/887165081 conn(0x7fc3540fd120 msgr2=0x7fc354106120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 shutdown_connections 2024-03-22T02:13:57.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.205+0000 7fc35b4fb640 1 -- 172.21.15.28:0/887165081 wait complete. 2024-03-22T02:13:57.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 -- 172.21.15.28:0/1396023708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530101ba0 msgr2=0x7ff530101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 --2- 172.21.15.28:0/1396023708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530101ba0 0x7ff530101fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff528009900 tx=0x7ff52802bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 -- 172.21.15.28:0/1396023708 shutdown_connections 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 --2- 172.21.15.28:0/1396023708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530103750 0x7ff530109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 --2- 172.21.15.28:0/1396023708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 0x7ff530103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 --2- 172.21.15.28:0/1396023708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530101ba0 0x7ff530101fc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.322+0000 7ff534cfa640 1 -- 172.21.15.28:0/1396023708 >> 172.21.15.28:0/1396023708 conn(0x7ff5300fd110 msgr2=0x7ff5300ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- 172.21.15.28:0/1396023708 shutdown_connections 2024-03-22T02:13:57.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- 172.21.15.28:0/1396023708 wait complete. 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 Processor -- start 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- start start 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 0x7ff530197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 0x7ff530198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff53010d7f0 con 0x7ff530103750 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff53010d670 con 0x7ff530101ba0 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.323+0000 7ff534cfa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff53010d970 con 0x7ff530102d90 2024-03-22T02:13:57.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 0x7ff530197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60402/0 (socket says 172.21.15.28:60402) 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 -- 172.21.15.28:0/3024578288 learned_addr learned my addr 172.21.15.28:0/3024578288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52dd9b640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 0x7ff530198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 -- 172.21.15.28:0/3024578288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 msgr2=0x7ff530198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 0x7ff530198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 -- 172.21.15.28:0/3024578288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 msgr2=0x7ff530197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 0x7ff530197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 -- 172.21.15.28:0/3024578288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53019fb30 con 0x7ff530103750 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52e59c640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 0x7ff530197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff52ed9d640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff52000eb10 tx=0x7ff52000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520004b20 con 0x7ff530103750 2024-03-22T02:13:57.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff520004cc0 con 0x7ff530103750 2024-03-22T02:13:57.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff520010760 con 0x7ff530103750 2024-03-22T02:13:57.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53019fe20 con 0x7ff530103750 2024-03-22T02:13:57.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.324+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5301a0290 con 0x7ff530103750 2024-03-22T02:13:57.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.325+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff530108fe0 con 0x7ff530103750 2024-03-22T02:13:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.326+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7ff520005020 con 0x7ff530103750 2024-03-22T02:13:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.327+0000 7ff5277fe640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 0x7ff51c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.327+0000 7ff52e59c640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 0x7ff51c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.327+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff5200cfef0 con 0x7ff530103750 2024-03-22T02:13:57.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.327+0000 7ff52e59c640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 0x7ff51c0a5470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff528000c00 tx=0x7ff528005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.331+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff52009c7b0 con 0x7ff530103750 2024-03-22T02:13:57.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.514+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff53002d600 con 0x7ff530103750 2024-03-22T02:13:57.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.515+0000 7ff5277fe640 1 -- 172.21.15.28:0/3024578288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7ff5200bae20 con 0x7ff530103750 2024-03-22T02:13:57.515 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:13:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.516+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 msgr2=0x7ff51c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.516+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 0x7ff51c0a5470 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff528000c00 tx=0x7ff528005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.516+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 msgr2=0x7ff53019f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.516+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff52000eb10 tx=0x7ff52000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 shutdown_connections 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff51c0a2fb0 0x7ff51c0a5470 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff530103750 0x7ff53019f4c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff530102d90 0x7ff530198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 --2- 172.21.15.28:0/3024578288 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff530101ba0 0x7ff530197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 >> 172.21.15.28:0/3024578288 conn(0x7ff5300fd110 msgr2=0x7ff5301061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 shutdown_connections 2024-03-22T02:13:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.517+0000 7ff534cfa640 1 -- 172.21.15.28:0/3024578288 wait complete. 2024-03-22T02:13:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:57 smithi028 ceph-mon[31916]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:57 smithi028 ceph-mon[31916]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1961798038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4104950076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:57 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/887165081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:13:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 -- 172.21.15.28:0/2364987149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c105020 msgr2=0x7f677c107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 --2- 172.21.15.28:0/2364987149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c105020 0x7f677c107410 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f6770009ca0 tx=0x7f677002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 -- 172.21.15.28:0/2364987149 shutdown_connections 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 --2- 172.21.15.28:0/2364987149 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c107950 0x7f677c109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 --2- 172.21.15.28:0/2364987149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c105020 0x7f677c107410 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 --2- 172.21.15.28:0/2364987149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 0x7f677c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 -- 172.21.15.28:0/2364987149 >> 172.21.15.28:0/2364987149 conn(0x7f677c0fc040 msgr2=0x7f677c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 -- 172.21.15.28:0/2364987149 shutdown_connections 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.634+0000 7f678141c640 1 -- 172.21.15.28:0/2364987149 wait complete. 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 Processor -- start 2024-03-22T02:13:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 -- start start 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 0x7f677c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 0x7f677c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10d830 con 0x7f677c107950 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10d6b0 con 0x7f677c105020 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f678141c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f677c10d9b0 con 0x7f677c1026f0 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 0x7f677c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 0x7f677c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60408/0 (socket says 172.21.15.28:60408) 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 -- 172.21.15.28:0/2948877788 learned_addr learned my addr 172.21.15.28:0/2948877788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 -- 172.21.15.28:0/2948877788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 msgr2=0x7f677c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 0x7f677c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 -- 172.21.15.28:0/2948877788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 msgr2=0x7f677c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 0x7f677c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677b7fe640 1 -- 172.21.15.28:0/2948877788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677c19fb20 con 0x7f677c107950 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.635+0000 7f677a7fc640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 0x7f677c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f677b7fe640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f676c00e970 tx=0x7f676c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c00ccc0 con 0x7f677c107950 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677c19fe10 con 0x7f677c107950 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f676c00ce60 con 0x7f677c107950 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676c010490 con 0x7f677c107950 2024-03-22T02:13:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.636+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677c1a0280 con 0x7f677c107950 2024-03-22T02:13:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.637+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677c0fdb30 con 0x7f677c107950 2024-03-22T02:13:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.638+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f676c005020 con 0x7f677c107950 2024-03-22T02:13:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.639+0000 7f6763fff640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 0x7f67640a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.639+0000 7f677affd640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 0x7f67640a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.639+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f676c0cfdd0 con 0x7f677c107950 2024-03-22T02:13:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.639+0000 7f677affd640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 0x7f67640a5470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6774009bb0 tx=0x7f6774005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.644+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f676c09c690 con 0x7f677c107950 2024-03-22T02:13:57.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:57 smithi120 ceph-mon[36803]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:57 smithi120 ceph-mon[36803]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1961798038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:57.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4104950076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:57 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/887165081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:57 smithi083 ceph-mon[36340]: from='client.15549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:57 smithi083 ceph-mon[36340]: from='client.15555 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T02:13:57.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1961798038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:57.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4104950076' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:57 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/887165081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:13:57.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.829+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f677c02d5b0 con 0x7f677c107950 2024-03-22T02:13:57.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.830+0000 7f6763fff640 1 -- 172.21.15.28:0/2948877788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f676c09bf90 con 0x7f677c107950 2024-03-22T02:13:57.830 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 msgr2=0x7f67640a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 0x7f67640a5470 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f6774009bb0 tx=0x7f6774005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 msgr2=0x7f677c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f676c00e970 tx=0x7f676c00ed70 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 shutdown_connections 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67640a2fb0 0x7f67640a5470 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f677c107950 0x7f677c19f4b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f677c105020 0x7f677c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 --2- 172.21.15.28:0/2948877788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f677c1026f0 0x7f677c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 >> 172.21.15.28:0/2948877788 conn(0x7f677c0fc040 msgr2=0x7f677c103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 shutdown_connections 2024-03-22T02:13:57.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.832+0000 7f678141c640 1 -- 172.21.15.28:0/2948877788 wait complete. 2024-03-22T02:13:57.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 -- 172.21.15.28:0/2543705601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100ab0 msgr2=0x7f132c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 --2- 172.21.15.28:0/2543705601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100ab0 0x7f132c100ed0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f131c009900 tx=0x7f131c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 -- 172.21.15.28:0/2543705601 shutdown_connections 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 --2- 172.21.15.28:0/2543705601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c102660 0x7f132c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 --2- 172.21.15.28:0/2543705601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 0x7f132c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 --2- 172.21.15.28:0/2543705601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c100ab0 0x7f132c100ed0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 -- 172.21.15.28:0/2543705601 >> 172.21.15.28:0/2543705601 conn(0x7f132c0fc040 msgr2=0x7f132c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:57.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 -- 172.21.15.28:0/2543705601 shutdown_connections 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.946+0000 7f133191e640 1 -- 172.21.15.28:0/2543705601 wait complete. 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 Processor -- start 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 -- start start 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c100ab0 0x7f132c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 0x7f132c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c720 con 0x7f132c102660 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c5a0 con 0x7f132c100ab0 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f133191e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f132c10c8a0 con 0x7f132c101ca0 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60418/0 (socket says 172.21.15.28:60418) 2024-03-22T02:13:57.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 -- 172.21.15.28:0/4131067873 learned_addr learned my addr 172.21.15.28:0/4131067873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f13227fc640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 0x7f132c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132affd640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c100ab0 0x7f132c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 -- 172.21.15.28:0/4131067873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 msgr2=0x7f132c198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 0x7f132c198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 -- 172.21.15.28:0/4131067873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c100ab0 msgr2=0x7f132c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c100ab0 0x7f132c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.947+0000 7f132b7fe640 1 -- 172.21.15.28:0/4131067873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c19fb20 con 0x7f132c102660 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f132b7fe640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f132400eb10 tx=0x7f132400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324004b20 con 0x7f132c102660 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1324004cc0 con 0x7f132c102660 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1324010760 con 0x7f132c102660 2024-03-22T02:13:57.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c19fe10 con 0x7f132c102660 2024-03-22T02:13:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.948+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1a0280 con 0x7f132c102660 2024-03-22T02:13:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.949+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c1a0540 con 0x7f132c102660 2024-03-22T02:13:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.950+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f1324010900 con 0x7f132c102660 2024-03-22T02:13:57.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.950+0000 7f133091c640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 0x7f13180a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.951+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f13240d0130 con 0x7f132c102660 2024-03-22T02:13:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.951+0000 7f132affd640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 0x7f13180a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.951+0000 7f132affd640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 0x7f13180a5470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f131c000c00 tx=0x7f131c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:57.954+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1324053740 con 0x7f132c102660 2024-03-22T02:13:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.176+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f132c1a0710 con 0x7f132c102660 2024-03-22T02:13:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.177+0000 7f133091c640 1 -- 172.21.15.28:0/4131067873 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11866 (secure 0 0 0) 0x7f132409c9f0 con 0x7f132c102660 2024-03-22T02:13:58.178 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:58.178 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:13:50.358406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:13:58.179 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-03-22T02:13:58.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 msgr2=0x7f13180a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 0x7f13180a5470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f131c000c00 tx=0x7f131c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 msgr2=0x7f132c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f132400eb10 tx=0x7f132400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 shutdown_connections 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13180a2fb0 0x7f13180a5470 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f132c102660 0x7f132c19f4b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f132c101ca0 0x7f132c198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 --2- 172.21.15.28:0/4131067873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f132c100ab0 0x7f132c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 >> 172.21.15.28:0/4131067873 conn(0x7f132c0fc040 msgr2=0x7f132c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 shutdown_connections 2024-03-22T02:13:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.179+0000 7f133191e640 1 -- 172.21.15.28:0/4131067873 wait complete. 2024-03-22T02:13:58.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 -- 172.21.15.28:0/2130735630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 msgr2=0x7f0a80103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 --2- 172.21.15.28:0/2130735630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a80103220 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009900 tx=0x7f0a7402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 -- 172.21.15.28:0/2130735630 shutdown_connections 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 --2- 172.21.15.28:0/2130735630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80103760 0x7f0a80109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 --2- 172.21.15.28:0/2130735630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a80103220 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 --2- 172.21.15.28:0/2130735630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80101bb0 0x7f0a80101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 -- 172.21.15.28:0/2130735630 >> 172.21.15.28:0/2130735630 conn(0x7f0a800fd140 msgr2=0x7f0a800ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 -- 172.21.15.28:0/2130735630 shutdown_connections 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.294+0000 7f0a85d32640 1 -- 172.21.15.28:0/2130735630 wait complete. 2024-03-22T02:13:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 Processor -- start 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 -- start start 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80101bb0 0x7f0a80197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80103760 0x7f0a8019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8010d810 con 0x7f0a80102da0 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8010d690 con 0x7f0a80101bb0 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a85d32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8010d990 con 0x7f0a80103760 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a7effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60436/0 (socket says 172.21.15.28:60436) 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a7effd640 1 -- 172.21.15.28:0/3618413683 learned_addr learned my addr 172.21.15.28:0/3618413683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a7ffff640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80103760 0x7f0a8019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.295+0000 7f0a7f7fe640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80101bb0 0x7f0a80197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 -- 172.21.15.28:0/3618413683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80103760 msgr2=0x7f0a8019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80103760 0x7f0a8019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 -- 172.21.15.28:0/3618413683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80101bb0 msgr2=0x7f0a80197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80101bb0 0x7f0a80197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 -- 172.21.15.28:0/3618413683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8019fad0 con 0x7f0a80102da0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7effd640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009230 tx=0x7f0a7402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74041070 con 0x7f0a80102da0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a74004450 con 0x7f0a80102da0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8019fd60 con 0x7f0a80102da0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74003e20 con 0x7f0a80102da0 2024-03-22T02:13:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.296+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a801a0240 con 0x7f0a80102da0 2024-03-22T02:13:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.297+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a80108ed0 con 0x7f0a80102da0 2024-03-22T02:13:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.298+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f0a74044070 con 0x7f0a80102da0 2024-03-22T02:13:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.298+0000 7f0a7cff9640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 0x7f0a680a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.299+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0a740f1330 con 0x7f0a80102da0 2024-03-22T02:13:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.299+0000 7f0a7f7fe640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 0x7f0a680a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.299+0000 7f0a7f7fe640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 0x7f0a680a54f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a80102c00 tx=0x7f0a70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.303+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a740bd6b0 con 0x7f0a80102da0 2024-03-22T02:13:58.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.489+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a8002d620 con 0x7f0a80102da0 2024-03-22T02:13:58.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.491+0000 7f0a7cff9640 1 -- 172.21.15.28:0/3618413683 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+24903 (secure 0 0 0) 0x7f0a740a80c0 con 0x7f0a80102da0 2024-03-22T02:13:58.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:13:58.491 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:13:29.222546+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:13:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 msgr2=0x7f0a680a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 0x7f0a680a54f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0a80102c00 tx=0x7f0a70005f90 comp rx=0 tx=0).stop 2024-03-22T02:13:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 msgr2=0x7f0a801983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0a74009230 tx=0x7f0a7402c650 comp rx=0 tx=0).stop 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 shutdown_connections 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a680a3030 0x7f0a680a54f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a80103760 0x7f0a8019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a80102da0 0x7f0a801983f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 --2- 172.21.15.28:0/3618413683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a80101bb0 0x7f0a80197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 >> 172.21.15.28:0/3618413683 conn(0x7f0a800fd140 msgr2=0x7f0a801060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 shutdown_connections 2024-03-22T02:13:58.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.493+0000 7f0a85d32640 1 -- 172.21.15.28:0/3618413683 wait complete. 2024-03-22T02:13:58.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:58 smithi028 ceph-mon[31916]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:13:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3024578288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2948877788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4131067873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:58.505 DEBUG:tasks.fwd_scrub:Yielding 2024-03-22T02:13:58.506 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-22T02:13:58.517 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:13:58.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-22T02:13:58.517 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-22T02:13:58.518 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:13:58.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:13:58.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:13:58.521 INFO:teuthology.run_tasks:Running task workunit... 2024-03-22T02:13:58.536 INFO:tasks.workunit:Pulling workunits from ref c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:58.536 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-22T02:13:58.536 INFO:tasks.workunit:timeout=3h 2024-03-22T02:13:58.536 INFO:tasks.workunit:cleanup=True 2024-03-22T02:13:58.537 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:58.551 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-03-22 02:11:56.419638540 +0000 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-03-22 02:11:56.419638540 +0000 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-03-22 02:13:49.299887726 +0000 2024-03-22T02:13:58.552 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-03-22 02:11:56.419638540 +0000 2024-03-22T02:13:58.552 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-22T02:13:58.553 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-22T02:13:58.624 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- 172.21.15.28:0/2879577308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0101c90 msgr2=0x7f7dc0102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/2879577308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0101c90 0x7f7dc0102110 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009900 tx=0x7f7db402bc30 comp rx=0 tx=0).stop 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- 172.21.15.28:0/2879577308 shutdown_connections 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/2879577308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0102650 0x7f7dc0108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/2879577308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0101c90 0x7f7dc0102110 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/2879577308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- 172.21.15.28:0/2879577308 >> 172.21.15.28:0/2879577308 conn(0x7f7dc00fc030 msgr2=0x7f7dc00fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- 172.21.15.28:0/2879577308 shutdown_connections 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- 172.21.15.28:0/2879577308 wait complete. 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 Processor -- start 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- start start 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0101c90 0x7f7dc0198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 0x7f7dc019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc010c700 con 0x7f7dc0100aa0 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc010c580 con 0x7f7dc0102650 2024-03-22T02:13:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.627+0000 7f7dc79c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc010c880 con 0x7f7dc0101c90 2024-03-22T02:13:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5f66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 0x7f7dc019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc4f64640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0101c90 0x7f7dc0198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60448/0 (socket says 172.21.15.28:60448) 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 -- 172.21.15.28:0/4263809022 learned_addr learned my addr 172.21.15.28:0/4263809022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 -- 172.21.15.28:0/4263809022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0101c90 msgr2=0x7f7dc0198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0101c90 0x7f7dc0198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 -- 172.21.15.28:0/4263809022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 msgr2=0x7f7dc019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 0x7f7dc019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 -- 172.21.15.28:0/4263809022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dc019faf0 con 0x7f7dc0100aa0 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 0x7f7dc019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc5765640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7db800e9c0 tx=0x7f7db800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dbe7fc640 1 -- 172.21.15.28:0/4263809022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db800cde0 con 0x7f7dc0100aa0 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dbe7fc640 1 -- 172.21.15.28:0/4263809022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db8010910 con 0x7f7dc0100aa0 2024-03-22T02:13:58.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dbe7fc640 1 -- 172.21.15.28:0/4263809022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db8018a00 con 0x7f7dc0100aa0 2024-03-22T02:13:58.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dc019fde0 con 0x7f7dc0100aa0 2024-03-22T02:13:58.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.628+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dc01a0250 con 0x7f7dc0100aa0 2024-03-22T02:13:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.630+0000 7f7dbe7fc640 1 -- 172.21.15.28:0/4263809022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f7db801f020 con 0x7f7dc0100aa0 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 Processor -- start 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 -- start start 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 0x55823d1fafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55823d15f570 con 0x55823d200720 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55823d107e10 con 0x55823d1fabc0 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55823d15b520 con 0x55823d1fb5b0 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.631+0000 7f7dbe7fc640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 0x7f7da80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc4f64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 0x7f7da80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc4f64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60464/0 (socket says 172.21.15.28:60464) 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58978/0 (socket says 172.21.15.28:58978) 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc4f64640 1 -- 172.21.15.28:0/1772183099 learned_addr learned my addr 172.21.15.28:0/1772183099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 0x55823d1fafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dbe7fc640 1 -- 172.21.15.28:0/4263809022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7db80cfb50 con 0x7f7dc0100aa0 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 -- 172.21.15.28:0/1772183099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 msgr2=0x55823d1fafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 0x55823d1fafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 -- 172.21.15.28:0/1772183099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 msgr2=0x55823d202bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 -- 172.21.15.28:0/1772183099 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55823cf3b7e0 con 0x55823d1fb5b0 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 0x55823d1fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5765640 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7db80cd090 tx=0x7f7db806e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc497c5bc7e4790b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 0x7f7da80a54f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55823d1fbe10 tx=0x7f7db0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7da77fe640 1 -- 172.21.15.28:0/1772183099 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db80120f0 con 0x55823d1fb5b0 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7da77fe640 1 -- 172.21.15.28:0/1772183099 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db809c3b0 con 0x55823d1fb5b0 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.632+0000 7f7da77fe640 1 -- 172.21.15.28:0/1772183099 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db80cd2e0 con 0x55823d1fb5b0 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1772183099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 msgr2=0x55823d2000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7db80cd090 tx=0x7f7db806e9e0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1772183099 shutdown_connections 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d202bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fb5b0 0x55823d2000b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1772183099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fabc0 0x55823d1fafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1772183099 >> 172.21.15.28:0/1772183099 conn(0x55823d1f6130 msgr2=0x55823d1f6530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1772183099 shutdown_connections 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1772183099 wait complete. 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.633+0000 7f7dca5b1740 1 -- start start 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 Processor -- start 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d204da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d2052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d20c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55823d2036d0 con 0x55823d200720 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55823d11b110 con 0x55823d1fb5b0 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55823d203850 con 0x55823d1fabc0 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dca5b1740 10 client.0 _ll_register_callbacks cb 0x55823d157be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d204da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d20c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d204da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58982/0 (socket says 172.21.15.28:58982) 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5765640 1 -- 172.21.15.28:0/2618319893 learned_addr learned my addr 172.21.15.28:0/2618319893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d2052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 -- 172.21.15.28:0/2618319893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 msgr2=0x55823d204da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d204da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 -- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 msgr2=0x55823d2052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d2052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc5f66640 1 -- 172.21.15.28:0/2618319893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55823cf9bf70 con 0x55823d200720 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.634+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d2052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d20c320 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7db00099e0 tx=0x7f7db0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459149aef4b5278b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0019040 con 0x55823d200720 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7db0004b20 con 0x55823d200720 2024-03-22T02:13:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db809c3b0 con 0x55823d200720 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7db0034050 con 0x55823d200720 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 10 client.15609 fetch_fsmap learned FSMap version 25 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55823ce33170 con 0x55823d200720 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db809c3b0 con 0x55823d200720 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7db00051e0 con 0x55823d200720 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 10 client.15609 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 10 client.15609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 20 client.15609 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d210d80 0x55823d2131c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 4 client.15609 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.635+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55823d16cab0 con 0x55823d210d80 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.636+0000 7f7dc5765640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d210d80 0x55823d2131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.636+0000 7f7da4ff9640 10 client.15609 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.636+0000 7f7dc5765640 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d210d80 0x55823d2131c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3d5a4edaf2af0fd server_cookie=55630e38deeaa5a9 in_seq=0 out_seq=0 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7da4ff9640 1 -- 172.21.15.28:0/2618319893 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55823d16cab0 con 0x55823d210d80 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7da4ff9640 10 client.15609 handle_command_reply: tid=0 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 client.15609 shutdown 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 msgr2=0x55823d20c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d20c320 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7db00099e0 tx=0x7f7db0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 shutdown_connections 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d210d80 0x55823d2131c0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d20c320 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d2052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/2618319893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d204da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 >> 172.21.15.28:0/2618319893 conn(0x55823d1f6130 msgr2=0x55823d1f53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7da4ff9640 0 client.15609 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 shutdown_connections 2024-03-22T02:13:58.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 1 -- 172.21.15.28:0/2618319893 wait complete. 2024-03-22T02:13:58.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.637+0000 7f7dca5b1740 20 client.15609 trim_cache size 0 max 16384 2024-03-22T02:13:58.646 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 Processor -- start 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 -- start start 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d1f4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d20c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01ad200 con 0x55823d200720 2024-03-22T02:13:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01aaad0 con 0x55823d1fb5b0 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01a8920 con 0x55823d1fabc0 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc5f66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d1f4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc5f66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60476/0 (socket says 172.21.15.28:60476) 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d1f4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58984/0 (socket says 172.21.15.28:58984) 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc5f66640 1 -- 172.21.15.28:0/3970094430 learned_addr learned my addr 172.21.15.28:0/3970094430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.655+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d20c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 -- 172.21.15.28:0/3970094430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 msgr2=0x55823d1f4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d1f4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 -- 172.21.15.28:0/3970094430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 msgr2=0x55823d20c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d20c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 -- 172.21.15.28:0/3970094430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55823d1fb990 con 0x55823d200720 2024-03-22T02:13:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d20c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7db000fe30 tx=0x7f7db0010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24d6f782a6f64c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7da6ffd640 1 -- 172.21.15.28:0/3970094430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0019040 con 0x55823d200720 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7da6ffd640 1 -- 172.21.15.28:0/3970094430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db001fc00 con 0x55823d200720 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.656+0000 7f7da6ffd640 1 -- 172.21.15.28:0/3970094430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0005240 con 0x55823d200720 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- 172.21.15.28:0/3970094430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 msgr2=0x55823d1f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7db000fe30 tx=0x7f7db0010800 comp rx=0 tx=0).stop 2024-03-22T02:13:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- 172.21.15.28:0/3970094430 shutdown_connections 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d200720 0x55823d1f50a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d1fb5b0 0x55823d20c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/3970094430 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d1f4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- 172.21.15.28:0/3970094430 >> 172.21.15.28:0/3970094430 conn(0x55823d1f6130 msgr2=0x55823d1f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- 172.21.15.28:0/3970094430 shutdown_connections 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- 172.21.15.28:0/3970094430 wait complete. 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- start start 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 Processor -- start 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d209280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 0x55823d2750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01a8f90 con 0x55823d1fb5b0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01aa400 con 0x55823d200720 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dc01a9660 con 0x55823d1fabc0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.657+0000 7f7dca5b1740 10 client.0 _ll_register_callbacks cb 0x7f7db0021410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc4f64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc4f64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60480/0 (socket says 172.21.15.28:60480) 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5765640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d209280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 0x55823d2750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc4f64640 1 -- 172.21.15.28:0/1889816138 learned_addr learned my addr 172.21.15.28:0/1889816138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 -- 172.21.15.28:0/1889816138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 msgr2=0x55823d209280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d209280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 -- 172.21.15.28:0/1889816138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 msgr2=0x55823d2097c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 -- 172.21.15.28:0/1889816138 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55823d275670 con 0x55823d200720 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc4f64640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dc5f66640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 0x55823d2750b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7db0006010 tx=0x7f7db00056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11612da64073b91c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7da4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0045020 con 0x55823d200720 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7db001f920 con 0x55823d200720 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7db001fc00 con 0x55823d200720 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7db0050050 con 0x55823d200720 2024-03-22T02:13:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.658+0000 7f7dca5b1740 10 client.25457 fetch_fsmap learned FSMap version 25 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55823d2769e0 con 0x55823d200720 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db001fc00 con 0x55823d200720 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7db00416b0 con 0x55823d200720 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 10 client.25457 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 10 client.25457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 20 client.25457 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d277350 0x55823d279770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 4 client.25457 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:13:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55823d166e00 con 0x55823d277350 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dc5765640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d277350 0x55823d279770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7dc5765640 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d277350 0x55823d279770 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9a02f1f006d3725 server_cookie=eaa415944a7cbb99 in_seq=0 out_seq=0 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.659+0000 7f7da4ff9640 10 client.25457 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7da4ff9640 1 -- 172.21.15.28:0/1889816138 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55823d166e00 con 0x55823d277350 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7da4ff9640 10 client.25457 handle_command_reply: tid=0 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 client.25457 shutdown 2024-03-22T02:13:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 msgr2=0x55823d2750b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 0x55823d2750b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7db0006010 tx=0x7f7db00056d0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 shutdown_connections 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55823d277350 0x55823d279770 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55823d200720 0x55823d2750b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55823d1fb5b0 0x55823d2097c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 --2- 172.21.15.28:0/1889816138 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55823d1fabc0 0x55823d209280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 >> 172.21.15.28:0/1889816138 conn(0x55823d1f6130 msgr2=0x55823d203c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.660+0000 7f7da4ff9640 0 client.25457 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 shutdown_connections 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dca5b1740 1 -- 172.21.15.28:0/1889816138 wait complete. 2024-03-22T02:13:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dca5b1740 20 client.25457 trim_cache size 0 max 16384 2024-03-22T02:13:58.671 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:58.671 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:13:58.671 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fbb4abef-8d68-49bc-8498-6a71674379e8", 2024-03-22T02:13:58.671 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:13:58.671 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:58.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 msgr2=0x7f7da80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 0x7f7da80a54f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55823d1fbe10 tx=0x7f7db0004a40 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 msgr2=0x7f7dc0197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7db800e9c0 tx=0x7f7db800ee80 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 shutdown_connections 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7da80a3030 0x7f7da80a54f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dc0102650 0x7f7dc019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dc0101c90 0x7f7dc0198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 --2- 172.21.15.28:0/4263809022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dc0100aa0 0x7f7dc0197f20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 >> 172.21.15.28:0/4263809022 conn(0x7f7dc00fc030 msgr2=0x7f7dc00fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 shutdown_connections 2024-03-22T02:13:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.661+0000 7f7dc79c9640 1 -- 172.21.15.28:0/4263809022 wait complete. 2024-03-22T02:13:58.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:13:58.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:13:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:58 smithi120 ceph-mon[36803]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:13:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3024578288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2948877788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4131067873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:58 smithi083 ceph-mon[36340]: pgmap v369: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:13:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3024578288' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2948877788' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:13:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4131067873' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:13:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- 172.21.15.28:0/2389959463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 msgr2=0x7f6374108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 --2- 172.21.15.28:0/2389959463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f6374108ed0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f636c00bcc0 tx=0x7f636c009260 comp rx=0 tx=0).stop 2024-03-22T02:13:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- 172.21.15.28:0/2389959463 shutdown_connections 2024-03-22T02:13:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 --2- 172.21.15.28:0/2389959463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f6374108ed0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 --2- 172.21.15.28:0/2389959463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374101c80 0x7f6374102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 --2- 172.21.15.28:0/2389959463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374100a90 0x7f6374100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- 172.21.15.28:0/2389959463 >> 172.21.15.28:0/2389959463 conn(0x7f63740fc020 msgr2=0x7f63740fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- 172.21.15.28:0/2389959463 shutdown_connections 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- 172.21.15.28:0/2389959463 wait complete. 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 Processor -- start 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.829+0000 7f637a44e640 1 -- start start 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 0x7f6374195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f637410b610 con 0x7f6374102640 2024-03-22T02:13:58.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f637410b490 con 0x7f6374100a90 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f637a44e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f637410b790 con 0x7f6374101c80 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63737fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f6373fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 0x7f6374195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60502/0 (socket says 172.21.15.28:60502) 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63737fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59012/0 (socket says 172.21.15.28:59012) 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 -- 172.21.15.28:0/1731661305 learned_addr learned my addr 172.21.15.28:0/1731661305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 -- 172.21.15.28:0/1731661305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 msgr2=0x7f63741961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 -- 172.21.15.28:0/1731661305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 msgr2=0x7f6374195c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 0x7f6374195c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 -- 172.21.15.28:0/1731661305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637419d870 con 0x7f6374102640 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63737fe640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f6373fff640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 0x7f6374195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.830+0000 7f63789eb640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f636c035fd0 tx=0x7f636c035b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.831+0000 7f63717fa640 1 -- 172.21.15.28:0/1731661305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c043070 con 0x7f6374102640 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.831+0000 7f63717fa640 1 -- 172.21.15.28:0/1731661305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636c004040 con 0x7f6374102640 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.831+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f637419db00 con 0x7f6374102640 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.831+0000 7f63717fa640 1 -- 172.21.15.28:0/1731661305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c03f680 con 0x7f6374102640 2024-03-22T02:13:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.831+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f637419e050 con 0x7f6374102640 2024-03-22T02:13:58.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f63717fa640 1 -- 172.21.15.28:0/1731661305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f636c046070 con 0x7f6374102640 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f637d03b740 1 Processor -- start 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f637d03b740 1 -- start start 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f63717fa640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 0x7f635c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f637d03b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.833+0000 7f637d03b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56864920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f637d03b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed56864f90 0x55ed56867420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f637d03b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed567c48c0 con 0x55ed5685f430 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f637d03b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ed5676cc20 con 0x55ed56864f90 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f637d03b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ed567c0870 con 0x55ed5685fe20 2024-03-22T02:13:58.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f63737fe640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 0x7f635c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f63717fa640 1 -- 172.21.15.28:0/1731661305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f636c0f1de0 con 0x7f6374102640 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60504/0 (socket says 172.21.15.28:60504) 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 -- 172.21.15.28:0/2286603504 learned_addr learned my addr 172.21.15.28:0/2286603504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f63737fe640 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56864920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f63789eb640 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed56864f90 0x55ed56867420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 -- 172.21.15.28:0/2286603504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 msgr2=0x55ed56864920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56864920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 -- 172.21.15.28:0/2286603504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed56864f90 msgr2=0x55ed56867420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed56864f90 0x55ed56867420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 -- 172.21.15.28:0/2286603504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed56791150 con 0x55ed5685f430 2024-03-22T02:13:58.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f63737fe640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 0x7f635c0a5470 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f636801ae50 tx=0x7f636801a450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f6373fff640 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f63640114f0 tx=0x7f63640119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd066b778d003f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f635a7fc640 1 -- 172.21.15.28:0/2286603504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636401e070 con 0x55ed5685f430 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f635a7fc640 1 -- 172.21.15.28:0/2286603504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636400fb00 con 0x55ed5685f430 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.834+0000 7f635a7fc640 1 -- 172.21.15.28:0/2286603504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364019b80 con 0x55ed5685f430 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- 172.21.15.28:0/2286603504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 msgr2=0x55ed5685f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f63640114f0 tx=0x7f63640119b0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- 172.21.15.28:0/2286603504 shutdown_connections 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed56864f90 0x55ed56867420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56864920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 --2- 172.21.15.28:0/2286603504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685f430 0x55ed5685f850 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- 172.21.15.28:0/2286603504 >> 172.21.15.28:0/2286603504 conn(0x55ed5685a9a0 msgr2=0x55ed5685ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- 172.21.15.28:0/2286603504 shutdown_connections 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- 172.21.15.28:0/2286603504 wait complete. 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.835+0000 7f637d03b740 1 -- start start 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 Processor -- start 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed56869960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56869ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ed56867f40 con 0x55ed56864f90 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ed56780130 con 0x55ed5685f430 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ed568680c0 con 0x55ed5685fe20 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f637d03b740 10 client.0 _ll_register_callbacks cb 0x55ed5677fb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed56869960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63737fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56869ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60512/0 (socket says 172.21.15.28:60512) 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63789eb640 1 -- 172.21.15.28:0/1848986235 learned_addr learned my addr 172.21.15.28:0/1848986235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 -- 172.21.15.28:0/1848986235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 msgr2=0x55ed56869ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56869ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 -- 172.21.15.28:0/1848986235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 msgr2=0x55ed56870ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 -- 172.21.15.28:0/1848986235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed565a27d0 con 0x55ed5685f430 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63789eb640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f6373fff640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed56869960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f63640114c0 tx=0x7f636400f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1e1ea36a63da70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.836+0000 7f63537fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:13:58.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636401e040 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f6364016040 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636400fb00 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f636402f050 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 10 client.25469 fetch_fsmap learned FSMap version 25 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ed564d83f0 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636400fb00 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f63640046c0 con 0x55ed5685f430 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 10 client.25469 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 10 client.25469 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 20 client.25469 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed56875540 0x55ed56877990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 4 client.25469 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed567d1e00 con 0x55ed56875540 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.837+0000 7f63737fe640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed56875540 0x55ed56877990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.838+0000 7f63737fe640 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed56875540 0x55ed56877990 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a8d34090e735b24 server_cookie=559fb4c079aa1f92 in_seq=0 out_seq=0 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.838+0000 7f63537fe640 10 client.25469 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.838+0000 7f63537fe640 1 -- 172.21.15.28:0/1848986235 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed567d1e00 con 0x55ed56875540 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.838+0000 7f63537fe640 10 client.25469 handle_command_reply: tid=0 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.838+0000 7f637d03b740 1 client.25469 shutdown 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 msgr2=0x55ed56869960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed56869960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f63640114c0 tx=0x7f636400f8d0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 shutdown_connections 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed56875540 0x55ed56877990 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed56870ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685fe20 0x55ed56869ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 --2- 172.21.15.28:0/1848986235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed56869960 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f63537fe640 0 client.25469 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 >> 172.21.15.28:0/1848986235 conn(0x55ed5685a9a0 msgr2=0x55ed56859c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 shutdown_connections 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 1 -- 172.21.15.28:0/1848986235 wait complete. 2024-03-22T02:13:58.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.839+0000 7f637d03b740 20 client.25469 trim_cache size 0 max 16384 2024-03-22T02:13:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 Processor -- start 2024-03-22T02:13:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 -- start start 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed568686c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed56864f90 0x55ed56859a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63741aafc0 con 0x55ed5685fe20 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a8890 con 0x55ed5685f430 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f637d03b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a66e0 con 0x55ed56864f90 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60522/0 (socket says 172.21.15.28:60522) 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 -- 172.21.15.28:0/2690543431 learned_addr learned my addr 172.21.15.28:0/2690543431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f6373fff640 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed568686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63789eb640 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed56864f90 0x55ed56859a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 -- 172.21.15.28:0/2690543431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed56864f90 msgr2=0x55ed56859a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed56864f90 0x55ed56859a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 -- 172.21.15.28:0/2690543431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 msgr2=0x55ed568686c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed568686c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.858+0000 7f63737fe640 1 -- 172.21.15.28:0/2690543431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed564980f0 con 0x55ed5685fe20 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f63737fe640 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55ed56860220 tx=0x7f6368021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c0c1b578de2b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f6359ffb640 1 -- 172.21.15.28:0/2690543431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63680044d0 con 0x55ed5685fe20 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f6359ffb640 1 -- 172.21.15.28:0/2690543431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636802d050 con 0x55ed5685fe20 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f6359ffb640 1 -- 172.21.15.28:0/2690543431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636800b810 con 0x55ed5685fe20 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 -- 172.21.15.28:0/2690543431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 msgr2=0x55ed56870e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x55ed56860220 tx=0x7f6368021000 comp rx=0 tx=0).stop 2024-03-22T02:13:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 -- 172.21.15.28:0/2690543431 shutdown_connections 2024-03-22T02:13:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed56864f90 0x55ed56859a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed5685fe20 0x55ed56870e00 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 --2- 172.21.15.28:0/2690543431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685f430 0x55ed568686c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 -- 172.21.15.28:0/2690543431 >> 172.21.15.28:0/2690543431 conn(0x55ed5685a9a0 msgr2=0x55ed56859370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 -- 172.21.15.28:0/2690543431 shutdown_connections 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.859+0000 7f637d03b740 1 -- 172.21.15.28:0/2690543431 wait complete. 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 -- start start 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 Processor -- start 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685f430 0x55ed5686e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685fe20 0x55ed5686ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a6d50 con 0x55ed56864f90 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a81c0 con 0x55ed5685fe20 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63741a7420 con 0x55ed5685f430 2024-03-22T02:13:58.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f637d03b740 10 client.0 _ll_register_callbacks cb 0x7f636801a9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f63789eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60528/0 (socket says 172.21.15.28:60528) 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f63789eb640 1 -- 172.21.15.28:0/3182593620 learned_addr learned my addr 172.21.15.28:0/3182593620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f63737fe640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685fe20 0x55ed5686ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.860+0000 7f6373fff640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685f430 0x55ed5686e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 -- 172.21.15.28:0/3182593620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685f430 msgr2=0x55ed5686e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685f430 0x55ed5686e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 -- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685fe20 msgr2=0x55ed5686ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685fe20 0x55ed5686ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 -- 172.21.15.28:0/3182593620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed56870220 con 0x55ed56864f90 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63789eb640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f636c0bdd50 tx=0x7f636c0ba390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fee603c363f446 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:13:58.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c0d7070 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f636c004b40 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f636802d050 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f636c04d040 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f637d03b740 10 client.15633 fetch_fsmap learned FSMap version 25 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ed568dbae0 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f636802d050 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f636c0bd530 con 0x55ed56864f90 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f637d03b740 10 client.15633 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.861+0000 7f637d03b740 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 20 client.15633 populate_metadata read hostname 'smithi028' 2024-03-22T02:13:58.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed568dc510 0x55ed568de930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 4 client.15633 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ed56868da0 con 0x55ed568dc510 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f6373fff640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed568dc510 0x55ed568de930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f6373fff640 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed568dc510 0x55ed568de930 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7819b33fde2953f0 server_cookie=498f0c6276b14e06 in_seq=0 out_seq=0 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f63537fe640 10 client.15633 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f63537fe640 1 -- 172.21.15.28:0/3182593620 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ed56868da0 con 0x55ed568dc510 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f63537fe640 10 client.15633 handle_command_reply: tid=0 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 client.15633 shutdown 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 msgr2=0x55ed5686fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f636c0bdd50 tx=0x7f636c0ba390 comp rx=0 tx=0).stop 2024-03-22T02:13:58.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 shutdown_connections 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ed568dc510 0x55ed568de930 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ed56864f90 0x55ed5686fc30 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ed5685fe20 0x55ed5686ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 --2- 172.21.15.28:0/3182593620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed5685f430 0x55ed5686e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f63537fe640 0 client.15633 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 >> 172.21.15.28:0/3182593620 conn(0x55ed5685a9a0 msgr2=0x55ed56859c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.862+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 shutdown_connections 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637d03b740 1 -- 172.21.15.28:0/3182593620 wait complete. 2024-03-22T02:13:58.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637d03b740 20 client.15633 trim_cache size 0 max 16384 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "fbb4abef-8d68-49bc-8498-6a71674379e8": { 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "fbb4abef-8d68-49bc-8498-6a71674379e8", 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:13:58.873 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 msgr2=0x7f635c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 0x7f635c0a5470 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f636801ae50 tx=0x7f636801a450 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 msgr2=0x7f637419d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f636c035fd0 tx=0x7f636c035b70 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 shutdown_connections 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f635c0a2fb0 0x7f635c0a5470 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6374102640 0x7f637419d1d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6374101c80 0x7f63741961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 --2- 172.21.15.28:0/1731661305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6374100a90 0x7f6374195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:13:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 >> 172.21.15.28:0/1731661305 conn(0x7f63740fc020 msgr2=0x7f6374104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:13:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 shutdown_connections 2024-03-22T02:13:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:13:58.863+0000 7f637a44e640 1 -- 172.21.15.28:0/1731661305 wait complete. 2024-03-22T02:13:58.876 INFO:tasks.cephfs.filesystem:scrub status for tag:fbb4abef-8d68-49bc-8498-6a71674379e8 - {'path': '/', 'tag': 'fbb4abef-8d68-49bc-8498-6a71674379e8', 'options': 'recursive,force'} 2024-03-22T02:13:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:59 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3618413683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:59 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:13:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:59 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:13:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:59 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:13:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:13:59 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:13:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:59 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3618413683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:59 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:13:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:59 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:13:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:59 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:13:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:13:59 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:13:59.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:59 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3618413683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:13:59.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:59 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:13:59.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:59 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:13:59.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:59 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:13:59.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:13:59 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:00 smithi120 ceph-mon[36803]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:14:00.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:00 smithi083 ceph-mon[36340]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:14:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:00 smithi028 ceph-mon[31916]: pgmap v370: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-22T02:14:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:02 smithi120 ceph-mon[36803]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T02:14:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:02 smithi083 ceph-mon[36340]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T02:14:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:02 smithi028 ceph-mon[31916]: pgmap v371: 129 pgs: 129 active+clean; 652 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-03-22T02:14:03.406 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:03 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:14:03.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:03 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:14:03.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:03 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: pgmap v372: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:04 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: pgmap v372: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-22T02:14:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:04 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: pgmap v372: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:04 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:14:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:14:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:14:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:14:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:14:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:14:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:14:06.649 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:06 smithi120 ceph-mon[36803]: pgmap v373: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:06.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:06 smithi083 ceph-mon[36340]: pgmap v373: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:06 smithi028 ceph-mon[31916]: pgmap v373: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:08 smithi120 ceph-mon[36803]: pgmap v374: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-22T02:14:08.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:08 smithi083 ceph-mon[36340]: pgmap v374: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-22T02:14:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:08 smithi028 ceph-mon[31916]: pgmap v374: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2024-03-22T02:14:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:10 smithi120 ceph-mon[36803]: pgmap v375: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:10 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:14:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:10 smithi083 ceph-mon[36340]: pgmap v375: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:10 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:14:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:10 smithi028 ceph-mon[31916]: pgmap v375: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:10 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 -- 172.21.15.28:0/2219545936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0105050 msgr2=0x7fcca0107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 --2- 172.21.15.28:0/2219545936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0105050 0x7fcca0107440 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009900 tx=0x7fcc9002bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 -- 172.21.15.28:0/2219545936 shutdown_connections 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 --2- 172.21.15.28:0/2219545936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 0x7fcca0109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 --2- 172.21.15.28:0/2219545936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0105050 0x7fcca0107440 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 --2- 172.21.15.28:0/2219545936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0102720 0x7fcca0104b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 -- 172.21.15.28:0/2219545936 >> 172.21.15.28:0/2219545936 conn(0x7fcca00fc030 msgr2=0x7fcca00fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.662+0000 7fcca7a39640 1 -- 172.21.15.28:0/2219545936 shutdown_connections 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 -- 172.21.15.28:0/2219545936 wait complete. 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 Processor -- start 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 -- start start 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0105050 0x7fcca01006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 0x7fcca0100c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010d840 con 0x7fcca0102720 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010d6c0 con 0x7fcca0107980 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca7a39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010d9c0 con 0x7fcca0105050 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53602/0 (socket says 172.21.15.28:53602) 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.663+0000 7fcca57d5640 1 -- 172.21.15.28:0/1211878006 learned_addr learned my addr 172.21.15.28:0/1211878006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 0x7fcca0100c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0105050 0x7fcca01006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 -- 172.21.15.28:0/1211878006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0105050 msgr2=0x7fcca01006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0105050 0x7fcca01006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 -- 172.21.15.28:0/1211878006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 msgr2=0x7fcca0100c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 0x7fcca0100c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 -- 172.21.15.28:0/1211878006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca01014a0 con 0x7fcca0102720 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca57d5640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00ea60 tx=0x7fcc9c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcc967fc640 1 -- 172.21.15.28:0/1211878006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c00cde0 con 0x7fcca0102720 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcc967fc640 1 -- 172.21.15.28:0/1211878006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc9c0035f0 con 0x7fcca0102720 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca01a1f90 con 0x7fcca0102720 2024-03-22T02:14:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcc967fc640 1 -- 172.21.15.28:0/1211878006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c010670 con 0x7fcca0102720 2024-03-22T02:14:11.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.664+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca01a23f0 con 0x7fcca0102720 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.667+0000 7fcc967fc640 1 -- 172.21.15.28:0/1211878006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fcc9c0037a0 con 0x7fcca0102720 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.667+0000 7fccaa61d740 1 Processor -- start 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.667+0000 7fccaa61d740 1 -- start start 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.667+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae1d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae204d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fccaa61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55905ad7d040 con 0x55905ae184e0 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fccaa61d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55905ad259e0 con 0x55905ae1e040 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fccaa61d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55905ad78ff0 con 0x55905ae18ed0 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcc967fc640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 0x7fcc8c0a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae1d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53612/0 (socket says 172.21.15.28:53612) 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 -- 172.21.15.28:0/945397246 learned_addr learned my addr 172.21.15.28:0/945397246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 0x7fcc8c0a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcc967fc640 1 -- 172.21.15.28:0/1211878006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fcc9c0cfc20 con 0x7fcca0102720 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 -- 172.21.15.28:0/945397246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 msgr2=0x55905ae204d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae204d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 -- 172.21.15.28:0/945397246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 msgr2=0x55905ae18900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 -- 172.21.15.28:0/945397246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905aa91210 con 0x55905ae18ed0 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca57d5640 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae1d9d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0cd1c0 tx=0x7fcc9c06eb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=976dbfdc6ee9b21c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.668+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 0x7fcc8c0a55a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55905ae19730 tx=0x7fcc98009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fcc8b7fe640 1 -- 172.21.15.28:0/945397246 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0669a0 con 0x55905ae18ed0 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fcc8b7fe640 1 -- 172.21.15.28:0/945397246 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc9c09c4e0 con 0x55905ae18ed0 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fcc8b7fe640 1 -- 172.21.15.28:0/945397246 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c003b30 con 0x55905ae18ed0 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 -- 172.21.15.28:0/945397246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 msgr2=0x55905ae1d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae1d9d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0cd1c0 tx=0x7fcc9c06eb10 comp rx=0 tx=0).stop 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 -- 172.21.15.28:0/945397246 shutdown_connections 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae1d9d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 --2- 172.21.15.28:0/945397246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae18900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 -- 172.21.15.28:0/945397246 >> 172.21.15.28:0/945397246 conn(0x55905ae13a50 msgr2=0x55905ae13e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 -- 172.21.15.28:0/945397246 shutdown_connections 2024-03-22T02:14:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.669+0000 7fccaa61d740 1 -- 172.21.15.28:0/945397246 wait complete. 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 -- start start 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 Processor -- start 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae22a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae29a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55905ae20ff0 con 0x55905ae184e0 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55905ad38db0 con 0x55905ae1e040 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55905ae21170 con 0x55905ae18ed0 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fccaa61d740 10 client.0 _ll_register_callbacks cb 0x55905aa90a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53614/0 (socket says 172.21.15.28:53614) 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fcca4fd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae22a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fcca57d5640 1 -- 172.21.15.28:0/3352774983 learned_addr learned my addr 172.21.15.28:0/3352774983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.670+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae29a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 -- 172.21.15.28:0/3352774983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 msgr2=0x55905ae22a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae22a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 -- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 msgr2=0x55905ae29a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae29a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 -- 172.21.15.28:0/3352774983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905ab5d620 con 0x55905ae184e0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcca57d5640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c098240 tx=0x7fcc9c09ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38ac4ecc8e4a6ce server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcc88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c09f070 con 0x55905ae184e0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fcc9c0bab50 con 0x55905ae184e0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc9c09c4e0 con 0x55905ae184e0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcc9c09bde0 con 0x55905ae184e0 2024-03-22T02:14:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fccaa61d740 10 client.15645 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55905ae2cdc0 con 0x55905ae184e0 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.671+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc9c09c4e0 con 0x55905ae184e0 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcc9c097b30 con 0x55905ae184e0 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 10 client.15645 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 10 client.15645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 20 client.15645 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae2e110 0x55905ae30500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 4 client.15645 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55905ad8a580 con 0x55905ae2e110 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae2e110 0x55905ae30500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fcc88ff9640 10 client.15645 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.672+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae2e110 0x55905ae30500 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a521c1eb02000581 server_cookie=7a015e5177aabd73 in_seq=0 out_seq=0 2024-03-22T02:14:11.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fcc88ff9640 1 -- 172.21.15.28:0/3352774983 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55905ad8a580 con 0x55905ae2e110 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fcc88ff9640 10 client.15645 handle_command_reply: tid=0 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 client.15645 shutdown 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 msgr2=0x55905ae22540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c098240 tx=0x7fcc9c09ff30 comp rx=0 tx=0).stop 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 shutdown_connections 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae2e110 0x55905ae30500 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae1e040 0x55905ae29a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae18ed0 0x55905ae22a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 --2- 172.21.15.28:0/3352774983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae184e0 0x55905ae22540 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fcc88ff9640 0 client.15645 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.673+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 >> 172.21.15.28:0/3352774983 conn(0x55905ae13a50 msgr2=0x55905ae12c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.674+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 shutdown_connections 2024-03-22T02:14:11.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.674+0000 7fccaa61d740 1 -- 172.21.15.28:0/3352774983 wait complete. 2024-03-22T02:14:11.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.674+0000 7fccaa61d740 20 client.15645 trim_cache size 0 max 16384 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 Processor -- start 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 -- start start 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae184e0 0x55905ae229e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 0x55905ae29a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01af400 con 0x55905ae1e040 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01accd0 con 0x55905ae18ed0 2024-03-22T02:14:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fccaa61d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01aab20 con 0x55905ae184e0 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fcca5fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fcca5fd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57414/0 (socket says 172.21.15.28:57414) 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fcca5fd6640 1 -- 172.21.15.28:0/1596718772 learned_addr learned my addr 172.21.15.28:0/1596718772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 0x55905ae29a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.689+0000 7fcca57d5640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae184e0 0x55905ae229e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 -- 172.21.15.28:0/1596718772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae184e0 msgr2=0x55905ae229e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae184e0 0x55905ae229e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 -- 172.21.15.28:0/1596718772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 msgr2=0x55905ae29a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 0x55905ae29a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 -- 172.21.15.28:0/1596718772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55905ae1d1e0 con 0x55905ae1e040 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 0x55905ae29a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc98011980 tx=0x7fcc98008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b549c2f72619878b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcc8affd640 1 -- 172.21.15.28:0/1596718772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc980045c0 con 0x55905ae1e040 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcc8affd640 1 -- 172.21.15.28:0/1596718772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc98004760 con 0x55905ae1e040 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.690+0000 7fcc8affd640 1 -- 172.21.15.28:0/1596718772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc98005840 con 0x55905ae1e040 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- 172.21.15.28:0/1596718772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 msgr2=0x55905ae173c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcc98011980 tx=0x7fcc98008000 comp rx=0 tx=0).stop 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- 172.21.15.28:0/1596718772 shutdown_connections 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae1e040 0x55905ae173c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae18ed0 0x55905ae29a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- 172.21.15.28:0/1596718772 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae184e0 0x55905ae229e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- 172.21.15.28:0/1596718772 >> 172.21.15.28:0/1596718772 conn(0x55905ae13a50 msgr2=0x55905ae16cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- 172.21.15.28:0/1596718772 shutdown_connections 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- 172.21.15.28:0/1596718772 wait complete. 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 -- start start 2024-03-22T02:14:11.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 Processor -- start 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.691+0000 7fccaa61d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae1e040 0x55905ae28610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fccaa61d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01ab190 con 0x55905ae18ed0 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fccaa61d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01ac600 con 0x55905ae184e0 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fccaa61d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcca01ab860 con 0x55905ae1e040 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fccaa61d740 10 client.0 _ll_register_callbacks cb 0x7fcc98011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca4fd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca4fd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57418/0 (socket says 172.21.15.28:57418) 2024-03-22T02:14:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41486/0 (socket says 172.21.15.28:41486) 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca4fd4640 1 -- 172.21.15.28:0/4091175042 learned_addr learned my addr 172.21.15.28:0/4091175042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca5fd6640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae1e040 0x55905ae28610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 -- 172.21.15.28:0/4091175042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae1e040 msgr2=0x55905ae28610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae1e040 0x55905ae28610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 -- 172.21.15.28:0/4091175042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 msgr2=0x55905ae27600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 -- 172.21.15.28:0/4091175042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55905ae28c00 con 0x55905ae184e0 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcca57d5640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c09bd70 tx=0x7fcc9c009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31db24970d128f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcc88ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c09f070 con 0x55905ae184e0 2024-03-22T02:14:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc98004760 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fcc9c097b30 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.692+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fcc9c097d30 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 10 client.25493 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55905ae28c00 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc98004760 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fcc9c0ba030 con 0x55905ae184e0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 10 client.25493 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 10 client.25493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 20 client.25493 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae94810 0x55905ae96c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 4 client.25493 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55905ad848d0 con 0x55905ae94810 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae94810 0x55905ae96c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fcc88ff9640 10 client.25493 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.693+0000 7fcca4fd4640 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae94810 0x55905ae96c30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5d19e7732b27145 server_cookie=60c57ea608e7397e in_seq=0 out_seq=0 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fcc88ff9640 1 -- 172.21.15.28:0/4091175042 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55905ad848d0 con 0x55905ae94810 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fcc88ff9640 10 client.25493 handle_command_reply: tid=0 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 client.25493 shutdown 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 msgr2=0x55905ae270c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c09bd70 tx=0x7fcc9c009590 comp rx=0 tx=0).stop 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 shutdown_connections 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55905ae94810 0x55905ae96c30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55905ae1e040 0x55905ae28610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fcc88ff9640 0 client.25493 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55905ae18ed0 0x55905ae27600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 --2- 172.21.15.28:0/4091175042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55905ae184e0 0x55905ae270c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 >> 172.21.15.28:0/4091175042 conn(0x55905ae13a50 msgr2=0x55905ae17540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 shutdown_connections 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 1 -- 172.21.15.28:0/4091175042 wait complete. 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.704+0000 7fccaa61d740 20 client.25493 trim_cache size 0 max 16384 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:14:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 msgr2=0x7fcc8c0a55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 0x7fcc8c0a55a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55905ae19730 tx=0x7fcc98009500 comp rx=0 tx=0).stop 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 msgr2=0x7fcca0102090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c00ea60 tx=0x7fcc9c00ef20 comp rx=0 tx=0).stop 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 shutdown_connections 2024-03-22T02:14:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc8c0a30e0 0x7fcc8c0a55a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcca0107980 0x7fcca0100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcca0105050 0x7fcca01006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 --2- 172.21.15.28:0/1211878006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca0102720 0x7fcca0102090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 >> 172.21.15.28:0/1211878006 conn(0x7fcca00fc030 msgr2=0x7fcca0103080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 shutdown_connections 2024-03-22T02:14:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:11.705+0000 7fcca7a39640 1 -- 172.21.15.28:0/1211878006 wait complete. 2024-03-22T02:14:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:12 smithi120 ceph-mon[36803]: pgmap v376: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-22T02:14:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:12 smithi083 ceph-mon[36340]: pgmap v376: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-22T02:14:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:12 smithi028 ceph-mon[31916]: pgmap v376: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-22T02:14:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:14 smithi120 ceph-mon[36803]: pgmap v377: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-22T02:14:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:14 smithi083 ceph-mon[36340]: pgmap v377: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-22T02:14:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:14 smithi028 ceph-mon[31916]: pgmap v377: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-22T02:14:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:16 smithi120 ceph-mon[36803]: pgmap v378: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-22T02:14:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:16 smithi083 ceph-mon[36340]: pgmap v378: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-22T02:14:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:16 smithi028 ceph-mon[31916]: pgmap v378: 129 pgs: 129 active+clean; 657 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 767 B/s wr, 8 op/s 2024-03-22T02:14:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:18 smithi120 ceph-mon[36803]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:18 smithi083 ceph-mon[36340]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:18.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:18 smithi028 ceph-mon[31916]: pgmap v379: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:20 smithi120 ceph-mon[36803]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:20.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:20 smithi083 ceph-mon[36340]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:20 smithi028 ceph-mon[31916]: pgmap v380: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:22 smithi120 ceph-mon[36803]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-22T02:14:22.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:22 smithi083 ceph-mon[36340]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-22T02:14:22.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:22 smithi028 ceph-mon[31916]: pgmap v381: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-03-22T02:14:24.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:24 smithi120 ceph-mon[36803]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-22T02:14:24.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:24 smithi083 ceph-mon[36340]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-22T02:14:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.802+0000 7f5313215640 1 -- 172.21.15.28:0/1336717735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 msgr2=0x7f530c100f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.802+0000 7f5313215640 1 --2- 172.21.15.28:0/1336717735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c100f20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f530002bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:24.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 -- 172.21.15.28:0/1336717735 shutdown_connections 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 --2- 172.21.15.28:0/1336717735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c1026b0 0x7f530c108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 --2- 172.21.15.28:0/1336717735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c101cf0 0x7f530c102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 --2- 172.21.15.28:0/1336717735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c100f20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 -- 172.21.15.28:0/1336717735 >> 172.21.15.28:0/1336717735 conn(0x7f530c0fc070 msgr2=0x7f530c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 -- 172.21.15.28:0/1336717735 shutdown_connections 2024-03-22T02:14:24.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 -- 172.21.15.28:0/1336717735 wait complete. 2024-03-22T02:14:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.803+0000 7f5313215640 1 Processor -- start 2024-03-22T02:14:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 -- start start 2024-03-22T02:14:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c06af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 0x7f530c19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 0x7f530c19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f530c10c750 con 0x7f530c1026b0 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f530c10c5d0 con 0x7f530c100b00 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5313215640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f530c10c8d0 con 0x7f530c101cf0 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f53117b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 0x7f530c19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 0x7f530c19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f53117b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 0x7f530c19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56570/0 (socket says 172.21.15.28:56570) 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f53117b2640 1 -- 172.21.15.28:0/2277327886 learned_addr learned my addr 172.21.15.28:0/2277327886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f5310fb1640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c06af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 -- 172.21.15.28:0/2277327886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 msgr2=0x7f530c06af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c06af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 -- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 msgr2=0x7f530c19f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 0x7f530c19f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 -- 172.21.15.28:0/2277327886 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c19fa70 con 0x7f530c101cf0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.804+0000 7f530bfff640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 0x7f530c19cb70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f52f800e9f0 tx=0x7f52f800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.805+0000 7f5309ffb640 1 -- 172.21.15.28:0/2277327886 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f800ccc0 con 0x7f530c101cf0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.805+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c19fd60 con 0x7f530c101cf0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.805+0000 7f5309ffb640 1 -- 172.21.15.28:0/2277327886 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8010910 con 0x7f530c101cf0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.805+0000 7f5309ffb640 1 -- 172.21.15.28:0/2277327886 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8018960 con 0x7f530c101cf0 2024-03-22T02:14:24.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.805+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c1a0270 con 0x7f530c101cf0 2024-03-22T02:14:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.807+0000 7f5309ffb640 1 -- 172.21.15.28:0/2277327886 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f52f8010ac0 con 0x7f530c101cf0 2024-03-22T02:14:24.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 Processor -- start 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 -- start start 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5309ffb640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 0x7f52fc0a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c8597469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597494d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f53117b2640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 0x7f52fc0a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c8596a6040 con 0x55c8597414e0 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c85964e9e0 con 0x55c859741ed0 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5315e01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c8596a1ff0 con 0x55c859747040 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5309ffb640 1 -- 172.21.15.28:0/2277327886 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f52f80cfd60 con 0x7f530c101cf0 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56584/0 (socket says 172.21.15.28:56584) 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 -- 172.21.15.28:0/488392963 learned_addr learned my addr 172.21.15.28:0/488392963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f53117b2640 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f530bfff640 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c8597469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 -- 172.21.15.28:0/488392963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 msgr2=0x55c8597494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 -- 172.21.15.28:0/488392963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 msgr2=0x55c8597469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.808+0000 7f5310fb1640 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c8597469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f5310fb1640 1 -- 172.21.15.28:0/488392963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8593ba210 con 0x55c8597414e0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f53117b2640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 0x7f52fc0a5520 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f53040049d0 tx=0x7f5304011c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f5310fb1640 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f5300009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f343ec22137bb53 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f52f6ffd640 1 -- 172.21.15.28:0/488392963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300005d50 con 0x55c8597414e0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f52f6ffd640 1 -- 172.21.15.28:0/488392963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530002cb80 con 0x55c8597414e0 2024-03-22T02:14:24.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.809+0000 7f52f6ffd640 1 -- 172.21.15.28:0/488392963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300004920 con 0x55c8597414e0 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- 172.21.15.28:0/488392963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 msgr2=0x55c859741900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f5300009560 comp rx=0 tx=0).stop 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- 172.21.15.28:0/488392963 shutdown_connections 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c8597469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 --2- 172.21.15.28:0/488392963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c859741900 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- 172.21.15.28:0/488392963 >> 172.21.15.28:0/488392963 conn(0x55c85973ca50 msgr2=0x55c85973ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- 172.21.15.28:0/488392963 shutdown_connections 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- 172.21.15.28:0/488392963 wait complete. 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 -- start start 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.810+0000 7f5315e01740 1 Processor -- start 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c85974b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c859749f90 con 0x55c8597414e0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c859661db0 con 0x55c859741ed0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c85974a110 con 0x55c859747040 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 10 client.0 _ll_register_callbacks cb 0x55c8593b9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56592/0 (socket says 172.21.15.28:56592) 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 -- 172.21.15.28:0/3552348767 learned_addr learned my addr 172.21.15.28:0/3552348767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f53117b2640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f530bfff640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c85974b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 -- 172.21.15.28:0/3552348767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 msgr2=0x55c8597529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 -- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 msgr2=0x55c85974b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c85974b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 -- 172.21.15.28:0/3552348767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c859486620 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5310fb1640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f530002c170 tx=0x7f5300045b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2025e71d303949a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f52ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300005d50 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5300045d20 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.811+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f530002cb80 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5300056050 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 10 client.15663 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c859755d40 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f530002cb80 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f530004d070 con 0x55c8597414e0 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 10 client.15663 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:24.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 20 client.15663 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c859759480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 4 client.15663 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8596b3580 con 0x55c859757090 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.812+0000 7f530bfff640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c859759480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.813+0000 7f52ebfff640 10 client.15663 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.813+0000 7f530bfff640 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c859759480 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75191bd532820c93 server_cookie=7c140ed82166db9a in_seq=0 out_seq=0 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.813+0000 7f52ebfff640 1 -- 172.21.15.28:0/3552348767 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c8596b3580 con 0x55c859757090 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.813+0000 7f52ebfff640 10 client.15663 handle_command_reply: tid=0 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 client.15663 shutdown 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 msgr2=0x55c85974b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f530002c170 tx=0x7f5300045b30 comp rx=0 tx=0).stop 2024-03-22T02:14:24.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 shutdown_connections 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c859759480 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859747040 0x55c8597529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f52ebfff640 0 client.15663 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859741ed0 0x55c85974b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 --2- 172.21.15.28:0/3552348767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974b450 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 >> 172.21.15.28:0/3552348767 conn(0x55c85973ca50 msgr2=0x55c85973bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 shutdown_connections 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 1 -- 172.21.15.28:0/3552348767 wait complete. 2024-03-22T02:14:24.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.814+0000 7f5315e01740 20 client.15663 trim_cache size 0 max 16384 2024-03-22T02:14:24.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:24 smithi028 ceph-mon[31916]: pgmap v382: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-03-22T02:14:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 Processor -- start 2024-03-22T02:14:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 -- start start 2024-03-22T02:14:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859752990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c85974aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1ad1a0 con 0x55c8597414e0 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1aaa70 con 0x55c859747040 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5315e01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1a88c0 con 0x55c859741ed0 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56606/0 (socket says 172.21.15.28:56606) 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 -- 172.21.15.28:0/2896798179 learned_addr learned my addr 172.21.15.28:0/2896798179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f530bfff640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859752990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f53117b2640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c85974aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 -- 172.21.15.28:0/2896798179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 msgr2=0x55c859752990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859752990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 -- 172.21.15.28:0/2896798179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 msgr2=0x55c85974aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c85974aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f5310fb1640 1 -- 172.21.15.28:0/2896798179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8597422b0 con 0x55c8597414e0 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.829+0000 7f530bfff640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859752990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5310fb1640 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f5300005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2a1eb145e5282f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f52f67fc640 1 -- 172.21.15.28:0/2896798179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300009520 con 0x55c8597414e0 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f52f67fc640 1 -- 172.21.15.28:0/2896798179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5300053070 con 0x55c8597414e0 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f52f67fc640 1 -- 172.21.15.28:0/2896798179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300003ef0 con 0x55c8597414e0 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 -- 172.21.15.28:0/2896798179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 msgr2=0x55c85973bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f5300009900 tx=0x7f5300005fd0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 -- 172.21.15.28:0/2896798179 shutdown_connections 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c85974aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859752990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 --2- 172.21.15.28:0/2896798179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85973bc50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 -- 172.21.15.28:0/2896798179 >> 172.21.15.28:0/2896798179 conn(0x55c85973ca50 msgr2=0x55c859740480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.830+0000 7f5315e01740 1 -- 172.21.15.28:0/2896798179 shutdown_connections 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 -- 172.21.15.28:0/2896798179 wait complete. 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 -- start start 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 Processor -- start 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859750120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c859751130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1a8f30 con 0x55c8597414e0 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1aa3a0 con 0x55c859747040 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f530c1a9600 con 0x55c859741ed0 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5315e01740 10 client.0 _ll_register_callbacks cb 0x7f52f8010eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5310fb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56616/0 (socket says 172.21.15.28:56616) 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f5310fb1640 1 -- 172.21.15.28:0/92772813 learned_addr learned my addr 172.21.15.28:0/92772813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f530bfff640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859750120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.831+0000 7f53117b2640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c859751130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 -- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 msgr2=0x55c859751130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c859751130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 -- 172.21.15.28:0/92772813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 msgr2=0x55c85974fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 -- 172.21.15.28:0/92772813 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c859751720 con 0x55c859741ed0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f5310fb1640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f530bfff640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859750120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f52f80cd2a0 tx=0x7f52f80bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f4249225497235f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f809bf70 con 0x55c859741ed0 2024-03-22T02:14:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f52f80985e0 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5300053070 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f52f8082050 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f5315e01740 10 client.25498 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c859752aa0 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5300053070 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.832+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f52f80b8070 con 0x55c859741ed0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 10 client.25498 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 10 client.25498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 20 client.25498 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c8597bf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 4 client.25498 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c8596ad8d0 con 0x55c859757090 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5310fb1640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c8597bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f52ebfff640 10 client.25498 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.833+0000 7f5310fb1640 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c8597bf870 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd37ecdb70622cf1 server_cookie=349d3023a51499a in_seq=0 out_seq=0 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f52ebfff640 1 -- 172.21.15.28:0/92772813 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c8596ad8d0 con 0x55c859757090 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f52ebfff640 10 client.25498 handle_command_reply: tid=0 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 client.25498 shutdown 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 msgr2=0x55c859750120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859750120 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f52f80cd2a0 tx=0x7f52f80bc040 comp rx=0 tx=0).stop 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 shutdown_connections 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c859757090 0x55c8597bf870 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c859747040 0x55c859751130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c859741ed0 0x55c859750120 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 --2- 172.21.15.28:0/92772813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8597414e0 0x55c85974fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f52ebfff640 0 client.25498 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 >> 172.21.15.28:0/92772813 conn(0x55c85973ca50 msgr2=0x55c85973bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 shutdown_connections 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 1 -- 172.21.15.28:0/92772813 wait complete. 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.836+0000 7f5315e01740 20 client.25498 trim_cache size 0 max 16384 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:14:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:14:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 msgr2=0x7f52fc0a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 0x7f52fc0a5520 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f53040049d0 tx=0x7f5304011c80 comp rx=0 tx=0).stop 2024-03-22T02:14:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 msgr2=0x7f530c19cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 0x7f530c19cb70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f52f800e9f0 tx=0x7f52f800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 shutdown_connections 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52fc0a3060 0x7f52fc0a5520 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530c1026b0 0x7f530c19f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f530c101cf0 0x7f530c19cb70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 --2- 172.21.15.28:0/2277327886 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f530c100b00 0x7f530c06af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 >> 172.21.15.28:0/2277327886 conn(0x7f530c0fc070 msgr2=0x7f530c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 shutdown_connections 2024-03-22T02:14:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:24.837+0000 7f5313215640 1 -- 172.21.15.28:0/2277327886 wait complete. 2024-03-22T02:14:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:26 smithi120 ceph-mon[36803]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:26.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:26 smithi083 ceph-mon[36340]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:26 smithi028 ceph-mon[31916]: pgmap v383: 129 pgs: 129 active+clean; 658 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-22T02:14:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:28 smithi120 ceph-mon[36803]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-22T02:14:28.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:28 smithi083 ceph-mon[36340]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-22T02:14:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:28 smithi028 ceph-mon[31916]: pgmap v384: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 8 op/s 2024-03-22T02:14:28.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:14:28.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:14:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.996+0000 7f3d67fe2640 1 -- 172.21.15.28:0/2038155083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60102630 msgr2=0x7f3d60104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.996+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/2038155083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60102630 0x7f3d60104a20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3d58009900 tx=0x7f3d5802bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- 172.21.15.28:0/2038155083 shutdown_connections 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/2038155083 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60107890 0x7f3d60109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/2038155083 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/2038155083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60102630 0x7f3d60104a20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- 172.21.15.28:0/2038155083 >> 172.21.15.28:0/2038155083 conn(0x7f3d600fc040 msgr2=0x7f3d600fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- 172.21.15.28:0/2038155083 shutdown_connections 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- 172.21.15.28:0/2038155083 wait complete. 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 Processor -- start 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- start start 2024-03-22T02:14:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 0x7f3d6019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6010d850 con 0x7f3d60107890 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6010d6d0 con 0x7f3d60104f60 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.997+0000 7f3d67fe2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6010d9d0 con 0x7f3d60102630 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 0x7f3d6019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d65d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d65d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44586/0 (socket says 172.21.15.28:44586) 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d65d7e640 1 -- 172.21.15.28:0/1374449143 learned_addr learned my addr 172.21.15.28:0/1374449143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 -- 172.21.15.28:0/1374449143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 msgr2=0x7f3d60197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 -- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 msgr2=0x7f3d6019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 0x7f3d6019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 -- 172.21.15.28:0/1374449143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6019fb10 con 0x7f3d60104f60 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6657f640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 0x7f3d6019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d6557d640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60198430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400e9c0 tx=0x7f3d5400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d5effd640 1 -- 172.21.15.28:0/1374449143 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5400ccc0 con 0x7f3d60104f60 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6019fe00 con 0x7f3d60104f60 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d5effd640 1 -- 172.21.15.28:0/1374449143 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d54010910 con 0x7f3d60104f60 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d5effd640 1 -- 172.21.15.28:0/1374449143 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d540188c0 con 0x7f3d60104f60 2024-03-22T02:14:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:28.998+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d601a02c0 con 0x7f3d60104f60 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.001+0000 7f3d5effd640 1 -- 172.21.15.28:0/1374449143 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f3d54010ac0 con 0x7f3d60104f60 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.001+0000 7f3d6abc7740 1 Processor -- start 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 -- start start 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x561806456920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806456f90 0x561806459420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5618063b68c0 con 0x561806451430 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d5effd640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 0x7f3d480a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56180635ec20 con 0x561806451e20 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618063b2870 con 0x561806456f90 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6557d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806456f90 0x561806459420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56650/0 (socket says 172.21.15.28:56650) 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6657f640 1 -- 172.21.15.28:0/262014201 learned_addr learned my addr 172.21.15.28:0/262014201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x561806456920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6657f640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 0x7f3d480a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d5effd640 1 -- 172.21.15.28:0/1374449143 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d540cfcc0 con 0x7f3d60104f60 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 -- 172.21.15.28:0/262014201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806456f90 msgr2=0x561806459420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806456f90 0x561806459420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 -- 172.21.15.28:0/262014201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 msgr2=0x561806451850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d65d7e640 1 -- 172.21.15.28:0/262014201 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561806383150 con 0x561806451e20 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.002+0000 7f3d6657f640 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.003+0000 7f3d6657f640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 0x7f3d480a5470 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3d500099e0 tx=0x7f3d50009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.003+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x561806456920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c7b0 tx=0x7f3d5802cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffff0e83842b96e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.003+0000 7f3d47fff640 1 -- 172.21.15.28:0/262014201 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58042070 con 0x561806451e20 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.003+0000 7f3d47fff640 1 -- 172.21.15.28:0/262014201 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d58004ba0 con 0x561806451e20 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.003+0000 7f3d47fff640 1 -- 172.21.15.28:0/262014201 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58005230 con 0x561806451e20 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- 172.21.15.28:0/262014201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 msgr2=0x561806456920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x561806456920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c7b0 tx=0x7f3d5802cd30 comp rx=0 tx=0).stop 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- 172.21.15.28:0/262014201 shutdown_connections 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806456f90 0x561806459420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x561806456920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/262014201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x561806451850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- 172.21.15.28:0/262014201 >> 172.21.15.28:0/262014201 conn(0x56180644c9a0 msgr2=0x56180644cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- 172.21.15.28:0/262014201 shutdown_connections 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- 172.21.15.28:0/262014201 wait complete. 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 -- start start 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.004+0000 7f3d6abc7740 1 Processor -- start 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x56180645b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x56180645be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561806459ee0 con 0x561806456f90 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561806372130 con 0x561806451e20 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56180645a060 con 0x561806451430 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 10 client.0 _ll_register_callbacks cb 0x561806371b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d65d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x56180645b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56658/0 (socket says 172.21.15.28:56658) 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 -- 172.21.15.28:0/2647089846 learned_addr learned my addr 172.21.15.28:0/2647089846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6557d640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x56180645be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 -- 172.21.15.28:0/2647089846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 msgr2=0x56180645b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x56180645b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 -- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 msgr2=0x56180645be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x56180645be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 -- 172.21.15.28:0/2647089846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618061947d0 con 0x561806456f90 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6557d640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x56180645be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6657f640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d50003ec0 tx=0x7f3d50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133a74a17894b08f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d457fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50004d20 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d50020670 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.005+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d58004ba0 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3d50033050 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5618060ca3f0 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d58004ba0 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3d50027030 con 0x561806456f90 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 20 client.15681 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561806467830 0x561806469cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 4 client.15681 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:29.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618063c3e00 con 0x561806467830 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.006+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561806467830 0x561806469cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561806467830 0x561806469cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caeaf316e2526a48 server_cookie=ac9f21575412280d in_seq=0 out_seq=0 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d457fa640 10 client.15681 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d457fa640 1 -- 172.21.15.28:0/2647089846 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618063c3e00 con 0x561806467830 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d457fa640 10 client.15681 handle_command_reply: tid=0 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 client.15681 shutdown 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 msgr2=0x561806462e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3d50003ec0 tx=0x7f3d50004a40 comp rx=0 tx=0).stop 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 shutdown_connections 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561806467830 0x561806469cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806456f90 0x561806462e90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806451e20 0x56180645be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d457fa640 0 client.15681 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2647089846 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x56180645b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.007+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 >> 172.21.15.28:0/2647089846 conn(0x56180644c9a0 msgr2=0x56180644bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.008+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 shutdown_connections 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.008+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2647089846 wait complete. 2024-03-22T02:14:29.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.008+0000 7f3d6abc7740 20 client.15681 trim_cache size 0 max 16384 2024-03-22T02:14:29.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 Processor -- start 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 -- start start 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x561806453320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x561806462e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601ad290 con 0x561806451e20 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601aab60 con 0x561806456f90 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601a89b0 con 0x561806451430 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6557d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6557d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56666/0 (socket says 172.21.15.28:56666) 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6557d640 1 -- 172.21.15.28:0/2026386193 learned_addr learned my addr 172.21.15.28:0/2026386193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x561806453320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.025+0000 7f3d6657f640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x561806462e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 -- 172.21.15.28:0/2026386193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 msgr2=0x561806453320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x561806453320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 -- 172.21.15.28:0/2026386193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 msgr2=0x561806462e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x561806462e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 -- 172.21.15.28:0/2026386193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56180608a0f0 con 0x561806451e20 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6657f640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x561806462e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x561806453320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d6557d640 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d54097c10 tx=0x7f3d540b5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f872d6c69536182 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d477fe640 1 -- 172.21.15.28:0/2026386193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5405fe60 con 0x561806451e20 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d477fe640 1 -- 172.21.15.28:0/2026386193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5409bf30 con 0x561806451e20 2024-03-22T02:14:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.026+0000 7f3d477fe640 1 -- 172.21.15.28:0/2026386193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54026040 con 0x561806451e20 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2026386193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 msgr2=0x561806453860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3d54097c10 tx=0x7f3d540b5040 comp rx=0 tx=0).stop 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2026386193 shutdown_connections 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x561806462e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451e20 0x561806453860 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/2026386193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451430 0x561806453320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2026386193 >> 172.21.15.28:0/2026386193 conn(0x56180644c9a0 msgr2=0x56180645a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2026386193 shutdown_connections 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- 172.21.15.28:0/2026386193 wait complete. 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- start start 2024-03-22T02:14:29.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 Processor -- start 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451e20 0x5618064604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x5618064614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601a9020 con 0x561806451430 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601aa490 con 0x561806456f90 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d601a96f0 con 0x561806451e20 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d6abc7740 10 client.0 _ll_register_callbacks cb 0x7f3d50003c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.027+0000 7f3d65d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56670/0 (socket says 172.21.15.28:56670) 2024-03-22T02:14:29.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 -- 172.21.15.28:0/4169711540 learned_addr learned my addr 172.21.15.28:0/4169711540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6657f640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x5618064614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6557d640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451e20 0x5618064604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 -- 172.21.15.28:0/4169711540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451e20 msgr2=0x5618064604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451e20 0x5618064604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 -- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 msgr2=0x5618064614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x5618064614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 -- 172.21.15.28:0/4169711540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561806461ac0 con 0x561806451430 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6657f640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x5618064614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d65d7e640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c8c0 tx=0x7f3d58005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338b24a099f77010 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d457fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d58042070 con 0x561806451430 2024-03-22T02:14:29.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d58004450 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d5409bf30 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3d58009520 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6abc7740 10 client.15687 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561806462e40 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.028+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5409bf30 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3d58004ef0 con 0x561806451430 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 10 client.15687 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 20 client.15687 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618064cdbb0 0x5618064cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 4 client.15687 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5618063be150 con 0x5618064cdbb0 2024-03-22T02:14:29.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6557d640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618064cdbb0 0x5618064cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d457fa640 10 client.15687 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6557d640 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618064cdbb0 0x5618064cffa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb518f988211eca0 server_cookie=28244e302e81d659 in_seq=0 out_seq=0 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d457fa640 1 -- 172.21.15.28:0/4169711540 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5618063be150 con 0x5618064cdbb0 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d457fa640 10 client.15687 handle_command_reply: tid=0 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.029+0000 7f3d6abc7740 1 client.15687 shutdown 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 msgr2=0x56180645ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3d5802c8c0 tx=0x7f3d58005d50 comp rx=0 tx=0).stop 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 shutdown_connections 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618064cdbb0 0x5618064cffa0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561806456f90 0x5618064614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d457fa640 0 client.15687 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561806451e20 0x5618064604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 --2- 172.21.15.28:0/4169711540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561806451430 0x56180645ff80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 >> 172.21.15.28:0/4169711540 conn(0x56180644c9a0 msgr2=0x56180645a400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 shutdown_connections 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 1 -- 172.21.15.28:0/4169711540 wait complete. 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d6abc7740 20 client.15687 trim_cache size 0 max 16384 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:14:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:14:29.041 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:14:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 msgr2=0x7f3d480a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 0x7f3d480a5470 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3d500099e0 tx=0x7f3d50009500 comp rx=0 tx=0).stop 2024-03-22T02:14:29.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 msgr2=0x7f3d60198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60198430 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400e9c0 tx=0x7f3d5400ee80 comp rx=0 tx=0).stop 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 shutdown_connections 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d480a2fb0 0x7f3d480a5470 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d60107890 0x7f3d6019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d60104f60 0x7f3d60198430 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 --2- 172.21.15.28:0/1374449143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d60102630 0x7f3d60197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 >> 172.21.15.28:0/1374449143 conn(0x7f3d600fc040 msgr2=0x7f3d60103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 shutdown_connections 2024-03-22T02:14:29.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.030+0000 7f3d67fe2640 1 -- 172.21.15.28:0/1374449143 wait complete. 2024-03-22T02:14:29.043 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:14:29.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 -- 172.21.15.28:0/288374528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac101bb0 msgr2=0x7f07ac101fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 --2- 172.21.15.28:0/288374528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac101bb0 0x7f07ac101fd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f07a8009900 tx=0x7f07a802bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 -- 172.21.15.28:0/288374528 shutdown_connections 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 --2- 172.21.15.28:0/288374528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac103760 0x7f07ac109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 --2- 172.21.15.28:0/288374528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 0x7f07ac103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 --2- 172.21.15.28:0/288374528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac101bb0 0x7f07ac101fd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 -- 172.21.15.28:0/288374528 >> 172.21.15.28:0/288374528 conn(0x7f07ac0fd140 msgr2=0x7f07ac0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 -- 172.21.15.28:0/288374528 shutdown_connections 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.145+0000 7f07b3513640 1 -- 172.21.15.28:0/288374528 wait complete. 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 Processor -- start 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 -- start start 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 0x7f07ac198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac103760 0x7f07ac19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d820 con 0x7f07ac101bb0 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d6a0 con 0x7f07ac102da0 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b3513640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f07ac10d9a0 con 0x7f07ac103760 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56686/0 (socket says 172.21.15.28:56686) 2024-03-22T02:14:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 -- 172.21.15.28:0/1210764429 learned_addr learned my addr 172.21.15.28:0/1210764429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b0aae640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 0x7f07ac198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b1ab0640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac103760 0x7f07ac19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 -- 172.21.15.28:0/1210764429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac103760 msgr2=0x7f07ac19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac103760 0x7f07ac19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 -- 172.21.15.28:0/1210764429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 msgr2=0x7f07ac198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 0x7f07ac198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 -- 172.21.15.28:0/1210764429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ac19faf0 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.146+0000 7f07b12af640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8041070 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f07a8004450 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07ac19fd80 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8034440 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07ac1a01c0 con 0x7f07ac101bb0 2024-03-22T02:14:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.147+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07ac108ef0 con 0x7f07ac101bb0 2024-03-22T02:14:29.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.149+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f07a8044070 con 0x7f07ac101bb0 2024-03-22T02:14:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.149+0000 7f07a27fc640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 0x7f07980a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.149+0000 7f07b0aae640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 0x7f07980a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.149+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f07a80f1520 con 0x7f07ac101bb0 2024-03-22T02:14:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.150+0000 7f07b0aae640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 0x7f07980a5470 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f07ac1993e0 tx=0x7f079c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.154+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f07a80bd930 con 0x7f07ac101bb0 2024-03-22T02:14:29.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.379+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07ac02d3b0 con 0x7f07ac101bb0 2024-03-22T02:14:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.379+0000 7f07a27fc640 1 -- 172.21.15.28:0/1210764429 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11866 (secure 0 0 0) 0x7f07a80c19d0 con 0x7f07ac101bb0 2024-03-22T02:14:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:14:29.380 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:13:50.358406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:14:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-03-22T02:14:29.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 msgr2=0x7f07980a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 0x7f07980a5470 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f07ac1993e0 tx=0x7f079c005f90 comp rx=0 tx=0).stop 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 msgr2=0x7f07ac197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07a8000c00 tx=0x7f07a802c640 comp rx=0 tx=0).stop 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 shutdown_connections 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f07980a2fb0 0x7f07980a5470 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f07ac103760 0x7f07ac19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f07ac102da0 0x7f07ac198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 --2- 172.21.15.28:0/1210764429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ac101bb0 0x7f07ac197ed0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 >> 172.21.15.28:0/1210764429 conn(0x7f07ac0fd140 msgr2=0x7f07ac1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 shutdown_connections 2024-03-22T02:14:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.382+0000 7f07b3513640 1 -- 172.21.15.28:0/1210764429 wait complete. 2024-03-22T02:14:29.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 -- 172.21.15.28:0/3632034576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc103770 msgr2=0x7fc4cc10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 --2- 172.21.15.28:0/3632034576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc103770 0x7fc4cc10a000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009ca0 tx=0x7fc4c802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 -- 172.21.15.28:0/3632034576 shutdown_connections 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 --2- 172.21.15.28:0/3632034576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc103770 0x7fc4cc10a000 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 --2- 172.21.15.28:0/3632034576 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc102db0 0x7fc4cc103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 --2- 172.21.15.28:0/3632034576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc101bc0 0x7fc4cc101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 -- 172.21.15.28:0/3632034576 >> 172.21.15.28:0/3632034576 conn(0x7fc4cc0fd130 msgr2=0x7fc4cc0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 -- 172.21.15.28:0/3632034576 shutdown_connections 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.498+0000 7fc4d3415640 1 -- 172.21.15.28:0/3632034576 wait complete. 2024-03-22T02:14:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 Processor -- start 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 -- start start 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc101bc0 0x7fc4cc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc103770 0x7fc4cc19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc10d840 con 0x7fc4cc102db0 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc10d6c0 con 0x7fc4cc101bc0 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d3415640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc10d9c0 con 0x7fc4cc103770 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc101bc0 0x7fc4cc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56700/0 (socket says 172.21.15.28:56700) 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d19b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc103770 0x7fc4cc19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 -- 172.21.15.28:0/1573554998 learned_addr learned my addr 172.21.15.28:0/1573554998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 -- 172.21.15.28:0/1573554998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc103770 msgr2=0x7fc4cc19f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc103770 0x7fc4cc19f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 -- 172.21.15.28:0/1573554998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc101bc0 msgr2=0x7fc4cc197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc101bc0 0x7fc4cc197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 -- 172.21.15.28:0/1573554998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4cc19fa70 con 0x7fc4cc102db0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc00e9f0 tx=0x7fc4bc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4c27fc640 1 -- 172.21.15.28:0/1573554998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc00cd80 con 0x7fc4cc102db0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.499+0000 7fc4c27fc640 1 -- 172.21.15.28:0/1573554998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4bc010910 con 0x7fc4cc102db0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.500+0000 7fc4c27fc640 1 -- 172.21.15.28:0/1573554998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc018990 con 0x7fc4cc102db0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.500+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4cc19fd60 con 0x7fc4cc102db0 2024-03-22T02:14:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.500+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4cc1a0270 con 0x7fc4cc102db0 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4c27fc640 1 -- 172.21.15.28:0/1573554998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fc4bc010ac0 con 0x7fc4cc102db0 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 Processor -- start 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 -- start start 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5078820 0x5596a5078c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 0x5596a5080810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5596a4fdd8d0 con 0x5596a5079210 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5596a4f85e00 con 0x5596a507e380 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4c27fc640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 0x7fc4b80a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5596a4fd9880 con 0x5596a5078820 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56708/0 (socket says 172.21.15.28:56708) 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d11b1640 1 -- 172.21.15.28:0/1506845404 learned_addr learned my addr 172.21.15.28:0/1506845404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5078820 0x5596a5078c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4c27fc640 1 -- 172.21.15.28:0/1573554998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc4bc0cfbd0 con 0x7fc4cc102db0 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.502+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 0x5596a5080810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 0x7fc4b80a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 -- 172.21.15.28:0/1506845404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5078820 msgr2=0x5596a5078c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5078820 0x5596a5078c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 -- 172.21.15.28:0/1506845404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 msgr2=0x5596a5080810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 0x5596a5080810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 -- 172.21.15.28:0/1506845404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a4cf13f0 con 0x5596a5079210 2024-03-22T02:14:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 0x5596a5080810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 0x7fc4b80a54f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005e30 tx=0x7fc4c8032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009ca0 tx=0x7fc4c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6342909fc3ed49 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4b77fe640 1 -- 172.21.15.28:0/1506845404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4004220 con 0x5596a5079210 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4b77fe640 1 -- 172.21.15.28:0/1506845404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c40043c0 con 0x5596a5079210 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.503+0000 7fc4b77fe640 1 -- 172.21.15.28:0/1506845404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4019890 con 0x5596a5079210 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1506845404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 msgr2=0x5596a507dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009ca0 tx=0x7fc4c4011810 comp rx=0 tx=0).stop 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1506845404 shutdown_connections 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a507e380 0x5596a5080810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a507dd10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1506845404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5078820 0x5596a5078c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1506845404 >> 172.21.15.28:0/1506845404 conn(0x5596a5073d90 msgr2=0x5596a5074190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1506845404 shutdown_connections 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1506845404 wait complete. 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 -- start start 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.504+0000 7fc4d5ffe740 1 Processor -- start 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5082d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 0x5596a5083260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5596a5081330 con 0x5596a507e380 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5596a4f99300 con 0x5596a5078820 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5596a50814b0 con 0x5596a5079210 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 10 client.0 _ll_register_callbacks cb 0x5596a4fd4e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56718/0 (socket says 172.21.15.28:56718) 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5082d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 0x5596a5083260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 -- 172.21.15.28:0/1022877325 learned_addr learned my addr 172.21.15.28:0/1022877325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 -- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 msgr2=0x5596a5083260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 0x5596a5083260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 -- 172.21.15.28:0/1022877325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 msgr2=0x5596a5082d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5082d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 -- 172.21.15.28:0/1022877325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a4dbb7d0 con 0x5596a507e380 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 0x5596a5083260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009430 tx=0x7fc4c802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafc47d6e5b33a43 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8054060 con 0x5596a507e380 2024-03-22T02:14:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc4c8004250 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.505+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4c40043c0 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc4c8039040 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 10 client.15705 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596a508d680 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c40043c0 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc4c804f470 con 0x5596a507e380 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 10 client.15705 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 10 client.15705 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 20 client.15705 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a508edc0 0x5596a5091220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 4 client.15705 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596a4feae10 con 0x5596a508edc0 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.506+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a508edc0 0x5596a5091220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4b4ff9640 10 client.15705 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a508edc0 0x5596a5091220 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27786ae10780b3d3 server_cookie=8a160ea6b3e1c8d0 in_seq=0 out_seq=0 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/1022877325 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5596a4feae10 con 0x5596a508edc0 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4b4ff9640 10 client.15705 handle_command_reply: tid=0 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 client.15705 shutdown 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 msgr2=0x5596a508a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009430 tx=0x7fc4c802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 shutdown_connections 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a507e380 0x5596a508a2a0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a5079210 0x5596a5083260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a508edc0 0x5596a5091220 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/1022877325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5082d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4b4ff9640 0 client.15705 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:14:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.507+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 >> 172.21.15.28:0/1022877325 conn(0x5596a5073d90 msgr2=0x5596a5072fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.508+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 shutdown_connections 2024-03-22T02:14:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.508+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/1022877325 wait complete. 2024-03-22T02:14:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.508+0000 7fc4d5ffe740 20 client.15705 trim_cache size 0 max 16384 2024-03-22T02:14:29.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 Processor -- start 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 -- start start 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5079210 0x5596a508a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5095ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1ad250 con 0x5596a5078820 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1aab20 con 0x5596a5079210 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1a8970 con 0x5596a507e380 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d11b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56730/0 (socket says 172.21.15.28:56730) 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5079210 0x5596a508a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d11b1640 1 -- 172.21.15.28:0/2457030972 learned_addr learned my addr 172.21.15.28:0/2457030972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.521+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5095ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 -- 172.21.15.28:0/2457030972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 msgr2=0x5596a5095ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5095ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 -- 172.21.15.28:0/2457030972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5079210 msgr2=0x5596a508a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5079210 0x5596a508a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 -- 172.21.15.28:0/2457030972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a50783e0 con 0x5596a5078820 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc4c401b040 tx=0x7fc4c4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30869066e3028bf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4b6ffd640 1 -- 172.21.15.28:0/2457030972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4030070 con 0x5596a5078820 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4b6ffd640 1 -- 172.21.15.28:0/2457030972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4005040 con 0x5596a5078820 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4b6ffd640 1 -- 172.21.15.28:0/2457030972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4003a30 con 0x5596a5078820 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/2457030972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 msgr2=0x5596a5077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc4c401b040 tx=0x7fc4c4005f90 comp rx=0 tx=0).stop 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/2457030972 shutdown_connections 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5095ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5079210 0x5596a508a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/2457030972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5078820 0x5596a5077700 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/2457030972 >> 172.21.15.28:0/2457030972 conn(0x5596a5073d90 msgr2=0x5596a5076ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/2457030972 shutdown_connections 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.522+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/2457030972 wait complete. 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 -- start start 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 Processor -- start 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5087ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5089010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1a8fe0 con 0x5596a5079210 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1aa450 con 0x5596a5078820 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4cc1a96b0 con 0x5596a507e380 2024-03-22T02:14:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d5ffe740 10 client.0 _ll_register_callbacks cb 0x7fc4c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d09b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56736/0 (socket says 172.21.15.28:56736) 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d09b0640 1 -- 172.21.15.28:0/3715200448 learned_addr learned my addr 172.21.15.28:0/3715200448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5087ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d19b2640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5089010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d09b0640 1 -- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 msgr2=0x5596a5089010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.523+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5089010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d09b0640 1 -- 172.21.15.28:0/3715200448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 msgr2=0x5596a5087ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5087ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d09b0640 1 -- 172.21.15.28:0/3715200448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a5089600 con 0x5596a5079210 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d09b0640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0982a0 tx=0x7fc4bc0bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5a8475a1f476fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc018d30 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc4bc09b290 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4c4005040 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc4bc09b490 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 10 client.15711 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596a50f5250 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4005040 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc4bc097b90 con 0x5596a5079210 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 10 client.15711 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 10 client.15711 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 20 client.15711 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a50f5c80 0x5596a50f80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 4 client.15711 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.524+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596a4fe5160 con 0x5596a50f5c80 2024-03-22T02:14:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a50f5c80 0x5596a50f80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4b4ff9640 10 client.15711 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d11b1640 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a50f5c80 0x5596a50f80a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f200db616486190e server_cookie=c26957149a060155 in_seq=0 out_seq=0 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4b4ff9640 1 -- 172.21.15.28:0/3715200448 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596a4fe5160 con 0x5596a50f5c80 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4b4ff9640 10 client.15711 handle_command_reply: tid=0 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 client.15711 shutdown 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 msgr2=0x5596a5088000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc0982a0 tx=0x7fc4bc0bc040 comp rx=0 tx=0).stop 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 shutdown_connections 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5596a507e380 0x5596a5089010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596a5079210 0x5596a5088000 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5596a50f5c80 0x5596a50f80a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 --2- 172.21.15.28:0/3715200448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596a5078820 0x5596a5087ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4b4ff9640 0 client.15711 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:14:29.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 >> 172.21.15.28:0/3715200448 conn(0x5596a5073d90 msgr2=0x5596a5077880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 shutdown_connections 2024-03-22T02:14:29.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 1 -- 172.21.15.28:0/3715200448 wait complete. 2024-03-22T02:14:29.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.525+0000 7fc4d5ffe740 20 client.15711 trim_cache size 0 max 16384 2024-03-22T02:14:29.533 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 msgr2=0x7fc4b80a54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 0x7fc4b80a54f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8005e30 tx=0x7fc4c8032590 comp rx=0 tx=0).stop 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 msgr2=0x7fc4cc198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc00e9f0 tx=0x7fc4bc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 shutdown_connections 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4b80a3030 0x7fc4b80a54f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4cc103770 0x7fc4cc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4cc102db0 0x7fc4cc198390 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 --2- 172.21.15.28:0/1573554998 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4cc101bc0 0x7fc4cc197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 >> 172.21.15.28:0/1573554998 conn(0x7fc4cc0fd130 msgr2=0x7fc4cc1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 shutdown_connections 2024-03-22T02:14:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.526+0000 7fc4d3415640 1 -- 172.21.15.28:0/1573554998 wait complete. 2024-03-22T02:14:29.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:14:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 -- 172.21.15.28:0/2414639279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20101b60 msgr2=0x7f3d20101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 --2- 172.21.15.28:0/2414639279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20101b60 0x7f3d20101fe0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009900 tx=0x7f3d1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 -- 172.21.15.28:0/2414639279 shutdown_connections 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 --2- 172.21.15.28:0/2414639279 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20102520 0x7f3d20108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 --2- 172.21.15.28:0/2414639279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20101b60 0x7f3d20101fe0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 --2- 172.21.15.28:0/2414639279 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 -- 172.21.15.28:0/2414639279 >> 172.21.15.28:0/2414639279 conn(0x7f3d200fbf00 msgr2=0x7f3d200fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 -- 172.21.15.28:0/2414639279 shutdown_connections 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.642+0000 7f3d24da2640 1 -- 172.21.15.28:0/2414639279 wait complete. 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 Processor -- start 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 -- start start 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20104900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 0x7f3d20103480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c5f0 con 0x7f3d20102520 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c470 con 0x7f3d20100970 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.643+0000 7f3d24da2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c770 con 0x7f3d20101b60 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20104900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 0x7f3d20103480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44750/0 (socket says 172.21.15.28:44750) 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20104900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42510/0 (socket says 172.21.15.28:42510) 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 -- 172.21.15.28:0/3563722723 learned_addr learned my addr 172.21.15.28:0/3563722723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 -- 172.21.15.28:0/3563722723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 msgr2=0x7f3d20104900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20104900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 -- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 msgr2=0x7f3d20103480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 0x7f3d20103480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 -- 172.21.15.28:0/3563722723 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d20103d10 con 0x7f3d20101b60 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 0x7f3d20103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1effd640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009230 tx=0x7f3d1402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1cff9640 1 -- 172.21.15.28:0/3563722723 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14041070 con 0x7f3d20101b60 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1cff9640 1 -- 172.21.15.28:0/3563722723 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d14004450 con 0x7f3d20101b60 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d201a1e40 con 0x7f3d20101b60 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d1cff9640 1 -- 172.21.15.28:0/3563722723 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14003e20 con 0x7f3d20101b60 2024-03-22T02:14:29.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.644+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d201a22a0 con 0x7f3d20101b60 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.647+0000 7f3d1cff9640 1 -- 172.21.15.28:0/3563722723 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f3d14044070 con 0x7f3d20101b60 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.647+0000 7f3d28be0740 1 Processor -- start 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 -- start start 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce62e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce68570 0x559afce6aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559afcdc78d0 con 0x559afce62a10 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559afcd6fe00 con 0x559afce63400 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d28be0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559afcdc3880 con 0x559afce68570 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1cff9640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 0x7f3d080a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42522/0 (socket says 172.21.15.28:42522) 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/611589940 learned_addr learned my addr 172.21.15.28:0/611589940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce62e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1effd640 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce68570 0x559afce6aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/611589940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce68570 msgr2=0x559afce6aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce68570 0x559afce6aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 0x7f3d080a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1cff9640 1 -- 172.21.15.28:0/3563722723 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d140f1370 con 0x7f3d20101b60 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/611589940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 msgr2=0x559afce62e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce62e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.648+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/611589940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559afcadb3f0 con 0x559afce63400 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009ca0 tx=0x7f3d10011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecba985469e70ae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 0x7f3d080a5470 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559afce63c60 tx=0x7f3d18004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d05ffb640 1 -- 172.21.15.28:0/611589940 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10004220 con 0x559afce63400 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d05ffb640 1 -- 172.21.15.28:0/611589940 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d100043c0 con 0x559afce63400 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d05ffb640 1 -- 172.21.15.28:0/611589940 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10019930 con 0x559afce63400 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 -- 172.21.15.28:0/611589940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 msgr2=0x559afce67f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009ca0 tx=0x7f3d10011810 comp rx=0 tx=0).stop 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 -- 172.21.15.28:0/611589940 shutdown_connections 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce68570 0x559afce6aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce67f00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 --2- 172.21.15.28:0/611589940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce62e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.649+0000 7f3d28be0740 1 -- 172.21.15.28:0/611589940 >> 172.21.15.28:0/611589940 conn(0x559afce5df80 msgr2=0x559afce5e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- 172.21.15.28:0/611589940 shutdown_connections 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- 172.21.15.28:0/611589940 wait complete. 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- start start 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 Processor -- start 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce6cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 0x559afce6d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce68570 0x559afce74480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559afce6b520 con 0x559afce62a10 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559afcd83300 con 0x559afce68570 2024-03-22T02:14:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559afce6b6a0 con 0x559afce63400 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d28be0740 10 client.0 _ll_register_callbacks cb 0x559afcdbee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce6cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d1effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 0x559afce6d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce6cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56772/0 (socket says 172.21.15.28:56772) 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/2381318459 learned_addr learned my addr 172.21.15.28:0/2381318459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.650+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce68570 0x559afce74480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 -- 172.21.15.28:0/2381318459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce68570 msgr2=0x559afce74480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce68570 0x559afce74480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 -- 172.21.15.28:0/2381318459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 msgr2=0x559afce6cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce6cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 -- 172.21.15.28:0/2381318459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559afcba57d0 con 0x559afce63400 2024-03-22T02:14:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 0x559afce6d440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d140eea20 tx=0x7f3d140b2bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76ed1489b2457fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1404b070 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d140c14b0 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d100043c0 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3d14004c60 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d28be0740 10 client.25552 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.651+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559afce6b590 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d140bd6f0 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3d140bd950 con 0x559afce63400 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 10 client.25552 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 10 client.25552 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 20 client.25552 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce78ed0 0x559afce7b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 4 client.25552 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f3d100043c0 con 0x559afce78ed0 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce78ed0 0x559afce7b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce78ed0 0x559afce7b370 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95b549f48b41e348 server_cookie=b2865838d9fd6fca in_seq=0 out_seq=0 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.652+0000 7f3cfeffd640 10 client.25552 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2381318459 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f3d100043c0 con 0x559afce78ed0 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3cfeffd640 10 client.25552 handle_command_reply: tid=0 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 client.25552 shutdown 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 msgr2=0x559afce6d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 0x559afce6d440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d140eea20 tx=0x7f3d140b2bd0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 shutdown_connections 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce78ed0 0x559afce7b370 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce68570 0x559afce74480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce63400 0x559afce6d440 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3cfeffd640 0 client.25552 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2381318459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce6cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 >> 172.21.15.28:0/2381318459 conn(0x559afce5df80 msgr2=0x559afce5d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 shutdown_connections 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.653+0000 7f3d28be0740 1 -- 172.21.15.28:0/2381318459 wait complete. 2024-03-22T02:14:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.654+0000 7f3d28be0740 20 client.25552 trim_cache size 0 max 16384 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.666+0000 7f3d28be0740 1 Processor -- start 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 -- start start 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce62a10 0x559afce74440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce7fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201af260 con 0x559afce5df80 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201acb30 con 0x559afce63400 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d28be0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201aa980 con 0x559afce62a10 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56778/0 (socket says 172.21.15.28:56778) 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/3090773859 learned_addr learned my addr 172.21.15.28:0/3090773859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1effd640 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce62a10 0x559afce74440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce7fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/3090773859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce62a10 msgr2=0x559afce74440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce62a10 0x559afce74440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/3090773859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 msgr2=0x559afce7fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce7fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 -- 172.21.15.28:0/3090773859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559afce637e0 con 0x559afce5df80 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3d10003fd0 tx=0x7f3d10011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f2cbc31e770f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d057fa640 1 -- 172.21.15.28:0/3090773859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1002bd70 con 0x559afce5df80 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d057fa640 1 -- 172.21.15.28:0/3090773859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10005040 con 0x559afce5df80 2024-03-22T02:14:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.667+0000 7f3d057fa640 1 -- 172.21.15.28:0/3090773859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1002b820 con 0x559afce5df80 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- 172.21.15.28:0/3090773859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 msgr2=0x559afce618f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3d10003fd0 tx=0x7f3d10011810 comp rx=0 tx=0).stop 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- 172.21.15.28:0/3090773859 shutdown_connections 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce7fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce62a10 0x559afce74440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 --2- 172.21.15.28:0/3090773859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce5df80 0x559afce618f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- 172.21.15.28:0/3090773859 >> 172.21.15.28:0/3090773859 conn(0x559afce641e0 msgr2=0x559afce611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- 172.21.15.28:0/3090773859 shutdown_connections 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- 172.21.15.28:0/3090773859 wait complete. 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 -- start start 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.668+0000 7f3d28be0740 1 Processor -- start 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce5df80 0x559afce74070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce72c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201aaff0 con 0x559afce62a10 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201ac460 con 0x559afce63400 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d201ab6c0 con 0x559afce5df80 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 10 client.0 _ll_register_callbacks cb 0x7f3d140b92d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56794/0 (socket says 172.21.15.28:56794) 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 -- 172.21.15.28:0/2780701914 learned_addr learned my addr 172.21.15.28:0/2780701914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce5df80 0x559afce74070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1ffff640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce72c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 -- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce5df80 msgr2=0x559afce74070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce5df80 0x559afce74070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 -- 172.21.15.28:0/2780701914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 msgr2=0x559afce72c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce72c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 -- 172.21.15.28:0/2780701914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559afce731f0 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d1effd640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d14004e00 tx=0x7f3d14009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ae3fdd6d0e8226 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3cfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1404b070 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3d140d6600 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.669+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d10005040 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3d140b7050 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 10 client.15729 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559afcee00a0 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d10005040 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3d140d6260 con 0x559afce62a10 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 10 client.15729 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 10 client.15729 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:14:29.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 20 client.15729 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce68570 0x559afcee2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 4 client.15729 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559afcdd4e10 con 0x559afce68570 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce68570 0x559afcee2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3cfeffd640 10 client.15729 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.670+0000 7f3d1f7fe640 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce68570 0x559afcee2900 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=da1f715ba41c48d3 server_cookie=d1a9b70273b895c6 in_seq=0 out_seq=0 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3cfeffd640 1 -- 172.21.15.28:0/2780701914 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559afcdd4e10 con 0x559afce68570 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3cfeffd640 10 client.15729 handle_command_reply: tid=0 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 client.15729 shutdown 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 msgr2=0x559afce72740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3d14004e00 tx=0x7f3d14009260 comp rx=0 tx=0).stop 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 shutdown_connections 2024-03-22T02:14:29.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559afce68570 0x559afcee2900 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559afce63400 0x559afce72c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3cfeffd640 0 client.15729 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559afce62a10 0x559afce72740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 --2- 172.21.15.28:0/2780701914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559afce5df80 0x559afce74070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 >> 172.21.15.28:0/2780701914 conn(0x559afce641e0 msgr2=0x559afce61a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 shutdown_connections 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 1 -- 172.21.15.28:0/2780701914 wait complete. 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d28be0740 20 client.15729 trim_cache size 0 max 16384 2024-03-22T02:14:29.679 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:14:29.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 msgr2=0x7f3d080a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 0x7f3d080a5470 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x559afce63c60 tx=0x7f3d18004ab0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 msgr2=0x7f3d20102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.671+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009230 tx=0x7f3d1402c650 comp rx=0 tx=0).stop 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 shutdown_connections 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d080a2fb0 0x7f3d080a5470 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d20102520 0x7f3d20103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d20101b60 0x7f3d20102f40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 --2- 172.21.15.28:0/3563722723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d20100970 0x7f3d20104900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 >> 172.21.15.28:0/3563722723 conn(0x7f3d200fbf00 msgr2=0x7f3d200fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 shutdown_connections 2024-03-22T02:14:29.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.672+0000 7f3d24da2640 1 -- 172.21.15.28:0/3563722723 wait complete. 2024-03-22T02:14:29.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:29 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1210764429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:14:29.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:14:29.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:29 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1210764429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:14:29.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:29 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1210764429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- 172.21.15.28:0/170469949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608101c80 msgr2=0x7f4608102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- 172.21.15.28:0/170469949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608101c80 0x7f4608102100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009900 tx=0x7f45f802bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- 172.21.15.28:0/170469949 shutdown_connections 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- 172.21.15.28:0/170469949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608102640 0x7f4608108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- 172.21.15.28:0/170469949 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608101c80 0x7f4608102100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- 172.21.15.28:0/170469949 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608100a90 0x7f4608100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- 172.21.15.28:0/170469949 >> 172.21.15.28:0/170469949 conn(0x7f46080fc020 msgr2=0x7f46080fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- 172.21.15.28:0/170469949 shutdown_connections 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- 172.21.15.28:0/170469949 wait complete. 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 Processor -- start 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- start start 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 0x7f460819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 0x7f460819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608102640 0x7f46081a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f460810c6f0 con 0x7f4608100a90 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f460810c570 con 0x7f4608101c80 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.787+0000 7f4610713640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f460810c870 con 0x7f4608102640 2024-03-22T02:14:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 0x7f460819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 0x7f460819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460ecb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608102640 0x7f46081a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 0x7f460819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56808/0 (socket says 172.21.15.28:56808) 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460e4af640 1 -- 172.21.15.28:0/3141448623 learned_addr learned my addr 172.21.15.28:0/3141448623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 -- 172.21.15.28:0/3141448623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608102640 msgr2=0x7f46081a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608102640 0x7f46081a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 -- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 msgr2=0x7f460819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 0x7f460819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 -- 172.21.15.28:0/3141448623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46081a1d40 con 0x7f4608101c80 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f460dcae640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 0x7f460819a660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009230 tx=0x7f45f802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f45ff7fe640 1 -- 172.21.15.28:0/3141448623 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f8041070 con 0x7f4608101c80 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f45ff7fe640 1 -- 172.21.15.28:0/3141448623 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45f8004450 con 0x7f4608101c80 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46081a1fd0 con 0x7f4608101c80 2024-03-22T02:14:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f45ff7fe640 1 -- 172.21.15.28:0/3141448623 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f8003d80 con 0x7f4608101c80 2024-03-22T02:14:29.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.788+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46081a2430 con 0x7f4608101c80 2024-03-22T02:14:29.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.790+0000 7f45ff7fe640 1 -- 172.21.15.28:0/3141448623 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f45f8044070 con 0x7f4608101c80 2024-03-22T02:14:29.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 Processor -- start 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 -- start start 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 0x558d23096c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d2309bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d22ffb8d0 con 0x558d2309c380 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558d22fa3e00 con 0x558d23097210 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f46133de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558d22ff7880 con 0x558d23096820 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460dcae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460ecb0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 0x558d23096c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460dcae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56812/0 (socket says 172.21.15.28:56812) 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460dcae640 1 -- 172.21.15.28:0/3708784210 learned_addr learned my addr 172.21.15.28:0/3708784210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460e4af640 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d2309bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f45ff7fe640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 0x7f45f40a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460ecb0640 1 -- 172.21.15.28:0/3708784210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 msgr2=0x558d2309bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d2309bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.791+0000 7f460ecb0640 1 -- 172.21.15.28:0/3708784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 msgr2=0x558d2309e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460ecb0640 1 -- 172.21.15.28:0/3708784210 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d22d0f3f0 con 0x558d23096820 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460dcae640 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 0x7f45f40a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f45ff7fe640 1 -- 172.21.15.28:0/3141448623 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f45f80f12b0 con 0x7f4608101c80 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 0x558d23096c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4600011490 tx=0x7f4600011950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=879a5a912ef8b78d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f45fcff9640 1 -- 172.21.15.28:0/3708784210 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4600017070 con 0x558d23096820 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f45fcff9640 1 -- 172.21.15.28:0/3708784210 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460000fc20 con 0x558d23096820 2024-03-22T02:14:29.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f45fcff9640 1 -- 172.21.15.28:0/3708784210 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460001c970 con 0x558d23096820 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.792+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 0x7f45f40a5470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4600023000 tx=0x7f4600024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- 172.21.15.28:0/3708784210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 msgr2=0x558d23096c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 0x558d23096c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4600011490 tx=0x7f4600011950 comp rx=0 tx=0).stop 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- 172.21.15.28:0/3708784210 shutdown_connections 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d2309c380 0x558d2309e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d2309bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- 172.21.15.28:0/3708784210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23096820 0x558d23096c40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- 172.21.15.28:0/3708784210 >> 172.21.15.28:0/3708784210 conn(0x558d23091d90 msgr2=0x558d23092190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- 172.21.15.28:0/3708784210 shutdown_connections 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- 172.21.15.28:0/3708784210 wait complete. 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- start start 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 Processor -- start 2024-03-22T02:14:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d230a1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 0x558d230a82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558d2309f330 con 0x558d23096820 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558d22fb7300 con 0x558d23097210 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558d2309f4b0 con 0x558d2309c380 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f46133de740 10 client.0 _ll_register_callbacks cb 0x558d22ff2e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56828/0 (socket says 172.21.15.28:56828) 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.793+0000 7f460e4af640 1 -- 172.21.15.28:0/3564102427 learned_addr learned my addr 172.21.15.28:0/3564102427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460dcae640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d230a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 0x558d230a82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 -- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 msgr2=0x558d230a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d230a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 -- 172.21.15.28:0/3564102427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 msgr2=0x558d230a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 -- 172.21.15.28:0/3564102427 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d22dd97d0 con 0x558d2309c380 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460e4af640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f460ecb0640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 0x558d230a82d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4600009ab0 tx=0x7f4600009d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ce53fd6a9efd871 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f45f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4600017040 con 0x558d2309c380 2024-03-22T02:14:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f460000fc20 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f460000f740 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f460001c9c0 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f46133de740 10 client.25570 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.794+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558d230ab700 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460000fc20 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f460001b070 con 0x558d2309c380 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 10 client.25570 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 10 client.25570 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 20 client.25570 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d230acb40 0x558d230aef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 4 client.25570 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d23008e10 con 0x558d230acb40 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f460e4af640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d230acb40 0x558d230aef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f460e4af640 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d230acb40 0x558d230aef90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b40be608224d430 server_cookie=30d99937f84ea798 in_seq=0 out_seq=0 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.795+0000 7f45f1ffb640 10 client.25570 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f45f1ffb640 1 -- 172.21.15.28:0/3564102427 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558d23008e10 con 0x558d230acb40 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f45f1ffb640 10 client.25570 handle_command_reply: tid=0 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 client.25570 shutdown 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 msgr2=0x558d230a82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 0x558d230a82d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4600009ab0 tx=0x7f4600009d50 comp rx=0 tx=0).stop 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 shutdown_connections 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d230acb40 0x558d230aef90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d2309c380 0x558d230a82d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d23097210 0x558d230a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f45f1ffb640 0 client.25570 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 --2- 172.21.15.28:0/3564102427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 >> 172.21.15.28:0/3564102427 conn(0x558d23091d90 msgr2=0x558d23090fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 shutdown_connections 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.796+0000 7f46133de740 1 -- 172.21.15.28:0/3564102427 wait complete. 2024-03-22T02:14:29.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.797+0000 7f46133de740 20 client.25570 trim_cache size 0 max 16384 2024-03-22T02:14:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 Processor -- start 2024-03-22T02:14:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 -- start start 2024-03-22T02:14:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a8290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23090c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46081af4b0 con 0x558d23096820 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f46081acd80 con 0x558d2309c380 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f46133de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46081aabd0 con 0x558d23097210 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56842/0 (socket says 172.21.15.28:56842) 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460dcae640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 -- 172.21.15.28:0/2910602828 learned_addr learned my addr 172.21.15.28:0/2910602828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460ecb0640 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23090c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 -- 172.21.15.28:0/2910602828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 msgr2=0x558d230a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 -- 172.21.15.28:0/2910602828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 msgr2=0x558d23090c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23090c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 -- 172.21.15.28:0/2910602828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d23096460 con 0x558d23096820 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.810+0000 7f460e4af640 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x558d230af7a0 tx=0x7f4604005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50c6024aaca458b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f45f3fff640 1 -- 172.21.15.28:0/2910602828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460401f110 con 0x558d23096820 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f45f3fff640 1 -- 172.21.15.28:0/2910602828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4604023020 con 0x558d23096820 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f45f3fff640 1 -- 172.21.15.28:0/2910602828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604010cd0 con 0x558d23096820 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- 172.21.15.28:0/2910602828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 msgr2=0x558d23090740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x558d230af7a0 tx=0x7f4604005f90 comp rx=0 tx=0).stop 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- 172.21.15.28:0/2910602828 shutdown_connections 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23090c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 --2- 172.21.15.28:0/2910602828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d23090740 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- 172.21.15.28:0/2910602828 >> 172.21.15.28:0/2910602828 conn(0x558d23091d90 msgr2=0x558d230a0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- 172.21.15.28:0/2910602828 shutdown_connections 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- 172.21.15.28:0/2910602828 wait complete. 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 -- start start 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.811+0000 7f46133de740 1 Processor -- start 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23110e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46081ab240 con 0x558d23096820 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f46081ac6b0 con 0x558d2309c380 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f46081ab910 con 0x558d23097210 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 10 client.0 _ll_register_callbacks cb 0x7f45f80f4c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56854/0 (socket says 172.21.15.28:56854) 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 -- 172.21.15.28:0/260928359 learned_addr learned my addr 172.21.15.28:0/260928359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460dcae640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460ecb0640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23110e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 -- 172.21.15.28:0/260928359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 msgr2=0x558d230a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 -- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 msgr2=0x558d23110e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23110e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 -- 172.21.15.28:0/260928359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d23111470 con 0x558d23096820 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f460e4af640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4604012fd0 tx=0x7f4604012200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a32b94167786abc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f45f1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4604010a50 con 0x558d23096820 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4604010bf0 con 0x558d23096820 2024-03-22T02:14:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.812+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4604023020 con 0x558d23096820 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f460403a050 con 0x558d23096820 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 10 client.15747 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558d23112710 con 0x558d23096820 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4604023020 con 0x558d23096820 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f460401f510 con 0x558d23096820 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 10 client.15747 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 20 client.15747 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d231130b0 0x558d231154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 4 client.15747 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558d23003160 con 0x558d231130b0 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f460dcae640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d231130b0 0x558d231154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:29.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f45f1ffb640 10 client.15747 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.813+0000 7f460dcae640 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d231130b0 0x558d231154d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fa87109d88a339c server_cookie=2cf720ac752868c9 in_seq=0 out_seq=0 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f45f1ffb640 1 -- 172.21.15.28:0/260928359 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558d23003160 con 0x558d231130b0 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f45f1ffb640 10 client.15747 handle_command_reply: tid=0 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 client.15747 shutdown 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 msgr2=0x558d230a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4604012fd0 tx=0x7f4604012200 comp rx=0 tx=0).stop 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 shutdown_connections 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558d231130b0 0x558d231154d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558d2309c380 0x558d23110e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f45f1ffb640 0 client.15747 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558d23097210 0x558d230a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 --2- 172.21.15.28:0/260928359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558d23096820 0x558d230a52a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 >> 172.21.15.28:0/260928359 conn(0x558d23091d90 msgr2=0x558d23094ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 shutdown_connections 2024-03-22T02:14:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 1 -- 172.21.15.28:0/260928359 wait complete. 2024-03-22T02:14:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f46133de740 20 client.15747 trim_cache size 0 max 16384 2024-03-22T02:14:29.823 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 msgr2=0x7f45f40a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 0x7f45f40a5470 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4600023000 tx=0x7f4600024000 comp rx=0 tx=0).stop 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 msgr2=0x7f460819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 0x7f460819a660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45f8009230 tx=0x7f45f802c650 comp rx=0 tx=0).stop 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 shutdown_connections 2024-03-22T02:14:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45f40a2fb0 0x7f45f40a5470 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4608102640 0x7f46081a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4608101c80 0x7f460819a660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 --2- 172.21.15.28:0/3141448623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4608100a90 0x7f460819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.814+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 >> 172.21.15.28:0/3141448623 conn(0x7f46080fc020 msgr2=0x7f46080fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.815+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 shutdown_connections 2024-03-22T02:14:29.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:29.815+0000 7f4610713640 1 -- 172.21.15.28:0/3141448623 wait complete. 2024-03-22T02:14:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:30 smithi120 ceph-mon[36803]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:30.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:30 smithi083 ceph-mon[36340]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:30 smithi028 ceph-mon[31916]: pgmap v385: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:30.827 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:14:30.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:14:30.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:14:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- 172.21.15.28:0/1394371820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8102660 msgr2=0x7f3ad8108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 --2- 172.21.15.28:0/1394371820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8102660 0x7f3ad8108ef0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009ca0 tx=0x7f3ad002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- 172.21.15.28:0/1394371820 shutdown_connections 2024-03-22T02:14:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 --2- 172.21.15.28:0/1394371820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8102660 0x7f3ad8108ef0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 --2- 172.21.15.28:0/1394371820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 --2- 172.21.15.28:0/1394371820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- 172.21.15.28:0/1394371820 >> 172.21.15.28:0/1394371820 conn(0x7f3ad80fc040 msgr2=0x7f3ad80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- 172.21.15.28:0/1394371820 shutdown_connections 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- 172.21.15.28:0/1394371820 wait complete. 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 Processor -- start 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.946+0000 7f3adf17a640 1 -- start start 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8102660 0x7f3ad819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810d810 con 0x7f3ad8101ca0 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810d690 con 0x7f3ad8100ab0 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad810d990 con 0x7f3ad8102660 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adcf16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adcf16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42636/0 (socket says 172.21.15.28:42636) 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56874/0 (socket says 172.21.15.28:56874) 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adcf16640 1 -- 172.21.15.28:0/4293069638 learned_addr learned my addr 172.21.15.28:0/4293069638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3add717640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8102660 0x7f3ad819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4293069638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8102660 msgr2=0x7f3ad819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8102660 0x7f3ad819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4293069638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 msgr2=0x7f3ad8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4293069638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad819fa70 con 0x7f3ad8101ca0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad5ffb640 1 -- 172.21.15.28:0/4293069638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400ccc0 con 0x7f3ad8101ca0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad819fd60 con 0x7f3ad8101ca0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad5ffb640 1 -- 172.21.15.28:0/4293069638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac4010910 con 0x7f3ad8101ca0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.947+0000 7f3ad5ffb640 1 -- 172.21.15.28:0/4293069638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4018960 con 0x7f3ad8101ca0 2024-03-22T02:14:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.948+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad81a0250 con 0x7f3ad8101ca0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.950+0000 7f3ad5ffb640 1 -- 172.21.15.28:0/4293069638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f3ac4010ac0 con 0x7f3ad8101ca0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 Processor -- start 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 -- start start 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab08bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a77aa65570 con 0x55a77ab06720 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a77aa0de10 con 0x55a77ab015b0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a77aa61520 con 0x55a77ab00bc0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab08bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44868/0 (socket says 172.21.15.28:44868) 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 -- 172.21.15.28:0/2419002072 learned_addr learned my addr 172.21.15.28:0/2419002072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3adcf16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ad5ffb640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 0x7f3ac80a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 -- 172.21.15.28:0/2419002072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 msgr2=0x55a77ab060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 -- 172.21.15.28:0/2419002072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 msgr2=0x55a77ab08bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab08bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 -- 172.21.15.28:0/2419002072 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77a8417e0 con 0x55a77ab00bc0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3ad5ffb640 1 -- 172.21.15.28:0/4293069638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3ac40cfd00 con 0x7f3ad8101ca0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.951+0000 7f3add717640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 0x7f3ac80a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3add717640 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009ca0 tx=0x7f3ad0005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c46af2d983b6c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ac2ffd640 1 -- 172.21.15.28:0/2419002072 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0039040 con 0x55a77ab00bc0 2024-03-22T02:14:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ac2ffd640 1 -- 172.21.15.28:0/2419002072 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ad0003e50 con 0x55a77ab00bc0 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ac2ffd640 1 -- 172.21.15.28:0/2419002072 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0034440 con 0x55a77ab00bc0 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 -- 172.21.15.28:0/2419002072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 msgr2=0x55a77ab00fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3add717640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 0x7f3ac80a5470 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3ad004c000 tx=0x7f3ad0045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009ca0 tx=0x7f3ad0005f70 comp rx=0 tx=0).stop 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 -- 172.21.15.28:0/2419002072 shutdown_connections 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab08bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/2419002072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab00fe0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.952+0000 7f3ae1d66740 1 -- 172.21.15.28:0/2419002072 >> 172.21.15.28:0/2419002072 conn(0x55a77aafc130 msgr2=0x55a77aafc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- 172.21.15.28:0/2419002072 shutdown_connections 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- 172.21.15.28:0/2419002072 wait complete. 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- start start 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 Processor -- start 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a77ab096d0 con 0x55a77ab06720 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a77aa21110 con 0x55a77ab015b0 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a77ab09850 con 0x55a77ab00bc0 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ae1d66740 10 client.0 _ll_register_callbacks cb 0x55a77aa5dbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3adcf16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3add717640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3adcf16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44884/0 (socket says 172.21.15.28:44884) 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3add717640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56884/0 (socket says 172.21.15.28:56884) 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42660/0 (socket says 172.21.15.28:42660) 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.953+0000 7f3adcf16640 1 -- 172.21.15.28:0/3682452802 learned_addr learned my addr 172.21.15.28:0/3682452802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 -- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 msgr2=0x55a77ab0b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 -- 172.21.15.28:0/3682452802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 msgr2=0x55a77ab123b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 -- 172.21.15.28:0/3682452802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77a8a1f70 con 0x55a77ab00bc0 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3add717640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:30.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3acc012700 tx=0x7f3acc012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b5dbb40ffec1a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3acc00f8c0 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3acc00fa60 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ad0003e50 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3acc025050 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ae1d66740 10 client.25588 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.954+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a77a739170 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ad0003e50 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3acc010430 con 0x55a77ab00bc0 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 10 client.25588 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 10 client.25588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 20 client.25588 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:30.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab16d90 0x55a77ab19230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 4 client.25588 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a77aa72ab0 con 0x55a77ab16d90 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab16d90 0x55a77ab19230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ab7fff640 10 client.25588 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.955+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab16d90 0x55a77ab19230 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1c28972ce950c4e server_cookie=356fa39b92fe978f in_seq=0 out_seq=0 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3682452802 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a77aa72ab0 con 0x55a77ab16d90 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ab7fff640 10 client.25588 handle_command_reply: tid=0 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 client.25588 shutdown 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 msgr2=0x55a77ab0ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3acc012700 tx=0x7f3acc012bc0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 shutdown_connections 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab16d90 0x55a77ab19230 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab06720 0x55a77ab123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab015b0 0x55a77ab0b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3682452802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0ae30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ab7fff640 0 client.25588 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 >> 172.21.15.28:0/3682452802 conn(0x55a77aafc130 msgr2=0x55a77aafb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 shutdown_connections 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3682452802 wait complete. 2024-03-22T02:14:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.956+0000 7f3ae1d66740 20 client.25588 trim_cache size 0 max 16384 2024-03-22T02:14:30.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 Processor -- start 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 -- start start 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab01fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab02a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81ad1f0 con 0x55a77ab015b0 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81aaac0 con 0x55a77ab06720 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81a8910 con 0x55a77ab00bc0 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56888/0 (socket says 172.21.15.28:56888) 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4033769562 learned_addr learned my addr 172.21.15.28:0/4033769562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3adcf16640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab01fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.974+0000 7f3add717640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab02a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4033769562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 msgr2=0x55a77ab01fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab01fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4033769562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 msgr2=0x55a77ab02a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab02a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 -- 172.21.15.28:0/4033769562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a77ab00800 con 0x55a77ab015b0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3add717640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab02a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c6a0 tx=0x7f3ac40d2260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934a3acc8f8e073d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ac27fc640 1 -- 172.21.15.28:0/4033769562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40d1040 con 0x55a77ab015b0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ac27fc640 1 -- 172.21.15.28:0/4033769562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac40a48d0 con 0x55a77ab015b0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ac27fc640 1 -- 172.21.15.28:0/4033769562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40a4440 con 0x55a77ab015b0 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 -- 172.21.15.28:0/4033769562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 msgr2=0x55a77ab02510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400c6a0 tx=0x7f3ac40d2260 comp rx=0 tx=0).stop 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 -- 172.21.15.28:0/4033769562 shutdown_connections 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab02a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab02510 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/4033769562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab01fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.975+0000 7f3ae1d66740 1 -- 172.21.15.28:0/4033769562 >> 172.21.15.28:0/4033769562 conn(0x55a77aafc130 msgr2=0x55a77aaffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- 172.21.15.28:0/4033769562 shutdown_connections 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- 172.21.15.28:0/4033769562 wait complete. 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- start start 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 Processor -- start 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab0ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81a8f80 con 0x55a77ab015b0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81aa3f0 con 0x55a77ab06720 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad81a9650 con 0x55a77ab00bc0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ae1d66740 10 client.0 _ll_register_callbacks cb 0x7f3acc009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56902/0 (socket says 172.21.15.28:56902) 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3ad7fff640 1 -- 172.21.15.28:0/3147873681 learned_addr learned my addr 172.21.15.28:0/3147873681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.976+0000 7f3add717640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab0ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 -- 172.21.15.28:0/3147873681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 msgr2=0x55a77ab0eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 -- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 msgr2=0x55a77ab0ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab0ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 -- 172.21.15.28:0/3147873681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a77ab10590 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ad7fff640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40a43a0 tx=0x7f3ac4009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc518300a5a31c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac40d1030 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3ac40a4640 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ac40a48d0 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ac409bf70 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ae1d66740 10 client.15765 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a77ab11900 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.977+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ac40a48d0 con 0x55a77ab015b0 2024-03-22T02:14:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3ac4018c90 con 0x55a77ab015b0 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 10 client.15765 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 20 client.15765 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab7d790 0x55a77ab7fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 4 client.15765 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a77aa6ce00 con 0x55a77ab7d790 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab7d790 0x55a77ab7fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3adcf16640 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab7d790 0x55a77ab7fbb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eadc9264c2bfec server_cookie=dacebe1c0b0eb906 in_seq=0 out_seq=0 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.978+0000 7f3ab7fff640 10 client.15765 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ab7fff640 1 -- 172.21.15.28:0/3147873681 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a77aa6ce00 con 0x55a77ab7d790 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ab7fff640 10 client.15765 handle_command_reply: tid=0 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 client.15765 shutdown 2024-03-22T02:14:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 msgr2=0x55a77ab0f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f3ac40a43a0 tx=0x7f3ac4009700 comp rx=0 tx=0).stop 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 shutdown_connections 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a77ab7d790 0x55a77ab7fbb0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a77ab06720 0x55a77ab0ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a77ab015b0 0x55a77ab0f030 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 --2- 172.21.15.28:0/3147873681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a77ab00bc0 0x55a77ab0eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ab7fff640 0 client.15765 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 >> 172.21.15.28:0/3147873681 conn(0x55a77aafc130 msgr2=0x55a77aafab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 shutdown_connections 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.979+0000 7f3ae1d66740 1 -- 172.21.15.28:0/3147873681 wait complete. 2024-03-22T02:14:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3ae1d66740 20 client.15765 trim_cache size 0 max 16384 2024-03-22T02:14:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:14:30.989 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:14:30.989 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c28f7788-2f52-4408-a731-a366288a79fa", 2024-03-22T02:14:30.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:14:30.989 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 msgr2=0x7f3ac80a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 0x7f3ac80a5470 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3ad004c000 tx=0x7f3ad0045000 comp rx=0 tx=0).stop 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 msgr2=0x7f3ad8198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 shutdown_connections 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3ac80a2fb0 0x7f3ac80a5470 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3ad8102660 0x7f3ad819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ad8101ca0 0x7f3ad8198390 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 --2- 172.21.15.28:0/4293069638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3ad8100ab0 0x7f3ad8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 >> 172.21.15.28:0/4293069638 conn(0x7f3ad80fc040 msgr2=0x7f3ad8104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 shutdown_connections 2024-03-22T02:14:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:30.980+0000 7f3adf17a640 1 -- 172.21.15.28:0/4293069638 wait complete. 2024-03-22T02:14:30.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:14:30.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:14:31.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 -- 172.21.15.28:0/232538085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 msgr2=0x7f1228100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 --2- 172.21.15.28:0/232538085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f1228100ec0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f1218009900 tx=0x7f121802bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 -- 172.21.15.28:0/232538085 shutdown_connections 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 --2- 172.21.15.28:0/232538085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228102650 0x7f1228108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 --2- 172.21.15.28:0/232538085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228101c90 0x7f1228102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 --2- 172.21.15.28:0/232538085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f1228100ec0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 -- 172.21.15.28:0/232538085 >> 172.21.15.28:0/232538085 conn(0x7f12280fc030 msgr2=0x7f12280fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 -- 172.21.15.28:0/232538085 shutdown_connections 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.096+0000 7f122cd81640 1 -- 172.21.15.28:0/232538085 wait complete. 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 Processor -- start 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 -- start start 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 0x7f122819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 0x7f122819f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f122810c6f0 con 0x7f1228100aa0 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f122810c570 con 0x7f1228102650 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122cd81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f122810c870 con 0x7f1228101c90 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56916/0 (socket says 172.21.15.28:56916) 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f122659c640 1 -- 172.21.15.28:0/2304318064 learned_addr learned my addr 172.21.15.28:0/2304318064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f121ffff640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 0x7f122819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.097+0000 7f1226d9d640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 0x7f122819f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 -- 172.21.15.28:0/2304318064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 msgr2=0x7f122819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 0x7f122819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 -- 172.21.15.28:0/2304318064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 msgr2=0x7f122819f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 0x7f122819f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 -- 172.21.15.28:0/2304318064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122819fb70 con 0x7f1228100aa0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f121ffff640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 0x7f122819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f1226d9d640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 0x7f122819f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122659c640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f121802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f121f7fe640 1 -- 172.21.15.28:0/2304318064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218041070 con 0x7f1228100aa0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f122819fe00 con 0x7f1228100aa0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f121f7fe640 1 -- 172.21.15.28:0/2304318064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1218004450 con 0x7f1228100aa0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f121f7fe640 1 -- 172.21.15.28:0/2304318064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1218003df0 con 0x7f1228100aa0 2024-03-22T02:14:31.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.098+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281a0260 con 0x7f1228100aa0 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.100+0000 7f121f7fe640 1 -- 172.21.15.28:0/2304318064 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f1218044070 con 0x7f1228100aa0 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 Processor -- start 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 -- start start 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f121f7fe640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 0x7f12140a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a75b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a75e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55897a6bb8c0 con 0x55897a756430 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55897a663c20 con 0x55897a75bf90 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122f969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55897a6b7870 con 0x55897a756e20 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f1226d9d640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 0x7f12140a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56930/0 (socket says 172.21.15.28:56930) 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 -- 172.21.15.28:0/834529530 learned_addr learned my addr 172.21.15.28:0/834529530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f121f7fe640 1 -- 172.21.15.28:0/2304318064 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f12180f1340 con 0x7f1228100aa0 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f121ffff640 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a75b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f1226d9d640 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a75e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 -- 172.21.15.28:0/834529530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 msgr2=0x55897a75b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a75b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 -- 172.21.15.28:0/834529530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 msgr2=0x55897a75e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a75e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.101+0000 7f122659c640 1 -- 172.21.15.28:0/834529530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897a688150 con 0x55897a756430 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.102+0000 7f1226d9d640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 0x7f12140a5470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1220009b80 tx=0x7f1220011710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.102+0000 7f122659c640 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f12180eea20 tx=0x7f1218003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b72aa079c279c1f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.102+0000 7f121cff9640 1 -- 172.21.15.28:0/834529530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121804b070 con 0x55897a756430 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.102+0000 7f121cff9640 1 -- 172.21.15.28:0/834529530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12180bdda0 con 0x55897a756430 2024-03-22T02:14:31.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.102+0000 7f121cff9640 1 -- 172.21.15.28:0/834529530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12180c17f0 con 0x55897a756430 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- 172.21.15.28:0/834529530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 msgr2=0x55897a756850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f12180eea20 tx=0x7f1218003920 comp rx=0 tx=0).stop 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- 172.21.15.28:0/834529530 shutdown_connections 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a75e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a75b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- 172.21.15.28:0/834529530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a756850 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- 172.21.15.28:0/834529530 >> 172.21.15.28:0/834529530 conn(0x55897a7519a0 msgr2=0x55897a751da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- 172.21.15.28:0/834529530 shutdown_connections 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- 172.21.15.28:0/834529530 wait complete. 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- start start 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 Processor -- start 2024-03-22T02:14:31.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756430 0x55897a760670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756e20 0x55897a760bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55897a75ed00 con 0x55897a75bf90 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55897a677130 con 0x55897a756e20 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55897a75ee80 con 0x55897a756430 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122f969740 10 client.0 _ll_register_callbacks cb 0x55897a676b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f1226d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f1226d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56944/0 (socket says 172.21.15.28:56944) 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f1226d9d640 1 -- 172.21.15.28:0/1280913326 learned_addr learned my addr 172.21.15.28:0/1280913326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f122659c640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756430 0x55897a760670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.103+0000 7f121ffff640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756e20 0x55897a760bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 -- 172.21.15.28:0/1280913326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756430 msgr2=0x55897a760670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756430 0x55897a760670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 -- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756e20 msgr2=0x55897a760bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756e20 0x55897a760bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 -- 172.21.15.28:0/1280913326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897a4997d0 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f1226d9d640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f12200098d0 tx=0x7f122001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb60eaade21e4167 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220024070 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f122001f660 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12180bdda0 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1220033050 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f122f969740 10 client.15777 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55897a3cf3f0 con 0x55897a75bf90 2024-03-22T02:14:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12180bdda0 con 0x55897a75bf90 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1220028900 con 0x55897a75bf90 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.104+0000 7f122f969740 10 client.15777 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122f969740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122f969740 20 client.15777 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a76c530 0x55897a76e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122f969740 4 client.15777 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55897a6c8e00 con 0x55897a76c530 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122659c640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a76c530 0x55897a76e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f122659c640 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a76c530 0x55897a76e9b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=695a11f78b3fa03 server_cookie=cfe95f99056452fb in_seq=0 out_seq=0 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.105+0000 7f120dffb640 10 client.15777 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f120dffb640 1 -- 172.21.15.28:0/1280913326 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55897a6c8e00 con 0x55897a76c530 2024-03-22T02:14:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f120dffb640 10 client.15777 handle_command_reply: tid=0 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 client.15777 shutdown 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 msgr2=0x55897a767bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f12200098d0 tx=0x7f122001d000 comp rx=0 tx=0).stop 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 shutdown_connections 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a76c530 0x55897a76e9b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a75bf90 0x55897a767bf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756e20 0x55897a760bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 --2- 172.21.15.28:0/1280913326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756430 0x55897a760670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f120dffb640 0 client.15777 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 >> 172.21.15.28:0/1280913326 conn(0x55897a7519a0 msgr2=0x55897a750c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 shutdown_connections 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 1 -- 172.21.15.28:0/1280913326 wait complete. 2024-03-22T02:14:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.106+0000 7f122f969740 20 client.15777 trim_cache size 0 max 16384 2024-03-22T02:14:31.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.123+0000 7f122f969740 1 Processor -- start 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.123+0000 7f122f969740 1 -- start start 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 0x55897a75fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a75bf90 0x55897a750370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12281ad330 con 0x55897a756e20 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f12281aac00 con 0x55897a756430 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122f969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a8a50 con 0x55897a75bf90 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56950/0 (socket says 172.21.15.28:56950) 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 -- 172.21.15.28:0/1900374834 learned_addr learned my addr 172.21.15.28:0/1900374834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122659c640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 0x55897a75fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f1226d9d640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a75bf90 0x55897a750370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 -- 172.21.15.28:0/1900374834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a75bf90 msgr2=0x55897a750370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a75bf90 0x55897a750370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 -- 172.21.15.28:0/1900374834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 msgr2=0x55897a75fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 0x55897a75fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 -- 172.21.15.28:0/1900374834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897a38f0f0 con 0x55897a756e20 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f122659c640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 0x55897a75fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f121ffff640 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f122806aa80 tx=0x7f1210005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6520b7c1f97f140e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f120ffff640 1 -- 172.21.15.28:0/1900374834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210022070 con 0x55897a756e20 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f120ffff640 1 -- 172.21.15.28:0/1900374834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12100120d0 con 0x55897a756e20 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.124+0000 7f120ffff640 1 -- 172.21.15.28:0/1900374834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1210003850 con 0x55897a756e20 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- 172.21.15.28:0/1900374834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 msgr2=0x55897a767bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f122806aa80 tx=0x7f1210005f90 comp rx=0 tx=0).stop 2024-03-22T02:14:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- 172.21.15.28:0/1900374834 shutdown_connections 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a75bf90 0x55897a750370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756e20 0x55897a767bb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- 172.21.15.28:0/1900374834 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a756430 0x55897a75fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- 172.21.15.28:0/1900374834 >> 172.21.15.28:0/1900374834 conn(0x55897a7519a0 msgr2=0x55897a760670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- 172.21.15.28:0/1900374834 shutdown_connections 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- 172.21.15.28:0/1900374834 wait complete. 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- start start 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 Processor -- start 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a765970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a766910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a90c0 con 0x55897a756430 2024-03-22T02:14:31.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f12281aa530 con 0x55897a75bf90 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f12281a9790 con 0x55897a756e20 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.125+0000 7f122f969740 10 client.0 _ll_register_callbacks cb 0x7f1220003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56956/0 (socket says 172.21.15.28:56956) 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 -- 172.21.15.28:0/64676415 learned_addr learned my addr 172.21.15.28:0/64676415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f121ffff640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a765970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f1226d9d640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a766910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 -- 172.21.15.28:0/64676415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 msgr2=0x55897a765970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a765970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 -- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 msgr2=0x55897a766910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a766910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 -- 172.21.15.28:0/64676415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897a766f00 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122659c640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f121802cec0 tx=0x7f12180057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7971618c1feac45 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f120dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f121804b040 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f12180035f0 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12100120d0 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f12180bdda0 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122f969740 10 client.15783 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55897a7d2a40 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.126+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12100120d0 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f12180eeaa0 con 0x55897a756430 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 10 client.15783 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:31.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 10 client.15783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 20 client.15783 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a7d3220 0x55897a7d5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 4 client.15783 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55897a6c3150 con 0x55897a7d3220 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f121ffff640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a7d3220 0x55897a7d5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f120dffb640 10 client.15783 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f121ffff640 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a7d3220 0x55897a7d5640 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40cc79175d130eb6 server_cookie=1b950e253f81cf in_seq=0 out_seq=0 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f120dffb640 1 -- 172.21.15.28:0/64676415 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55897a6c3150 con 0x55897a7d3220 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f120dffb640 10 client.15783 handle_command_reply: tid=0 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 client.15783 shutdown 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 msgr2=0x55897a765430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f121802cec0 tx=0x7f12180057b0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 shutdown_connections 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55897a7d3220 0x55897a7d5640 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55897a75bf90 0x55897a766910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55897a756e20 0x55897a765970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f120dffb640 0 client.15783 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 --2- 172.21.15.28:0/64676415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55897a756430 0x55897a765430 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.127+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 >> 172.21.15.28:0/64676415 conn(0x55897a7519a0 msgr2=0x55897a760670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 shutdown_connections 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122f969740 1 -- 172.21.15.28:0/64676415 wait complete. 2024-03-22T02:14:31.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122f969740 20 client.15783 trim_cache size 0 max 16384 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "c28f7788-2f52-4408-a731-a366288a79fa": { 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c28f7788-2f52-4408-a731-a366288a79fa", 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:14:31.137 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:14:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 msgr2=0x7f12140a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 0x7f12140a5470 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f1220009b80 tx=0x7f1220011710 comp rx=0 tx=0).stop 2024-03-22T02:14:31.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 msgr2=0x7f122806ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f1218000c00 tx=0x7f121802c640 comp rx=0 tx=0).stop 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 shutdown_connections 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f12140a2fb0 0x7f12140a5470 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1228102650 0x7f122819f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1228101c90 0x7f122819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 --2- 172.21.15.28:0/2304318064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1228100aa0 0x7f122806ad10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 >> 172.21.15.28:0/2304318064 conn(0x7f12280fc030 msgr2=0x7f1228105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 shutdown_connections 2024-03-22T02:14:31.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:14:31.128+0000 7f122cd81640 1 -- 172.21.15.28:0/2304318064 wait complete. 2024-03-22T02:14:31.140 INFO:tasks.cephfs.filesystem:scrub status for tag:c28f7788-2f52-4408-a731-a366288a79fa - {'path': '/', 'tag': 'c28f7788-2f52-4408-a731-a366288a79fa', 'options': 'recursive,force'} 2024-03-22T02:14:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:32 smithi120 ceph-mon[36803]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:14:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:32 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:14:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:32 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:32 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:14:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:32 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:32.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:32 smithi083 ceph-mon[36340]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:14:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:32 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:14:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:32 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:32 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:14:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:32 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:32 smithi028 ceph-mon[31916]: pgmap v386: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-22T02:14:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:32 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:14:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:32 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:32 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:14:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:32 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:14:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:34 smithi120 ceph-mon[36803]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:14:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:34 smithi083 ceph-mon[36340]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:14:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:34 smithi028 ceph-mon[31916]: pgmap v387: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:14:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:35 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:14:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:35 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:14:35.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:35 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:14:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:36 smithi120 ceph-mon[36803]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:36 smithi083 ceph-mon[36340]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:36 smithi028 ceph-mon[31916]: pgmap v388: 129 pgs: 129 active+clean; 659 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 -- 172.21.15.28:0/1378092023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370100ab0 msgr2=0x7f3370100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 --2- 172.21.15.28:0/1378092023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370100ab0 0x7f3370100ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3360009900 tx=0x7f336002bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 -- 172.21.15.28:0/1378092023 shutdown_connections 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 --2- 172.21.15.28:0/1378092023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f3370108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 --2- 172.21.15.28:0/1378092023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370101ca0 0x7f3370102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 --2- 172.21.15.28:0/1378092023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370100ab0 0x7f3370100ed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 -- 172.21.15.28:0/1378092023 >> 172.21.15.28:0/1378092023 conn(0x7f33700fc020 msgr2=0x7f33700fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 -- 172.21.15.28:0/1378092023 shutdown_connections 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.927+0000 7f3376866640 1 -- 172.21.15.28:0/1378092023 wait complete. 2024-03-22T02:14:37.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 Processor -- start 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 -- start start 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 0x7f3370197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 0x7f33701983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f337019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f337010c700 con 0x7f3370102660 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f337010c580 con 0x7f3370101ca0 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3376866640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f337010c880 con 0x7f3370100ab0 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3374e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f337019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 0x7f33701983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3374e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f337019f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57336/0 (socket says 172.21.15.28:57336) 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f3374e03640 1 -- 172.21.15.28:0/1067872578 learned_addr learned my addr 172.21.15.28:0/1067872578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.928+0000 7f336ffff640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 0x7f3370197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 -- 172.21.15.28:0/1067872578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 msgr2=0x7f3370197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 0x7f3370197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 -- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 msgr2=0x7f337019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f337019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 -- 172.21.15.28:0/1067872578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337019fa90 con 0x7f3370101ca0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336ffff640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 0x7f3370197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336f7fe640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 0x7f33701983b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f336400e9f0 tx=0x7f336400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336d7fa640 1 -- 172.21.15.28:0/1067872578 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336400cd80 con 0x7f3370101ca0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336d7fa640 1 -- 172.21.15.28:0/1067872578 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33640035f0 con 0x7f3370101ca0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337019fd80 con 0x7f3370101ca0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f336d7fa640 1 -- 172.21.15.28:0/1067872578 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3364010690 con 0x7f3370101ca0 2024-03-22T02:14:37.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.929+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701a0270 con 0x7f3370101ca0 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.931+0000 7f337944a740 1 Processor -- start 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.931+0000 7f337944a740 1 -- start start 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.931+0000 7f336d7fa640 1 -- 172.21.15.28:0/1067872578 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7f3364003b30 con 0x7f3370101ca0 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3357754e0 0x55e335775900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e33577d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e3356da040 con 0x55e3357754e0 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e3356829e0 con 0x55e335775ed0 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f337944a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e3356d5ff0 con 0x55e33577b040 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f3374e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3357754e0 0x55e335775900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46378/0 (socket says 172.21.15.28:46378) 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 -- 172.21.15.28:0/3181587327 learned_addr learned my addr 172.21.15.28:0/3181587327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336f7fe640 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e33577d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 -- 172.21.15.28:0/3181587327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 msgr2=0x55e33577d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e33577d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 -- 172.21.15.28:0/3181587327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3357754e0 msgr2=0x55e335775900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3357754e0 0x55e335775900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 -- 172.21.15.28:0/3181587327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3353ee210 con 0x55e335775ed0 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336f7fe640 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e33577d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336d7fa640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 0x7f33580a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f336ffff640 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3360009900 tx=0x7f336002cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de7279a6e15b13f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f3374e03640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 0x7f33580a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f33567fc640 1 -- 172.21.15.28:0/3181587327 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360005d50 con 0x55e335775ed0 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f33567fc640 1 -- 172.21.15.28:0/3181587327 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3360005520 con 0x55e335775ed0 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.932+0000 7f33567fc640 1 -- 172.21.15.28:0/3181587327 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360004b80 con 0x55e335775ed0 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f336d7fa640 1 -- 172.21.15.28:0/1067872578 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f33640cfc40 con 0x7f3370101ca0 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- 172.21.15.28:0/3181587327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 msgr2=0x55e33577a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f3374e03640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 0x7f33580a5520 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55e335776730 tx=0x7f3368009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3360009900 tx=0x7f336002cd30 comp rx=0 tx=0).stop 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- 172.21.15.28:0/3181587327 shutdown_connections 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e33577d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e33577a9d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 --2- 172.21.15.28:0/3181587327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e3357754e0 0x55e335775900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- 172.21.15.28:0/3181587327 >> 172.21.15.28:0/3181587327 conn(0x55e335770a50 msgr2=0x55e335770e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- 172.21.15.28:0/3181587327 shutdown_connections 2024-03-22T02:14:37.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- 172.21.15.28:0/3181587327 wait complete. 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.933+0000 7f337944a740 1 -- start start 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 Processor -- start 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e33577f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335786a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e33577df90 con 0x55e335775ed0 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e335695db0 con 0x55e3357754e0 2024-03-22T02:14:37.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e33577e110 con 0x55e33577b040 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f337944a740 10 client.0 _ll_register_callbacks cb 0x55e3353eda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57348/0 (socket says 172.21.15.28:57348) 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 -- 172.21.15.28:0/520132726 learned_addr learned my addr 172.21.15.28:0/520132726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336ffff640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e33577f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f3374e03640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335786a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 -- 172.21.15.28:0/520132726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 msgr2=0x55e335786a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335786a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 -- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 msgr2=0x55e33577f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e33577f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.934+0000 7f336f7fe640 1 -- 172.21.15.28:0/520132726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3354ba620 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f336f7fe640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3364098260 tx=0x7f3364098160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22b1f0111573d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336409c500 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3364098470 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3360005520 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3364009430 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 10 client.15795 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e335789df0 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3360005520 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f33640bab70 con 0x55e335775ed0 2024-03-22T02:14:37.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 10 client.15795 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.935+0000 7f337944a740 20 client.15795 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e33578b510 0x55e33578d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f337944a740 4 client.15795 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3356e7580 con 0x55e33578b510 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f336ffff640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e33578b510 0x55e33578d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f334f7fe640 10 client.15795 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.936+0000 7f336ffff640 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e33578b510 0x55e33578d920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1bab88e6c6d0830 server_cookie=9cf96ea503bad741 in_seq=0 out_seq=0 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f334f7fe640 1 -- 172.21.15.28:0/520132726 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e3356e7580 con 0x55e33578b510 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f334f7fe640 10 client.15795 handle_command_reply: tid=0 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 client.15795 shutdown 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 msgr2=0x55e33577f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3364098260 tx=0x7f3364098160 comp rx=0 tx=0).stop 2024-03-22T02:14:37.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 shutdown_connections 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e33578b510 0x55e33578d920 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335786a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f334f7fe640 0 client.15795 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e33577f9d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 --2- 172.21.15.28:0/520132726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e33577f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 >> 172.21.15.28:0/520132726 conn(0x55e335770a50 msgr2=0x55e33576fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 shutdown_connections 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 1 -- 172.21.15.28:0/520132726 wait complete. 2024-03-22T02:14:37.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.937+0000 7f337944a740 20 client.15795 trim_cache size 0 max 16384 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.952+0000 7f337944a740 1 Processor -- start 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.952+0000 7f337944a740 1 -- start start 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3357754e0 0x55e335777080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e3357869d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33701ad290 con 0x55e33577b040 2024-03-22T02:14:37.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33701aab60 con 0x55e335775ed0 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f337944a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33701a89b0 con 0x55e3357754e0 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e3357869d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57354/0 (socket says 172.21.15.28:57354) 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 -- 172.21.15.28:0/1555151139 learned_addr learned my addr 172.21.15.28:0/1555151139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f336ffff640 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3357754e0 0x55e335777080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 -- 172.21.15.28:0/1555151139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3357754e0 msgr2=0x55e335777080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3357754e0 0x55e335777080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 -- 172.21.15.28:0/1555151139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 msgr2=0x55e3357869d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e3357869d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 -- 172.21.15.28:0/1555151139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e33577a1a0 con 0x55e33577b040 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f336f7fe640 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e3357869d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:37.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3374e03640 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f336800fa30 tx=0x7f3368017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3360f36f77549a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3355ffb640 1 -- 172.21.15.28:0/1555151139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368004b20 con 0x55e33577b040 2024-03-22T02:14:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3355ffb640 1 -- 172.21.15.28:0/1555151139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368004cc0 con 0x55e33577b040 2024-03-22T02:14:37.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.953+0000 7f3355ffb640 1 -- 172.21.15.28:0/1555151139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368005360 con 0x55e33577b040 2024-03-22T02:14:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- 172.21.15.28:0/1555151139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 msgr2=0x55e335792220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f336800fa30 tx=0x7f3368017000 comp rx=0 tx=0).stop 2024-03-22T02:14:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- 172.21.15.28:0/1555151139 shutdown_connections 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e33577b040 0x55e335792220 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e335775ed0 0x55e3357869d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 --2- 172.21.15.28:0/1555151139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e3357754e0 0x55e335777080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- 172.21.15.28:0/1555151139 >> 172.21.15.28:0/1555151139 conn(0x55e335770a50 msgr2=0x55e335776970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- 172.21.15.28:0/1555151139 shutdown_connections 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- 172.21.15.28:0/1555151139 wait complete. 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.954+0000 7f337944a740 1 -- start start 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 Processor -- start 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e3357864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335785020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f33701a9020 con 0x55e335775ed0 2024-03-22T02:14:37.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f33701aa490 con 0x55e3357754e0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f33701a96f0 con 0x55e33577b040 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f337944a740 10 client.0 _ll_register_callbacks cb 0x7f3364003e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57358/0 (socket says 172.21.15.28:57358) 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 -- 172.21.15.28:0/2185889034 learned_addr learned my addr 172.21.15.28:0/2185889034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336ffff640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e3357864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f3374e03640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335785020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 -- 172.21.15.28:0/2185889034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 msgr2=0x55e335785020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335785020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 -- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 msgr2=0x55e3357864d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e3357864d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.955+0000 7f336f7fe640 1 -- 172.21.15.28:0/2185889034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e335785560 con 0x55e335775ed0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f336f7fe640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f336409bb00 tx=0x7f336409bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c181c6255b9204 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:37.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336409b940 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3364097b50 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3368004cc0 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f336409c500 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 10 client.15801 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55e3357f2500 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3368004cc0 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f33640a10e0 con 0x55e335775ed0 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 10 client.15801 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 20 client.15801 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e3357f2d20 0x55e3357f5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:37.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 4 client.15801 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:14:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.956+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e3356e18d0 con 0x55e3357f2d20 2024-03-22T02:14:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.957+0000 7f336ffff640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e3357f2d20 0x55e3357f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.957+0000 7f334f7fe640 10 client.15801 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.957+0000 7f336ffff640 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e3357f2d20 0x55e3357f5140 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee8bd1ed80dafdfa server_cookie=e6d7d6622d1bdd40 in_seq=0 out_seq=0 2024-03-22T02:14:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f334f7fe640 1 -- 172.21.15.28:0/2185889034 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e3356e18d0 con 0x55e3357f2d20 2024-03-22T02:14:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f334f7fe640 10 client.15801 handle_command_reply: tid=0 2024-03-22T02:14:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 client.15801 shutdown 2024-03-22T02:14:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 msgr2=0x55e335784ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f336409bb00 tx=0x7f336409bd00 comp rx=0 tx=0).stop 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 shutdown_connections 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e3357f2d20 0x55e3357f5140 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e33577b040 0x55e335785020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e335775ed0 0x55e335784ae0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 --2- 172.21.15.28:0/2185889034 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e3357754e0 0x55e3357864d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f334f7fe640 0 client.15801 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 >> 172.21.15.28:0/2185889034 conn(0x55e335770a50 msgr2=0x55e335777200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 shutdown_connections 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 1 -- 172.21.15.28:0/2185889034 wait complete. 2024-03-22T02:14:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.963+0000 7f337944a740 20 client.15801 trim_cache size 0 max 16384 2024-03-22T02:14:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:14:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:14:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:14:37.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 msgr2=0x7f33580a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 0x7f33580a5520 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55e335776730 tx=0x7f3368009500 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 msgr2=0x7f33701983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 0x7f33701983b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f336400e9f0 tx=0x7f336400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 shutdown_connections 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f33580a3060 0x7f33580a5520 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3370102660 0x7f337019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3370101ca0 0x7f33701983b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 --2- 172.21.15.28:0/1067872578 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3370100ab0 0x7f3370197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 >> 172.21.15.28:0/1067872578 conn(0x7f33700fc020 msgr2=0x7f33700fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 shutdown_connections 2024-03-22T02:14:37.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:37.964+0000 7f3376866640 1 -- 172.21.15.28:0/1067872578 wait complete. 2024-03-22T02:14:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:38 smithi120 ceph-mon[36803]: pgmap v389: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:38.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:38 smithi083 ceph-mon[36340]: pgmap v389: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:38 smithi028 ceph-mon[31916]: pgmap v389: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-22T02:14:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:40 smithi120 ceph-mon[36803]: pgmap v390: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-22T02:14:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:40 smithi083 ceph-mon[36340]: pgmap v390: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-22T02:14:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:40 smithi028 ceph-mon[31916]: pgmap v390: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1023 B/s wr, 7 op/s 2024-03-22T02:14:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:42 smithi120 ceph-mon[36803]: pgmap v391: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-22T02:14:42.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:42 smithi083 ceph-mon[36340]: pgmap v391: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-22T02:14:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:42 smithi028 ceph-mon[31916]: pgmap v391: 129 pgs: 129 active+clean; 669 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-22T02:14:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:44 smithi120 ceph-mon[36803]: pgmap v392: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-22T02:14:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:44 smithi083 ceph-mon[36340]: pgmap v392: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-22T02:14:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:44 smithi028 ceph-mon[31916]: pgmap v392: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-22T02:14:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:46 smithi120 ceph-mon[36803]: pgmap v393: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:46 smithi083 ceph-mon[36340]: pgmap v393: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:46 smithi028 ceph-mon[31916]: pgmap v393: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:48 smithi120 ceph-mon[36803]: pgmap v394: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:48 smithi083 ceph-mon[36340]: pgmap v394: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:48 smithi028 ceph-mon[31916]: pgmap v394: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-03-22T02:14:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:50 smithi120 ceph-mon[36803]: pgmap v395: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-22T02:14:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:50 smithi083 ceph-mon[36340]: pgmap v395: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-22T02:14:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:50 smithi028 ceph-mon[31916]: pgmap v395: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-22T02:14:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 -- 172.21.15.28:0/2091801178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8101ca0 msgr2=0x7fd0a8102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 --2- 172.21.15.28:0/2091801178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8101ca0 0x7fd0a8102120 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd098009900 tx=0x7fd09802bc30 comp rx=0 tx=0).stop 2024-03-22T02:14:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 -- 172.21.15.28:0/2091801178 shutdown_connections 2024-03-22T02:14:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 --2- 172.21.15.28:0/2091801178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8102660 0x7fd0a8108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 --2- 172.21.15.28:0/2091801178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8101ca0 0x7fd0a8102120 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 --2- 172.21.15.28:0/2091801178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ab0 0x7fd0a8100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.089+0000 7fd0afa50640 1 -- 172.21.15.28:0/2091801178 >> 172.21.15.28:0/2091801178 conn(0x7fd0a80fc020 msgr2=0x7fd0a80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.090+0000 7fd0afa50640 1 -- 172.21.15.28:0/2091801178 shutdown_connections 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.090+0000 7fd0afa50640 1 -- 172.21.15.28:0/2091801178 wait complete. 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.090+0000 7fd0afa50640 1 Processor -- start 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.090+0000 7fd0afa50640 1 -- start start 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.090+0000 7fd0afa50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102660 0x7fd0a81985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a819d5f0 0x7fd0a819f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810d850 con 0x7fd0a819d5f0 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810d6d0 con 0x7fd0a8100ab0 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a810d9d0 con 0x7fd0a8102660 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52466/0 (socket says 172.21.15.28:52466) 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1726199086 learned_addr learned my addr 172.21.15.28:0/1726199086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0adfed640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a819d5f0 0x7fd0a819f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102660 0x7fd0a81985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1726199086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102660 msgr2=0x7fd0a81985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102660 0x7fd0a81985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a819d5f0 msgr2=0x7fd0a819f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a819d5f0 0x7fd0a819f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1726199086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a81a0050 con 0x7fd0a8100ab0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e9c0 tx=0x7fd0a400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd09e7fc640 1 -- 172.21.15.28:0/1726199086 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a400cd80 con 0x7fd0a8100ab0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a81a0340 con 0x7fd0a8100ab0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.091+0000 7fd09e7fc640 1 -- 172.21.15.28:0/1726199086 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a4010910 con 0x7fd0a8100ab0 2024-03-22T02:14:51.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.092+0000 7fd09e7fc640 1 -- 172.21.15.28:0/1726199086 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4018980 con 0x7fd0a8100ab0 2024-03-22T02:14:51.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.092+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a81a08b0 con 0x7fd0a8100ab0 2024-03-22T02:14:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.094+0000 7fd09e7fc640 1 -- 172.21.15.28:0/1726199086 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 19) ==== 137905+0+0 (secure 0 0 0) 0x7fd0a401f020 con 0x7fd0a8100ab0 2024-03-22T02:14:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 Processor -- start 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 -- start start 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5573e9153040 con 0x5573e91f4040 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5573e90fb9e0 con 0x5573e91ee4e0 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0b2630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573e914eff0 con 0x5573e91eeed0 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52474/0 (socket says 172.21.15.28:52474) 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 -- 172.21.15.28:0/3802491728 learned_addr learned my addr 172.21.15.28:0/3802491728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:51.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 -- 172.21.15.28:0/3802491728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 msgr2=0x5573e91f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 -- 172.21.15.28:0/3802491728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 msgr2=0x5573e91f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91f64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd09e7fc640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 0x7fd0940a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.095+0000 7fd0adfed640 1 -- 172.21.15.28:0/3802491728 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e8e67210 con 0x5573e91ee4e0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd09e7fc640 1 -- 172.21.15.28:0/1726199086 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0a40cfa70 con 0x7fd0a8100ab0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd0adfed640 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0011440 tx=0x7fd0a0011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c386031d98585813 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd0adfed640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 0x7fd0940a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd0937fe640 1 -- 172.21.15.28:0/3802491728 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a001e070 con 0x5573e91ee4e0 2024-03-22T02:14:51.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd0937fe640 1 -- 172.21.15.28:0/3802491728 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a0003d50 con 0x5573e91ee4e0 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.096+0000 7fd0937fe640 1 -- 172.21.15.28:0/3802491728 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000fd10 con 0x5573e91ee4e0 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- 172.21.15.28:0/3802491728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 msgr2=0x5573e91ee900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0adfed640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 0x7fd0940a5520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0010010 tx=0x7fd0a0019c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0011440 tx=0x7fd0a0011900 comp rx=0 tx=0).stop 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- 172.21.15.28:0/3802491728 shutdown_connections 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 --2- 172.21.15.28:0/3802491728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ee900 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- 172.21.15.28:0/3802491728 >> 172.21.15.28:0/3802491728 conn(0x5573e91e9a50 msgr2=0x5573e91e9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- 172.21.15.28:0/3802491728 shutdown_connections 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- 172.21.15.28:0/3802491728 wait complete. 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 -- start start 2024-03-22T02:14:51.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.097+0000 7fd0b2630740 1 Processor -- start 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91ff9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5573e91f6ff0 con 0x5573e91f4040 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5573e910edb0 con 0x5573e91ee4e0 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5573e91f7170 con 0x5573e91eeed0 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52488/0 (socket says 172.21.15.28:52488) 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0adfed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91ff9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1748781197 learned_addr learned my addr 172.21.15.28:0/1748781197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0b2630740 10 client.0 _ll_register_callbacks cb 0x5573e8e66a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1748781197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 msgr2=0x5573e91f8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1748781197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 msgr2=0x5573e91ff9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91ff9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1748781197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e8f33620 con 0x5573e91ee4e0 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.098+0000 7fd0adfed640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0a409bbc0 tx=0x7fd0a4010f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976d8d5e55049fb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a409b770 con 0x5573e91ee4e0 2024-03-22T02:14:51.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0a0003d50 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0a40a41b0 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0a409b590 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 10 client.25655 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573e9202dd0 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a40d7040 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd0a409b080 con 0x5573e91ee4e0 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 10 client.25655 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 10 client.25655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 20 client.25655 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e9206870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 4 client.25655 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:14:51.107 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd0a0003d50 con 0x5573e9204460 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.099+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e9206870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.100+0000 7fd090ff9640 10 client.25655 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.100+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e9206870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4af30eaf8ac760b5 server_cookie=336be8d715580b31 in_seq=0 out_seq=0 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.100+0000 7fd090ff9640 1 -- 172.21.15.28:0/1748781197 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fd0a0003d50 con 0x5573e9204460 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.100+0000 7fd090ff9640 10 client.25655 handle_command_reply: tid=0 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.100+0000 7fd0b2630740 1 client.25655 shutdown 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 msgr2=0x5573e91f84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0a409bbc0 tx=0x7fd0a4010f10 comp rx=0 tx=0).stop 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 shutdown_connections 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e9206870 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd090ff9640 0 client.25655 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91f4040 0x5573e91ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91eeed0 0x5573e91f8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1748781197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91f84e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 >> 172.21.15.28:0/1748781197 conn(0x5573e91e9a50 msgr2=0x5573e91e8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 shutdown_connections 2024-03-22T02:14:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 1 -- 172.21.15.28:0/1748781197 wait complete. 2024-03-22T02:14:51.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.101+0000 7fd0b2630740 20 client.25655 trim_cache size 0 max 16384 2024-03-22T02:14:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 Processor -- start 2024-03-22T02:14:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 -- start start 2024-03-22T02:14:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ed3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91f4040 0x5573e91f80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81ad7d0 con 0x5573e91eeed0 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81ab0a0 con 0x5573e91ee4e0 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0b2630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81a8ef0 con 0x5573e91f4040 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48024/0 (socket says 172.21.15.28:48024) 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ed3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0acfeb640 1 -- 172.21.15.28:0/347646310 learned_addr learned my addr 172.21.15.28:0/347646310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0adfed640 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91f4040 0x5573e91f80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/347646310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91f4040 msgr2=0x5573e91f80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91f4040 0x5573e91f80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/347646310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 msgr2=0x5573e91ff9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/347646310 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573e91e40f0 con 0x5573e91ee4e0 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.116+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.117+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ed3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c6a0 tx=0x7fd0a40cd470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=816ffb6f380acb66 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.117+0000 7fd092ffd640 1 -- 172.21.15.28:0/347646310 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a409b770 con 0x5573e91ee4e0 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.117+0000 7fd092ffd640 1 -- 172.21.15.28:0/347646310 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a40b9030 con 0x5573e91ee4e0 2024-03-22T02:14:51.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.117+0000 7fd092ffd640 1 -- 172.21.15.28:0/347646310 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40a4640 con 0x5573e91ee4e0 2024-03-22T02:14:51.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- 172.21.15.28:0/347646310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 msgr2=0x5573e91ed3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ed3c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c6a0 tx=0x7fd0a40cd470 comp rx=0 tx=0).stop 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- 172.21.15.28:0/347646310 shutdown_connections 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91f4040 0x5573e91f80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 --2- 172.21.15.28:0/347646310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91ee4e0 0x5573e91ed3c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- 172.21.15.28:0/347646310 >> 172.21.15.28:0/347646310 conn(0x5573e91e9a50 msgr2=0x5573e91eccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- 172.21.15.28:0/347646310 shutdown_connections 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- 172.21.15.28:0/347646310 wait complete. 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 -- start start 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.118+0000 7fd0b2630740 1 Processor -- start 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 0x5573e91fc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91fcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91f4040 0x5573e9268740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91fcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81a9560 con 0x5573e91eeed0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0acfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91fcee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48040/0 (socket says 172.21.15.28:48040) 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0acfeb640 1 -- 172.21.15.28:0/1119721996 learned_addr learned my addr 172.21.15.28:0/1119721996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 0x5573e91fc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81aa9d0 con 0x5573e91f4040 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a81a9c30 con 0x5573e91ee4e0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0b2630740 10 client.0 _ll_register_callbacks cb 0x7fd0a40d32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0adfed640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91f4040 0x5573e9268740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91f4040 msgr2=0x5573e9268740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91f4040 0x5573e9268740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1119721996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 msgr2=0x5573e91fcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91fcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573e9268d30 con 0x5573e91ee4e0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd0ad7ec640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 0x5573e91fc9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4098090 tx=0x7fd0a409c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42a25fefda0894ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.119+0000 7fd090ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a409b770 con 0x5573e91ee4e0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0a409b080 con 0x5573e91ee4e0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0a40b9030 con 0x5573e91ee4e0 2024-03-22T02:14:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0a409b2c0 con 0x5573e91ee4e0 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 10 client.25648 fetch_fsmap learned FSMap version 25 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5573e9269fe0 con 0x5573e91ee4e0 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a409fd80 con 0x5573e91ee4e0 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd0a409aaf0 con 0x5573e91ee4e0 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 10 client.25648 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 10 client.25648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 20 client.25648 populate_metadata read hostname 'smithi028' 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e926c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 4 client.25648 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd0a40b9030 con 0x5573e9204460 2024-03-22T02:14:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.120+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e926c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:14:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.121+0000 7fd0acfeb640 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e926c8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83b5a1ea045dde49 server_cookie=302c37700d270b16 in_seq=0 out_seq=0 2024-03-22T02:14:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.121+0000 7fd090ff9640 10 client.25648 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd090ff9640 1 -- 172.21.15.28:0/1119721996 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd0a40b9030 con 0x5573e9204460 2024-03-22T02:14:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd090ff9640 10 client.25648 handle_command_reply: tid=0 2024-03-22T02:14:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 client.25648 shutdown 2024-03-22T02:14:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 msgr2=0x5573e91fc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 0x5573e91fc9a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4098090 tx=0x7fd0a409c430 comp rx=0 tx=0).stop 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 shutdown_connections 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5573e9204460 0x5573e926c8b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5573e91f4040 0x5573e9268740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573e91eeed0 0x5573e91fcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 --2- 172.21.15.28:0/1119721996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5573e91ee4e0 0x5573e91fc9a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 >> 172.21.15.28:0/1119721996 conn(0x5573e91e9a50 msgr2=0x5573e91f80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.124+0000 7fd090ff9640 0 client.25648 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 shutdown_connections 2024-03-22T02:14:51.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0b2630740 1 -- 172.21.15.28:0/1119721996 wait complete. 2024-03-22T02:14:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0b2630740 20 client.25648 trim_cache size 0 max 16384 2024-03-22T02:14:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:14:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:14:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:14:51.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 msgr2=0x7fd0940a5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 0x7fd0940a5520 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0010010 tx=0x7fd0a0019c30 comp rx=0 tx=0).stop 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 msgr2=0x7fd0a81980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400e9c0 tx=0x7fd0a400edc0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 shutdown_connections 2024-03-22T02:14:51.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0940a3060 0x7fd0940a5520 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a819d5f0 0x7fd0a819f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102660 0x7fd0a81985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 --2- 172.21.15.28:0/1726199086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8100ab0 0x7fd0a81980a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 >> 172.21.15.28:0/1726199086 conn(0x7fd0a80fc020 msgr2=0x7fd0a81050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.125+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 shutdown_connections 2024-03-22T02:14:51.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:14:51.126+0000 7fd0afa50640 1 -- 172.21.15.28:0/1726199086 wait complete. 2024-03-22T02:14:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:52 smithi120 ceph-mon[36803]: pgmap v396: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:52 smithi083 ceph-mon[36340]: pgmap v396: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:52 smithi028 ceph-mon[31916]: pgmap v396: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2024-03-22T02:14:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:54 smithi120 ceph-mon[36803]: pgmap v397: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-22T02:14:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:54 smithi083 ceph-mon[36340]: pgmap v397: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-22T02:14:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:54 smithi028 ceph-mon[31916]: pgmap v397: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-22T02:14:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:56 smithi120 ceph-mon[36803]: pgmap v398: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:14:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:56 smithi083 ceph-mon[36340]: pgmap v398: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:14:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:56 smithi028 ceph-mon[31916]: pgmap v398: 129 pgs: 129 active+clean; 670 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-22T02:14:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:58 smithi120 ceph-mon[36803]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:58 smithi083 ceph-mon[36340]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:58 smithi028 ceph-mon[31916]: pgmap v399: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-22T02:14:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:14:59 smithi120 ceph-mon[36803]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-22T02:14:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:14:59 smithi083 ceph-mon[36340]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-22T02:14:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:14:59 smithi028 ceph-mon[31916]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-03-22T02:15:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:00 smithi120 ceph-mon[36803]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:00.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:00 smithi083 ceph-mon[36340]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:00 smithi028 ceph-mon[31916]: pgmap v400: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:01.141 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:15:01.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/765252809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74102610 msgr2=0x7f8b74104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/765252809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74102610 0x7f8b74104a00 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009900 tx=0x7f8b7002bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/765252809 shutdown_connections 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/765252809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b74107870 0x7f8b74109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/765252809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74104f40 0x7f8b74107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/765252809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74102610 0x7f8b74104a00 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/765252809 >> 172.21.15.28:0/765252809 conn(0x7f8b740fc020 msgr2=0x7f8b740fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/765252809 shutdown_connections 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/765252809 wait complete. 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 Processor -- start 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.260+0000 7f8b7b6fb640 1 -- start start 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74107870 0x7f8b741986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 0x7f8b7419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7410d840 con 0x7f8b74104f40 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7410d6c0 con 0x7f8b7419d710 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b7b6fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7410d9c0 con 0x7f8b74107870 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79497640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79497640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48772/0 (socket says 172.21.15.28:48772) 2024-03-22T02:15:01.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79497640 1 -- 172.21.15.28:0/2181546974 learned_addr learned my addr 172.21.15.28:0/2181546974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b78c96640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74107870 0x7f8b741986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 0x7f8b7419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 -- 172.21.15.28:0/2181546974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74107870 msgr2=0x7f8b741986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74107870 0x7f8b741986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 -- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 msgr2=0x7f8b74198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 -- 172.21.15.28:0/2181546974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b74100400 con 0x7f8b7419d710 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79497640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.261+0000 7f8b79c98640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 0x7f8b7419fb00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00ec50 tx=0x7f8b6c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.262+0000 7f8b6a7fc640 1 -- 172.21.15.28:0/2181546974 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c003b70 con 0x7f8b7419d710 2024-03-22T02:15:01.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.262+0000 7f8b6a7fc640 1 -- 172.21.15.28:0/2181546974 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b6c003d10 con 0x7f8b7419d710 2024-03-22T02:15:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.262+0000 7f8b6a7fc640 1 -- 172.21.15.28:0/2181546974 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c010890 con 0x7f8b7419d710 2024-03-22T02:15:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.262+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b741006f0 con 0x7f8b7419d710 2024-03-22T02:15:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.262+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b74100b60 con 0x7f8b7419d710 2024-03-22T02:15:01.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.265+0000 7f8b6a7fc640 1 -- 172.21.15.28:0/2181546974 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8b6c003ec0 con 0x7f8b7419d710 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.265+0000 7f8b7e3bf740 1 Processor -- start 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 -- start start 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9430 0x5622a53b9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 0x5622a53c1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622a531e8c0 con 0x5622a53b9430 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5622a52c6c20 con 0x5622a53bef90 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5622a531a870 con 0x5622a53b9e20 2024-03-22T02:15:01.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b78c96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9430 0x5622a53b9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60204/0 (socket says 172.21.15.28:60204) 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 -- 172.21.15.28:0/4279196594 learned_addr learned my addr 172.21.15.28:0/4279196594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b78c96640 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 0x5622a53c1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 -- 172.21.15.28:0/4279196594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 msgr2=0x5622a53c1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 0x5622a53c1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 -- 172.21.15.28:0/4279196594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9430 msgr2=0x5622a53b9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9430 0x5622a53b9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b79497640 1 -- 172.21.15.28:0/4279196594 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a52eb150 con 0x5622a53b9e20 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.266+0000 7f8b78c96640 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 0x5622a53c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:15:01.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b6a7fc640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 0x7f8b600a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b79497640 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009230 tx=0x7f8b7002c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7f6d44fdc01cd8d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b78c96640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 0x7f8b600a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b6a7fc640 1 -- 172.21.15.28:0/2181546974 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8b6c0cfde0 con 0x7f8b7419d710 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b5f7fe640 1 -- 172.21.15.28:0/4279196594 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70042070 con 0x5622a53b9e20 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b5f7fe640 1 -- 172.21.15.28:0/4279196594 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7002c5d0 con 0x5622a53b9e20 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.267+0000 7f8b5f7fe640 1 -- 172.21.15.28:0/4279196594 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70005170 con 0x5622a53b9e20 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/4279196594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 msgr2=0x5622a53be920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b78c96640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 0x7f8b600a52c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009710 tx=0x7f8b64009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8b70009230 tx=0x7f8b7002c910 comp rx=0 tx=0).stop 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/4279196594 shutdown_connections 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53bef90 0x5622a53c1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53be920 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/4279196594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9430 0x5622a53b9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/4279196594 >> 172.21.15.28:0/4279196594 conn(0x5622a53b49a0 msgr2=0x5622a53b4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/4279196594 shutdown_connections 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/4279196594 wait complete. 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 -- start start 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.268+0000 7f8b7e3bf740 1 Processor -- start 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 0x5622a53c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53bef90 0x5622a53cae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5622a53c1ee0 con 0x5622a53b9e20 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5622a52da130 con 0x5622a53b9430 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5622a53c2060 con 0x5622a53bef90 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b7e3bf740 10 client.0 _ll_register_callbacks cb 0x5622a52d9b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b78c96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 0x5622a53c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:58856/0 (socket says 172.21.15.28:58856) 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 -- 172.21.15.28:0/3431410343 learned_addr learned my addr 172.21.15.28:0/3431410343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79c98640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53bef90 0x5622a53cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 -- 172.21.15.28:0/3431410343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53bef90 msgr2=0x5622a53cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53bef90 0x5622a53cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 -- 172.21.15.28:0/3431410343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 msgr2=0x5622a53c3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 0x5622a53c3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b79497640 1 -- 172.21.15.28:0/3431410343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a50fc7d0 con 0x5622a53b9430 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.269+0000 7f8b78c96640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 0x5622a53c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b79497640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8b70005ab0 tx=0x7f8b70004df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fdf91b0be2f2984 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70042070 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8b70005320 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b7002c5d0 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8b70005d50 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 10 client.25670 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5622a50323f0 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7002c5d0 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8b70046400 con 0x5622a53b9430 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 10 client.25670 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 10 client.25670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 20 client.25670 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a53d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 4 client.25670 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:01.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.270+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5622a532be00 con 0x5622a53cf280 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.271+0000 7f8b78c96640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a53d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.271+0000 7f8b5cff9640 10 client.25670 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.271+0000 7f8b78c96640 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a53d1670 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=288ea582b7ee4c1f server_cookie=9ef73d3cf7d0593c in_seq=0 out_seq=0 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b5cff9640 1 -- 172.21.15.28:0/3431410343 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5622a532be00 con 0x5622a53cf280 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b5cff9640 10 client.25670 handle_command_reply: tid=0 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 client.25670 shutdown 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 msgr2=0x5622a53c3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8b70005ab0 tx=0x7f8b70004df0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 shutdown_connections 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a53d1670 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b5cff9640 0 client.25670 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:01.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53bef90 0x5622a53cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53b9e20 0x5622a53c3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/3431410343 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c3910 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 >> 172.21.15.28:0/3431410343 conn(0x5622a53b49a0 msgr2=0x5622a53b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 shutdown_connections 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/3431410343 wait complete. 2024-03-22T02:15:01.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.272+0000 7f8b7e3bf740 20 client.25670 trim_cache size 0 max 16384 2024-03-22T02:15:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 Processor -- start 2024-03-22T02:15:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 -- start start 2024-03-22T02:15:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53b8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53cad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53d5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741af8d0 con 0x5622a53bef90 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741ad1a0 con 0x5622a53b9430 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741aaff0 con 0x5622a53b9e20 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53d5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b78c96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53d5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48806/0 (socket says 172.21.15.28:48806) 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79c98640 1 -- 172.21.15.28:0/1061342120 learned_addr learned my addr 172.21.15.28:0/1061342120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b78c96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53cad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60234/0 (socket says 172.21.15.28:60234) 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 -- 172.21.15.28:0/1061342120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 msgr2=0x5622a53cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 -- 172.21.15.28:0/1061342120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 msgr2=0x5622a53d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53d5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.290+0000 7f8b79497640 1 -- 172.21.15.28:0/1061342120 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5622a4ff20f0 con 0x5622a53b9430 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b79497640 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53b8310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b70046820 tx=0x7f8b70005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f6d2768f0b8813d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b5effd640 1 -- 172.21.15.28:0/1061342120 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70042040 con 0x5622a53b9430 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b5effd640 1 -- 172.21.15.28:0/1061342120 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b70046400 con 0x5622a53b9430 2024-03-22T02:15:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b5effd640 1 -- 172.21.15.28:0/1061342120 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b700044e0 con 0x5622a53b9430 2024-03-22T02:15:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/1061342120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 msgr2=0x5622a53b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.291+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53b8310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b70046820 tx=0x7f8b70005fd0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/1061342120 shutdown_connections 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53d5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/1061342120 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53b8310 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/1061342120 >> 172.21.15.28:0/1061342120 conn(0x5622a53b49a0 msgr2=0x5622a53b7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/1061342120 shutdown_connections 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/1061342120 wait complete. 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- start start 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 Processor -- start 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741ab660 con 0x5622a53bef90 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741acad0 con 0x5622a53b9430 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741abd30 con 0x5622a53b9e20 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b79c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b79497640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.292+0000 7f8b7e3bf740 10 client.0 _ll_register_callbacks cb 0x7f8b70004a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b79c98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48816/0 (socket says 172.21.15.28:48816) 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b79c98640 1 -- 172.21.15.28:0/542287660 learned_addr learned my addr 172.21.15.28:0/542287660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 -- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 msgr2=0x5622a53c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 -- 172.21.15.28:0/542287660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 msgr2=0x5622a53c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 -- 172.21.15.28:0/542287660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5622a53c9f00 con 0x5622a53b9e20 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b79497640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b79c98640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b78c96640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53c8900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5622a53d1d70 tx=0x7f8b64004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7217dae3afe96d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b5cff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64030030 con 0x5622a53b9e20 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b70046400 con 0x5622a53b9e20 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8b6402c430 con 0x5622a53b9e20 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8b64039050 con 0x5622a53b9e20 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b7e3bf740 10 client.25666 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.293+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5622a5435490 con 0x5622a53b9e20 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.294+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b70046400 con 0x5622a53b9e20 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8b64005020 con 0x5622a53b9e20 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 10 client.25666 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 20 client.25666 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a5437e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 4 client.25666 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5622a5326150 con 0x5622a53cf280 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b79497640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a5437e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b5cff9640 10 client.25666 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b79497640 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a5437e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3130da263abef1 server_cookie=a65fcd531dc58ad7 in_seq=0 out_seq=0 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b5cff9640 1 -- 172.21.15.28:0/542287660 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5622a5326150 con 0x5622a53cf280 2024-03-22T02:15:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b5cff9640 10 client.25666 handle_command_reply: tid=0 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 client.25666 shutdown 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 msgr2=0x5622a53c8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53c8900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x5622a53d1d70 tx=0x7f8b64004730 comp rx=0 tx=0).stop 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 shutdown_connections 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5622a53cf280 0x5622a5437e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.295+0000 7f8b5cff9640 0 client.25666 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622a53bef90 0x5622a53c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5622a53b9e20 0x5622a53c8900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 --2- 172.21.15.28:0/542287660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5622a53b9430 0x5622a53c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 >> 172.21.15.28:0/542287660 conn(0x5622a53b49a0 msgr2=0x5622a53b8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 shutdown_connections 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 1 -- 172.21.15.28:0/542287660 wait complete. 2024-03-22T02:15:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7e3bf740 20 client.25666 trim_cache size 0 max 16384 2024-03-22T02:15:01.303 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:15:01.303 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:15:01.303 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 msgr2=0x7f8b600a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 0x7f8b600a52c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009710 tx=0x7f8b64009870 comp rx=0 tx=0).stop 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 msgr2=0x7f8b7419fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 0x7f8b7419fb00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00ec50 tx=0x7f8b6c00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 shutdown_connections 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b600a2e00 0x7f8b600a52c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b7419d710 0x7f8b7419fb00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b74107870 0x7f8b741986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 --2- 172.21.15.28:0/2181546974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b74104f40 0x7f8b74198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 >> 172.21.15.28:0/2181546974 conn(0x7f8b740fc020 msgr2=0x7f8b741030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.296+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 shutdown_connections 2024-03-22T02:15:01.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.297+0000 7f8b7b6fb640 1 -- 172.21.15.28:0/2181546974 wait complete. 2024-03-22T02:15:01.309 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:15:01.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:15:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 -- 172.21.15.28:0/3435789109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4101ca0 msgr2=0x7f90f4102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 --2- 172.21.15.28:0/3435789109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4101ca0 0x7f90f4102120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f90e8009860 tx=0x7f90e802bb70 comp rx=0 tx=0).stop 2024-03-22T02:15:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 -- 172.21.15.28:0/3435789109 shutdown_connections 2024-03-22T02:15:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 --2- 172.21.15.28:0/3435789109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4102660 0x7f90f4108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 --2- 172.21.15.28:0/3435789109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4101ca0 0x7f90f4102120 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 --2- 172.21.15.28:0/3435789109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 -- 172.21.15.28:0/3435789109 >> 172.21.15.28:0/3435789109 conn(0x7f90f40fc020 msgr2=0x7f90f40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 -- 172.21.15.28:0/3435789109 shutdown_connections 2024-03-22T02:15:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.412+0000 7f90f971e640 1 -- 172.21.15.28:0/3435789109 wait complete. 2024-03-22T02:15:01.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.413+0000 7f90f971e640 1 Processor -- start 2024-03-22T02:15:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.413+0000 7f90f971e640 1 -- start start 2024-03-22T02:15:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48848/0 (socket says 172.21.15.28:48848) 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f27fc640 1 -- 172.21.15.28:0/1093768113 learned_addr learned my addr 172.21.15.28:0/1093768113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60258/0 (socket says 172.21.15.28:60258) 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4102660 0x7f90f419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90f410c730 con 0x7f90f4101ca0 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f90f410c5b0 con 0x7f90f4102660 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f90f410c8b0 con 0x7f90f4100ab0 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f37fe640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4102660 0x7f90f419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 -- 172.21.15.28:0/1093768113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4102660 msgr2=0x7f90f419f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4102660 0x7f90f419f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 -- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 msgr2=0x7f90f419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f419fb20 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f27fc640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.414+0000 7f90f2ffd640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f90e400e9c0 tx=0x7f90e400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.415+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e400ccc0 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.415+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e40035f0 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.415+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e400ce60 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.415+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f419fe10 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.415+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f41a0220 con 0x7f90f4100ab0 2024-03-22T02:15:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.416+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f4107dd0 con 0x7f90f4100ab0 2024-03-22T02:15:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.417+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f90e40037a0 con 0x7f90f4100ab0 2024-03-22T02:15:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.418+0000 7f90dbfff640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 0x7f90dc0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.418+0000 7f90f27fc640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 0x7f90dc0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.418+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f90e40cfc60 con 0x7f90f4100ab0 2024-03-22T02:15:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.418+0000 7f90f27fc640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 0x7f90dc0a5460 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f90f40695f0 tx=0x7f90e802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.421+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90e409c680 con 0x7f90f4100ab0 2024-03-22T02:15:01.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.651+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90f402d700 con 0x7f90f4100ab0 2024-03-22T02:15:01.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.652+0000 7f90dbfff640 1 -- 172.21.15.28:0/1093768113 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11866 (secure 0 0 0) 0x7f90f402d700 con 0x7f90f4100ab0 2024-03-22T02:15:01.654 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:15:01.654 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:13:50.358406+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:15:01.654 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 msgr2=0x7f90dc0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 0x7f90dc0a5460 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f90f40695f0 tx=0x7f90e802e040 comp rx=0 tx=0).stop 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 msgr2=0x7f90f4069880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f90e400e9c0 tx=0x7f90e400ee80 comp rx=0 tx=0).stop 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 shutdown_connections 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f90dc0a2fa0 0x7f90dc0a5460 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f90f4102660 0x7f90f419f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90f4101ca0 0x7f90f419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 --2- 172.21.15.28:0/1093768113 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f4100ab0 0x7f90f4069880 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.654+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 >> 172.21.15.28:0/1093768113 conn(0x7f90f40fc020 msgr2=0x7f90f4104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.655+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 shutdown_connections 2024-03-22T02:15:01.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.655+0000 7f90f971e640 1 -- 172.21.15.28:0/1093768113 wait complete. 2024-03-22T02:15:01.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 -- 172.21.15.28:0/4022339013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824101c80 msgr2=0x7ff824102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 --2- 172.21.15.28:0/4022339013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824101c80 0x7ff824102100 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff810009900 tx=0x7ff81002bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 -- 172.21.15.28:0/4022339013 shutdown_connections 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 --2- 172.21.15.28:0/4022339013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824102640 0x7ff824108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 --2- 172.21.15.28:0/4022339013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824101c80 0x7ff824102100 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 --2- 172.21.15.28:0/4022339013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 -- 172.21.15.28:0/4022339013 >> 172.21.15.28:0/4022339013 conn(0x7ff8240fc020 msgr2=0x7ff8240fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 -- 172.21.15.28:0/4022339013 shutdown_connections 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.773+0000 7ff82aca4640 1 -- 172.21.15.28:0/4022339013 wait complete. 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 Processor -- start 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 -- start start 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824101c80 0x7ff824198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 0x7ff82419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff82410c6f0 con 0x7ff824102640 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff82410c570 con 0x7ff824101c80 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff82aca4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff82410c870 con 0x7ff824100a90 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff829241640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 0x7ff82419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff828a40640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff828a40640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38606/0 (socket says 172.21.15.28:38606) 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff828a40640 1 -- 172.21.15.28:0/1301975727 learned_addr learned my addr 172.21.15.28:0/1301975727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.774+0000 7ff823fff640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824101c80 0x7ff824198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 -- 172.21.15.28:0/1301975727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824101c80 msgr2=0x7ff824198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824101c80 0x7ff824198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 -- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 msgr2=0x7ff82419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 0x7ff82419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 -- 172.21.15.28:0/1301975727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82419fb00 con 0x7ff824100a90 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff829241640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 0x7ff82419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff828a40640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9c0 tx=0x7ff81800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff821ffb640 1 -- 172.21.15.28:0/1301975727 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81800cd80 con 0x7ff824100a90 2024-03-22T02:15:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff821ffb640 1 -- 172.21.15.28:0/1301975727 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8180035f0 con 0x7ff824100a90 2024-03-22T02:15:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff821ffb640 1 -- 172.21.15.28:0/1301975727 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff818010690 con 0x7ff824100a90 2024-03-22T02:15:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82419fdf0 con 0x7ff824100a90 2024-03-22T02:15:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.775+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8241a01a0 con 0x7ff824100a90 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 Processor -- start 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 -- start start 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeafd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defeb2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563defe0f8d0 con 0x563defeab210 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563defdb7e00 con 0x563defeaa820 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff82d969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563defe0b880 con 0x563defeb0380 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff821ffb640 1 -- 172.21.15.28:0/1301975727 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff8180037a0 con 0x7ff824100a90 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff823fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defeb2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff829241640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff829241640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34494/0 (socket says 172.21.15.28:34494) 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff829241640 1 -- 172.21.15.28:0/751604707 learned_addr learned my addr 172.21.15.28:0/751604707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.778+0000 7ff823fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defeb2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38618/0 (socket says 172.21.15.28:38618) 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 -- 172.21.15.28:0/751604707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 msgr2=0x563defeb2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defeb2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 -- 172.21.15.28:0/751604707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 msgr2=0x563defeafd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff828a40640 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeafd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeafd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 -- 172.21.15.28:0/751604707 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563defb233f0 con 0x563defeaa820 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff828a40640 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeafd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff829241640 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0117e0 tx=0x7ff81c011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=678dd16f012c098d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff821ffb640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 0x7ff8140a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff80effd640 1 -- 172.21.15.28:0/751604707 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c011310 con 0x563defeaa820 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff828a40640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 0x7ff8140a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff80effd640 1 -- 172.21.15.28:0/751604707 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81c003e50 con 0x563defeaa820 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.779+0000 7ff80effd640 1 -- 172.21.15.28:0/751604707 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c0045b0 con 0x563defeaa820 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff821ffb640 1 -- 172.21.15.28:0/1301975727 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff8180cfb60 con 0x7ff824100a90 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff828a40640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 0x7ff8140a5460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff8180cd5c0 tx=0x7ff818098120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 -- 172.21.15.28:0/751604707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 msgr2=0x563defeaac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff81c0117e0 tx=0x7ff81c011ca0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 -- 172.21.15.28:0/751604707 shutdown_connections 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defeb2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeafd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 --2- 172.21.15.28:0/751604707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 -- 172.21.15.28:0/751604707 >> 172.21.15.28:0/751604707 conn(0x563defea5d90 msgr2=0x563defea6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 -- 172.21.15.28:0/751604707 shutdown_connections 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.780+0000 7ff82d969740 1 -- 172.21.15.28:0/751604707 wait complete. 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 -- start start 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 Processor -- start 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeb8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defec0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563defeb7740 con 0x563defeab210 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563defdcb300 con 0x563defeaa820 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563defeb78c0 con 0x563defeb0380 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff82d969740 10 client.0 _ll_register_callbacks cb 0x563defe06e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff823fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff823fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39826/0 (socket says 172.21.15.28:39826) 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff829241640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeb8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff823fff640 1 -- 172.21.15.28:0/515364567 learned_addr learned my addr 172.21.15.28:0/515364567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.781+0000 7ff829241640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defec0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 -- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 msgr2=0x563defec0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defec0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 -- 172.21.15.28:0/515364567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 msgr2=0x563defeb8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeb8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 -- 172.21.15.28:0/515364567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563defbed7d0 con 0x563defeab210 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff829241640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeb8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff823fff640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff810036040 tx=0x7ff810004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa56535d69d5460 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff803fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810005d50 con 0x563defeab210 2024-03-22T02:15:01.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff810005230 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff81c003e50 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff810005430 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff82d969740 10 client.15852 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.782+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563defec37e0 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff81c003e50 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff810003710 con 0x563defeab210 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 10 client.15852 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 10 client.15852 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 20 client.15852 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563defec4ee0 0x563defec7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 4 client.15852 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563defe1ce10 con 0x563defec4ee0 2024-03-22T02:15:01.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.783+0000 7ff829241640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563defec4ee0 0x563defec7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff803fff640 10 client.15852 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff829241640 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563defec4ee0 0x563defec7340 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d187515e4fc4483b server_cookie=c943e661ec881d44 in_seq=0 out_seq=0 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff803fff640 1 -- 172.21.15.28:0/515364567 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563defe1ce10 con 0x563defec4ee0 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff803fff640 10 client.15852 handle_command_reply: tid=0 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff82d969740 1 client.15852 shutdown 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.784+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 msgr2=0x563defeb9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff810036040 tx=0x7ff810004270 comp rx=0 tx=0).stop 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 shutdown_connections 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeb0380 0x563defec0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeab210 0x563defeb9450 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563defec4ee0 0x563defec7340 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 --2- 172.21.15.28:0/515364567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeb8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 >> 172.21.15.28:0/515364567 conn(0x563defea5d90 msgr2=0x563defea4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff803fff640 0 client.15852 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 shutdown_connections 2024-03-22T02:15:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 1 -- 172.21.15.28:0/515364567 wait complete. 2024-03-22T02:15:01.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.785+0000 7ff82d969740 20 client.15852 trim_cache size 0 max 16384 2024-03-22T02:15:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.811+0000 7ff82d969740 1 Processor -- start 2024-03-22T02:15:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.811+0000 7ff82d969740 1 -- start start 2024-03-22T02:15:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defeac530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241ab0d0 con 0x563defeb0380 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241a89a0 con 0x563defeaa820 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff82d969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241a67f0 con 0x563defeab210 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff829241640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38636/0 (socket says 172.21.15.28:38636) 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 -- 172.21.15.28:0/1923052042 learned_addr learned my addr 172.21.15.28:0/1923052042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff829241640 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defeac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 -- 172.21.15.28:0/1923052042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 msgr2=0x563defeaac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 -- 172.21.15.28:0/1923052042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 msgr2=0x563defeac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defeac530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.812+0000 7ff823fff640 1 -- 172.21.15.28:0/1923052042 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563defeabaa0 con 0x563defeab210 2024-03-22T02:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.813+0000 7ff823fff640 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff810000c00 tx=0x7ff810009230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c7e4e1c7e7ce145 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.813+0000 7ff80e7fc640 1 -- 172.21.15.28:0/1923052042 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810005d50 con 0x563defeab210 2024-03-22T02:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.813+0000 7ff80e7fc640 1 -- 172.21.15.28:0/1923052042 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff810009520 con 0x563defeab210 2024-03-22T02:15:01.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.813+0000 7ff80e7fc640 1 -- 172.21.15.28:0/1923052042 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810034900 con 0x563defeab210 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- 172.21.15.28:0/1923052042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 msgr2=0x563defec0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff810000c00 tx=0x7ff810009230 comp rx=0 tx=0).stop 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- 172.21.15.28:0/1923052042 shutdown_connections 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defeac530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeab210 0x563defec0450 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- 172.21.15.28:0/1923052042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeaa820 0x563defeaac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- 172.21.15.28:0/1923052042 >> 172.21.15.28:0/1923052042 conn(0x563defea5d90 msgr2=0x563defeb8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- 172.21.15.28:0/1923052042 shutdown_connections 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- 172.21.15.28:0/1923052042 wait complete. 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- start start 2024-03-22T02:15:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 Processor -- start 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 0x563defebd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 0x563defebdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defebee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241a6e60 con 0x563defeb0380 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241a82d0 con 0x563defeab210 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8241a7530 con 0x563defeaa820 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.814+0000 7ff82d969740 10 client.0 _ll_register_callbacks cb 0x7ff818005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff829241640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 0x563defebd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff829241640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 0x563defebd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38638/0 (socket says 172.21.15.28:38638) 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff829241640 1 -- 172.21.15.28:0/605624598 learned_addr learned my addr 172.21.15.28:0/605624598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff829241640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defebee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 0x563defebdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 -- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 msgr2=0x563defebd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 0x563defebd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 -- 172.21.15.28:0/605624598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 msgr2=0x563defebee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defebee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 -- 172.21.15.28:0/605624598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563defebf430 con 0x563defeab210 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff829241640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defebee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff823fff640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 0x563defebdf10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff81002cfd0 tx=0x7ff810005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21aedeaa387e00fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff803fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.815+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff810005d50 con 0x563defeab210 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff810009520 con 0x563defeab210 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff810034820 con 0x563defeab210 2024-03-22T02:15:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff810034a20 con 0x563defeab210 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 10 client.25700 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563defec0730 con 0x563defeab210 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff810009520 con 0x563defeab210 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7ff810053900 con 0x563defeab210 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 10 client.25700 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 10 client.25700 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 20 client.25700 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563deff2b950 0x563deff2dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 4 client.25700 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.816+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563defe17160 con 0x563deff2b950 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.817+0000 7ff829241640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563deff2b950 0x563deff2dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.817+0000 7ff803fff640 10 client.25700 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.817+0000 7ff829241640 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563deff2b950 0x563deff2dd70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbc873d47b3a9a72 server_cookie=5f1fe03c4ef72f29 in_seq=0 out_seq=0 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff803fff640 1 -- 172.21.15.28:0/605624598 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563defe17160 con 0x563deff2b950 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff803fff640 10 client.25700 handle_command_reply: tid=0 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 client.25700 shutdown 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 msgr2=0x563defebdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 0x563defebdf10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff81002cfd0 tx=0x7ff810005a20 comp rx=0 tx=0).stop 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 shutdown_connections 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563defeb0380 0x563defebee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563deff2b950 0x563deff2dd70 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563defeab210 0x563defebdf10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 --2- 172.21.15.28:0/605624598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563defeaa820 0x563defebd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff803fff640 0 client.25700 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:01.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 >> 172.21.15.28:0/605624598 conn(0x563defea5d90 msgr2=0x563defeac530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 shutdown_connections 2024-03-22T02:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 1 -- 172.21.15.28:0/605624598 wait complete. 2024-03-22T02:15:01.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.818+0000 7ff82d969740 20 client.25700 trim_cache size 0 max 16384 2024-03-22T02:15:01.825 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 msgr2=0x7ff8140a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 0x7ff8140a5460 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff8180cd5c0 tx=0x7ff818098120 comp rx=0 tx=0).stop 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 msgr2=0x7ff824197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff81800e9c0 tx=0x7ff81800ee80 comp rx=0 tx=0).stop 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 shutdown_connections 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8140a2fa0 0x7ff8140a5460 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff824102640 0x7ff82419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff824101c80 0x7ff824198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 --2- 172.21.15.28:0/1301975727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff824100a90 0x7ff824197ee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 >> 172.21.15.28:0/1301975727 conn(0x7ff8240fc020 msgr2=0x7ff8240fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 shutdown_connections 2024-03-22T02:15:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.819+0000 7ff82aca4640 1 -- 172.21.15.28:0/1301975727 wait complete. 2024-03-22T02:15:01.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:15:01.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 -- 172.21.15.28:0/868982043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964102610 msgr2=0x7f3964104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 --2- 172.21.15.28:0/868982043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964102610 0x7f3964104a00 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3960009900 tx=0x7f396002bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:01.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 -- 172.21.15.28:0/868982043 shutdown_connections 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 --2- 172.21.15.28:0/868982043 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 0x7f3964109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 --2- 172.21.15.28:0/868982043 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964104f40 0x7f3964107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 --2- 172.21.15.28:0/868982043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964102610 0x7f3964104a00 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 -- 172.21.15.28:0/868982043 >> 172.21.15.28:0/868982043 conn(0x7f39640fc020 msgr2=0x7f39640fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 -- 172.21.15.28:0/868982043 shutdown_connections 2024-03-22T02:15:01.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.937+0000 7f396bd0c640 1 -- 172.21.15.28:0/868982043 wait complete. 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 Processor -- start 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 -- start start 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 0x7f396419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 0x7f39641a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f396410d880 con 0x7f3964104f40 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f396410d700 con 0x7f3964102610 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396bd0c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f396410da00 con 0x7f3964107870 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f39692a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 0x7f396419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f39692a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39884/0 (socket says 172.21.15.28:39884) 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f39692a7640 1 -- 172.21.15.28:0/153550613 learned_addr learned my addr 172.21.15.28:0/153550613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f396a2a9640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 0x7f39641a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 -- 172.21.15.28:0/153550613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 msgr2=0x7f39641a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 0x7f39641a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 -- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 msgr2=0x7f396419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f3969aa8640 1 -- 172.21.15.28:0/153550613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39641a1d30 con 0x7f3964102610 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.938+0000 7f39692a7640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f3969aa8640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 0x7f396419a110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3960036040 tx=0x7f396002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f395affd640 1 -- 172.21.15.28:0/153550613 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960043020 con 0x7f3964102610 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f395affd640 1 -- 172.21.15.28:0/153550613 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39600054e0 con 0x7f3964102610 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39641a1fc0 con 0x7f3964102610 2024-03-22T02:15:01.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f395affd640 1 -- 172.21.15.28:0/153550613 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960003c30 con 0x7f3964102610 2024-03-22T02:15:01.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.939+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39641a2430 con 0x7f3964102610 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 Processor -- start 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 -- start start 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 0x55922aa41d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa42380 0x55922aa44810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55922a9a18d0 con 0x55922aa3d210 2024-03-22T02:15:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55922a949e00 con 0x55922aa42380 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396e9d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55922a99d880 con 0x55922aa3c820 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f395affd640 1 -- 172.21.15.28:0/153550613 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3960046020 con 0x7f3964102610 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f3969aa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 0x55922aa41d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38666/0 (socket says 172.21.15.28:38666) 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f396a2a9640 1 -- 172.21.15.28:0/4217598239 learned_addr learned my addr 172.21.15.28:0/4217598239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.942+0000 7f39692a7640 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa42380 0x55922aa44810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 -- 172.21.15.28:0/4217598239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa42380 msgr2=0x55922aa44810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa42380 0x55922aa44810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 -- 172.21.15.28:0/4217598239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 msgr2=0x55922aa41d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 0x55922aa41d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 -- 172.21.15.28:0/4217598239 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922a6b53f0 con 0x55922aa3c820 2024-03-22T02:15:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f3969aa8640 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 0x55922aa41d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f395c010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8099a61a4c6ba0f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f395affd640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 0x7f39500a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f394ffff640 1 -- 172.21.15.28:0/4217598239 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c003b70 con 0x55922aa3c820 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f394ffff640 1 -- 172.21.15.28:0/4217598239 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c003d10 con 0x55922aa3c820 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f394ffff640 1 -- 172.21.15.28:0/4217598239 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c011ae0 con 0x55922aa3c820 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.943+0000 7f395affd640 1 -- 172.21.15.28:0/153550613 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f39600f1850 con 0x7f3964102610 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f3969aa8640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 0x7f39500a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f396e9d2740 1 -- 172.21.15.28:0/4217598239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 msgr2=0x55922aa3cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f395c009fc0 tx=0x7f395c010860 comp rx=0 tx=0).stop 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f3969aa8640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 0x7f39500a5460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f396002c7b0 tx=0x7f39600f7530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f396e9d2740 1 -- 172.21.15.28:0/4217598239 shutdown_connections 2024-03-22T02:15:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa42380 0x55922aa44810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.944+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3d210 0x55922aa41d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4217598239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa3cc40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 -- 172.21.15.28:0/4217598239 >> 172.21.15.28:0/4217598239 conn(0x55922aa37d90 msgr2=0x55922aa38190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 -- 172.21.15.28:0/4217598239 shutdown_connections 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 -- 172.21.15.28:0/4217598239 wait complete. 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 -- start start 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.945+0000 7f396e9d2740 1 Processor -- start 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa4f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa56b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55922aa45330 con 0x55922aa42380 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55922a95d300 con 0x55922aa3d210 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55922aa454b0 con 0x55922aa3c820 2024-03-22T02:15:01.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa4f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34558/0 (socket says 172.21.15.28:34558) 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa4f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38682/0 (socket says 172.21.15.28:38682) 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 -- 172.21.15.28:0/676352695 learned_addr learned my addr 172.21.15.28:0/676352695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396a2a9640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa56b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396e9d2740 10 client.0 _ll_register_callbacks cb 0x55922a998e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 -- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 msgr2=0x55922aa4f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa4f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 -- 172.21.15.28:0/676352695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 msgr2=0x55922aa56b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa56b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f39692a7640 1 -- 172.21.15.28:0/676352695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922a77f7d0 con 0x55922aa3d210 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.946+0000 7f396a2a9640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa56b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:15:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f39692a7640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3954012d20 tx=0x7f395400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3457d33b127af90b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954012110 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3954004050 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f395c003d10 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f39540122b0 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 10 client.25712 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55922aa59e80 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f395c003d10 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f39540104c0 con 0x55922aa3d210 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 10 client.25712 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 10 client.25712 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 20 client.25712 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.947+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aa5b640 0x55922aa5dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.948+0000 7f396e9d2740 4 client.25712 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.948+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55922a9aee10 con 0x55922aa5b640 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.948+0000 7f396a2a9640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aa5b640 0x55922aa5dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.948+0000 7f396a2a9640 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aa5b640 0x55922aa5dae0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ae1be6e71b3bb38e server_cookie=c49dd45f194f5aa1 in_seq=0 out_seq=0 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.948+0000 7f394d7fa640 10 client.25712 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f394d7fa640 1 -- 172.21.15.28:0/676352695 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55922a9aee10 con 0x55922aa5b640 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f394d7fa640 10 client.25712 handle_command_reply: tid=0 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 client.25712 shutdown 2024-03-22T02:15:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 msgr2=0x55922aa4fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3954012d20 tx=0x7f395400f790 comp rx=0 tx=0).stop 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 shutdown_connections 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aa5b640 0x55922aa5dae0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa56b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa4fac0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f394d7fa640 0 client.25712 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 --2- 172.21.15.28:0/676352695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3c820 0x55922aa4f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.949+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 >> 172.21.15.28:0/676352695 conn(0x55922aa37d90 msgr2=0x55922aa36fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.950+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 shutdown_connections 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.950+0000 7f396e9d2740 1 -- 172.21.15.28:0/676352695 wait complete. 2024-03-22T02:15:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.950+0000 7f396e9d2740 20 client.25712 trim_cache size 0 max 16384 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 Processor -- start 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 -- start start 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa56ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa42380 0x55922aa3b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39641ab060 con 0x55922aa3c820 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39641a8930 con 0x55922aa3d210 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396e9d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39641a6780 con 0x55922aa42380 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39914/0 (socket says 172.21.15.28:39914) 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396a2a9640 1 -- 172.21.15.28:0/429678954 learned_addr learned my addr 172.21.15.28:0/429678954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f39692a7640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa56ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.963+0000 7f396a2a9640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa42380 0x55922aa3b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 -- 172.21.15.28:0/429678954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa42380 msgr2=0x55922aa3b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa42380 0x55922aa3b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 -- 172.21.15.28:0/429678954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 msgr2=0x55922aa41d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 -- 172.21.15.28:0/429678954 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55922aa5e160 con 0x55922aa3d210 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f396a2a9640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f39692a7640 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa56ac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3954010930 tx=0x7f3954012910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d5c22a6e25f421b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f394f7fe640 1 -- 172.21.15.28:0/429678954 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395402e070 con 0x55922aa3d210 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f394f7fe640 1 -- 172.21.15.28:0/429678954 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39540055c0 con 0x55922aa3d210 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.964+0000 7f394f7fe640 1 -- 172.21.15.28:0/429678954 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3954003850 con 0x55922aa3d210 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- 172.21.15.28:0/429678954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 msgr2=0x55922aa56ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa56ac0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3954010930 tx=0x7f3954012910 comp rx=0 tx=0).stop 2024-03-22T02:15:01.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- 172.21.15.28:0/429678954 shutdown_connections 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa42380 0x55922aa3b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3d210 0x55922aa56ac0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 --2- 172.21.15.28:0/429678954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa3c820 0x55922aa41d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- 172.21.15.28:0/429678954 >> 172.21.15.28:0/429678954 conn(0x55922aa37d90 msgr2=0x55922aa3aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- 172.21.15.28:0/429678954 shutdown_connections 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- 172.21.15.28:0/429678954 wait complete. 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 -- start start 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.965+0000 7f396e9d2740 1 Processor -- start 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3c820 0x55922aa56370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3d210 0x55922aa568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39641a6df0 con 0x55922aa42380 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39641a8260 con 0x55922aa3c820 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39641a74c0 con 0x55922aa3d210 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396e9d2740 10 client.0 _ll_register_callbacks cb 0x7f3954009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3c820 0x55922aa56370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f39692a7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3d210 0x55922aa568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39924/0 (socket says 172.21.15.28:39924) 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 -- 172.21.15.28:0/4126479262 learned_addr learned my addr 172.21.15.28:0/4126479262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 -- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3d210 msgr2=0x55922aa568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3d210 0x55922aa568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 -- 172.21.15.28:0/4126479262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3c820 msgr2=0x55922aa56370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3c820 0x55922aa56370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 -- 172.21.15.28:0/4126479262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55922aa54fc0 con 0x55922aa42380 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f396a2a9640 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f395c009f90 tx=0x7f395c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f055d647e57a9179 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f394d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.966+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f395c011570 con 0x55922aa42380 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f395c011710 con 0x55922aa42380 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39540055c0 con 0x55922aa42380 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f395c011910 con 0x55922aa42380 2024-03-22T02:15:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 10 client.15876 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55922aac2340 con 0x55922aa42380 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39540055c0 con 0x55922aa42380 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f395c005600 con 0x55922aa42380 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 10 client.15876 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 10 client.15876 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 20 client.15876 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aac2b90 0x55922aac4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 4 client.15876 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.967+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55922a9a9160 con 0x55922aac2b90 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f39692a7640 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aac2b90 0x55922aac4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f39692a7640 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aac2b90 0x55922aac4fb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=daeb4990aeabe7f6 server_cookie=1c88bb363753602d in_seq=0 out_seq=0 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f394d7fa640 10 client.15876 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:01.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f394d7fa640 1 -- 172.21.15.28:0/4126479262 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55922a9a9160 con 0x55922aac2b90 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f394d7fa640 10 client.15876 handle_command_reply: tid=0 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 client.15876 shutdown 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 msgr2=0x55922aa54980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f395c009f90 tx=0x7f395c010860 comp rx=0 tx=0).stop 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 shutdown_connections 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55922aac2b90 0x55922aac4fb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55922aa42380 0x55922aa54980 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55922aa3d210 0x55922aa568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f394d7fa640 0 client.15876 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 --2- 172.21.15.28:0/4126479262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55922aa3c820 0x55922aa56370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 >> 172.21.15.28:0/4126479262 conn(0x55922aa37d90 msgr2=0x55922aa3b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 shutdown_connections 2024-03-22T02:15:01.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.968+0000 7f396e9d2740 1 -- 172.21.15.28:0/4126479262 wait complete. 2024-03-22T02:15:01.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396e9d2740 20 client.15876 trim_cache size 0 max 16384 2024-03-22T02:15:01.976 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 msgr2=0x7f39500a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 0x7f39500a5460 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f396002c7b0 tx=0x7f39600f7530 comp rx=0 tx=0).stop 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 msgr2=0x7f396419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 0x7f396419a110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3960036040 tx=0x7f396002c650 comp rx=0 tx=0).stop 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 shutdown_connections 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39500a2fa0 0x7f39500a5460 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3964107870 0x7f39641a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3964104f40 0x7f396419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:01.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 --2- 172.21.15.28:0/153550613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3964102610 0x7f396419a110 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3960036040 tx=0x7f396002c650 comp rx=0 tx=0).stop 2024-03-22T02:15:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 >> 172.21.15.28:0/153550613 conn(0x7f39640fc020 msgr2=0x7f3964101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 shutdown_connections 2024-03-22T02:15:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:01.969+0000 7f396bd0c640 1 -- 172.21.15.28:0/153550613 wait complete. 2024-03-22T02:15:01.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.086+0000 7f8f67e08640 1 -- 172.21.15.28:0/2369672758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 msgr2=0x7f8f60108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.086+0000 7f8f67e08640 1 --2- 172.21.15.28:0/2369672758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f60108ed0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009be0 tx=0x7f8f5802bef0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 -- 172.21.15.28:0/2369672758 shutdown_connections 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 --2- 172.21.15.28:0/2369672758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f60108ed0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 --2- 172.21.15.28:0/2369672758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60101c80 0x7f8f60102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 --2- 172.21.15.28:0/2369672758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60100a90 0x7f8f60100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 -- 172.21.15.28:0/2369672758 >> 172.21.15.28:0/2369672758 conn(0x7f8f600fc020 msgr2=0x7f8f600fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 -- 172.21.15.28:0/2369672758 shutdown_connections 2024-03-22T02:15:02.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 -- 172.21.15.28:0/2369672758 wait complete. 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 Processor -- start 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.087+0000 7f8f67e08640 1 -- start start 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 0x7f8f601983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f6019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6010c740 con 0x7f8f60100a90 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6010c5c0 con 0x7f8f60102640 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f67e08640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6010c8c0 con 0x7f8f60101c80 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39932/0 (socket says 172.21.15.28:39932) 2024-03-22T02:15:02.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 -- 172.21.15.28:0/487282283 learned_addr learned my addr 172.21.15.28:0/487282283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f653a3640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 0x7f8f601983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f663a5640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f6019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 -- 172.21.15.28:0/487282283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 msgr2=0x7f8f601983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 0x7f8f601983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 -- 172.21.15.28:0/487282283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 msgr2=0x7f8f6019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f6019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 -- 172.21.15.28:0/487282283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6019fac0 con 0x7f8f60100a90 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f653a3640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 0x7f8f601983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f663a5640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f6019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.088+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00e9c0 tx=0x7f8f5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.089+0000 7f8f56ffd640 1 -- 172.21.15.28:0/487282283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c00cd80 con 0x7f8f60100a90 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.089+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f6019fdb0 con 0x7f8f60100a90 2024-03-22T02:15:02.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.089+0000 7f8f56ffd640 1 -- 172.21.15.28:0/487282283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f5c0035f0 con 0x7f8f60100a90 2024-03-22T02:15:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.089+0000 7f8f56ffd640 1 -- 172.21.15.28:0/487282283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c010690 con 0x7f8f60100a90 2024-03-22T02:15:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.089+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f601a0270 con 0x7f8f60100a90 2024-03-22T02:15:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.092+0000 7f8f6a9f2740 1 Processor -- start 2024-03-22T02:15:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.092+0000 7f8f56ffd640 1 -- 172.21.15.28:0/487282283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8f5c0037a0 con 0x7f8f60100a90 2024-03-22T02:15:02.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.092+0000 7f8f6a9f2740 1 -- start start 2024-03-22T02:15:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.092+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.092+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 0x55dc28f1cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dc28e7c8d0 con 0x55dc28f1d380 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dc28e24e00 con 0x55dc28f18210 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dc28e78880 con 0x55dc28f17820 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f663a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f56ffd640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 0x7f8f4c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 0x55dc28f1cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f663a5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38708/0 (socket says 172.21.15.28:38708) 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f663a5640 1 -- 172.21.15.28:0/1060898713 learned_addr learned my addr 172.21.15.28:0/1060898713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39936/0 (socket says 172.21.15.28:39936) 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f56ffd640 1 -- 172.21.15.28:0/487282283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8f5c0cfbd0 con 0x7f8f60100a90 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f663a5640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 0x7f8f4c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 -- 172.21.15.28:0/1060898713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 msgr2=0x55dc28f17c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 -- 172.21.15.28:0/1060898713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 msgr2=0x55dc28f1cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 0x55dc28f1cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f653a3640 1 -- 172.21.15.28:0/1060898713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc28b903f0 con 0x55dc28f1d380 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 0x55dc28f1cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.093+0000 7f8f663a5640 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.094+0000 7f8f663a5640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 0x7f8f4c0a53e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55dc28f18a70 tx=0x7f8f5802bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.094+0000 7f8f653a3640 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009530 tx=0x7f8f500228e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed62d71d32a9cb04 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.094+0000 7f8f4bfff640 1 -- 172.21.15.28:0/1060898713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50004220 con 0x55dc28f1d380 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.094+0000 7f8f4bfff640 1 -- 172.21.15.28:0/1060898713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f500043c0 con 0x55dc28f1d380 2024-03-22T02:15:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.094+0000 7f8f4bfff640 1 -- 172.21.15.28:0/1060898713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f50027be0 con 0x55dc28f1d380 2024-03-22T02:15:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1060898713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 msgr2=0x55dc28f1f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009530 tx=0x7f8f500228e0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1060898713 shutdown_connections 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f1d380 0x55dc28f1f810 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f18210 0x55dc28f1cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1060898713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f17820 0x55dc28f17c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1060898713 >> 172.21.15.28:0/1060898713 conn(0x55dc28f12d90 msgr2=0x55dc28f13190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1060898713 shutdown_connections 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.095+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1060898713 wait complete. 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 -- start start 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 Processor -- start 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f223f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f27420 0x55dc28f29810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dc28f20330 con 0x55dc28f27420 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dc28e38300 con 0x55dc28f17820 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dc28f204b0 con 0x55dc28f18210 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f6a9f2740 10 client.0 _ll_register_callbacks cb 0x55dc28e73e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34618/0 (socket says 172.21.15.28:34618) 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f65ba4640 1 -- 172.21.15.28:0/2736294836 learned_addr learned my addr 172.21.15.28:0/2736294836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f663a5640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f27420 0x55dc28f29810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.096+0000 7f8f653a3640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f223f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 -- 172.21.15.28:0/2736294836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 msgr2=0x55dc28f223f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f223f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 -- 172.21.15.28:0/2736294836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f27420 msgr2=0x55dc28f29810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f27420 0x55dc28f29810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 -- 172.21.15.28:0/2736294836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc28c5a7d0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c004770 tx=0x7f8f5c09c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5175a715c083e884 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c09bef0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8f5c003b30 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f500043c0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f5c0a4470 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f6a9f2740 10 client.25730 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dc28f2cbf0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f500043c0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.097+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8f5c09adb0 con 0x55dc28f17820 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 10 client.25730 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 10 client.25730 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 20 client.25730 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f2e200 0x55dc28f306c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 4 client.25730 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc28e89e10 con 0x55dc28f2e200 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f653a3640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f2e200 0x55dc28f306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f497fa640 10 client.25730 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.098+0000 7f8f653a3640 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f2e200 0x55dc28f306c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0e1f74a23114ec server_cookie=c131a63440697a7 in_seq=0 out_seq=0 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f497fa640 1 -- 172.21.15.28:0/2736294836 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc28e89e10 con 0x55dc28f2e200 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f497fa640 10 client.25730 handle_command_reply: tid=0 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 client.25730 shutdown 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 msgr2=0x55dc28f21eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c004770 tx=0x7f8f5c09c940 comp rx=0 tx=0).stop 2024-03-22T02:15:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 shutdown_connections 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f2e200 0x55dc28f306c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f27420 0x55dc28f29810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f223f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f497fa640 0 client.25730 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2736294836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f21eb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 >> 172.21.15.28:0/2736294836 conn(0x55dc28f12d90 msgr2=0x55dc28f11fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.099+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 shutdown_connections 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.100+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2736294836 wait complete. 2024-03-22T02:15:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.100+0000 7f8f6a9f2740 20 client.25730 trim_cache size 0 max 16384 2024-03-22T02:15:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 Processor -- start 2024-03-22T02:15:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 -- start start 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f16700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 0x55dc28f34f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f601a29a0 con 0x55dc28f17820 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8f60073ad0 con 0x55dc28f27420 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8f60071920 con 0x55dc28f18210 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f65ba4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39960/0 (socket says 172.21.15.28:39960) 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f663a5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 0x55dc28f34f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f65ba4640 1 -- 172.21.15.28:0/1953285933 learned_addr learned my addr 172.21.15.28:0/1953285933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.113+0000 7f8f653a3640 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f16700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 -- 172.21.15.28:0/1953285933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 msgr2=0x55dc28f16700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f16700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 -- 172.21.15.28:0/1953285933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 msgr2=0x55dc28f22350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 -- 172.21.15.28:0/1953285933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc28f13b00 con 0x55dc28f27420 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f663a5640 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 0x55dc28f34f80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8f58004ab0 tx=0x7f8f58005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3d18f3836e1e11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f4b7fe640 1 -- 172.21.15.28:0/1953285933 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58034900 con 0x55dc28f27420 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f4b7fe640 1 -- 172.21.15.28:0/1953285933 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f58054070 con 0x55dc28f27420 2024-03-22T02:15:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.114+0000 7f8f4b7fe640 1 -- 172.21.15.28:0/1953285933 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5804fd00 con 0x55dc28f27420 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.115+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1953285933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 msgr2=0x55dc28f34f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.115+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 0x55dc28f34f80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8f58004ab0 tx=0x7f8f58005e50 comp rx=0 tx=0).stop 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1953285933 shutdown_connections 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f27420 0x55dc28f34f80 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f18210 0x55dc28f16700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/1953285933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f17820 0x55dc28f22350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1953285933 >> 172.21.15.28:0/1953285933 conn(0x55dc28f12d90 msgr2=0x55dc28f15ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1953285933 shutdown_connections 2024-03-22T02:15:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/1953285933 wait complete. 2024-03-22T02:15:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.116+0000 7f8f6a9f2740 1 -- start start 2024-03-22T02:15:02.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.117+0000 7f8f6a9f2740 1 Processor -- start 2024-03-22T02:15:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f41260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 0x55dc28f417a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 0x55dc28f3f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f60071f90 con 0x55dc28f18210 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8f60073400 con 0x55dc28f17820 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8f60072660 con 0x55dc28f27420 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f6a9f2740 10 client.0 _ll_register_callbacks cb 0x7f8f5c09bcc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f653a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 0x55dc28f417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f653a3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 0x55dc28f417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39964/0 (socket says 172.21.15.28:39964) 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f653a3640 1 -- 172.21.15.28:0/2550003407 learned_addr learned my addr 172.21.15.28:0/2550003407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 0x55dc28f3f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f41260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 -- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 msgr2=0x55dc28f41260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f41260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 -- 172.21.15.28:0/2550003407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 msgr2=0x55dc28f417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 0x55dc28f417a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 -- 172.21.15.28:0/2550003407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc28f401e0 con 0x55dc28f27420 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f663a5640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 0x55dc28f3f9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f58034fd0 tx=0x7f8f580098f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e647f08360f7961 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.118+0000 7f8f497fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5805c030 con 0x55dc28f27420 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8f58004d50 con 0x55dc28f27420 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f58054070 con 0x55dc28f27420 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f58050ed0 con 0x55dc28f27420 2024-03-22T02:15:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f6a9f2740 10 client.25726 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dc28f94ba0 con 0x55dc28f27420 2024-03-22T02:15:02.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.119+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f58054070 con 0x55dc28f27420 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8f5804f420 con 0x55dc28f27420 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 10 client.25726 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 10 client.25726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 20 client.25726 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f95540 0x55dc28f97960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 4 client.25726 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc28e84160 con 0x55dc28f95540 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f95540 0x55dc28f97960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f65ba4640 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f95540 0x55dc28f97960 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c5ee18a20d824d3 server_cookie=4b0adda94092e16c in_seq=0 out_seq=0 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.120+0000 7f8f497fa640 10 client.25726 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:02.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f497fa640 1 -- 172.21.15.28:0/2550003407 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc28e84160 con 0x55dc28f95540 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f497fa640 10 client.25726 handle_command_reply: tid=0 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 client.25726 shutdown 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 msgr2=0x55dc28f3f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 0x55dc28f3f9b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8f58034fd0 tx=0x7f8f580098f0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 shutdown_connections 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dc28f95540 0x55dc28f97960 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f497fa640 0 client.25726 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dc28f27420 0x55dc28f3f9b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dc28f18210 0x55dc28f417a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 --2- 172.21.15.28:0/2550003407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dc28f17820 0x55dc28f41260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 >> 172.21.15.28:0/2550003407 conn(0x55dc28f12d90 msgr2=0x55dc28f221d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 shutdown_connections 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 1 -- 172.21.15.28:0/2550003407 wait complete. 2024-03-22T02:15:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.121+0000 7f8f6a9f2740 20 client.25726 trim_cache size 0 max 16384 2024-03-22T02:15:02.127 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 msgr2=0x7f8f4c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 0x7f8f4c0a53e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55dc28f18a70 tx=0x7f8f5802bef0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 msgr2=0x7f8f60197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c00e9c0 tx=0x7f8f5c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 shutdown_connections 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8f4c0a2f20 0x7f8f4c0a53e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f60102640 0x7f8f6019f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f60101c80 0x7f8f601983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 --2- 172.21.15.28:0/487282283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f60100a90 0x7f8f60197ea0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 >> 172.21.15.28:0/487282283 conn(0x7f8f600fc020 msgr2=0x7f8f600fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 shutdown_connections 2024-03-22T02:15:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:02.122+0000 7f8f67e08640 1 -- 172.21.15.28:0/487282283 wait complete. 2024-03-22T02:15:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:02 smithi120 ceph-mon[36803]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-22T02:15:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:02 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1093768113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:02 smithi083 ceph-mon[36340]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-22T02:15:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:02 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1093768113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:02 smithi028 ceph-mon[31916]: pgmap v401: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-22T02:15:02.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:02 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1093768113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:03.138 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:15:03.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:15:03.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 -- 172.21.15.28:0/1113376796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 msgr2=0x7fc10410a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 --2- 172.21.15.28:0/1113376796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10410a150 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc009ca0 tx=0x7fc0fc02c140 comp rx=0 tx=0).stop 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 -- 172.21.15.28:0/1113376796 shutdown_connections 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 --2- 172.21.15.28:0/1113376796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10410a150 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 --2- 172.21.15.28:0/1113376796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc104103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 --2- 172.21.15.28:0/1113376796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc1041002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 -- 172.21.15.28:0/1113376796 >> 172.21.15.28:0/1113376796 conn(0x7fc1040fb7c0 msgr2=0x7fc1040fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 -- 172.21.15.28:0/1113376796 shutdown_connections 2024-03-22T02:15:03.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.252+0000 7fc10968f640 1 -- 172.21.15.28:0/1113376796 wait complete. 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.253+0000 7fc10968f640 1 Processor -- start 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.253+0000 7fc10968f640 1 -- start start 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.253+0000 7fc10968f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.253+0000 7fc10968f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.253+0000 7fc10968f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38742/0 (socket says 172.21.15.28:38742) 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34652/0 (socket says 172.21.15.28:34652) 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39968/0 (socket says 172.21.15.28:39968) 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 -- 172.21.15.28:0/857356692 learned_addr learned my addr 172.21.15.28:0/857356692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc10410b6b0 con 0x7fc1041038c0 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc10410b530 con 0x7fc1040ffe90 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc10410b830 con 0x7fc104102f00 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 -- 172.21.15.28:0/857356692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 msgr2=0x7fc104195c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 -- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 msgr2=0x7fc10419d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1027fc640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10419d870 con 0x7fc104102f00 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc102ffd640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.254+0000 7fc1037fe640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc1027fc640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800ea60 tx=0x7fc0f800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc0ebfff640 1 -- 172.21.15.28:0/857356692 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f800cde0 con 0x7fc104102f00 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc0ebfff640 1 -- 172.21.15.28:0/857356692 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f80035f0 con 0x7fc104102f00 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc0ebfff640 1 -- 172.21.15.28:0/857356692 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8010690 con 0x7fc104102f00 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10419db60 con 0x7fc104102f00 2024-03-22T02:15:03.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.255+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10419e050 con 0x7fc104102f00 2024-03-22T02:15:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.257+0000 7fc10c272740 1 Processor -- start 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 -- start start 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 0x556ac32fefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc0ebfff640 1 -- 172.21.15.28:0/857356692 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc0f80037a0 con 0x7fc104102f00 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32ff5b0 0x556ac33040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3306bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ac3263570 con 0x556ac32ff5b0 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ac320be10 con 0x556ac32febc0 2024-03-22T02:15:03.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc10c272740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ac325f520 con 0x556ac3304720 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3306bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 0x556ac32fefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3306bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38750/0 (socket says 172.21.15.28:38750) 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1027fc640 1 -- 172.21.15.28:0/3693648815 learned_addr learned my addr 172.21.15.28:0/3693648815 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32ff5b0 0x556ac33040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 -- 172.21.15.28:0/3693648815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 msgr2=0x556ac3306bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3306bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 -- 172.21.15.28:0/3693648815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32ff5b0 msgr2=0x556ac33040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32ff5b0 0x556ac33040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc1037fe640 1 -- 172.21.15.28:0/3693648815 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac303f7e0 con 0x556ac32febc0 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.258+0000 7fc0ebfff640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 0x7fc0ec0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc1037fe640 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 0x556ac32fefe0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc02c680 tx=0x7fc0fc005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b989a4c31122dfc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc0ebfff640 1 -- 172.21.15.28:0/857356692 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc0f80cfb10 con 0x7fc104102f00 2024-03-22T02:15:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc102ffd640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 0x7fc0ec0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc0e97fa640 1 -- 172.21.15.28:0/3693648815 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc043070 con 0x556ac32febc0 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc0e97fa640 1 -- 172.21.15.28:0/3693648815 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0fc003e50 con 0x556ac32febc0 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.259+0000 7fc0e97fa640 1 -- 172.21.15.28:0/3693648815 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc039040 con 0x556ac32febc0 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- 172.21.15.28:0/3693648815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 msgr2=0x556ac32fefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc102ffd640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 0x7fc0ec0a53e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x556ac32ff9b0 tx=0x7fc0f400f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 0x556ac32fefe0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc02c680 tx=0x7fc0fc005ef0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- 172.21.15.28:0/3693648815 shutdown_connections 2024-03-22T02:15:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3306bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32ff5b0 0x556ac33040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 --2- 172.21.15.28:0/3693648815 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32febc0 0x556ac32fefe0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- 172.21.15.28:0/3693648815 >> 172.21.15.28:0/3693648815 conn(0x556ac32fa130 msgr2=0x556ac32fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- 172.21.15.28:0/3693648815 shutdown_connections 2024-03-22T02:15:03.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- 172.21.15.28:0/3693648815 wait complete. 2024-03-22T02:15:03.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 -- start start 2024-03-22T02:15:03.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.260+0000 7fc10c272740 1 Processor -- start 2024-03-22T02:15:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32febc0 0x556ac3308e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac3304720 0x556ac3310340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ac3307670 con 0x556ac3304720 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc1027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34668/0 (socket says 172.21.15.28:34668) 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc1027fc640 1 -- 172.21.15.28:0/3530764683 learned_addr learned my addr 172.21.15.28:0/3530764683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32febc0 0x556ac3308e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac3304720 0x556ac3310340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ac321f110 con 0x556ac32ff5b0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ac33077f0 con 0x556ac32febc0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.261+0000 7fc10c272740 10 client.0 _ll_register_callbacks cb 0x556ac325bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 -- 172.21.15.28:0/3530764683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32febc0 msgr2=0x556ac3308e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32febc0 0x556ac3308e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 -- 172.21.15.28:0/3530764683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac3304720 msgr2=0x556ac3310340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac3304720 0x556ac3310340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac309ff70 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800d010 tx=0x7fc0f80d31e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b13883bf284846 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc0e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80d5030 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc0f80baba0 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0fc003e50 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0f809b970 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc10c272740 10 client.25748 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.262+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556ac2f37170 con 0x556ac32ff5b0 2024-03-22T02:15:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0fc003e50 con 0x556ac32ff5b0 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0f809b280 con 0x556ac32ff5b0 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 10 client.25748 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 20 client.25748 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac3314b20 0x556ac3316f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 4 client.25748 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac3270ab0 con 0x556ac3314b20 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.263+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac3314b20 0x556ac3316f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac3314b20 0x556ac3316f50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46e19b12ef92dbcc server_cookie=b7159f0cfca669bf in_seq=0 out_seq=0 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc0e6ff5640 10 client.25748 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3530764683 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ac3270ab0 con 0x556ac3314b20 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc0e6ff5640 10 client.25748 handle_command_reply: tid=0 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 client.25748 shutdown 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 msgr2=0x556ac3309370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800d010 tx=0x7fc0f80d31e0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 shutdown_connections 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac3314b20 0x556ac3316f50 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac3304720 0x556ac3310340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3309370 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 --2- 172.21.15.28:0/3530764683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32febc0 0x556ac3308e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.264+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 >> 172.21.15.28:0/3530764683 conn(0x556ac32fa130 msgr2=0x556ac32f93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.265+0000 7fc0e6ff5640 0 client.25748 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.265+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 shutdown_connections 2024-03-22T02:15:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.265+0000 7fc10c272740 1 -- 172.21.15.28:0/3530764683 wait complete. 2024-03-22T02:15:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.265+0000 7fc10c272740 20 client.25748 trim_cache size 0 max 16384 2024-03-22T02:15:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 Processor -- start 2024-03-22T02:15:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 -- start start 2024-03-22T02:15:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3310300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3300050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041ab050 con 0x556ac32febc0 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3300050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39988/0 (socket says 172.21.15.28:39988) 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1037fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3300050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38760/0 (socket says 172.21.15.28:38760) 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc102ffd640 1 -- 172.21.15.28:0/1248548747 learned_addr learned my addr 172.21.15.28:0/1248548747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.300+0000 7fc10c272740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041a8920 con 0x556ac32ff5b0 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041a6770 con 0x556ac3304720 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3310300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 -- 172.21.15.28:0/1248548747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 msgr2=0x556ac3300050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3300050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 -- 172.21.15.28:0/1248548747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 msgr2=0x556ac32fdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 -- 172.21.15.28:0/1248548747 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac32f9ea0 con 0x556ac32ff5b0 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc102ffd640 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc1027fc640 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3310300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8098720 tx=0x7fc0f8003ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ac363165830ddc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc0e8ff9640 1 -- 172.21.15.28:0/1248548747 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f80d5070 con 0x556ac32ff5b0 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc0e8ff9640 1 -- 172.21.15.28:0/1248548747 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f809c530 con 0x556ac32ff5b0 2024-03-22T02:15:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.301+0000 7fc0e8ff9640 1 -- 172.21.15.28:0/1248548747 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8099f70 con 0x556ac32ff5b0 2024-03-22T02:15:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 msgr2=0x556ac3310300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3310300 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8098720 tx=0x7fc0f8003ed0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 shutdown_connections 2024-03-22T02:15:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac3304720 0x556ac3300050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac32ff5b0 0x556ac3310300 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.302+0000 7fc10c272740 1 --2- 172.21.15.28:0/1248548747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac32fdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 >> 172.21.15.28:0/1248548747 conn(0x556ac32fa130 msgr2=0x556ac32fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 shutdown_connections 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 -- 172.21.15.28:0/1248548747 wait complete. 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 -- start start 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 Processor -- start 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc10c272740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32ff5b0 0x556ac330da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc102ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40002/0 (socket says 172.21.15.28:40002) 2024-03-22T02:15:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.303+0000 7fc102ffd640 1 -- 172.21.15.28:0/3644466173 learned_addr learned my addr 172.21.15.28:0/3644466173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 0x556ac330ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041a6de0 con 0x556ac32febc0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041a8250 con 0x556ac3304720 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc1041a74b0 con 0x556ac32ff5b0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32ff5b0 0x556ac330da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc10c272740 10 client.0 _ll_register_callbacks cb 0x7fc0f800e9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 -- 172.21.15.28:0/3644466173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32ff5b0 msgr2=0x556ac330da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32ff5b0 0x556ac330da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 -- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 msgr2=0x556ac330ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc1037fe640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 0x556ac330ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 0x556ac330ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac330f030 con 0x556ac32febc0 2024-03-22T02:15:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc1037fe640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 0x556ac330ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc102ffd640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc104069ea0 tx=0x7fc0f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cfa8169d0bc8a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.304+0000 7fc0e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f400b7d0 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc0f400b970 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0f809c530 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0f400bb70 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc10c272740 10 client.15912 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556ac3310400 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.305+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f400bd10 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0f4036070 con 0x556ac32febc0 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 10 client.15912 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:03.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 20 client.15912 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac337b450 0x556ac337d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 4 client.15912 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fc0f809c530 con 0x556ac337b450 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.306+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac337b450 0x556ac337d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.307+0000 7fc0e6ff5640 10 client.15912 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.307+0000 7fc1027fc640 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac337b450 0x556ac337d870 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=202f84471e6d35d6 server_cookie=97b130d705388a4a in_seq=0 out_seq=0 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.308+0000 7fc0e6ff5640 1 -- 172.21.15.28:0/3644466173 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fc0f809c530 con 0x556ac337b450 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.308+0000 7fc0e6ff5640 10 client.15912 handle_command_reply: tid=0 2024-03-22T02:15:03.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.308+0000 7fc10c272740 1 client.15912 shutdown 2024-03-22T02:15:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.308+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 msgr2=0x556ac330d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc104069ea0 tx=0x7fc0f4004730 comp rx=0 tx=0).stop 2024-03-22T02:15:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 shutdown_connections 2024-03-22T02:15:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ac337b450 0x556ac337d870 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ac3304720 0x556ac330ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ac32ff5b0 0x556ac330da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 --2- 172.21.15.28:0/3644466173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ac32febc0 0x556ac330d4f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 >> 172.21.15.28:0/3644466173 conn(0x556ac32fa130 msgr2=0x556ac32fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc0e6ff5640 0 client.15912 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 shutdown_connections 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 1 -- 172.21.15.28:0/3644466173 wait complete. 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.309+0000 7fc10c272740 20 client.15912 trim_cache size 0 max 16384 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:03.315 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:15:03.316 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8fad35d7-aa4a-45b8-95e8-ae13567a80b1", 2024-03-22T02:15:03.316 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:15:03.316 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:03.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 msgr2=0x7fc0ec0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 0x7fc0ec0a53e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x556ac32ff9b0 tx=0x7fc0f400f040 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 msgr2=0x7fc1041961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc0f800ea60 tx=0x7fc0f800ef20 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 shutdown_connections 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0ec0a2f20 0x7fc0ec0a53e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1041038c0 0x7fc10419d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc104102f00 0x7fc1041961a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 --2- 172.21.15.28:0/857356692 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc1040ffe90 0x7fc104195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 >> 172.21.15.28:0/857356692 conn(0x7fc1040fb7c0 msgr2=0x7fc104106200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 shutdown_connections 2024-03-22T02:15:03.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.310+0000 7fc10968f640 1 -- 172.21.15.28:0/857356692 wait complete. 2024-03-22T02:15:03.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:15:03.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 -- 172.21.15.28:0/881809572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc101c90 msgr2=0x7f35cc102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 --2- 172.21.15.28:0/881809572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc101c90 0x7f35cc102110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009900 tx=0x7f35c002bc60 comp rx=0 tx=0).stop 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 -- 172.21.15.28:0/881809572 shutdown_connections 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 --2- 172.21.15.28:0/881809572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc102650 0x7f35cc108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 --2- 172.21.15.28:0/881809572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc101c90 0x7f35cc102110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 --2- 172.21.15.28:0/881809572 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc100aa0 0x7f35cc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 -- 172.21.15.28:0/881809572 >> 172.21.15.28:0/881809572 conn(0x7f35cc0fc030 msgr2=0x7f35cc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 -- 172.21.15.28:0/881809572 shutdown_connections 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.435+0000 7f35d159a640 1 -- 172.21.15.28:0/881809572 wait complete. 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 Processor -- start 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 -- start start 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 0x7f35cc197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc102650 0x7f35cc19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc10d840 con 0x7f35cc100aa0 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc10d6c0 con 0x7f35cc101c90 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35d159a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc10d9c0 con 0x7f35cc102650 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35ca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 0x7f35cc197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 0x7f35cc197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40028/0 (socket says 172.21.15.28:40028) 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35ca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34698/0 (socket says 172.21.15.28:34698) 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.436+0000 7f35caffd640 1 -- 172.21.15.28:0/2645898330 learned_addr learned my addr 172.21.15.28:0/2645898330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc102650 0x7f35cc19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 -- 172.21.15.28:0/2645898330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc102650 msgr2=0x7f35cc19f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc102650 0x7f35cc19f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 -- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 msgr2=0x7f35cc197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 0x7f35cc197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 -- 172.21.15.28:0/2645898330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc19fac0 con 0x7f35cc101c90 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35c0000c00 tx=0x7f35c002ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35b3fff640 1 -- 172.21.15.28:0/2645898330 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0004d90 con 0x7f35cc101c90 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35b3fff640 1 -- 172.21.15.28:0/2645898330 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c0005410 con 0x7f35cc101c90 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35b3fff640 1 -- 172.21.15.28:0/2645898330 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c002e950 con 0x7f35cc101c90 2024-03-22T02:15:03.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc1098a0 con 0x7f35cc101c90 2024-03-22T02:15:03.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.437+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc109ce0 con 0x7f35cc101c90 2024-03-22T02:15:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.440+0000 7f35b3fff640 1 -- 172.21.15.28:0/2645898330 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f35c004b060 con 0x7f35cc101c90 2024-03-22T02:15:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.440+0000 7f35d4184740 1 Processor -- start 2024-03-22T02:15:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.440+0000 7f35d4184740 1 -- start start 2024-03-22T02:15:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c06850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c0b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c0e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd83b6b8c0 con 0x55dd83c06430 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd83b13c20 con 0x55dd83c0bf90 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35d4184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dd83b67870 con 0x55dd83c06e20 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35b3fff640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 0x7f35b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c06850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c06850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40030/0 (socket says 172.21.15.28:40030) 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35cb7fe640 1 -- 172.21.15.28:0/4122993740 learned_addr learned my addr 172.21.15.28:0/4122993740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c0b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c0e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 -- 172.21.15.28:0/4122993740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 msgr2=0x55dd83c0e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c0e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 -- 172.21.15.28:0/4122993740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 msgr2=0x55dd83c06850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c06850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 -- 172.21.15.28:0/4122993740 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd83b38150 con 0x55dd83c06e20 2024-03-22T02:15:03.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 0x7f35b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35b3fff640 1 -- 172.21.15.28:0/2645898330 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f35c00f48c0 con 0x7f35cc101c90 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.441+0000 7f35caffd640 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c0b920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0058c0 tx=0x7f35bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c47a4d2ce0db6e63 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35b17fa640 1 -- 172.21.15.28:0/4122993740 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc004220 con 0x55dd83c06e20 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35b17fa640 1 -- 172.21.15.28:0/4122993740 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35bc0043c0 con 0x55dd83c06e20 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35b17fa640 1 -- 172.21.15.28:0/4122993740 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc004e00 con 0x55dd83c06e20 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 0x7f35b40a53e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55dd83c07680 tx=0x7f35c4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 -- 172.21.15.28:0/4122993740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 msgr2=0x55dd83c0b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c0b920 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0058c0 tx=0x7f35bc011810 comp rx=0 tx=0).stop 2024-03-22T02:15:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 -- 172.21.15.28:0/4122993740 shutdown_connections 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c0e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c0b920 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 --2- 172.21.15.28:0/4122993740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c06850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.442+0000 7f35d4184740 1 -- 172.21.15.28:0/4122993740 >> 172.21.15.28:0/4122993740 conn(0x55dd83c019a0 msgr2=0x55dd83c01da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- 172.21.15.28:0/4122993740 shutdown_connections 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- 172.21.15.28:0/4122993740 wait complete. 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- start start 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 Processor -- start 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 0x55dd83c10ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c17e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dd83c0ef40 con 0x55dd83c06e20 2024-03-22T02:15:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dd83b27130 con 0x55dd83c06430 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dd83c0f0c0 con 0x55dd83c0bf90 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35d4184740 10 client.0 _ll_register_callbacks cb 0x55dd83b26b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34706/0 (socket says 172.21.15.28:34706) 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35caffd640 1 -- 172.21.15.28:0/2740108569 learned_addr learned my addr 172.21.15.28:0/2740108569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 0x55dd83c10ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.443+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c17e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 -- 172.21.15.28:0/2740108569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 msgr2=0x55dd83c17e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c17e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 -- 172.21.15.28:0/2740108569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 msgr2=0x55dd83c10ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 0x55dd83c10ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 -- 172.21.15.28:0/2740108569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd839497d0 con 0x55dd83c06430 2024-03-22T02:15:03.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 0x55dd83c10ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c17e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35caffd640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009620 tx=0x7f35bc005310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=976bc1d7e95c463 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc027030 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35bc0043c0 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f35bc0046c0 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f35bc034050 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35d4184740 10 client.25763 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.444+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd8387f3f0 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35bc0043c0 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f35bc019c00 con 0x55dd83c06430 2024-03-22T02:15:03.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 10 client.25763 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 20 client.25763 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c1ca20 0x55dd83c1ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 4 client.25763 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd83b78e00 con 0x55dd83c1ca20 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c1ca20 0x55dd83c1ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35aeff5640 10 client.25763 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.445+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c1ca20 0x55dd83c1ee90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc511a7af00166e9 server_cookie=30d54804c18215d6 in_seq=0 out_seq=0 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35aeff5640 1 -- 172.21.15.28:0/2740108569 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dd83b78e00 con 0x55dd83c1ca20 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35aeff5640 10 client.25763 handle_command_reply: tid=0 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 client.25763 shutdown 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 msgr2=0x55dd83c10980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009620 tx=0x7f35bc005310 comp rx=0 tx=0).stop 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 shutdown_connections 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c1ca20 0x55dd83c1ee90 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c17e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06e20 0x55dd83c10ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 --2- 172.21.15.28:0/2740108569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06430 0x55dd83c10980 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35aeff5640 0 client.25763 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 >> 172.21.15.28:0/2740108569 conn(0x55dd83c019a0 msgr2=0x55dd83c00c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 shutdown_connections 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 1 -- 172.21.15.28:0/2740108569 wait complete. 2024-03-22T02:15:03.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.446+0000 7f35d4184740 20 client.25763 trim_cache size 0 max 16384 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 Processor -- start 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 -- start start 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c05310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1ab0c0 con 0x55dd83c06430 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1a8990 con 0x55dd83c06e20 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35d4184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1a67e0 con 0x55dd83c0bf90 2024-03-22T02:15:03.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35ca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c05310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38822/0 (socket says 172.21.15.28:38822) 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35ca7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34716/0 (socket says 172.21.15.28:34716) 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.469+0000 7f35cb7fe640 1 -- 172.21.15.28:0/3853738011 learned_addr learned my addr 172.21.15.28:0/3853738011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 -- 172.21.15.28:0/3853738011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 msgr2=0x55dd83c17e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 -- 172.21.15.28:0/3853738011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 msgr2=0x55dd83c05310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c05310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 -- 172.21.15.28:0/3853738011 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd8383f0f0 con 0x55dd83c0bf90 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f35c40136c0 tx=0x7f35c4009be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b52f521f96bf191 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35b0ff9640 1 -- 172.21.15.28:0/3853738011 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4004b20 con 0x55dd83c0bf90 2024-03-22T02:15:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35b0ff9640 1 -- 172.21.15.28:0/3853738011 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c4004cc0 con 0x55dd83c0bf90 2024-03-22T02:15:03.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.470+0000 7f35b0ff9640 1 -- 172.21.15.28:0/3853738011 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c4005440 con 0x55dd83c0bf90 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 -- 172.21.15.28:0/3853738011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 msgr2=0x55dd83c0ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f35c40136c0 tx=0x7f35c4009be0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 -- 172.21.15.28:0/3853738011 shutdown_connections 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c0bf90 0x55dd83c0ffa0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c06e20 0x55dd83c17e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 --2- 172.21.15.28:0/3853738011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c05310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 -- 172.21.15.28:0/3853738011 >> 172.21.15.28:0/3853738011 conn(0x55dd83c019a0 msgr2=0x55dd83c04c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 -- 172.21.15.28:0/3853738011 shutdown_connections 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.471+0000 7f35d4184740 1 -- 172.21.15.28:0/3853738011 wait complete. 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 -- start start 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 Processor -- start 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c15020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c80dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1a6e50 con 0x55dd83c06430 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1a82c0 con 0x55dd83c0bf90 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1a7520 con 0x55dd83c06e20 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35d4184740 10 client.0 _ll_register_callbacks cb 0x7f35bc004f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40072/0 (socket says 172.21.15.28:40072) 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35ca7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c15020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 -- 172.21.15.28:0/3080522813 learned_addr learned my addr 172.21.15.28:0/3080522813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35cb7fe640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c80dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 -- 172.21.15.28:0/3080522813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 msgr2=0x55dd83c15020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c15020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 -- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 msgr2=0x55dd83c80dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c80dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35caffd640 1 -- 172.21.15.28:0/3080522813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd83c18080 con 0x55dd83c06430 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.472+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c15020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35caffd640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0058c0 tx=0x7f35bc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1847f1e9d5d2430 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35aeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:03.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35bc027070 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f35bc005380 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c4004cc0 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f35bc01d040 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35d4184740 10 client.15930 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd83c82300 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.473+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c4004cc0 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f35bc019430 con 0x55dd83c06430 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 10 client.15930 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 20 client.15930 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c82d20 0x55dd83c85140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 4 client.15930 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:15:03.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55dd83b73150 con 0x55dd83c82d20 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.474+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c82d20 0x55dd83c85140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.475+0000 7f35ca7fc640 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c82d20 0x55dd83c85140 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d908588401d3c00 server_cookie=d25e1df2788b27df in_seq=0 out_seq=0 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.475+0000 7f35aeff5640 10 client.15930 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.475+0000 7f35aeff5640 1 -- 172.21.15.28:0/3080522813 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55dd83b73150 con 0x55dd83c82d20 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.475+0000 7f35aeff5640 10 client.15930 handle_command_reply: tid=0 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 client.15930 shutdown 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 msgr2=0x55dd83c14ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0058c0 tx=0x7f35bc005f90 comp rx=0 tx=0).stop 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 shutdown_connections 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dd83c82d20 0x55dd83c85140 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dd83c0bf90 0x55dd83c80dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dd83c06e20 0x55dd83c15020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35aeff5640 0 client.15930 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 --2- 172.21.15.28:0/3080522813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd83c06430 0x55dd83c14ae0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 >> 172.21.15.28:0/3080522813 conn(0x55dd83c019a0 msgr2=0x55dd83c05490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 shutdown_connections 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 1 -- 172.21.15.28:0/3080522813 wait complete. 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.476+0000 7f35d4184740 20 client.15930 trim_cache size 0 max 16384 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stdout: "8fad35d7-aa4a-45b8-95e8-ae13567a80b1": { 2024-03-22T02:15:03.484 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:15:03.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8fad35d7-aa4a-45b8-95e8-ae13567a80b1", 2024-03-22T02:15:03.485 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:15:03.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:15:03.485 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:15:03.485 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 msgr2=0x7f35b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 0x7f35b40a53e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55dd83c07680 tx=0x7f35c4004ab0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 msgr2=0x7f35cc1983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35c0000c00 tx=0x7f35c002ed20 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 shutdown_connections 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f35b40a2f20 0x7f35b40a53e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f35cc102650 0x7f35cc19f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f35cc101c90 0x7f35cc1983e0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f35c0000c00 tx=0x7f35c002ed20 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 --2- 172.21.15.28:0/2645898330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35cc100aa0 0x7f35cc197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:03.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 >> 172.21.15.28:0/2645898330 conn(0x7f35cc0fc030 msgr2=0x7f35cc105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 shutdown_connections 2024-03-22T02:15:03.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:03.477+0000 7f35d159a640 1 -- 172.21.15.28:0/2645898330 wait complete. 2024-03-22T02:15:03.490 INFO:tasks.cephfs.filesystem:scrub status for tag:8fad35d7-aa4a-45b8-95e8-ae13567a80b1 - {'path': '/', 'tag': '8fad35d7-aa4a-45b8-95e8-ae13567a80b1', 'options': 'recursive,force'} 2024-03-22T02:15:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.262+0000 7fd18a935640 1 -- 172.21.15.28:0/978070210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184101de0 msgr2=0x7fd184108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.262+0000 7fd18a935640 1 --2- 172.21.15.28:0/978070210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184101de0 0x7fd184108670 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd17c009ca0 tx=0x7fd17c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 -- 172.21.15.28:0/978070210 shutdown_connections 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 --2- 172.21.15.28:0/978070210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184101de0 0x7fd184108670 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 --2- 172.21.15.28:0/978070210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101420 0x7fd1841018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 --2- 172.21.15.28:0/978070210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184100230 0x7fd184100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 -- 172.21.15.28:0/978070210 >> 172.21.15.28:0/978070210 conn(0x7fd1840fb7e0 msgr2=0x7fd1840fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 -- 172.21.15.28:0/978070210 shutdown_connections 2024-03-22T02:15:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.263+0000 7fd18a935640 1 -- 172.21.15.28:0/978070210 wait complete. 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.264+0000 7fd18a935640 1 Processor -- start 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.264+0000 7fd18a935640 1 -- start start 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 0x7fd184197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 0x7fd184198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101de0 0x7fd18419f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 0x7fd184197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 0x7fd184197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40088/0 (socket says 172.21.15.28:40088) 2024-03-22T02:15:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd183fff640 1 -- 172.21.15.28:0/1337413909 learned_addr learned my addr 172.21.15.28:0/1337413909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd18410d880 con 0x7fd184100230 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 0x7fd184198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd188ed2640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101de0 0x7fd18419f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd18410d700 con 0x7fd184101de0 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.265+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd18410da00 con 0x7fd184101420 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 -- 172.21.15.28:0/1337413909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101de0 msgr2=0x7fd18419f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101de0 0x7fd18419f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 -- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 msgr2=0x7fd184197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 0x7fd184197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18419f800 con 0x7fd184101420 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 0x7fd184198120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e9f0 tx=0x7fd17800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1817fa640 1 -- 172.21.15.28:0/1337413909 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17800ccc0 con 0x7fd184101420 2024-03-22T02:15:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1817fa640 1 -- 172.21.15.28:0/1337413909 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd178010910 con 0x7fd184101420 2024-03-22T02:15:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18419faf0 con 0x7fd184101420 2024-03-22T02:15:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd1817fa640 1 -- 172.21.15.28:0/1337413909 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd178018960 con 0x7fd184101420 2024-03-22T02:15:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.266+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18419ffd0 con 0x7fd184101420 2024-03-22T02:15:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.269+0000 7fd1817fa640 1 -- 172.21.15.28:0/1337413909 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd178010430 con 0x7fd184101420 2024-03-22T02:15:04.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 Processor -- start 2024-03-22T02:15:04.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 -- start start 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591ced0 0x5555c59219d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c59244d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555c5881040 con 0x5555c591ced0 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5555c58299e0 con 0x5555c591c4e0 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555c587cff0 con 0x5555c5922040 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34756/0 (socket says 172.21.15.28:34756) 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 -- 172.21.15.28:0/2884038454 learned_addr learned my addr 172.21.15.28:0/2884038454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd183fff640 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591ced0 0x5555c59219d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd1837fe640 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c59244d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 -- 172.21.15.28:0/2884038454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 msgr2=0x5555c59244d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c59244d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 -- 172.21.15.28:0/2884038454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591ced0 msgr2=0x5555c59219d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591ced0 0x5555c59219d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.270+0000 7fd188ed2640 1 -- 172.21.15.28:0/2884038454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c5595210 con 0x5555c591c4e0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd1817fa640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 0x7fd16c0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd188ed2640 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd17c037040 tx=0x7fd17c0093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c9f85a1677ea432 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd183fff640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 0x7fd16c0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd16a7fc640 1 -- 172.21.15.28:0/2884038454 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c043070 con 0x5555c591c4e0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd16a7fc640 1 -- 172.21.15.28:0/2884038454 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd17c032c90 con 0x5555c591c4e0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd16a7fc640 1 -- 172.21.15.28:0/2884038454 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17c039040 con 0x5555c591c4e0 2024-03-22T02:15:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.271+0000 7fd1817fa640 1 -- 172.21.15.28:0/1337413909 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd1780cfb00 con 0x7fd184101420 2024-03-22T02:15:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- 172.21.15.28:0/2884038454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 msgr2=0x5555c591c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd183fff640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 0x7fd16c0a5500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5555c591d2d0 tx=0x7fd174005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd17c037040 tx=0x7fd17c0093f0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- 172.21.15.28:0/2884038454 shutdown_connections 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c59244d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591ced0 0x5555c59219d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/2884038454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c591c900 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- 172.21.15.28:0/2884038454 >> 172.21.15.28:0/2884038454 conn(0x5555c5917a50 msgr2=0x5555c5917e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- 172.21.15.28:0/2884038454 shutdown_connections 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- 172.21.15.28:0/2884038454 wait complete. 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.272+0000 7fd18d5f3740 1 -- start start 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 Processor -- start 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c59269c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c5922040 0x5555c592da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555c5924f90 con 0x5555c5922040 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5555c583cdb0 con 0x5555c591c4e0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555c5925110 con 0x5555c591ced0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd18d5f3740 10 client.0 _ll_register_callbacks cb 0x5555c5594a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd183fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd183fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34758/0 (socket says 172.21.15.28:34758) 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd1837fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c59269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd183fff640 1 -- 172.21.15.28:0/1458272409 learned_addr learned my addr 172.21.15.28:0/1458272409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.273+0000 7fd188ed2640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c5922040 0x5555c592da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 -- 172.21.15.28:0/1458272409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 msgr2=0x5555c59269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c59269c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 -- 172.21.15.28:0/1458272409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c5922040 msgr2=0x5555c592da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c5922040 0x5555c592da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 -- 172.21.15.28:0/1458272409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c5661620 con 0x5555c591c4e0 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c59269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:15:04.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd183fff640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd17401e9e0 tx=0x7fd174004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78c9234ad73558d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd1637fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17402e060 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd174018030 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd17c032c90 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd174035050 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd18d5f3740 10 client.25781 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.274+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5555c5930de0 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd17c032c90 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd17401c4c0 con 0x5555c591c4e0 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 10 client.25781 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:04.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 10 client.25781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 20 client.25781 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c59324b0 0x5555c59348c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 4 client.25781 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555c588e580 con 0x5555c59324b0 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c59324b0 0x5555c59348c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.275+0000 7fd1637fe640 10 client.25781 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.276+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c59324b0 0x5555c59348c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa07d2ce47266dcc server_cookie=45aa59b8cf28a6e3 in_seq=0 out_seq=0 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.276+0000 7fd1637fe640 1 -- 172.21.15.28:0/1458272409 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555c588e580 con 0x5555c59324b0 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.276+0000 7fd1637fe640 10 client.25781 handle_command_reply: tid=0 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.276+0000 7fd18d5f3740 1 client.25781 shutdown 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 msgr2=0x5555c5926480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd17401e9e0 tx=0x7fd174004730 comp rx=0 tx=0).stop 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 shutdown_connections 2024-03-22T02:15:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c59324b0 0x5555c59348c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c5922040 0x5555c592da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd1637fe640 0 client.25781 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c59269c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1458272409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591c4e0 0x5555c5926480 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 >> 172.21.15.28:0/1458272409 conn(0x5555c5917a50 msgr2=0x5555c5916c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 shutdown_connections 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1458272409 wait complete. 2024-03-22T02:15:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.277+0000 7fd18d5f3740 20 client.25781 trim_cache size 0 max 16384 2024-03-22T02:15:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.305+0000 7fd18d5f3740 1 Processor -- start 2024-03-22T02:15:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 -- start start 2024-03-22T02:15:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 0x5555c592d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40120/0 (socket says 172.21.15.28:40120) 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c591dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 -- 172.21.15.28:0/346464316 learned_addr learned my addr 172.21.15.28:0/346464316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841ad060 con 0x5555c591c4e0 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841aa930 con 0x5555c591ced0 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841a8780 con 0x5555c5922040 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd1837fe640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 0x5555c592d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd188ed2640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c591dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 -- 172.21.15.28:0/346464316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 msgr2=0x5555c591dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c591dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 -- 172.21.15.28:0/346464316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 msgr2=0x5555c592d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 0x5555c592d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd183fff640 1 -- 172.21.15.28:0/346464316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555c591c8e0 con 0x5555c591c4e0 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.306+0000 7fd1837fe640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 0x5555c592d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.307+0000 7fd183fff640 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1740092e0 tx=0x7fd17401e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19218563eaf31656 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.307+0000 7fd169ffb640 1 -- 172.21.15.28:0/346464316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17401cd30 con 0x5555c591c4e0 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.307+0000 7fd169ffb640 1 -- 172.21.15.28:0/346464316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd174004020 con 0x5555c591c4e0 2024-03-22T02:15:04.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.307+0000 7fd169ffb640 1 -- 172.21.15.28:0/346464316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174003b30 con 0x5555c591c4e0 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 msgr2=0x5555c591d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1740092e0 tx=0x7fd17401e940 comp rx=0 tx=0).stop 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 shutdown_connections 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c5922040 0x5555c591dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c591ced0 0x5555c592d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/346464316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c591c4e0 0x5555c591d760 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 >> 172.21.15.28:0/346464316 conn(0x5555c5917a50 msgr2=0x5555c591b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 shutdown_connections 2024-03-22T02:15:04.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- 172.21.15.28:0/346464316 wait complete. 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.308+0000 7fd18d5f3740 1 -- start start 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 Processor -- start 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c592b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 0x5555c592cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd1837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd188ed2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 0x5555c592cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd188ed2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 0x5555c592cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40124/0 (socket says 172.21.15.28:40124) 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd183fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c592b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd1837fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34778/0 (socket says 172.21.15.28:34778) 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd188ed2640 1 -- 172.21.15.28:0/1259778059 learned_addr learned my addr 172.21.15.28:0/1259778059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841a8df0 con 0x5555c592cb70 2024-03-22T02:15:04.315 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841aa260 con 0x5555c5922040 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1841a94c0 con 0x5555c591ced0 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.309+0000 7fd18d5f3740 10 client.0 _ll_register_callbacks cb 0x7fd17401e870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 -- 172.21.15.28:0/1259778059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 msgr2=0x5555c592b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c592b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 -- 172.21.15.28:0/1259778059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 msgr2=0x5555c592cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 0x5555c592cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555c592d530 con 0x5555c5922040 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd183fff640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c592b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1837fe640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1780cd240 tx=0x7fd17800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d38d497cf3759e9f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1637fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17809ba10 con 0x5555c5922040 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd178098540 con 0x5555c5922040 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd174004020 con 0x5555c5922040 2024-03-22T02:15:04.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd17809b830 con 0x5555c5922040 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd18d5f3740 10 client.25787 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.310+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5555c5999140 con 0x5555c5922040 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.311+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1780d7040 con 0x5555c5922040 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fd178010ac0 con 0x5555c5922040 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 10 client.25787 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 10 client.25787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 20 client.25787 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c5999aa0 0x5555c599bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 4 client.25787 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd174004020 con 0x5555c5999aa0 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd183fff640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c5999aa0 0x5555c599bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:04.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd183fff640 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c5999aa0 0x5555c599bec0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11a37937c04905ef server_cookie=8aa497fdcd5d0188 in_seq=0 out_seq=0 2024-03-22T02:15:04.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.312+0000 7fd1637fe640 10 client.25787 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd1637fe640 1 -- 172.21.15.28:0/1259778059 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd174004020 con 0x5555c5999aa0 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd1637fe640 10 client.25787 handle_command_reply: tid=0 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 client.25787 shutdown 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 msgr2=0x5555c592bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd1780cd240 tx=0x7fd17800cf60 comp rx=0 tx=0).stop 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 shutdown_connections 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5555c5999aa0 0x5555c599bec0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555c592cb70 0x5555c592cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555c5922040 0x5555c592bb60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 --2- 172.21.15.28:0/1259778059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555c591ced0 0x5555c592b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 >> 172.21.15.28:0/1259778059 conn(0x5555c5917a50 msgr2=0x5555c59163c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd1637fe640 0 client.25787 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 shutdown_connections 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 1 -- 172.21.15.28:0/1259778059 wait complete. 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.317+0000 7fd18d5f3740 20 client.25787 trim_cache size 0 max 16384 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:15:04.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:15:04.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:15:04.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 msgr2=0x7fd16c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 0x7fd16c0a5500 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x5555c591d2d0 tx=0x7fd174005f90 comp rx=0 tx=0).stop 2024-03-22T02:15:04.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 msgr2=0x7fd184198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 0x7fd184198120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd17800e9f0 tx=0x7fd17800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 shutdown_connections 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd16c0a3040 0x7fd16c0a5500 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd184101de0 0x7fd18419f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd184101420 0x7fd184198120 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 --2- 172.21.15.28:0/1337413909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd184100230 0x7fd184197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 >> 172.21.15.28:0/1337413909 conn(0x7fd1840fb7e0 msgr2=0x7fd184104650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 shutdown_connections 2024-03-22T02:15:04.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:04.318+0000 7fd18a935640 1 -- 172.21.15.28:0/1337413909 wait complete. 2024-03-22T02:15:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:04 smithi120 ceph-mon[36803]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:04 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:15:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:04 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:04 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:15:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:04 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:04 smithi083 ceph-mon[36340]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:04 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:15:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:04 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:04 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:15:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:04 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:04 smithi028 ceph-mon[31916]: pgmap v402: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:04 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:15:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:04 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:04 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:15:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:04 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:15:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:15:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:05 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:15:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:15:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:05 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:05 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:15:06.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:15:06.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:15:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:15:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: pgmap v403: 129 pgs: 129 active+clean; 671 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-22T02:15:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:15:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:15:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:15:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:08 smithi120 ceph-mon[36803]: pgmap v404: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-22T02:15:08.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:08 smithi083 ceph-mon[36340]: pgmap v404: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-22T02:15:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:08 smithi028 ceph-mon[31916]: pgmap v404: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2024-03-22T02:15:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:10 smithi120 ceph-mon[36803]: pgmap v405: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-22T02:15:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:10 smithi083 ceph-mon[36340]: pgmap v405: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-22T02:15:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:10 smithi028 ceph-mon[31916]: pgmap v405: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-22T02:15:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:12 smithi120 ceph-mon[36803]: pgmap v406: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:12 smithi083 ceph-mon[36340]: pgmap v406: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:12 smithi028 ceph-mon[31916]: pgmap v406: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-22T02:15:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:14 smithi120 ceph-mon[36803]: pgmap v407: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-22T02:15:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:14 smithi083 ceph-mon[36340]: pgmap v407: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-22T02:15:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:14 smithi028 ceph-mon[31916]: pgmap v407: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-22T02:15:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:16 smithi120 ceph-mon[36803]: pgmap v408: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:16 smithi083 ceph-mon[36340]: pgmap v408: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:16 smithi028 ceph-mon[31916]: pgmap v408: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:16.995 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 96% (11794/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-22T02:15:17.193 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to 'c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e'. 2024-03-22T02:15:17.193 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.193 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-22T02:15:17.193 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-22T02:15:17.193 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-03-22T02:15:17.194 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at c3a04c0e75f Merge PR #54581 into wip-batrick-testing-20240321.172515 2024-03-22T02:15:17.198 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-22T02:15:17.265 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-22T02:15:17.267 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T02:15:17.267 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-22T02:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 -- 172.21.15.28:0/1866541111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 msgr2=0x7f7aa8108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/1866541111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa8108f10 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0009ca0 tx=0x7f7aa002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 -- 172.21.15.28:0/1866541111 shutdown_connections 2024-03-22T02:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/1866541111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa8108f10 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/1866541111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 0x7f7aa8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/1866541111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 0x7f7aa8100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 -- 172.21.15.28:0/1866541111 >> 172.21.15.28:0/1866541111 conn(0x7f7aa80fc040 msgr2=0x7f7aa80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 -- 172.21.15.28:0/1866541111 shutdown_connections 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.410+0000 7f7aacfb6640 1 -- 172.21.15.28:0/1866541111 wait complete. 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 Processor -- start 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 -- start start 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 0x7f7aa8197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 0x7f7aa8198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810c750 con 0x7f7aa8102680 2024-03-22T02:15:17.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810c5d0 con 0x7f7aa8100ad0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aacfb6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa810c8d0 con 0x7f7aa8101cc0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48582/0 (socket says 172.21.15.28:48582) 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/2268214407 learned_addr learned my addr 172.21.15.28:0/2268214407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 0x7f7aa8198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa659c640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 0x7f7aa8197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/2268214407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 msgr2=0x7f7aa8198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 0x7f7aa8198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/2268214407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 msgr2=0x7f7aa8197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 0x7f7aa8197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/2268214407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa819f9f0 con 0x7f7aa8102680 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.411+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0037040 tx=0x7f7aa0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.412+0000 7f7a977fe640 1 -- 172.21.15.28:0/2268214407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0041020 con 0x7f7aa8102680 2024-03-22T02:15:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.412+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa819fc80 con 0x7f7aa8102680 2024-03-22T02:15:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.412+0000 7f7a977fe640 1 -- 172.21.15.28:0/2268214407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa0034cd0 con 0x7f7aa8102680 2024-03-22T02:15:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.412+0000 7f7a977fe640 1 -- 172.21.15.28:0/2268214407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0039040 con 0x7f7aa8102680 2024-03-22T02:15:17.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.412+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa81a01c0 con 0x7f7aa8102680 2024-03-22T02:15:17.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7a977fe640 1 -- 172.21.15.28:0/2268214407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7aa0044020 con 0x7f7aa8102680 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 Processor -- start 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 -- start start 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7a977fe640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 0x7f7a900a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672e5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564867241040 con 0x5648672e12e0 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.414+0000 7f7aafb98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648671e99e0 con 0x5648672e6450 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aafb98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56486723cff0 con 0x5648672e08f0 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 0x7f7a900a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7a977fe640 1 -- 172.21.15.28:0/2268214407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7aa00f1610 con 0x7f7aa8102680 2024-03-22T02:15:17.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37008/0 (socket says 172.21.15.28:37008) 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa659c640 1 -- 172.21.15.28:0/1623410818 learned_addr learned my addr 172.21.15.28:0/1623410818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672e5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/1623410818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 msgr2=0x5648672e0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/1623410818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 msgr2=0x5648672e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672e5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 -- 172.21.15.28:0/1623410818 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564866f55210 con 0x5648672e6450 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa659c640 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672e88e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00ef070 tx=0x7f7aa0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad07016799c11d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 0x7f7a900a5480 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009fd0 tx=0x7f7a9c009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7a94ff9640 1 -- 172.21.15.28:0/1623410818 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00bd320 con 0x5648672e6450 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7a94ff9640 1 -- 172.21.15.28:0/1623410818 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa00be0c0 con 0x5648672e6450 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.415+0000 7f7a94ff9640 1 -- 172.21.15.28:0/1623410818 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa00ba030 con 0x5648672e6450 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- 172.21.15.28:0/1623410818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 msgr2=0x5648672e88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672e88e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7aa00ef070 tx=0x7f7aa0004a40 comp rx=0 tx=0).stop 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- 172.21.15.28:0/1623410818 shutdown_connections 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672e88e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672e5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 --2- 172.21.15.28:0/1623410818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e08f0 0x5648672e0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- 172.21.15.28:0/1623410818 >> 172.21.15.28:0/1623410818 conn(0x5648672d7a50 msgr2=0x5648672d7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- 172.21.15.28:0/1623410818 shutdown_connections 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- 172.21.15.28:0/1623410818 wait complete. 2024-03-22T02:15:17.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.416+0000 7f7aafb98740 1 -- start start 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 Processor -- start 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 0x5648672eaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648672e91c0 con 0x5648672e08f0 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648671fcdb0 con 0x5648672e12e0 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648672e9340 con 0x5648672e6450 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aafb98740 10 client.0 _ll_register_callbacks cb 0x564866f54a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48604/0 (socket says 172.21.15.28:48604) 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 -- 172.21.15.28:0/3649393508 learned_addr learned my addr 172.21.15.28:0/3649393508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:17.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 0x5648672eaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 -- 172.21.15.28:0/3649393508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 msgr2=0x5648672f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672f1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 -- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 msgr2=0x5648672eaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 0x5648672eaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 -- 172.21.15.28:0/3649393508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564867021620 con 0x5648672e08f0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 0x5648672eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7a98012720 tx=0x7f7a98012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb8d70c017c60f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7a8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98004030 con 0x5648672e08f0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.417+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7a980041d0 con 0x5648672e08f0 2024-03-22T02:15:17.427 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7aa00be0c0 con 0x5648672e08f0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7a98025050 con 0x5648672e08f0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 10 client.15960 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5648672f4ee0 con 0x5648672e08f0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7aa00be0c0 con 0x5648672e08f0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7a98003990 con 0x5648672e08f0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 10 client.15960 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 20 client.15960 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648672f65d0 0x5648672f8a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 4 client.15960 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56486724e580 con 0x5648672f65d0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.418+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648672f65d0 0x5648672f8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.428 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648672f65d0 0x5648672f8a70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79aff68eb80a3767 server_cookie=d1ee9639c5db05f in_seq=0 out_seq=0 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7a8dffb640 10 client.15960 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3649393508 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56486724e580 con 0x5648672f65d0 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7a8dffb640 10 client.15960 handle_command_reply: tid=0 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 client.15960 shutdown 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 msgr2=0x5648672ea520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7a98012720 tx=0x7f7a98012be0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 shutdown_connections 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648672f65d0 0x5648672f8a70 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e12e0 0x5648672eaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7a8dffb640 0 client.15960 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3649393508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ea520 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 >> 172.21.15.28:0/3649393508 conn(0x5648672d7a50 msgr2=0x5648672d6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 shutdown_connections 2024-03-22T02:15:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.419+0000 7f7aafb98740 1 -- 172.21.15.28:0/3649393508 wait complete. 2024-03-22T02:15:17.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.420+0000 7f7aafb98740 20 client.15960 trim_cache size 0 max 16384 2024-03-22T02:15:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 Processor -- start 2024-03-22T02:15:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 -- start start 2024-03-22T02:15:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e08f0 0x5648672ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672d6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81aaff0 con 0x5648672e12e0 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81a88c0 con 0x5648672e08f0 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aafb98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81a6710 con 0x5648672e6450 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aa5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aa5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48610/0 (socket says 172.21.15.28:48610) 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.434+0000 7f7aa5d9b640 1 -- 172.21.15.28:0/2447197878 learned_addr learned my addr 172.21.15.28:0/2447197878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa659c640 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e08f0 0x5648672ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672d6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 -- 172.21.15.28:0/2447197878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 msgr2=0x5648672d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672d6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 -- 172.21.15.28:0/2447197878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e08f0 msgr2=0x5648672ea800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e08f0 0x5648672ea800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 -- 172.21.15.28:0/2447197878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648672e1ad0 con 0x5648672e12e0 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5648672e1d50 tx=0x7f7a9c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57e8ad1b7a164a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7a8ffff640 1 -- 172.21.15.28:0/2447197878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c02c450 con 0x5648672e12e0 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7a8ffff640 1 -- 172.21.15.28:0/2447197878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c00fe70 con 0x5648672e12e0 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.435+0000 7f7a8ffff640 1 -- 172.21.15.28:0/2447197878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c005490 con 0x5648672e12e0 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- 172.21.15.28:0/2447197878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 msgr2=0x5648672f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x5648672e1d50 tx=0x7f7a9c021000 comp rx=0 tx=0).stop 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- 172.21.15.28:0/2447197878 shutdown_connections 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e6450 0x5648672d6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e12e0 0x5648672f1a60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- 172.21.15.28:0/2447197878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e08f0 0x5648672ea800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- 172.21.15.28:0/2447197878 >> 172.21.15.28:0/2447197878 conn(0x5648672d7a50 msgr2=0x5648672d63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- 172.21.15.28:0/2447197878 shutdown_connections 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- 172.21.15.28:0/2447197878 wait complete. 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- start start 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 Processor -- start 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 0x5648672ef550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81a6d80 con 0x5648672e08f0 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81a81f0 con 0x5648672e6450 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa81a7450 con 0x5648672e12e0 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aafb98740 10 client.0 _ll_register_callbacks cb 0x7f7a98009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:17.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aa5d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 0x5648672ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aa659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48612/0 (socket says 172.21.15.28:48612) 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aa659c640 1 -- 172.21.15.28:0/3371508474 learned_addr learned my addr 172.21.15.28:0/3371508474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.436+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 -- 172.21.15.28:0/3371508474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 msgr2=0x5648672ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 0x5648672ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 -- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 msgr2=0x5648672f04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672f04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 -- 172.21.15.28:0/3371508474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648672f0ab0 con 0x5648672e08f0 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa6d9d640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672f04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 0x5648672ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:17.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aa659c640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7a98003d80 tx=0x7f7a98003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec240fdab6be659 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a98022040 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7a98005280 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a9c00fe70 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7a98036050 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 10 client.15966 fetch_fsmap learned FSMap version 25 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5648672f1e30 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a9c00fe70 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f7a9801c460 con 0x5648672e08f0 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 10 client.15966 fetch_fsmap finished waiting for FSMap version 25 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 10 client.15966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.437+0000 7f7aafb98740 20 client.15966 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:17.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56486735d1b0 0x56486735f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7aafb98740 4 client.15966 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:15:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648672488d0 con 0x56486735d1b0 2024-03-22T02:15:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56486735d1b0 0x56486735f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7a8dffb640 10 client.15966 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:17.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.438+0000 7f7aa5d9b640 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56486735d1b0 0x56486735f5d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84dd222510672186 server_cookie=5aa8569e58fd475 in_seq=0 out_seq=0 2024-03-22T02:15:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7a8dffb640 1 -- 172.21.15.28:0/3371508474 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648672488d0 con 0x56486735d1b0 2024-03-22T02:15:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7a8dffb640 10 client.15966 handle_command_reply: tid=0 2024-03-22T02:15:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 client.15966 shutdown 2024-03-22T02:15:17.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 msgr2=0x5648672ef010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7a98003d80 tx=0x7f7a98003890 comp rx=0 tx=0).stop 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 shutdown_connections 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56486735d1b0 0x56486735f5d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648672e6450 0x5648672f04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648672e12e0 0x5648672ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7a8dffb640 0 client.15966 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 --2- 172.21.15.28:0/3371508474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648672e08f0 0x5648672ef010 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 >> 172.21.15.28:0/3371508474 conn(0x5648672d7a50 msgr2=0x5648672d6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 shutdown_connections 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 1 -- 172.21.15.28:0/3371508474 wait complete. 2024-03-22T02:15:17.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aafb98740 20 client.15966 trim_cache size 0 max 16384 2024-03-22T02:15:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:15:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:15:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:15:17.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 msgr2=0x7f7a900a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 0x7f7a900a5480 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009fd0 tx=0x7f7a9c009b80 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 msgr2=0x7f7aa819f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7aa0037040 tx=0x7f7aa0032f40 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 shutdown_connections 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7a900a2fc0 0x7f7a900a5480 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aa8102680 0x7f7aa819f380 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7aa8101cc0 0x7f7aa8198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 --2- 172.21.15.28:0/2268214407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7aa8100ad0 0x7f7aa8197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:17.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 >> 172.21.15.28:0/2268214407 conn(0x7f7aa80fc040 msgr2=0x7f7aa80fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.441+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 shutdown_connections 2024-03-22T02:15:17.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:17.442+0000 7f7aacfb6640 1 -- 172.21.15.28:0/2268214407 wait complete. 2024-03-22T02:15:18.024 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-22T02:15:18.090 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-22T02:15:18.128 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-22T02:15:18.130 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T02:15:18.130 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-22T02:15:18.169 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-22T02:15:18.172 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:15:18.172 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-22T02:15:18.227 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-03-22T02:15:18.228 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-03-22T02:15:18.229 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-03-22T02:15:18.288 INFO:tasks.workunit.client.0.smithi028.stderr:+ dbench 1 2024-03-22T02:15:18.300 INFO:tasks.workunit.client.0.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-22T02:15:18.300 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:15:18.300 INFO:tasks.workunit.client.0.smithi028.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-22T02:15:18.304 INFO:tasks.workunit.client.0.smithi028.stdout:0 of 1 processes prepared for launch 0 sec 2024-03-22T02:15:18.404 INFO:tasks.workunit.client.0.smithi028.stdout:1 of 1 processes prepared for launch 0 sec 2024-03-22T02:15:18.404 INFO:tasks.workunit.client.0.smithi028.stdout:releasing clients 2024-03-22T02:15:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:18 smithi120 ceph-mon[36803]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:18 smithi083 ceph-mon[36340]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:18 smithi028 ceph-mon[31916]: pgmap v409: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-22T02:15:19.404 INFO:tasks.workunit.client.0.smithi028.stdout: 1 3015 33.54 MB/sec warmup 1 sec latency 7.546 ms 2024-03-22T02:15:20.404 INFO:tasks.workunit.client.0.smithi028.stdout: 1 5538 22.78 MB/sec warmup 2 sec latency 16.182 ms 2024-03-22T02:15:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:20 smithi083 ceph-mon[36340]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-22T02:15:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:20 smithi028 ceph-mon[31916]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-22T02:15:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:20 smithi120 ceph-mon[36803]: pgmap v410: 129 pgs: 129 active+clean; 672 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-22T02:15:21.404 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6907 18.17 MB/sec warmup 3 sec latency 608.228 ms 2024-03-22T02:15:21.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:21 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:21.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:21 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:21 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:22.404 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6907 13.63 MB/sec warmup 4 sec latency 1608.280 ms 2024-03-22T02:15:22.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:22 smithi083 ceph-mon[36340]: pgmap v411: 129 pgs: 129 active+clean; 1.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 38 KiB/s wr, 6 op/s 2024-03-22T02:15:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:22 smithi028 ceph-mon[31916]: pgmap v411: 129 pgs: 129 active+clean; 1.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 38 KiB/s wr, 6 op/s 2024-03-22T02:15:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:22 smithi120 ceph-mon[36803]: pgmap v411: 129 pgs: 129 active+clean; 1.0 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 38 KiB/s wr, 6 op/s 2024-03-22T02:15:23.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6907 10.90 MB/sec warmup 5 sec latency 2608.373 ms 2024-03-22T02:15:24.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6907 9.08 MB/sec warmup 6 sec latency 3608.470 ms 2024-03-22T02:15:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:24 smithi083 ceph-mon[36340]: pgmap v412: 129 pgs: 129 active+clean; 2.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 226 KiB/s wr, 40 op/s 2024-03-22T02:15:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:24 smithi028 ceph-mon[31916]: pgmap v412: 129 pgs: 129 active+clean; 2.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 226 KiB/s wr, 40 op/s 2024-03-22T02:15:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:24 smithi120 ceph-mon[36803]: pgmap v412: 129 pgs: 129 active+clean; 2.9 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 226 KiB/s wr, 40 op/s 2024-03-22T02:15:25.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6929 7.80 MB/sec warmup 7 sec latency 4576.406 ms 2024-03-22T02:15:26.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6933 6.82 MB/sec warmup 8 sec latency 995.032 ms 2024-03-22T02:15:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:26 smithi028 ceph-mon[31916]: pgmap v413: 129 pgs: 129 active+clean; 3.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 331 KiB/s wr, 45 op/s 2024-03-22T02:15:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:26 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:26 smithi120 ceph-mon[36803]: pgmap v413: 129 pgs: 129 active+clean; 3.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 331 KiB/s wr, 45 op/s 2024-03-22T02:15:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:26 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:27.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:26 smithi083 ceph-mon[36340]: pgmap v413: 129 pgs: 129 active+clean; 3.8 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 331 KiB/s wr, 45 op/s 2024-03-22T02:15:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:26 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:15:27.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6933 6.06 MB/sec warmup 9 sec latency 1995.130 ms 2024-03-22T02:15:28.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 6933 5.46 MB/sec warmup 10 sec latency 2995.199 ms 2024-03-22T02:15:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:28 smithi028 ceph-mon[31916]: pgmap v414: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 343 KiB/s wr, 48 op/s 2024-03-22T02:15:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:28 smithi120 ceph-mon[36803]: pgmap v414: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 343 KiB/s wr, 48 op/s 2024-03-22T02:15:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:28 smithi083 ceph-mon[36340]: pgmap v414: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 343 KiB/s wr, 48 op/s 2024-03-22T02:15:29.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 7345 5.35 MB/sec warmup 11 sec latency 3778.273 ms 2024-03-22T02:15:30.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 9492 5.52 MB/sec warmup 12 sec latency 15.774 ms 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 -- 172.21.15.28:0/896349070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000103840 msgr2=0x7f100010a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 --2- 172.21.15.28:0/896349070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000103840 0x7f100010a0d0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009cd0 tx=0x7f0ff802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 -- 172.21.15.28:0/896349070 shutdown_connections 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 --2- 172.21.15.28:0/896349070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000103840 0x7f100010a0d0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 --2- 172.21.15.28:0/896349070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1000102e80 0x7f1000103300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 --2- 172.21.15.28:0/896349070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10000ffeb0 0x7f10001002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 -- 172.21.15.28:0/896349070 >> 172.21.15.28:0/896349070 conn(0x7f10000fb7e0 msgr2=0x7f10000fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 -- 172.21.15.28:0/896349070 shutdown_connections 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 -- 172.21.15.28:0/896349070 wait complete. 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.530+0000 7f100500e640 1 Processor -- start 2024-03-22T02:15:30.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 -- start start 2024-03-22T02:15:30.533 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 0x7f1000100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 0x7f1000101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f100010d880 con 0x7f1000102e80 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f100010d700 con 0x7f10000ffeb0 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f100500e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f100010da00 con 0x7f1000103840 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 0x7f1000100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 0x7f1000101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 0x7f1000100e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33686/0 (socket says 172.21.15.28:33686) 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffe59c640 1 -- 172.21.15.28:0/554252391 learned_addr learned my addr 172.21.15.28:0/554252391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:30.534 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 -- 172.21.15.28:0/554252391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 msgr2=0x7f10001027f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 -- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 msgr2=0x7f1000100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 0x7f1000100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0fff59e640 1 -- 172.21.15.28:0/554252391 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1000101c00 con 0x7f1000103840 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.531+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f0fff59e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 0x7f1000101370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8037040 tx=0x7f0ff80346d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f0fe7fff640 1 -- 172.21.15.28:0/554252391 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8041020 con 0x7f1000103840 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f0fe7fff640 1 -- 172.21.15.28:0/554252391 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff8032890 con 0x7f1000103840 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10001a1f30 con 0x7f1000103840 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f0fe7fff640 1 -- 172.21.15.28:0/554252391 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8039040 con 0x7f1000103840 2024-03-22T02:15:30.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.532+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10001a2390 con 0x7f1000103840 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f0fe7fff640 1 -- 172.21.15.28:0/554252391 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0ff8044020 con 0x7f1000103840 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f1007bf8740 1 Processor -- start 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f0fe7fff640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 0x7f0fe80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f1007bf8740 1 -- start start 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f0fe7fff640 1 -- 172.21.15.28:0/554252391 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0ff80f1410 con 0x7f1000103840 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.534+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 0x7f0fe80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c80719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c80744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7fd1040 con 0x55d6c806c4e0 2024-03-22T02:15:30.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7f799e0 con 0x55d6c8072040 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f1007bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7fccff0 con 0x55d6c806ced0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33694/0 (socket says 172.21.15.28:33694) 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 -- 172.21.15.28:0/2978507853 learned_addr learned my addr 172.21.15.28:0/2978507853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0fff59e640 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c80744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c80719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 0x7f0fe80a5480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1000102560 tx=0x7f0ff4011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 -- 172.21.15.28:0/2978507853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 msgr2=0x55d6c80719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c80719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 -- 172.21.15.28:0/2978507853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 msgr2=0x55d6c80744d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c80744d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 -- 172.21.15.28:0/2978507853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c7ce5210 con 0x55d6c806c4e0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00114f0 tx=0x7f0ff00118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64158bed74d6aa0f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0fe57fa640 1 -- 172.21.15.28:0/2978507853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff001e070 con 0x55d6c806c4e0 2024-03-22T02:15:30.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0fe57fa640 1 -- 172.21.15.28:0/2978507853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff00035f0 con 0x55d6c806c4e0 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.535+0000 7f0fe57fa640 1 -- 172.21.15.28:0/2978507853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff000f740 con 0x55d6c806c4e0 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- 172.21.15.28:0/2978507853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 msgr2=0x55d6c806c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00114f0 tx=0x7f0ff00118f0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- 172.21.15.28:0/2978507853 shutdown_connections 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c80744d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c80719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 --2- 172.21.15.28:0/2978507853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c806c900 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- 172.21.15.28:0/2978507853 >> 172.21.15.28:0/2978507853 conn(0x55d6c8067a50 msgr2=0x55d6c8067e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- 172.21.15.28:0/2978507853 shutdown_connections 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- 172.21.15.28:0/2978507853 wait complete. 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.536+0000 7f1007bf8740 1 -- start start 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 Processor -- start 2024-03-22T02:15:30.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c8076a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c807da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8074ff0 con 0x55d6c806c4e0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d6c7f8cdb0 con 0x55d6c8072040 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6c8075170 con 0x55d6c806ced0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f1007bf8740 10 client.0 _ll_register_callbacks cb 0x55d6c7ce4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33708/0 (socket says 172.21.15.28:33708) 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 -- 172.21.15.28:0/795531975 learned_addr learned my addr 172.21.15.28:0/795531975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c8076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0fff59e640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c807da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 -- 172.21.15.28:0/795531975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 msgr2=0x55d6c8076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c8076a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 -- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 msgr2=0x55d6c807da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c807da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 -- 172.21.15.28:0/795531975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c7db1620 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00058c0 tx=0x7f0ff0003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20edd8813a8b3d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0fe2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.537+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0027020 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0ff0005250 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ff00035f0 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ff0032050 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 10 client.15978 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d6c8080e00 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff00035f0 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ff000f4e0 con 0x55d6c806c4e0 2024-03-22T02:15:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 10 client.15978 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 20 client.15978 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c8082380 0x55d6c80847b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 4 client.15978 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6c7fde580 con 0x55d6c8082380 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.538+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c8082380 0x55d6c80847b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.539+0000 7f0fe2ff5640 10 client.15978 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.539+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c8082380 0x55d6c80847b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70ba01b0c4a3cc0a server_cookie=823c5456849af5a3 in_seq=0 out_seq=0 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/795531975 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6c7fde580 con 0x55d6c8082380 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f0fe2ff5640 10 client.15978 handle_command_reply: tid=0 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 client.15978 shutdown 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 msgr2=0x55d6c8076510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00058c0 tx=0x7f0ff0003a50 comp rx=0 tx=0).stop 2024-03-22T02:15:30.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 shutdown_connections 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c8082380 0x55d6c80847b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c8072040 0x55d6c807da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c8076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f0fe2ff5640 0 client.15978 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 --2- 172.21.15.28:0/795531975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806c4e0 0x55d6c8076510 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 >> 172.21.15.28:0/795531975 conn(0x55d6c8067a50 msgr2=0x55d6c8066c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 shutdown_connections 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 1 -- 172.21.15.28:0/795531975 wait complete. 2024-03-22T02:15:30.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.540+0000 7f1007bf8740 20 client.15978 trim_cache size 0 max 16384 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.555+0000 7f1007bf8740 1 Processor -- start 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.555+0000 7f1007bf8740 1 -- start start 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c806b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8072040 0x55d6c8075b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10001af480 con 0x55d6c806ced0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10001acd50 con 0x55d6c806c4e0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f1007bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10001aaba0 con 0x55d6c8072040 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33712/0 (socket says 172.21.15.28:33712) 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 -- 172.21.15.28:0/3281918511 learned_addr learned my addr 172.21.15.28:0/3281918511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0fff59e640 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8072040 0x55d6c8075b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c806b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 -- 172.21.15.28:0/3281918511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8072040 msgr2=0x55d6c8075b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8072040 0x55d6c8075b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 -- 172.21.15.28:0/3281918511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 msgr2=0x55d6c806b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c806b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 -- 172.21.15.28:0/3281918511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6c8084e30 con 0x55d6c806ced0 2024-03-22T02:15:30.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0ffe59c640 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55d6c806d940 tx=0x7f0ff401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b52e51b5135b76 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0fe4ff9640 1 -- 172.21.15.28:0/3281918511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4017040 con 0x55d6c806ced0 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0fe4ff9640 1 -- 172.21.15.28:0/3281918511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff4019050 con 0x55d6c806ced0 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.556+0000 7f0fe4ff9640 1 -- 172.21.15.28:0/3281918511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4005480 con 0x55d6c806ced0 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- 172.21.15.28:0/3281918511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 msgr2=0x55d6c807d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x55d6c806d940 tx=0x7f0ff401c000 comp rx=0 tx=0).stop 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- 172.21.15.28:0/3281918511 shutdown_connections 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c8072040 0x55d6c8075b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c806ced0 0x55d6c807d9b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 --2- 172.21.15.28:0/3281918511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c806b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- 172.21.15.28:0/3281918511 >> 172.21.15.28:0/3281918511 conn(0x55d6c8067a50 msgr2=0x55d6c806e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- 172.21.15.28:0/3281918511 shutdown_connections 2024-03-22T02:15:30.562 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- 172.21.15.28:0/3281918511 wait complete. 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 -- start start 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 Processor -- start 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.557+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c807ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c807b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10001ab210 con 0x55d6c8072040 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f10001ac680 con 0x55d6c806c4e0 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f10001ab8e0 con 0x55d6c806ced0 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 10 client.0 _ll_register_callbacks cb 0x7f0ff001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33726/0 (socket says 172.21.15.28:33726) 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0ffed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c807ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0ffe59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c807b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 -- 172.21.15.28:0/1310695520 learned_addr learned my addr 172.21.15.28:0/1310695520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 -- 172.21.15.28:0/1310695520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 msgr2=0x55d6c807b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c807b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 -- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 msgr2=0x55d6c807ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c807ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 -- 172.21.15.28:0/1310695520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6c807c8c0 con 0x55d6c8072040 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fff59e640 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8006010 tx=0x7f0ff80bd330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c120c4d5995058 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fe2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80d6060 con 0x55d6c8072040 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ff4019050 con 0x55d6c8072040 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0ff8003a00 con 0x55d6c8072040 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ff80bd730 con 0x55d6c8072040 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 10 client.15984 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:30.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.558+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d6c807dc40 con 0x55d6c8072040 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ff4019050 con 0x55d6c8072040 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ff803c030 con 0x55d6c8072040 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 10 client.15984 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 20 client.15984 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c80e8cf0 0x55d6c80eb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 4 client.15984 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d6c7fd88d0 con 0x55d6c80e8cf0 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c80e8cf0 0x55d6c80eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f0fe2ff5640 10 client.15984 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:30.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.559+0000 7f0ffed9d640 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c80e8cf0 0x55d6c80eb110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f929a30054980b0 server_cookie=61939393392e8dd3 in_seq=0 out_seq=0 2024-03-22T02:15:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f0fe2ff5640 1 -- 172.21.15.28:0/1310695520 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d6c7fd88d0 con 0x55d6c80e8cf0 2024-03-22T02:15:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f0fe2ff5640 10 client.15984 handle_command_reply: tid=0 2024-03-22T02:15:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 client.15984 shutdown 2024-03-22T02:15:30.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 msgr2=0x55d6c807c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8006010 tx=0x7f0ff80bd330 comp rx=0 tx=0).stop 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 shutdown_connections 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d6c80e8cf0 0x55d6c80eb110 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6c8072040 0x55d6c807c2d0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6c806ced0 0x55d6c807b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f0fe2ff5640 0 client.15984 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 --2- 172.21.15.28:0/1310695520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6c806c4e0 0x55d6c807ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 >> 172.21.15.28:0/1310695520 conn(0x55d6c8067a50 msgr2=0x55d6c80663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 shutdown_connections 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 1 -- 172.21.15.28:0/1310695520 wait complete. 2024-03-22T02:15:30.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.635+0000 7f1007bf8740 20 client.15984 trim_cache size 0 max 16384 2024-03-22T02:15:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:15:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:15:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:15:30.638 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 msgr2=0x7f0fe80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 0x7f0fe80a5480 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1000102560 tx=0x7f0ff4011810 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 msgr2=0x7f1000101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 0x7f1000101370 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8037040 tx=0x7f0ff80346d0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 shutdown_connections 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0fe80a2fc0 0x7f0fe80a5480 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1000103840 0x7f1000101370 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1000102e80 0x7f1000100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 --2- 172.21.15.28:0/554252391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f10000ffeb0 0x7f10001027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 >> 172.21.15.28:0/554252391 conn(0x7f10000fb7e0 msgr2=0x7f10001060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:30.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 shutdown_connections 2024-03-22T02:15:30.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:30.636+0000 7f100500e640 1 -- 172.21.15.28:0/554252391 wait complete. 2024-03-22T02:15:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:30 smithi028 ceph-mon[31916]: pgmap v415: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 344 KiB/s wr, 49 op/s 2024-03-22T02:15:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:30 smithi120 ceph-mon[36803]: pgmap v415: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 344 KiB/s wr, 49 op/s 2024-03-22T02:15:31.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:30 smithi083 ceph-mon[36340]: pgmap v415: 129 pgs: 129 active+clean; 3.9 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 344 KiB/s wr, 49 op/s 2024-03-22T02:15:31.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 10451 5.66 MB/sec warmup 13 sec latency 613.276 ms 2024-03-22T02:15:32.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 10451 5.26 MB/sec warmup 14 sec latency 1613.425 ms 2024-03-22T02:15:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:32 smithi028 ceph-mon[31916]: pgmap v416: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 514 KiB/s wr, 51 op/s 2024-03-22T02:15:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:32 smithi120 ceph-mon[36803]: pgmap v416: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 514 KiB/s wr, 51 op/s 2024-03-22T02:15:33.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:32 smithi083 ceph-mon[36340]: pgmap v416: 129 pgs: 129 active+clean; 5.7 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 514 KiB/s wr, 51 op/s 2024-03-22T02:15:33.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 10451 4.91 MB/sec warmup 15 sec latency 2613.542 ms 2024-03-22T02:15:33.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:15:33.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 -- 172.21.15.28:0/1624804165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28102e50 msgr2=0x7f4d281032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/1624804165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28102e50 0x7f4d281032d0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c009930 tx=0x7f4d1c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 -- 172.21.15.28:0/1624804165 shutdown_connections 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/1624804165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d28103810 0x7f4d2810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/1624804165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28102e50 0x7f4d281032d0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/1624804165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d280ffe80 0x7f4d281002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 -- 172.21.15.28:0/1624804165 >> 172.21.15.28:0/1624804165 conn(0x7f4d280fb790 msgr2=0x7f4d280fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 -- 172.21.15.28:0/1624804165 shutdown_connections 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.607+0000 7f4d2d84e640 1 -- 172.21.15.28:0/1624804165 wait complete. 2024-03-22T02:15:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 Processor -- start 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 -- start start 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 0x7f4d28197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d28102e50 0x7f4d28198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d840 con 0x7f4d28103810 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d6c0 con 0x7f4d280ffe80 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d2d84e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810d9c0 con 0x7f4d28102e50 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46410/0 (socket says 172.21.15.28:46410) 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 0x7f4d28197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 -- 172.21.15.28:0/3081571507 learned_addr learned my addr 172.21.15.28:0/3081571507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d28102e50 0x7f4d28198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 -- 172.21.15.28:0/3081571507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d28102e50 msgr2=0x7f4d28198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d28102e50 0x7f4d28198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.608+0000 7f4d277fe640 1 -- 172.21.15.28:0/3081571507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 msgr2=0x7f4d28197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 0x7f4d28197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d277fe640 1 -- 172.21.15.28:0/3081571507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2819fb00 con 0x7f4d28103810 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 0x7f4d28197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000e970 tx=0x7f4d2000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d2c84c640 1 -- 172.21.15.28:0/3081571507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2000ccc0 con 0x7f4d28103810 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d2c84c640 1 -- 172.21.15.28:0/3081571507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2000ce60 con 0x7f4d28103810 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d2819fdf0 con 0x7f4d28103810 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d2c84c640 1 -- 172.21.15.28:0/3081571507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20010490 con 0x7f4d28103810 2024-03-22T02:15:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.609+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d281a0260 con 0x7f4d28103810 2024-03-22T02:15:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.611+0000 7f4d2c84c640 1 -- 172.21.15.28:0/3081571507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4d20005020 con 0x7f4d28103810 2024-03-22T02:15:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 Processor -- start 2024-03-22T02:15:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 -- start start 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d2c84c640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 0x7f4d100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7216920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 0x55f4d7219420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d71768c0 con 0x55f4d7211430 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f4d711ec20 con 0x55f4d7216f90 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d30433740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f4d7172870 con 0x55f4d7211e20 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d2c84c640 1 -- 172.21.15.28:0/3081571507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4d200cfc10 con 0x7f4d28103810 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 0x7f4d100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46418/0 (socket says 172.21.15.28:46418) 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 -- 172.21.15.28:0/3258281825 learned_addr learned my addr 172.21.15.28:0/3258281825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 0x55f4d7219420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7216920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 -- 172.21.15.28:0/3258281825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 msgr2=0x55f4d7216920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7216920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 -- 172.21.15.28:0/3258281825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 msgr2=0x55f4d7219420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 0x55f4d7219420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 -- 172.21.15.28:0/3258281825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d7143150 con 0x55f4d7211430 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 0x55f4d7219420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 0x7f4d100a53e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c02efd0 tx=0x7f4d1c02cb60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.612+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4d180107c0 tx=0x7f4d18010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33108bbc3a72d48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d0dffb640 1 -- 172.21.15.28:0/3258281825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18016070 con 0x55f4d7211430 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d0dffb640 1 -- 172.21.15.28:0/3258281825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d1800edb0 con 0x55f4d7211430 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d0dffb640 1 -- 172.21.15.28:0/3258281825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18005780 con 0x55f4d7211430 2024-03-22T02:15:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 -- 172.21.15.28:0/3258281825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 msgr2=0x55f4d7211850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4d180107c0 tx=0x7f4d18010c80 comp rx=0 tx=0).stop 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 -- 172.21.15.28:0/3258281825 shutdown_connections 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7216f90 0x55f4d7219420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7216920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 --2- 172.21.15.28:0/3258281825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7211430 0x55f4d7211850 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 -- 172.21.15.28:0/3258281825 >> 172.21.15.28:0/3258281825 conn(0x55f4d720c9a0 msgr2=0x55f4d720cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 -- 172.21.15.28:0/3258281825 shutdown_connections 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.613+0000 7f4d30433740 1 -- 172.21.15.28:0/3258281825 wait complete. 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 -- start start 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 Processor -- start 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d721b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4d7219f40 con 0x55f4d7216f90 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f4d7132130 con 0x55f4d7211e20 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f4d721a0c0 con 0x55f4d7211430 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d30433740 10 client.0 _ll_register_callbacks cb 0x55f4d7131b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d721b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54962/0 (socket says 172.21.15.28:54962) 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d721b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59370/0 (socket says 172.21.15.28:59370) 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.614+0000 7f4d267fc640 1 -- 172.21.15.28:0/3966973609 learned_addr learned my addr 172.21.15.28:0/3966973609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 -- 172.21.15.28:0/3966973609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 msgr2=0x55f4d721b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d721b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 -- 172.21.15.28:0/3966973609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 msgr2=0x55f4d7222e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 -- 172.21.15.28:0/3966973609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d6f547d0 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0096c0 tx=0x7f4d1c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a941ee0c29e1ae46 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c0048d0 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d1800edb0 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4d1c004a70 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d1c05a050 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d30433740 10 client.25835 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d6e8a3f0 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.615+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d1800edb0 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d1c034440 con 0x55f4d7211e20 2024-03-22T02:15:33.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 10 client.25835 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 10 client.25835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 20 client.25835 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d7227810 0x55f4d7229c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 4 client.25835 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4d7183e00 con 0x55f4d7227810 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d7227810 0x55f4d7229c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d7227810 0x55f4d7229c70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49fa5fbe815f7b0c server_cookie=76d9bf2747c43a13 in_seq=0 out_seq=0 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.616+0000 7f4d06ffd640 10 client.25835 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3966973609 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4d7183e00 con 0x55f4d7227810 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d06ffd640 10 client.25835 handle_command_reply: tid=0 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 client.25835 shutdown 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 msgr2=0x55f4d721be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c0096c0 tx=0x7f4d1c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:33.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 shutdown_connections 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d7227810 0x55f4d7229c70 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d721be90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 --2- 172.21.15.28:0/3966973609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d721b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d06ffd640 0 client.25835 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 >> 172.21.15.28:0/3966973609 conn(0x55f4d720c9a0 msgr2=0x55f4d720bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 shutdown_connections 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 1 -- 172.21.15.28:0/3966973609 wait complete. 2024-03-22T02:15:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.617+0000 7f4d30433740 20 client.25835 trim_cache size 0 max 16384 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 Processor -- start 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 -- start start 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d7213380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d72138c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281ad2a0 con 0x55f4d7216f90 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281aab70 con 0x55f4d7211e20 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d30433740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281a89c0 con 0x55f4d7211430 2024-03-22T02:15:33.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d72138c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46434/0 (socket says 172.21.15.28:46434) 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d7213380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 -- 172.21.15.28:0/4177322723 learned_addr learned my addr 172.21.15.28:0/4177322723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 -- 172.21.15.28:0/4177322723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 msgr2=0x55f4d7213380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d7213380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 -- 172.21.15.28:0/4177322723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 msgr2=0x55f4d72138c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d72138c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 -- 172.21.15.28:0/4177322723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4d6e4a0f0 con 0x55f4d7216f90 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d267fc640 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d72138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d277fe640 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4d200982e0 tx=0x7f4d20098170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d5a4dbe11f8ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d0d7fa640 1 -- 172.21.15.28:0/4177322723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200a48e0 con 0x55f4d7216f90 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.636+0000 7f4d0d7fa640 1 -- 172.21.15.28:0/4177322723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d200bc0b0 con 0x55f4d7216f90 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d0d7fa640 1 -- 172.21.15.28:0/4177322723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d20005310 con 0x55f4d7216f90 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 -- 172.21.15.28:0/4177322723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 msgr2=0x55f4d7222e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4d200982e0 tx=0x7f4d20098170 comp rx=0 tx=0).stop 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 -- 172.21.15.28:0/4177322723 shutdown_connections 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7222e20 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211e20 0x55f4d72138c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 --2- 172.21.15.28:0/4177322723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211430 0x55f4d7213380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 -- 172.21.15.28:0/4177322723 >> 172.21.15.28:0/4177322723 conn(0x55f4d720c9a0 msgr2=0x55f4d721a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 -- 172.21.15.28:0/4177322723 shutdown_connections 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.637+0000 7f4d30433740 1 -- 172.21.15.28:0/4177322723 wait complete. 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 -- start start 2024-03-22T02:15:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 Processor -- start 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 0x55f4d7220740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281a9030 con 0x55f4d7216f90 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281aa4a0 con 0x55f4d7211430 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d281a9700 con 0x55f4d7211e20 2024-03-22T02:15:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d30433740 10 client.0 _ll_register_callbacks cb 0x7f4d1c004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d26ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 0x55f4d7220740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d267fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59392/0 (socket says 172.21.15.28:59392) 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d267fc640 1 -- 172.21.15.28:0/3727626354 learned_addr learned my addr 172.21.15.28:0/3727626354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.638+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46440/0 (socket says 172.21.15.28:46440) 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 -- 172.21.15.28:0/3727626354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 msgr2=0x55f4d7220c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 -- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 msgr2=0x55f4d7220740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 0x55f4d7220740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 -- 172.21.15.28:0/3727626354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4d7222280 con 0x55f4d7216f90 2024-03-22T02:15:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d267fc640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 0x55f4d7220740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d277fe640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4d200982e0 tx=0x7f4d20005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53220237eef8a80f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d200cd3b0 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4d200a4450 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d200bc0b0 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4d200a4650 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d30433740 10 client.16002 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4d728ddf0 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d200bc0b0 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4d20051070 con 0x55f4d7216f90 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d30433740 10 client.16002 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.639+0000 7f4d30433740 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 20 client.16002 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d728e770 0x55f4d7290b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 4 client.16002 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f4d717e150 con 0x55f4d728e770 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d728e770 0x55f4d7290b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d26ffd640 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d728e770 0x55f4d7290b90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed0e2d3ac766b6 server_cookie=4095f09fe57a7573 in_seq=0 out_seq=0 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d06ffd640 10 client.16002 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d06ffd640 1 -- 172.21.15.28:0/3727626354 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f4d717e150 con 0x55f4d728e770 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d06ffd640 10 client.16002 handle_command_reply: tid=0 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 client.16002 shutdown 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 msgr2=0x55f4d7221c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4d200982e0 tx=0x7f4d20005c70 comp rx=0 tx=0).stop 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 shutdown_connections 2024-03-22T02:15:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4d728e770 0x55f4d7290b90 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4d7216f90 0x55f4d7221c90 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4d7211e20 0x55f4d7220c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d06ffd640 0 client.16002 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 --2- 172.21.15.28:0/3727626354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4d7211430 0x55f4d7220740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 >> 172.21.15.28:0/3727626354 conn(0x55f4d720c9a0 msgr2=0x55f4d721a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 shutdown_connections 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.640+0000 7f4d30433740 1 -- 172.21.15.28:0/3727626354 wait complete. 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d30433740 20 client.16002 trim_cache size 0 max 16384 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:33.649 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:15:33.650 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:15:33.650 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 msgr2=0x7f4d100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 0x7f4d100a53e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c02efd0 tx=0x7f4d1c02cb60 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 msgr2=0x7f4d2819f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4d2000e970 tx=0x7f4d2000ed70 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 shutdown_connections 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d100a2f20 0x7f4d100a53e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d28103810 0x7f4d2819f490 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d28102e50 0x7f4d28198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 --2- 172.21.15.28:0/3081571507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d280ffe80 0x7f4d28197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 >> 172.21.15.28:0/3081571507 conn(0x7f4d280fb790 msgr2=0x7f4d28106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 shutdown_connections 2024-03-22T02:15:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.641+0000 7f4d2d84e640 1 -- 172.21.15.28:0/3081571507 wait complete. 2024-03-22T02:15:33.654 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:15:33.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:15:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 -- 172.21.15.28:0/3265474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 msgr2=0x7f2cf4101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3265474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4101220 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009900 tx=0x7f2cdc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 -- 172.21.15.28:0/3265474250 shutdown_connections 2024-03-22T02:15:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3265474250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4101760 0x7f2cf4109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3265474250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4101220 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3265474250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4102070 0x7f2cf4102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 -- 172.21.15.28:0/3265474250 >> 172.21.15.28:0/3265474250 conn(0x7f2cf40fb7d0 msgr2=0x7f2cf40fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 -- 172.21.15.28:0/3265474250 shutdown_connections 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.755+0000 7f2cf9466640 1 -- 172.21.15.28:0/3265474250 wait complete. 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 Processor -- start 2024-03-22T02:15:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 -- start start 2024-03-22T02:15:33.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4101760 0x7f2cf419cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4102070 0x7f2cf419f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410c770 con 0x7f2cf4100da0 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410c5f0 con 0x7f2cf4102070 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf9466640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf410c8f0 con 0x7f2cf4101760 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46462/0 (socket says 172.21.15.28:46462) 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf2ffd640 1 -- 172.21.15.28:0/3272804845 learned_addr learned my addr 172.21.15.28:0/3272804845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf27fc640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4101760 0x7f2cf419cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.756+0000 7f2cf37fe640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4102070 0x7f2cf419f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 -- 172.21.15.28:0/3272804845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4101760 msgr2=0x7f2cf419cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4101760 0x7f2cf419cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 -- 172.21.15.28:0/3272804845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4102070 msgr2=0x7f2cf419f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4102070 0x7f2cf419f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 -- 172.21.15.28:0/3272804845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf419fb90 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf2ffd640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9c0 tx=0x7f2cec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec00cd80 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf419fe20 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cec0035f0 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec010690 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.757+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf41a0230 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.758+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf4107f60 con 0x7f2cf4100da0 2024-03-22T02:15:33.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.759+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2cec0037a0 con 0x7f2cf4100da0 2024-03-22T02:15:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.760+0000 7f2cebfff640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 0x7f2ce00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.760+0000 7f2cf27fc640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 0x7f2ce00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.760+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2cec0cfb20 con 0x7f2cf4100da0 2024-03-22T02:15:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.760+0000 7f2cf27fc640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 0x7f2ce00a53e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc02c7b0 tx=0x7f2cdc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:33.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.764+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cec09c4e0 con 0x7f2cf4100da0 2024-03-22T02:15:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.988+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cf402d3b0 con 0x7f2cf4100da0 2024-03-22T02:15:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.989+0000 7f2cebfff640 1 -- 172.21.15.28:0/3272804845 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11869 (secure 0 0 0) 0x7f2cec09bde0 con 0x7f2cf4100da0 2024-03-22T02:15:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:15:33.991 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:15:24.687791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:15:33.991 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 msgr2=0x7f2ce00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 0x7f2ce00a53e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc02c7b0 tx=0x7f2cdc000bc0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 msgr2=0x7f2cf4100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00e9c0 tx=0x7f2cec00ee80 comp rx=0 tx=0).stop 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 shutdown_connections 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2ce00a2f20 0x7f2ce00a53e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2cf4102070 0x7f2cf419f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2cf4101760 0x7f2cf419cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 --2- 172.21.15.28:0/3272804845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cf4100da0 0x7f2cf4100800 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:33.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.991+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 >> 172.21.15.28:0/3272804845 conn(0x7f2cf40fb7d0 msgr2=0x7f2cf4105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:33.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.992+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 shutdown_connections 2024-03-22T02:15:33.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:33.992+0000 7f2cf9466640 1 -- 172.21.15.28:0/3272804845 wait complete. 2024-03-22T02:15:34.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 -- 172.21.15.28:0/437088624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 msgr2=0x7f5d44100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 --2- 172.21.15.28:0/437088624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44100ec0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009900 tx=0x7f5d3402bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 -- 172.21.15.28:0/437088624 shutdown_connections 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 --2- 172.21.15.28:0/437088624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 0x7f5d44108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 --2- 172.21.15.28:0/437088624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 0x7f5d44102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 --2- 172.21.15.28:0/437088624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44100ec0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 -- 172.21.15.28:0/437088624 >> 172.21.15.28:0/437088624 conn(0x7f5d440fc030 msgr2=0x7f5d440fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 -- 172.21.15.28:0/437088624 shutdown_connections 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.107+0000 7f5d493e9640 1 -- 172.21.15.28:0/437088624 wait complete. 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 Processor -- start 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 -- start start 2024-03-22T02:15:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 0x7f5d44198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 0x7f5d4419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410c6f0 con 0x7f5d44100aa0 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410c570 con 0x7f5d44101c90 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d493e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410c870 con 0x7f5d44102650 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46468/0 (socket says 172.21.15.28:46468) 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3789999052 learned_addr learned my addr 172.21.15.28:0/3789999052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 0x7f5d44198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d437fe640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 0x7f5d4419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3789999052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 msgr2=0x7f5d4419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 0x7f5d4419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3789999052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 msgr2=0x7f5d44198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 0x7f5d44198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.108+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3789999052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4419fb60 con 0x7f5d44100aa0 2024-03-22T02:15:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5d34000c00 tx=0x7f5d3402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d40ff9640 1 -- 172.21.15.28:0/3789999052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34041070 con 0x7f5d44100aa0 2024-03-22T02:15:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d40ff9640 1 -- 172.21.15.28:0/3789999052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d34004450 con 0x7f5d44100aa0 2024-03-22T02:15:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4419fdf0 con 0x7f5d44100aa0 2024-03-22T02:15:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d40ff9640 1 -- 172.21.15.28:0/3789999052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34003df0 con 0x7f5d44100aa0 2024-03-22T02:15:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.109+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d441a0220 con 0x7f5d44100aa0 2024-03-22T02:15:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 Processor -- start 2024-03-22T02:15:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d40ff9640 1 -- 172.21.15.28:0/3789999052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5d34044070 con 0x7f5d44100aa0 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 -- start start 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 0x562173b58c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b5dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562173abd8d0 con 0x562173b59210 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562173a65e00 con 0x562173b5e380 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562173ab9880 con 0x562173b58820 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b5dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d437fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 0x562173b58c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d437fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 0x562173b58c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59428/0 (socket says 172.21.15.28:59428) 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d437fe640 1 -- 172.21.15.28:0/292478795 learned_addr learned my addr 172.21.15.28:0/292478795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d40ff9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 0x7f5d300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d3bfff640 1 -- 172.21.15.28:0/292478795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 msgr2=0x562173b58c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 0x562173b58c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.112+0000 7f5d3bfff640 1 -- 172.21.15.28:0/292478795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 msgr2=0x562173b5dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b5dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d3bfff640 1 -- 172.21.15.28:0/292478795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621737d13f0 con 0x562173b5e380 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d437fe640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 0x7f5d300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d40ff9640 1 -- 172.21.15.28:0/3789999052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5d340f1290 con 0x7f5d44100aa0 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b60810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009bf0 tx=0x7f5d2c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85316e394bb35ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d397fa640 1 -- 172.21.15.28:0/292478795 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c01e070 con 0x562173b5e380 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d397fa640 1 -- 172.21.15.28:0/292478795 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d2c009da0 con 0x562173b5e380 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.113+0000 7f5d397fa640 1 -- 172.21.15.28:0/292478795 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c018870 con 0x562173b5e380 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/292478795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 msgr2=0x562173b60810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d437fe640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 0x7f5d300a53e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x562173b59a70 tx=0x7f5d3c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b60810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009bf0 tx=0x7f5d2c011810 comp rx=0 tx=0).stop 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/292478795 shutdown_connections 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b60810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b5dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/292478795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b58820 0x562173b58c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/292478795 >> 172.21.15.28:0/292478795 conn(0x562173b53d90 msgr2=0x562173b54190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/292478795 shutdown_connections 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/292478795 wait complete. 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- start start 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 Processor -- start 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b632a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b6a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562173b61330 con 0x562173b58820 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562173a79300 con 0x562173b5e380 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562173b614b0 con 0x562173b59210 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d4bfd3740 10 client.0 _ll_register_callbacks cb 0x562173ab4e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.114+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d437fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b6a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46488/0 (socket says 172.21.15.28:46488) 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 -- 172.21.15.28:0/1711781212 learned_addr learned my addr 172.21.15.28:0/1711781212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b632a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 -- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 msgr2=0x562173b632a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b632a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 -- 172.21.15.28:0/1711781212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 msgr2=0x562173b6a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b6a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 -- 172.21.15.28:0/1711781212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56217389b7d0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d437fe640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b6a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5d340ee980 tx=0x7f5d340f4960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f5f3e882ec4402 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3404b030 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5d340b9cb0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d2c009da0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5d340b98e0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d4bfd3740 10 client.16020 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562173b6d650 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.115+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d2c009da0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5d340a6bd0 con 0x562173b58820 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 10 client.16020 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 10 client.16020 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 20 client.16020 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173b6ece0 0x562173b71170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 4 client.16020 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562173acae10 con 0x562173b6ece0 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173b6ece0 0x562173b71170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173b6ece0 0x562173b71170 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f71296536e4ffe35 server_cookie=31ca1f87d2906baf in_seq=0 out_seq=0 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.116+0000 7f5d2a7fc640 10 client.16020 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1711781212 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562173acae10 con 0x562173b6ece0 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d2a7fc640 10 client.16020 handle_command_reply: tid=0 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 client.16020 shutdown 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 msgr2=0x562173b62d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f5d340ee980 tx=0x7f5d340f4960 comp rx=0 tx=0).stop 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 shutdown_connections 2024-03-22T02:15:34.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173b6ece0 0x562173b71170 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b6a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b632a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1711781212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b62d60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d2a7fc640 0 client.16020 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 >> 172.21.15.28:0/1711781212 conn(0x562173b53d90 msgr2=0x562173b52fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.117+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 shutdown_connections 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.118+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1711781212 wait complete. 2024-03-22T02:15:34.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.118+0000 7f5d4bfd3740 20 client.16020 trim_cache size 0 max 16384 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 Processor -- start 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 -- start start 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b6a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b57700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441ad320 con 0x562173b58820 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441aabf0 con 0x562173b5e380 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441a8a40 con 0x562173b59210 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46492/0 (socket says 172.21.15.28:46492) 2024-03-22T02:15:34.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3622874808 learned_addr learned my addr 172.21.15.28:0/3622874808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b6a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.131+0000 7f5d437fe640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b57700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3622874808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 msgr2=0x562173b6a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b6a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3622874808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 msgr2=0x562173b57700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b57700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 -- 172.21.15.28:0/3622874808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562173b595f0 con 0x562173b58820 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b6a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d3402c7b0 tx=0x7f5d34003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8515cedaa129eeb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d38ff9640 1 -- 172.21.15.28:0/3622874808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3404b070 con 0x562173b58820 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d38ff9640 1 -- 172.21.15.28:0/3622874808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d340bdd40 con 0x562173b58820 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.132+0000 7f5d38ff9640 1 -- 172.21.15.28:0/3622874808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d340bcfa0 con 0x562173b58820 2024-03-22T02:15:34.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/3622874808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 msgr2=0x562173b63200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5d3402c7b0 tx=0x7f5d34003f90 comp rx=0 tx=0).stop 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/3622874808 shutdown_connections 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b5e380 0x562173b57700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b59210 0x562173b6a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/3622874808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b58820 0x562173b63200 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/3622874808 >> 172.21.15.28:0/3622874808 conn(0x562173b53d90 msgr2=0x562173b56ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/3622874808 shutdown_connections 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/3622874808 wait complete. 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- start start 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 Processor -- start 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 0x562173b67aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b5e380 0x562173b68ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441a90b0 con 0x562173b59210 2024-03-22T02:15:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441aa520 con 0x562173b58820 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d441a9780 con 0x562173b5e380 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d4bfd3740 10 client.0 _ll_register_callbacks cb 0x7f5d340f4f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46502/0 (socket says 172.21.15.28:46502) 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d3bfff640 1 -- 172.21.15.28:0/1614262516 learned_addr learned my addr 172.21.15.28:0/1614262516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 0x562173b67aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.133+0000 7f5d437fe640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b5e380 0x562173b68ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 -- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b5e380 msgr2=0x562173b68ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b5e380 0x562173b68ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 -- 172.21.15.28:0/1614262516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 msgr2=0x562173b67aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 0x562173b67aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 -- 172.21.15.28:0/1614262516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562173b695e0 con 0x562173b59210 2024-03-22T02:15:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 0x562173b67aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d3bfff640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x562173b54d00 tx=0x7f5d2c009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c895d104a6a20e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d2a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c02d070 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5d2c009d60 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d340bdd40 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5d2c039050 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d4bfd3740 10 client.16026 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562173bd5230 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.134+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d340bdd40 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5d2c0264c0 con 0x562173b59210 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 10 client.16026 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 10 client.16026 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 20 client.16026 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173bd5a70 0x562173bd7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 4 client.16026 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562173ac5160 con 0x562173bd5a70 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173bd5a70 0x562173bd7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d42ffd640 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173bd5a70 0x562173bd7e90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=98a8f2f2952bc06 server_cookie=9212d0f85a61f2b6 in_seq=0 out_seq=0 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d2a7fc640 10 client.16026 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d2a7fc640 1 -- 172.21.15.28:0/1614262516 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562173ac5160 con 0x562173bd5a70 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d2a7fc640 10 client.16026 handle_command_reply: tid=0 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 client.16026 shutdown 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 msgr2=0x562173b67fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x562173b54d00 tx=0x7f5d2c009280 comp rx=0 tx=0).stop 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 shutdown_connections 2024-03-22T02:15:34.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562173b5e380 0x562173b68ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562173b59210 0x562173b67fe0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x562173bd5a70 0x562173bd7e90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 --2- 172.21.15.28:0/1614262516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562173b58820 0x562173b67aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 >> 172.21.15.28:0/1614262516 conn(0x562173b53d90 msgr2=0x562173b62ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.135+0000 7f5d2a7fc640 0 client.16026 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 shutdown_connections 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d4bfd3740 1 -- 172.21.15.28:0/1614262516 wait complete. 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d4bfd3740 20 client.16026 trim_cache size 0 max 16384 2024-03-22T02:15:34.144 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 msgr2=0x7f5d300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 0x7f5d300a53e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x562173b59a70 tx=0x7f5d3c004a40 comp rx=0 tx=0).stop 2024-03-22T02:15:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 msgr2=0x7f5d44197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5d34000c00 tx=0x7f5d3402c640 comp rx=0 tx=0).stop 2024-03-22T02:15:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 shutdown_connections 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d300a2f20 0x7f5d300a53e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d44102650 0x7f5d4419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d44101c90 0x7f5d44198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 --2- 172.21.15.28:0/3789999052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d44100aa0 0x7f5d44197f40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 >> 172.21.15.28:0/3789999052 conn(0x7f5d440fc030 msgr2=0x7f5d4406a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 shutdown_connections 2024-03-22T02:15:34.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.136+0000 7f5d493e9640 1 -- 172.21.15.28:0/3789999052 wait complete. 2024-03-22T02:15:34.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.250+0000 7f2700a4e640 1 -- 172.21.15.28:0/1360862921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc102640 msgr2=0x7f26fc108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.250+0000 7f2700a4e640 1 --2- 172.21.15.28:0/1360862921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc102640 0x7f26fc108ed0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009ca0 tx=0x7f26f402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- 172.21.15.28:0/1360862921 shutdown_connections 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- 172.21.15.28:0/1360862921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc102640 0x7f26fc108ed0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- 172.21.15.28:0/1360862921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 0x7f26fc102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- 172.21.15.28:0/1360862921 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc100a90 0x7f26fc100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- 172.21.15.28:0/1360862921 >> 172.21.15.28:0/1360862921 conn(0x7f26fc0fc020 msgr2=0x7f26fc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- 172.21.15.28:0/1360862921 shutdown_connections 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- 172.21.15.28:0/1360862921 wait complete. 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 Processor -- start 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- start start 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 0x7f26fc1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc102640 0x7f26fc19e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc10c710 con 0x7f26fc100a90 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc10c590 con 0x7f26fc102640 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.251+0000 7f2700a4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc10c890 con 0x7f26fc101c80 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46520/0 (socket says 172.21.15.28:46520) 2024-03-22T02:15:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 -- 172.21.15.28:0/3379753801 learned_addr learned my addr 172.21.15.28:0/3379753801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 0x7f26fc1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fad9d640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc102640 0x7f26fc19e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 -- 172.21.15.28:0/3379753801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 msgr2=0x7f26fc1972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 0x7f26fc1972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 -- 172.21.15.28:0/3379753801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc102640 msgr2=0x7f26fc19e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc102640 0x7f26fc19e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 -- 172.21.15.28:0/3379753801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc19e9b0 con 0x7f26fc100a90 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26fa59c640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00e9c0 tx=0x7f26ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26eb7fe640 1 -- 172.21.15.28:0/3379753801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec00cd80 con 0x7f26fc100a90 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26eb7fe640 1 -- 172.21.15.28:0/3379753801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ec0035f0 con 0x7f26fc100a90 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc19eca0 con 0x7f26fc100a90 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f26eb7fe640 1 -- 172.21.15.28:0/3379753801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec010690 con 0x7f26fc100a90 2024-03-22T02:15:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.252+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc19f190 con 0x7f26fc100a90 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.254+0000 7f26eb7fe640 1 -- 172.21.15.28:0/3379753801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f26ec0037a0 con 0x7f26fc100a90 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 Processor -- start 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26eb7fe640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 0x7f26e40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26fad9d640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 0x7f26e40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 -- start start 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26eb7fe640 1 -- 172.21.15.28:0/3379753801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f26ec0cfbf0 con 0x7f26fc100a90 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5c50d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5c53810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555f5bb08d0 con 0x5555f5c4b820 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5555f5b58e00 con 0x5555f5c51380 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f2703636740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555f5bac880 con 0x5555f5c4c210 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46536/0 (socket says 172.21.15.28:46536) 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26fa59c640 1 -- 172.21.15.28:0/383150791 learned_addr learned my addr 172.21.15.28:0/383150791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.255+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5c50d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fad9d640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5c53810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fad9d640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 0x7f26e40a5460 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f26f4005e30 tx=0x7f26f4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 -- 172.21.15.28:0/383150791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 msgr2=0x5555f5c50d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5c50d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 -- 172.21.15.28:0/383150791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 msgr2=0x5555f5c53810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5c53810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 -- 172.21.15.28:0/383150791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f58c43f0 con 0x5555f5c4b820 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fad9d640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5c53810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5c50d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26fa59c640 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f26ec09bea0 tx=0x7f26ec09bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bfa15acef642622 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26e8ff9640 1 -- 172.21.15.28:0/383150791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0b4020 con 0x5555f5c4b820 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26e8ff9640 1 -- 172.21.15.28:0/383150791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ec09bf10 con 0x5555f5c4b820 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.256+0000 7f26e8ff9640 1 -- 172.21.15.28:0/383150791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec003c50 con 0x5555f5c4b820 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- 172.21.15.28:0/383150791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 msgr2=0x5555f5c4bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f26ec09bea0 tx=0x7f26ec09bb50 comp rx=0 tx=0).stop 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- 172.21.15.28:0/383150791 shutdown_connections 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5c53810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5c50d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- 172.21.15.28:0/383150791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5c4bc40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- 172.21.15.28:0/383150791 >> 172.21.15.28:0/383150791 conn(0x5555f5c46d90 msgr2=0x5555f5c47190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- 172.21.15.28:0/383150791 shutdown_connections 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- 172.21.15.28:0/383150791 wait complete. 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- start start 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 Processor -- start 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c55d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c562b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5555f5c54330 con 0x5555f5c51380 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5555f5b6c300 con 0x5555f5c4c210 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.257+0000 7f2703636740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5555f5c544b0 con 0x5555f5c4b820 2024-03-22T02:15:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f2703636740 10 client.0 _ll_register_callbacks cb 0x5555f5ba7e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46542/0 (socket says 172.21.15.28:46542) 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 -- 172.21.15.28:0/749911824 learned_addr learned my addr 172.21.15.28:0/749911824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c55d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 -- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 msgr2=0x5555f5c55d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c55d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 -- 172.21.15.28:0/749911824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 msgr2=0x5555f5c562b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c562b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 -- 172.21.15.28:0/749911824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f598e7d0 con 0x5555f5c51380 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fa59c640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c55d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26fad9d640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009a60 tx=0x7f26f402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473778e878e93c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f404f070 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f26f4004040 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26ec09bf10 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f26f405b050 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f2703636740 10 client.16038 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.258+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5555f5c606d0 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26ec09bf10 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f26f403f960 con 0x5555f5c51380 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 10 client.16038 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 10 client.16038 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:15:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 20 client.16038 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5c641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 4 client.16038 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555f5bbde10 con 0x5555f5c61d60 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f26fa59c640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5c641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f26e1ffb640 10 client.16038 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.259+0000 7f26fa59c640 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5c641e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0a272964f9b3771 server_cookie=606151f72914c181 in_seq=0 out_seq=0 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f26e1ffb640 1 -- 172.21.15.28:0/749911824 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555f5bbde10 con 0x5555f5c61d60 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f26e1ffb640 10 client.16038 handle_command_reply: tid=0 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 client.16038 shutdown 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 msgr2=0x5555f5c5d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009a60 tx=0x7f26f402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 shutdown_connections 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5c641e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c5d2f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c562b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f26e1ffb640 0 client.16038 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 --2- 172.21.15.28:0/749911824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c55d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 >> 172.21.15.28:0/749911824 conn(0x5555f5c46d90 msgr2=0x5555f5c45fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 shutdown_connections 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 1 -- 172.21.15.28:0/749911824 wait complete. 2024-03-22T02:15:34.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.260+0000 7f2703636740 20 client.16038 trim_cache size 0 max 16384 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.273+0000 7f2703636740 1 Processor -- start 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.273+0000 7f2703636740 1 -- start start 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c45740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c45c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1ac170 con 0x5555f5c51380 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1a9a40 con 0x5555f5c4c210 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f2703636740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1a7890 con 0x5555f5c4b820 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46546/0 (socket says 172.21.15.28:46546) 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 -- 172.21.15.28:0/2414993726 learned_addr learned my addr 172.21.15.28:0/2414993726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fa59c640 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c45740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c45c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 -- 172.21.15.28:0/2414993726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 msgr2=0x5555f5c45740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c45740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 -- 172.21.15.28:0/2414993726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 msgr2=0x5555f5c45c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c45c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 -- 172.21.15.28:0/2414993726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555f5c4caa0 con 0x5555f5c51380 2024-03-22T02:15:34.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26fad9d640 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f26f403fd00 tx=0x7f26f4032a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb651af3c808287a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26e3fff640 1 -- 172.21.15.28:0/2414993726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4064070 con 0x5555f5c51380 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26e3fff640 1 -- 172.21.15.28:0/2414993726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26f4044070 con 0x5555f5c51380 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.274+0000 7f26e3fff640 1 -- 172.21.15.28:0/2414993726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f404f050 con 0x5555f5c51380 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- 172.21.15.28:0/2414993726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 msgr2=0x5555f5c68aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f26f403fd00 tx=0x7f26f4032a60 comp rx=0 tx=0).stop 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- 172.21.15.28:0/2414993726 shutdown_connections 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c51380 0x5555f5c68aa0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c4c210 0x5555f5c45c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 --2- 172.21.15.28:0/2414993726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4b820 0x5555f5c45740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- 172.21.15.28:0/2414993726 >> 172.21.15.28:0/2414993726 conn(0x5555f5c46d90 msgr2=0x5555f5c47170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- 172.21.15.28:0/2414993726 shutdown_connections 2024-03-22T02:15:34.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- 172.21.15.28:0/2414993726 wait complete. 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 -- start start 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 Processor -- start 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.275+0000 7f2703636740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5cc6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5cc8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1a7f00 con 0x5555f5c4b820 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1a9370 con 0x5555f5c51380 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc1a85d0 con 0x5555f5c4c210 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 10 client.0 _ll_register_callbacks cb 0x7f26ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46556/0 (socket says 172.21.15.28:46556) 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 -- 172.21.15.28:0/309187272 learned_addr learned my addr 172.21.15.28:0/309187272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5cc6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fad9d640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5cc8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 -- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 msgr2=0x5555f5cc6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5cc6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 -- 172.21.15.28:0/309187272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 msgr2=0x5555f5cc8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5cc8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 -- 172.21.15.28:0/309187272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555f5cc9340 con 0x5555f5c4b820 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26fa59c640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f26ec004770 tx=0x7f26ec099f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5472b98d1e4ac5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0b4040 con 0x5555f5c4b820 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f26ec0984f0 con 0x5555f5c4b820 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26f4044070 con 0x5555f5c4b820 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f26ec0a0060 con 0x5555f5c4b820 2024-03-22T02:15:34.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 10 client.16044 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.276+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5555f5cca6b0 con 0x5555f5c4b820 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26f4044070 con 0x5555f5c4b820 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f26ec081020 con 0x5555f5c4b820 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 10 client.16044 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 10 client.16044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 20 client.16044 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5cccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 4 client.16044 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555f5bb8160 con 0x5555f5c61d60 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5cccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26f9d9b640 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5cccbf0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e3aa77000a37981 server_cookie=2d1f9e2d39157197 in_seq=0 out_seq=0 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26e1ffb640 10 client.16044 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26e1ffb640 1 -- 172.21.15.28:0/309187272 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555f5bb8160 con 0x5555f5c61d60 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f26e1ffb640 10 client.16044 handle_command_reply: tid=0 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.277+0000 7f2703636740 1 client.16044 shutdown 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 msgr2=0x5555f5cc40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f26ec004770 tx=0x7f26ec099f20 comp rx=0 tx=0).stop 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 shutdown_connections 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5555f5c61d60 0x5555f5cccbf0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5555f5c51380 0x5555f5cc8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5555f5c4c210 0x5555f5cc6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f26e1ffb640 0 client.16044 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 --2- 172.21.15.28:0/309187272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5555f5c4b820 0x5555f5cc40c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 >> 172.21.15.28:0/309187272 conn(0x5555f5c46d90 msgr2=0x5555f5c49ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 shutdown_connections 2024-03-22T02:15:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 1 -- 172.21.15.28:0/309187272 wait complete. 2024-03-22T02:15:34.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2703636740 20 client.16044 trim_cache size 0 max 16384 2024-03-22T02:15:34.287 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 msgr2=0x7f26e40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 0x7f26e40a5460 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f26f4005e30 tx=0x7f26f4032590 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 msgr2=0x7f26fc196da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f26ec00e9c0 tx=0x7f26ec00ee80 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 shutdown_connections 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26e40a2fa0 0x7f26e40a5460 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f26fc102640 0x7f26fc19e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26fc101c80 0x7f26fc1972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 --2- 172.21.15.28:0/3379753801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26fc100a90 0x7f26fc196da0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 >> 172.21.15.28:0/3379753801 conn(0x7f26fc0fc020 msgr2=0x7f26fc104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 shutdown_connections 2024-03-22T02:15:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.278+0000 7f2700a4e640 1 -- 172.21.15.28:0/3379753801 wait complete. 2024-03-22T02:15:34.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:15:34.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- 172.21.15.28:0/374824881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0102140 msgr2=0x7f0ce0102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 --2- 172.21.15.28:0/374824881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0102140 0x7f0ce0102560 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009900 tx=0x7f0cd002bc80 comp rx=0 tx=0).stop 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- 172.21.15.28:0/374824881 shutdown_connections 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 --2- 172.21.15.28:0/374824881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce01016c0 0x7f0ce0108f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 --2- 172.21.15.28:0/374824881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce0100d00 0x7f0ce0101180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 --2- 172.21.15.28:0/374824881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0102140 0x7f0ce0102560 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- 172.21.15.28:0/374824881 >> 172.21.15.28:0/374824881 conn(0x7f0ce00fb790 msgr2=0x7f0ce00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- 172.21.15.28:0/374824881 shutdown_connections 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- 172.21.15.28:0/374824881 wait complete. 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 Processor -- start 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.394+0000 7f0ce6972640 1 -- start start 2024-03-22T02:15:34.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce01016c0 0x7f0ce01972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce0102140 0x7f0ce019e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010c730 con 0x7f0ce0100d00 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010c5b0 con 0x7f0ce0102140 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce6972640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce010c8b0 con 0x7f0ce01016c0 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46572/0 (socket says 172.21.15.28:46572) 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 -- 172.21.15.28:0/2506353201 learned_addr learned my addr 172.21.15.28:0/2506353201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce01016c0 0x7f0ce01972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce0102140 0x7f0ce019e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 -- 172.21.15.28:0/2506353201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce01016c0 msgr2=0x7f0ce01972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce01016c0 0x7f0ce01972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 -- 172.21.15.28:0/2506353201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce0102140 msgr2=0x7f0ce019e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce0102140 0x7f0ce019e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 -- 172.21.15.28:0/2506353201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce019e9d0 con 0x7f0ce0100d00 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.395+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005ab0 tx=0x7f0cd002c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.396+0000 7f0cdd7fa640 1 -- 172.21.15.28:0/2506353201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0041070 con 0x7f0ce0100d00 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.396+0000 7f0cdd7fa640 1 -- 172.21.15.28:0/2506353201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0004450 con 0x7f0ce0100d00 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.396+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce019ec60 con 0x7f0ce0100d00 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.396+0000 7f0cdd7fa640 1 -- 172.21.15.28:0/2506353201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0034440 con 0x7f0ce0100d00 2024-03-22T02:15:34.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.396+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce019f150 con 0x7f0ce0100d00 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0cdd7fa640 1 -- 172.21.15.28:0/2506353201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0cd0044070 con 0x7f0ce0100d00 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 Processor -- start 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 -- start start 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 0x55fc0e0fff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e05d8d0 con 0x55fc0e100590 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e005e00 con 0x55fc0e0fb420 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.398+0000 7f0ce9630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e059880 con 0x55fc0e0faa30 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 0x55fc0e0fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46578/0 (socket says 172.21.15.28:46578) 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/782673792 learned_addr learned my addr 172.21.15.28:0/782673792 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdd7fa640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 0x7f0cc80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/782673792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 msgr2=0x55fc0e0fae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/782673792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 msgr2=0x55fc0e0fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 0x55fc0e0fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/782673792 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc0dd713f0 con 0x55fc0e100590 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdffff640 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 0x55fc0e0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 0x7f0cc80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdd7fa640 1 -- 172.21.15.28:0/2506353201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0cd00f1280 con 0x7f0ce0100d00 2024-03-22T02:15:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009c60 tx=0x7f0cd4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff819dcb3066b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cc67fc640 1 -- 172.21.15.28:0/782673792 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4004220 con 0x55fc0e100590 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cc67fc640 1 -- 172.21.15.28:0/782673792 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd40043c0 con 0x55fc0e100590 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.399+0000 7f0cc67fc640 1 -- 172.21.15.28:0/782673792 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4004e60 con 0x55fc0e100590 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- 172.21.15.28:0/782673792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 msgr2=0x55fc0e102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4009c60 tx=0x7f0cd4011810 comp rx=0 tx=0).stop 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 0x7f0cc80a53e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009fd0 tx=0x7f0cd8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- 172.21.15.28:0/782673792 shutdown_connections 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e102a20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0fb420 0x55fc0e0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 --2- 172.21.15.28:0/782673792 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- 172.21.15.28:0/782673792 >> 172.21.15.28:0/782673792 conn(0x55fc0e0f5fa0 msgr2=0x55fc0e0f63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- 172.21.15.28:0/782673792 shutdown_connections 2024-03-22T02:15:34.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- 172.21.15.28:0/782673792 wait complete. 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.400+0000 7f0ce9630740 1 -- start start 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 Processor -- start 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e104f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 0x55fc0e10c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e103540 con 0x55fc0e0fb420 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e019300 con 0x55fc0e0faa30 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fc0e1036c0 con 0x55fc0e100590 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce9630740 10 client.0 _ll_register_callbacks cb 0x55fc0e054e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46590/0 (socket says 172.21.15.28:46590) 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/476129767 learned_addr learned my addr 172.21.15.28:0/476129767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 0x55fc0e10c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdffff640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e104f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/476129767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 msgr2=0x55fc0e10c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 0x55fc0e10c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 msgr2=0x55fc0e104f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e104f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/476129767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc0de3b7d0 con 0x55fc0e0fb420 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 0x55fc0e10c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdffff640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0cd401b040 tx=0x7f0cd4019a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d10d028d96148c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.401+0000 7f0cbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4027030 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0cd40046c0 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd40043c0 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cd4034050 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 10 client.16056 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc0e10f870 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd40043c0 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cd4023b20 con 0x55fc0e0fb420 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 10 client.16056 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 20 client.16056 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e1131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 4 client.16056 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:34.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc0e06ae10 con 0x55fc0e110d60 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.402+0000 7f0cdffff640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e1131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0cdffff640 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e1131e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=591b91c25d900bb9 server_cookie=41051cf1e94b3b99 in_seq=0 out_seq=0 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0cbf7fe640 10 client.16056 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/476129767 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fc0e06ae10 con 0x55fc0e110d60 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0cbf7fe640 10 client.16056 handle_command_reply: tid=0 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 client.16056 shutdown 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 msgr2=0x55fc0e105470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0cd401b040 tx=0x7f0cd4019a70 comp rx=0 tx=0).stop 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 shutdown_connections 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e1131e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0cbf7fe640 0 client.16056 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.403+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e100590 0x55fc0e10c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e105470 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 1 --2- 172.21.15.28:0/476129767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 >> 172.21.15.28:0/476129767 conn(0x55fc0e0f5fa0 msgr2=0x55fc0e0f2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 shutdown_connections 2024-03-22T02:15:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 1 -- 172.21.15.28:0/476129767 wait complete. 2024-03-22T02:15:34.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.404+0000 7f0ce9630740 20 client.16056 trim_cache size 0 max 16384 2024-03-22T02:15:34.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 10451 4.60 MB/sec warmup 16 sec latency 3613.596 ms 2024-03-22T02:15:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 Processor -- start 2024-03-22T02:15:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 -- start start 2024-03-22T02:15:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e100590 0x55fc0e0f9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01aaf70 con 0x55fc0e0fb420 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01a8840 con 0x55fc0e100590 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce9630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01a6690 con 0x55fc0e0faa30 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46604/0 (socket says 172.21.15.28:46604) 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/3150464104 learned_addr learned my addr 172.21.15.28:0/3150464104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e100590 0x55fc0e0f9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdffff640 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/3150464104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 msgr2=0x55fc0e0fc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/3150464104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e100590 msgr2=0x55fc0e0f9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e100590 0x55fc0e0f9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.417+0000 7f0cdf7fe640 1 -- 172.21.15.28:0/3150464104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc0e103420 con 0x55fc0e0fb420 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4023ef0 tx=0x7f0cd4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698fcdee19f40552 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0cc5ffb640 1 -- 172.21.15.28:0/3150464104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4027070 con 0x55fc0e0fb420 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0cc5ffb640 1 -- 172.21.15.28:0/3150464104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd402e0b0 con 0x55fc0e0fb420 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0cc5ffb640 1 -- 172.21.15.28:0/3150464104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd401d040 con 0x55fc0e0fb420 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 -- 172.21.15.28:0/3150464104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 msgr2=0x55fc0e10c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4023ef0 tx=0x7f0cd4009fd0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 -- 172.21.15.28:0/3150464104 shutdown_connections 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e100590 0x55fc0e0f9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e0fb420 0x55fc0e10c470 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 --2- 172.21.15.28:0/3150464104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0faa30 0x55fc0e0fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 -- 172.21.15.28:0/3150464104 >> 172.21.15.28:0/3150464104 conn(0x55fc0e0f5fa0 msgr2=0x55fc0e0f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 -- 172.21.15.28:0/3150464104 shutdown_connections 2024-03-22T02:15:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.418+0000 7f0ce9630740 1 -- 172.21.15.28:0/3150464104 wait complete. 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 -- start start 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 Processor -- start 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e1099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0fb420 0x55fc0e109ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01a6d00 con 0x55fc0e100590 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01a8170 con 0x55fc0e0faa30 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01a73d0 con 0x55fc0e0fb420 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce9630740 10 client.0 _ll_register_callbacks cb 0x7f0cd00f4c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce4f0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce4f0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46612/0 (socket says 172.21.15.28:46612) 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0ce4f0f640 1 -- 172.21.15.28:0/2265913354 learned_addr learned my addr 172.21.15.28:0/2265913354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0cdf7fe640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0fb420 0x55fc0e109ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.419+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e1099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 -- 172.21.15.28:0/2265913354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0fb420 msgr2=0x55fc0e109ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0fb420 0x55fc0e109ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 -- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 msgr2=0x55fc0e1099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e1099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 -- 172.21.15.28:0/2265913354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc0e10b450 con 0x55fc0e100590 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce4f0f640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0cd801e5b0 tx=0x7f0cd8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe66a1458e354753 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd802e020 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f0cd801b670 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd402e0b0 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cd8039050 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 10 client.16062 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fc0e10c7c0 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd402e0b0 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cd802a8c0 con 0x55fc0e100590 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 10 client.16062 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 10 client.16062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 20 client.16062 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:34.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e179a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 4 client.16062 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.420+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fc0e065160 con 0x55fc0e110d60 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e179a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cdffff640 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e179a60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7a8170cba64a78 server_cookie=7ffac7375270c810 in_seq=0 out_seq=0 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cbf7fe640 10 client.16062 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cbf7fe640 1 -- 172.21.15.28:0/2265913354 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fc0e065160 con 0x55fc0e110d60 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cbf7fe640 10 client.16062 handle_command_reply: tid=0 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 client.16062 shutdown 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 msgr2=0x55fc0e10ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0cd801e5b0 tx=0x7f0cd8004ab0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 shutdown_connections 2024-03-22T02:15:34.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fc0e110d60 0x55fc0e179a60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fc0e100590 0x55fc0e10ae60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fc0e0fb420 0x55fc0e109ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 --2- 172.21.15.28:0/2265913354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fc0e0faa30 0x55fc0e1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0cbf7fe640 0 client.16062 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 >> 172.21.15.28:0/2265913354 conn(0x55fc0e0f5fa0 msgr2=0x55fc0e0f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 shutdown_connections 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 1 -- 172.21.15.28:0/2265913354 wait complete. 2024-03-22T02:15:34.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.421+0000 7f0ce9630740 20 client.16062 trim_cache size 0 max 16384 2024-03-22T02:15:34.432 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:15:34.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 msgr2=0x7f0cc80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 0x7f0cc80a53e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009fd0 tx=0x7f0cd8004a40 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 msgr2=0x7f0ce0196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005ab0 tx=0x7f0cd002c630 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 shutdown_connections 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0cc80a2f20 0x7f0cc80a53e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ce0102140 0x7f0ce019e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ce01016c0 0x7f0ce01972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 --2- 172.21.15.28:0/2506353201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce0100d00 0x7f0ce0196db0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 >> 172.21.15.28:0/2506353201 conn(0x7f0ce00fb790 msgr2=0x7f0ce01050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 shutdown_connections 2024-03-22T02:15:34.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:34.422+0000 7f0ce6972640 1 -- 172.21.15.28:0/2506353201 wait complete. 2024-03-22T02:15:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:34 smithi028 ceph-mon[31916]: pgmap v417: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1001 KiB/s wr, 84 op/s 2024-03-22T02:15:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:34 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3272804845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:34 smithi120 ceph-mon[36803]: pgmap v417: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1001 KiB/s wr, 84 op/s 2024-03-22T02:15:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:34 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3272804845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:35.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:34 smithi083 ceph-mon[36340]: pgmap v417: 129 pgs: 129 active+clean; 11 MiB data, 381 MiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 1001 KiB/s wr, 84 op/s 2024-03-22T02:15:35.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:34 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3272804845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:15:35.405 INFO:tasks.workunit.client.0.smithi028.stdout: 1 10473 4.33 MB/sec warmup 17 sec latency 4582.546 ms 2024-03-22T02:15:35.435 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:15:35.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:15:35.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:15:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- 172.21.15.28:0/1386500926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c101c80 msgr2=0x7fa42c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 --2- 172.21.15.28:0/1386500926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c101c80 0x7fa42c102100 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fa414009900 tx=0x7fa41402bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- 172.21.15.28:0/1386500926 shutdown_connections 2024-03-22T02:15:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 --2- 172.21.15.28:0/1386500926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 0x7fa42c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 --2- 172.21.15.28:0/1386500926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c101c80 0x7fa42c102100 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 --2- 172.21.15.28:0/1386500926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c100a90 0x7fa42c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- 172.21.15.28:0/1386500926 >> 172.21.15.28:0/1386500926 conn(0x7fa42c0fc020 msgr2=0x7fa42c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- 172.21.15.28:0/1386500926 shutdown_connections 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- 172.21.15.28:0/1386500926 wait complete. 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 Processor -- start 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.556+0000 7fa431839640 1 -- start start 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 0x7fa42c1035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c710 con 0x7fa42c100a90 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c590 con 0x7fa42c101c80 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa431839640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c10c890 con 0x7fa42c102640 2024-03-22T02:15:35.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55204/0 (socket says 172.21.15.28:55204) 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46630/0 (socket says 172.21.15.28:46630) 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 -- 172.21.15.28:0/768560137 learned_addr learned my addr 172.21.15.28:0/768560137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 0x7fa42c1035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 -- 172.21.15.28:0/768560137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 msgr2=0x7fa42c1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 0x7fa42c1035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 -- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 msgr2=0x7fa42c1049b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 -- 172.21.15.28:0/768560137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa42c103dc0 con 0x7fa42c101c80 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa42affd640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa423fff640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa414009230 tx=0x7fa41402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.557+0000 7fa430837640 1 -- 172.21.15.28:0/768560137 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414041070 con 0x7fa42c101c80 2024-03-22T02:15:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.558+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42c1a1f70 con 0x7fa42c101c80 2024-03-22T02:15:35.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.558+0000 7fa430837640 1 -- 172.21.15.28:0/768560137 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa414004450 con 0x7fa42c101c80 2024-03-22T02:15:35.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.558+0000 7fa430837640 1 -- 172.21.15.28:0/768560137 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414003e20 con 0x7fa42c101c80 2024-03-22T02:15:35.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.558+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42c1a23e0 con 0x7fa42c101c80 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.560+0000 7fa430837640 1 -- 172.21.15.28:0/768560137 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa414044070 con 0x7fa42c101c80 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.560+0000 7fa434425740 1 Processor -- start 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.560+0000 7fa434425740 1 -- start start 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.560+0000 7fa430837640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 0x7fa4180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba1910bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dba186d570 con 0x55dba1908bc0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dba1815e10 con 0x55dba19095b0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa434425740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dba1869520 con 0x55dba190e720 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 0x7fa4180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa430837640 1 -- 172.21.15.28:0/768560137 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa4140f13d0 con 0x7fa42c101c80 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46636/0 (socket says 172.21.15.28:46636) 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 -- 172.21.15.28:0/346719748 learned_addr learned my addr 172.21.15.28:0/346719748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa423fff640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba1910bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 -- 172.21.15.28:0/346719748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 msgr2=0x55dba1910bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba1910bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 -- 172.21.15.28:0/346719748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 msgr2=0x55dba190e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 -- 172.21.15.28:0/346719748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba16497e0 con 0x55dba1908bc0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa423fff640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 0x7fa4180a53e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa424005fd0 tx=0x7fa424009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa42affd640 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0114f0 tx=0x7fa41c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=842c0bfbbedf7c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa421ffb640 1 -- 172.21.15.28:0/346719748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c01e070 con 0x55dba1908bc0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa421ffb640 1 -- 172.21.15.28:0/346719748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41c00faa0 con 0x55dba1908bc0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.561+0000 7fa421ffb640 1 -- 172.21.15.28:0/346719748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c019ad0 con 0x55dba1908bc0 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- 172.21.15.28:0/346719748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 msgr2=0x55dba1908fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0114f0 tx=0x7fa41c0119b0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- 172.21.15.28:0/346719748 shutdown_connections 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba1910bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 --2- 172.21.15.28:0/346719748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1908fe0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- 172.21.15.28:0/346719748 >> 172.21.15.28:0/346719748 conn(0x55dba1904130 msgr2=0x55dba1904530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- 172.21.15.28:0/346719748 shutdown_connections 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- 172.21.15.28:0/346719748 wait complete. 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.562+0000 7fa434425740 1 -- start start 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 Processor -- start 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba191a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dba19116d0 con 0x55dba190e720 2024-03-22T02:15:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55dba1829110 con 0x55dba19095b0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55dba1911850 con 0x55dba1908bc0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa434425740 10 client.0 _ll_register_callbacks cb 0x55dba1865be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba191a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55222/0 (socket says 172.21.15.28:55222) 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59614/0 (socket says 172.21.15.28:59614) 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa423fff640 1 -- 172.21.15.28:0/2835636671 learned_addr learned my addr 172.21.15.28:0/2835636671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 -- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 msgr2=0x55dba1913290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 -- 172.21.15.28:0/2835636671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 msgr2=0x55dba191a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba191a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42affd640 1 -- 172.21.15.28:0/2835636671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba16a9f70 con 0x55dba1908bc0 2024-03-22T02:15:35.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa423fff640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.563+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba191a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa42affd640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0114c0 tx=0x7fa41c00f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8b3bcba9dbcea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c029060 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa41c00faa0 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa41c0052c0 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa41c030050 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 10 client.25906 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dba1541170 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41c00faa0 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa41c003800 con 0x55dba1908bc0 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 10 client.25906 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 10 client.25906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 20 client.25906 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba191e910 0x55dba1920db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 4 client.25906 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.564+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dba187aab0 con 0x55dba191e910 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.565+0000 7fa423fff640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba191e910 0x55dba1920db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.565+0000 7fa423fff640 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba191e910 0x55dba1920db0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b903e440791160 server_cookie=49792f0e401d161 in_seq=0 out_seq=0 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.565+0000 7fa40affd640 10 client.25906 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa40affd640 1 -- 172.21.15.28:0/2835636671 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dba187aab0 con 0x55dba191e910 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa40affd640 10 client.25906 handle_command_reply: tid=0 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 client.25906 shutdown 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 msgr2=0x55dba1912d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0114c0 tx=0x7fa41c00f870 comp rx=0 tx=0).stop 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 shutdown_connections 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba191e910 0x55dba1920db0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba191a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1913290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 --2- 172.21.15.28:0/2835636671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1912d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa40affd640 0 client.25906 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 >> 172.21.15.28:0/2835636671 conn(0x55dba1904130 msgr2=0x55dba19033f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 shutdown_connections 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.566+0000 7fa434425740 1 -- 172.21.15.28:0/2835636671 wait complete. 2024-03-22T02:15:35.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.567+0000 7fa434425740 20 client.25906 trim_cache size 0 max 16384 2024-03-22T02:15:35.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 Processor -- start 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 -- start start 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba190abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1af490 con 0x55dba1908bc0 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1acd60 con 0x55dba19095b0 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa434425740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1aabb0 con 0x55dba190e720 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46644/0 (socket says 172.21.15.28:46644) 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 -- 172.21.15.28:0/1805028569 learned_addr learned my addr 172.21.15.28:0/1805028569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba190abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa423fff640 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 -- 172.21.15.28:0/1805028569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 msgr2=0x55dba190abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba190abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 -- 172.21.15.28:0/1805028569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 msgr2=0x55dba190a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 -- 172.21.15.28:0/1805028569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dba1909990 con 0x55dba1908bc0 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.585+0000 7fa42affd640 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa41c001eb0 tx=0x7fa41c004fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79abecc626129d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa4217fa640 1 -- 172.21.15.28:0/1805028569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c029050 con 0x55dba1908bc0 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa4217fa640 1 -- 172.21.15.28:0/1805028569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41c016020 con 0x55dba1908bc0 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa4217fa640 1 -- 172.21.15.28:0/1805028569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c01e070 con 0x55dba1908bc0 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 -- 172.21.15.28:0/1805028569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 msgr2=0x55dba1907c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa41c001eb0 tx=0x7fa41c004fd0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 -- 172.21.15.28:0/1805028569 shutdown_connections 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba190e720 0x55dba190abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba190a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 --2- 172.21.15.28:0/1805028569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba1908bc0 0x55dba1907c20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 -- 172.21.15.28:0/1805028569 >> 172.21.15.28:0/1805028569 conn(0x55dba1904130 msgr2=0x55dba1909fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 -- 172.21.15.28:0/1805028569 shutdown_connections 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.586+0000 7fa434425740 1 -- 172.21.15.28:0/1805028569 wait complete. 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 -- start start 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 Processor -- start 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1917ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1ab220 con 0x55dba190e720 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1ac690 con 0x55dba19095b0 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa42c1ab8f0 con 0x55dba1908bc0 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa434425740 10 client.0 _ll_register_callbacks cb 0x7fa41c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa423fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1917ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59634/0 (socket says 172.21.15.28:59634) 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42affd640 1 -- 172.21.15.28:0/2782562123 learned_addr learned my addr 172.21.15.28:0/2782562123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46648/0 (socket says 172.21.15.28:46648) 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42affd640 1 -- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 msgr2=0x55dba1917ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.587+0000 7fa42affd640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1917ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa42affd640 1 -- 172.21.15.28:0/2782562123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 msgr2=0x55dba1918ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa42affd640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa42affd640 1 -- 172.21.15.28:0/2782562123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dba19194c0 con 0x55dba1908bc0 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa42b7fe640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa42affd640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0095f0 tx=0x7fa41c0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=460104d441c52faf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:15:35.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c029070 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fa41c0054c0 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa41c016020 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa41c045050 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 10 client.25912 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dba1984fb0 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41c016020 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa41c03f4e0 con 0x55dba1908bc0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 10 client.25912 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 20 client.25912 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba1985870 0x55dba1987c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 4 client.25912 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.588+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dba1874e00 con 0x55dba1985870 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.589+0000 7fa423fff640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba1985870 0x55dba1987c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.589+0000 7fa40affd640 10 client.25912 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.589+0000 7fa423fff640 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba1985870 0x55dba1987c90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58e9e1f36273f27 server_cookie=68f4bf6214b02c3b in_seq=0 out_seq=0 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa40affd640 1 -- 172.21.15.28:0/2782562123 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dba1874e00 con 0x55dba1985870 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa40affd640 10 client.25912 handle_command_reply: tid=0 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 client.25912 shutdown 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 msgr2=0x55dba1917980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa41c0095f0 tx=0x7fa41c0038a0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 shutdown_connections 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55dba1985870 0x55dba1987c90 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dba190e720 0x55dba1918ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55dba19095b0 0x55dba1917ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 --2- 172.21.15.28:0/2782562123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55dba1908bc0 0x55dba1917980 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa40affd640 0 client.25912 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 >> 172.21.15.28:0/2782562123 conn(0x55dba1904130 msgr2=0x55dba190a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 shutdown_connections 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 1 -- 172.21.15.28:0/2782562123 wait complete. 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.591+0000 7fa434425740 20 client.25912 trim_cache size 0 max 16384 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "33314238-32cc-4efe-85bf-a41bfdd68fa1", 2024-03-22T02:15:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:15:35.599 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 msgr2=0x7fa4180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 0x7fa4180a53e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa424005fd0 tx=0x7fa424009e00 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 msgr2=0x7fa42c103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa414009230 tx=0x7fa41402c650 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 shutdown_connections 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4180a2f20 0x7fa4180a53e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa42c102640 0x7fa42c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa42c101c80 0x7fa42c103060 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 --2- 172.21.15.28:0/768560137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa42c100a90 0x7fa42c1049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 >> 172.21.15.28:0/768560137 conn(0x7fa42c0fc020 msgr2=0x7fa42c0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 shutdown_connections 2024-03-22T02:15:35.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.592+0000 7fa431839640 1 -- 172.21.15.28:0/768560137 wait complete. 2024-03-22T02:15:35.604 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:15:35.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:15:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 -- 172.21.15.28:0/2024033158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48104f50 msgr2=0x7f9d48107340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 --2- 172.21.15.28:0/2024033158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48104f50 0x7f9d48107340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009900 tx=0x7f9d3802bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 -- 172.21.15.28:0/2024033158 shutdown_connections 2024-03-22T02:15:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 --2- 172.21.15.28:0/2024033158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 --2- 172.21.15.28:0/2024033158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48104f50 0x7f9d48107340 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 --2- 172.21.15.28:0/2024033158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48102620 0x7f9d48104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 -- 172.21.15.28:0/2024033158 >> 172.21.15.28:0/2024033158 conn(0x7f9d480fc030 msgr2=0x7f9d480fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 -- 172.21.15.28:0/2024033158 shutdown_connections 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 -- 172.21.15.28:0/2024033158 wait complete. 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.711+0000 7f9d4d165640 1 Processor -- start 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 -- start start 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48104f50 0x7f9d481005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810d830 con 0x7f9d48102620 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810d6b0 con 0x7f9d48107880 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4d165640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4810d9b0 con 0x7f9d48104f50 2024-03-22T02:15:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46664/0 (socket says 172.21.15.28:46664) 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 -- 172.21.15.28:0/370144687 learned_addr learned my addr 172.21.15.28:0/370144687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48104f50 0x7f9d481005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4759e640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 -- 172.21.15.28:0/370144687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48104f50 msgr2=0x7f9d481005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48104f50 0x7f9d481005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 -- 172.21.15.28:0/370144687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 msgr2=0x7f9d48100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 -- 172.21.15.28:0/370144687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d481013a0 con 0x7f9d48102620 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d4759e640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.712+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00e9f0 tx=0x7f9d3c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.713+0000 7f9d2ffff640 1 -- 172.21.15.28:0/370144687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c00cd80 con 0x7f9d48102620 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.713+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d481a1fc0 con 0x7f9d48102620 2024-03-22T02:15:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.713+0000 7f9d2ffff640 1 -- 172.21.15.28:0/370144687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c005500 con 0x7f9d48102620 2024-03-22T02:15:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.713+0000 7f9d2ffff640 1 -- 172.21.15.28:0/370144687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c010690 con 0x7f9d48102620 2024-03-22T02:15:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.713+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d481a23d0 con 0x7f9d48102620 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.715+0000 7f9d2ffff640 1 -- 172.21.15.28:0/370144687 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9d3c0046c0 con 0x7f9d48102620 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.715+0000 7f9d4fe2c740 1 Processor -- start 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.715+0000 7f9d4fe2c740 1 -- start start 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd399920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd39c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617fd2f98c0 con 0x5617fd394430 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5617fd2a1c20 con 0x5617fd394e20 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617fd2f5870 con 0x5617fd399f90 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d2ffff640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 0x7f9d300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46680/0 (socket says 172.21.15.28:46680) 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 -- 172.21.15.28:0/3821490699 learned_addr learned my addr 172.21.15.28:0/3821490699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd399920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4659c640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd39c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 0x7f9d300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d2ffff640 1 -- 172.21.15.28:0/370144687 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9d3c0cfc40 con 0x7f9d48102620 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3821490699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 msgr2=0x5617fd39c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd39c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3821490699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 msgr2=0x5617fd394850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3821490699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fd2c6150 con 0x5617fd394e20 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd399920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0cd300 tx=0x7f9d3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92a49b0d9eef1c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.716+0000 7f9d4759e640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 0x7f9d300a53e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x5617fd395680 tx=0x7f9d40004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d2d7fa640 1 -- 172.21.15.28:0/3821490699 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0a4970 con 0x5617fd394e20 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d2d7fa640 1 -- 172.21.15.28:0/3821490699 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c09c660 con 0x5617fd394e20 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d2d7fa640 1 -- 172.21.15.28:0/3821490699 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c09c1c0 con 0x5617fd394e20 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3821490699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 msgr2=0x5617fd399920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd399920 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0cd300 tx=0x7f9d3c004a40 comp rx=0 tx=0).stop 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3821490699 shutdown_connections 2024-03-22T02:15:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd39c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd399920 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3821490699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd394850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3821490699 >> 172.21.15.28:0/3821490699 conn(0x5617fd38f9a0 msgr2=0x5617fd38fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3821490699 shutdown_connections 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.717+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3821490699 wait complete. 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 -- start start 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 Processor -- start 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394430 0x5617fd39e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd39ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617fd39ce80 con 0x5617fd399f90 2024-03-22T02:15:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5617fd2b5130 con 0x5617fd394e20 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617fd39d000 con 0x5617fd394430 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4fe2c740 10 client.0 _ll_register_callbacks cb 0x5617fd2b4b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46694/0 (socket says 172.21.15.28:46694) 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 -- 172.21.15.28:0/2801914759 learned_addr learned my addr 172.21.15.28:0/2801914759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394430 0x5617fd39e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4659c640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd39ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 -- 172.21.15.28:0/2801914759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394430 msgr2=0x5617fd39e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394430 0x5617fd39e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 -- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 msgr2=0x5617fd39ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd39ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.718+0000 7f9d4759e640 1 -- 172.21.15.28:0/2801914759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fd0d77d0 con 0x5617fd399f90 2024-03-22T02:15:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4759e640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000fe10 tx=0x7f9d40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a13b1177d76b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40026020 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9d400223f0 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d3c09c660 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d40031050 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 10 client.16092 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5617fd00d3f0 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c09c660 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9d4002aa20 con 0x5617fd399f90 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 10 client.16092 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 10 client.16092 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 20 client.16092 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd3aa820 0x5617fd3acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 4 client.16092 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.719+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617fd306e00 con 0x5617fd3aa820 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.720+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd3aa820 0x5617fd3acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.720+0000 7f9d2aff5640 10 client.16092 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.720+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd3aa820 0x5617fd3acc80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b277b8c0da4c529 server_cookie=78336d5d8fd63911 in_seq=0 out_seq=0 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d2aff5640 1 -- 172.21.15.28:0/2801914759 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617fd306e00 con 0x5617fd3aa820 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d2aff5640 10 client.16092 handle_command_reply: tid=0 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 client.16092 shutdown 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 msgr2=0x5617fd3a5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000fe10 tx=0x7f9d40004a40 comp rx=0 tx=0).stop 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 shutdown_connections 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd3aa820 0x5617fd3acc80 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd399f90 0x5617fd3a5dd0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd39ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/2801914759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394430 0x5617fd39e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d2aff5640 0 client.16092 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 >> 172.21.15.28:0/2801914759 conn(0x5617fd38f9a0 msgr2=0x5617fd38ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 shutdown_connections 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/2801914759 wait complete. 2024-03-22T02:15:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.721+0000 7f9d4fe2c740 20 client.16092 trim_cache size 0 max 16384 2024-03-22T02:15:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 Processor -- start 2024-03-22T02:15:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 -- start start 2024-03-22T02:15:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394e20 0x5617fd3a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd399f90 0x5617fd393310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481af500 con 0x5617fd394430 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481acdd0 con 0x5617fd399f90 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481aac20 con 0x5617fd394e20 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46696/0 (socket says 172.21.15.28:46696) 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 -- 172.21.15.28:0/4053253511 learned_addr learned my addr 172.21.15.28:0/4053253511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4659c640 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394e20 0x5617fd3a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d4759e640 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd399f90 0x5617fd393310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 -- 172.21.15.28:0/4053253511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394e20 msgr2=0x5617fd3a5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394e20 0x5617fd3a5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 -- 172.21.15.28:0/4053253511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd399f90 msgr2=0x5617fd393310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd399f90 0x5617fd393310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.740+0000 7f9d46d9d640 1 -- 172.21.15.28:0/4053253511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617fcfcd0f0 con 0x5617fd394430 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c09a4b0 tx=0x7f9d3c098200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f66ad2a7e4d2c6c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d2cff9640 1 -- 172.21.15.28:0/4053253511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c09bd30 con 0x5617fd394430 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d2cff9640 1 -- 172.21.15.28:0/4053253511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c09c660 con 0x5617fd394430 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d2cff9640 1 -- 172.21.15.28:0/4053253511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0a00b0 con 0x5617fd394430 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/4053253511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 msgr2=0x5617fd39de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c09a4b0 tx=0x7f9d3c098200 comp rx=0 tx=0).stop 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/4053253511 shutdown_connections 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd399f90 0x5617fd393310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd394e20 0x5617fd3a5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/4053253511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd39de70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/4053253511 >> 172.21.15.28:0/4053253511 conn(0x5617fd38f9a0 msgr2=0x5617fd392c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/4053253511 shutdown_connections 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.741+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/4053253511 wait complete. 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 -- start start 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 Processor -- start 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd3a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd3a44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481ab290 con 0x5617fd394430 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481ac700 con 0x5617fd394e20 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9d481ab960 con 0x5617fd399f90 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4fe2c740 10 client.0 _ll_register_callbacks cb 0x7f9d40004930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46704/0 (socket says 172.21.15.28:46704) 2024-03-22T02:15:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3877595503 learned_addr learned my addr 172.21.15.28:0/3877595503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4659c640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd3a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d4759e640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd3a44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3877595503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 msgr2=0x5617fd3a44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd3a44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 msgr2=0x5617fd3a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd3a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 -- 172.21.15.28:0/3877595503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617fd3a49f0 con 0x5617fd394430 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d46d9d640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00cfd0 tx=0x7f9d3c09c0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961804322d19739a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.742+0000 7f9d2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c09bd30 con 0x5617fd394430 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9d3c0a44e0 con 0x5617fd394430 2024-03-22T02:15:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d3c09c660 con 0x5617fd394430 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9d3c09baa0 con 0x5617fd394430 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 10 client.16098 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5617fd4117d0 con 0x5617fd394430 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c09c660 con 0x5617fd394430 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9d3c0ab020 con 0x5617fd394430 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 10 client.16098 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 10 client.16098 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 20 client.16098 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd411c20 0x5617fd414010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 4 client.16098 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5617fd39e640 con 0x5617fd411c20 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.743+0000 7f9d4659c640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd411c20 0x5617fd414010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4659c640 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd411c20 0x5617fd414010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0b43d442f773a48 server_cookie=7b87443c6d0675f1 in_seq=0 out_seq=0 2024-03-22T02:15:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d2aff5640 10 client.16098 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d2aff5640 1 -- 172.21.15.28:0/3877595503 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5617fd39e640 con 0x5617fd411c20 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d2aff5640 10 client.16098 handle_command_reply: tid=0 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 client.16098 shutdown 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 msgr2=0x5617fd3a5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00cfd0 tx=0x7f9d3c09c0b0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 shutdown_connections 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617fd411c20 0x5617fd414010 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617fd399f90 0x5617fd3a44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617fd394e20 0x5617fd3a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 --2- 172.21.15.28:0/3877595503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617fd394430 0x5617fd3a5910 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d2aff5640 0 client.16098 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 >> 172.21.15.28:0/3877595503 conn(0x5617fd38f9a0 msgr2=0x5617fd392c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 shutdown_connections 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 1 -- 172.21.15.28:0/3877595503 wait complete. 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.744+0000 7f9d4fe2c740 20 client.16098 trim_cache size 0 max 16384 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "33314238-32cc-4efe-85bf-a41bfdd68fa1": { 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "33314238-32cc-4efe-85bf-a41bfdd68fa1", 2024-03-22T02:15:35.753 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:15:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:15:35.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:15:35.754 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 msgr2=0x7f9d300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 0x7f9d300a53e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x5617fd395680 tx=0x7f9d40004ab0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 msgr2=0x7f9d48101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00e9f0 tx=0x7f9d3c00edf0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 shutdown_connections 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9d300a2f20 0x7f9d300a53e0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9d48107880 0x7f9d48100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9d48104f50 0x7f9d481005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 --2- 172.21.15.28:0/370144687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d48102620 0x7f9d48101f90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 >> 172.21.15.28:0/370144687 conn(0x7f9d480fc030 msgr2=0x7f9d481030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 shutdown_connections 2024-03-22T02:15:35.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:15:35.745+0000 7f9d4d165640 1 -- 172.21.15.28:0/370144687 wait complete. 2024-03-22T02:15:35.757 INFO:tasks.cephfs.filesystem:scrub status for tag:33314238-32cc-4efe-85bf-a41bfdd68fa1 - {'path': '/', 'tag': '33314238-32cc-4efe-85bf-a41bfdd68fa1', 'options': 'recursive,force'} 2024-03-22T02:15:36.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 12424 4.65 MB/sec warmup 18 sec latency 15.798 ms 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: pgmap v418: 129 pgs: 129 active+clean; 26 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.4 MiB/s wr, 94 op/s 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:36.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:36 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: pgmap v418: 129 pgs: 129 active+clean; 26 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.4 MiB/s wr, 94 op/s 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:36 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: pgmap v418: 129 pgs: 129 active+clean; 26 MiB data, 384 MiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 2.4 MiB/s wr, 94 op/s 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:36 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:15:37.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 14001 4.88 MB/sec warmup 19 sec latency 593.992 ms 2024-03-22T02:15:38.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 14001 4.64 MB/sec warmup 20 sec latency 1594.058 ms 2024-03-22T02:15:38.817 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:38 smithi083 ceph-mon[36340]: pgmap v419: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 2.3 MiB/s wr, 102 op/s 2024-03-22T02:15:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:38 smithi028 ceph-mon[31916]: pgmap v419: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 2.3 MiB/s wr, 102 op/s 2024-03-22T02:15:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:38 smithi120 ceph-mon[36803]: pgmap v419: 129 pgs: 129 active+clean; 26 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 2.3 MiB/s wr, 102 op/s 2024-03-22T02:15:39.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 14001 4.42 MB/sec warmup 21 sec latency 2594.164 ms 2024-03-22T02:15:40.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 14021 4.22 MB/sec warmup 22 sec latency 3565.207 ms 2024-03-22T02:15:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:40 smithi028 ceph-mon[31916]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-22T02:15:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:40 smithi120 ceph-mon[36803]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-22T02:15:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:40 smithi083 ceph-mon[36340]: pgmap v420: 129 pgs: 129 active+clean; 26 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 118 op/s 2024-03-22T02:15:41.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 15890 4.47 MB/sec warmup 23 sec latency 35.646 ms 2024-03-22T02:15:42.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 17544 4.66 MB/sec warmup 24 sec latency 568.646 ms 2024-03-22T02:15:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:42 smithi028 ceph-mon[31916]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-22T02:15:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:42 smithi120 ceph-mon[36803]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-22T02:15:43.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:42 smithi083 ceph-mon[36340]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-03-22T02:15:43.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 17544 4.47 MB/sec warmup 25 sec latency 1568.719 ms 2024-03-22T02:15:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2093484273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101bc0 msgr2=0x7ff4d0101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2093484273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0101fe0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009900 tx=0x7ff4c402bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2093484273 shutdown_connections 2024-03-22T02:15:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2093484273 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2093484273 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0102db0 0x7ff4d0103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2093484273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0101fe0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2093484273 >> 172.21.15.28:0/2093484273 conn(0x7ff4d00fd150 msgr2=0x7ff4d00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2093484273 shutdown_connections 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.733+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2093484273 wait complete. 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 Processor -- start 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 -- start start 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d880 con 0x7ff4d0102db0 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d700 con 0x7ff4d0103770 2024-03-22T02:15:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d6e5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010da00 con 0x7ff4d0101bc0 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44576/0 (socket says 172.21.15.28:44576) 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4cffff640 1 -- 172.21.15.28:0/2885324536 learned_addr learned my addr 172.21.15.28:0/2885324536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.734+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 -- 172.21.15.28:0/2885324536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0101bc0 msgr2=0x7ff4d0197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 -- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 msgr2=0x7ff4d01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 -- 172.21.15.28:0/2885324536 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d019fad0 con 0x7ff4d0103770 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4cffff640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d019f460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009fd0 tx=0x7ff4c800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4cdffb640 1 -- 172.21.15.28:0/2885324536 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800ce20 con 0x7ff4d0103770 2024-03-22T02:15:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4cdffb640 1 -- 172.21.15.28:0/2885324536 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c8005020 con 0x7ff4d0103770 2024-03-22T02:15:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4cdffb640 1 -- 172.21.15.28:0/2885324536 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8010720 con 0x7ff4d0103770 2024-03-22T02:15:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d019fdc0 con 0x7ff4d0103770 2024-03-22T02:15:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.735+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d01a0230 con 0x7ff4d0103770 2024-03-22T02:15:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.737+0000 7ff4cdffb640 1 -- 172.21.15.28:0/2885324536 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff4c8005540 con 0x7ff4d0103770 2024-03-22T02:15:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4cdffb640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 0x7ff4c00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 0x7ff4c00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4cdffb640 1 -- 172.21.15.28:0/2885324536 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff4c80cfee0 con 0x7ff4d0103770 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 Processor -- start 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 -- start start 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5607f2c41040 con 0x5607f2ce2040 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5607f2be99e0 con 0x5607f2cdced0 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5607f2c3cff0 con 0x5607f2cdc4e0 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 0x7ff4c00a5480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff4c402c170 tx=0x7ff4c402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d53f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4d4bf8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.738+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:41334/0 (socket says 172.21.15.28:41334) 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 -- 172.21.15.28:0/289206163 learned_addr learned my addr 172.21.15.28:0/289206163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d53f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44582/0 (socket says 172.21.15.28:44582) 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 -- 172.21.15.28:0/289206163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 msgr2=0x5607f2ce19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 -- 172.21.15.28:0/289206163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 msgr2=0x5607f2ce44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 -- 172.21.15.28:0/289206163 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607f2955210 con 0x5607f2cdc4e0 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000c00 tx=0x7ff4c403f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81f18f7bac41fd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4baffd640 1 -- 172.21.15.28:0/289206163 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c402cd40 con 0x5607f2cdc4e0 2024-03-22T02:15:43.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4baffd640 1 -- 172.21.15.28:0/289206163 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c4050070 con 0x5607f2cdc4e0 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.739+0000 7ff4baffd640 1 -- 172.21.15.28:0/289206163 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004a40 con 0x5607f2cdc4e0 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- 172.21.15.28:0/289206163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 msgr2=0x5607f2cdc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000c00 tx=0x7ff4c403f000 comp rx=0 tx=0).stop 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- 172.21.15.28:0/289206163 shutdown_connections 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2ce2040 0x5607f2ce44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/289206163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdc900 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- 172.21.15.28:0/289206163 >> 172.21.15.28:0/289206163 conn(0x5607f2cd7a50 msgr2=0x5607f2cd7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- 172.21.15.28:0/289206163 shutdown_connections 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- 172.21.15.28:0/289206163 wait complete. 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 -- start start 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.740+0000 7ff4d9a40740 1 Processor -- start 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 0x5607f2ceda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5607f2ce4f90 con 0x5607f2cdc4e0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5607f2bfcdb0 con 0x5607f2cdced0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5607f2ce5110 con 0x5607f2ce2040 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d9a40740 10 client.0 _ll_register_callbacks cb 0x5607f2954a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d4bf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d4bf8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44590/0 (socket says 172.21.15.28:44590) 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d4bf8640 1 -- 172.21.15.28:0/1519534384 learned_addr learned my addr 172.21.15.28:0/1519534384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 0x5607f2ceda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 -- 172.21.15.28:0/1519534384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 msgr2=0x5607f2ceda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 0x5607f2ceda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 -- 172.21.15.28:0/1519534384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 msgr2=0x5607f2ce64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 -- 172.21.15.28:0/1519534384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607f2a21620 con 0x5607f2cdced0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 0x5607f2ceda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce6a20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5607f2cdd2d0 tx=0x7ff4bc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e9cdbd85dd8199b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.741+0000 7ff4affff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc005020 con 0x5607f2cdced0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4c4050070 con 0x5607f2cdced0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff4bc0051c0 con 0x5607f2cdced0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff4bc0053c0 con 0x5607f2cdced0 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 10 client.25949 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:43.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607f2cf0e90 con 0x5607f2cdced0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c4050070 con 0x5607f2cdced0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff4bc01c500 con 0x5607f2cdced0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 10 client.25949 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 10 client.25949 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 20 client.25949 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2cf24e0 0x5607f2cf4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 4 client.25949 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607f2c4e580 con 0x5607f2cf24e0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.742+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2cf24e0 0x5607f2cf4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.743+0000 7ff4affff640 10 client.25949 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.743+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2cf24e0 0x5607f2cf4980 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96580e0484a2c08f server_cookie=ba58881b2b4036c9 in_seq=0 out_seq=0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.743+0000 7ff4affff640 1 -- 172.21.15.28:0/1519534384 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5607f2c4e580 con 0x5607f2cf24e0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.743+0000 7ff4affff640 10 client.25949 handle_command_reply: tid=0 2024-03-22T02:15:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.743+0000 7ff4d9a40740 1 client.25949 shutdown 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 msgr2=0x5607f2ce6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce6a20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x5607f2cdd2d0 tx=0x7ff4bc00f790 comp rx=0 tx=0).stop 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 shutdown_connections 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2cf24e0 0x5607f2cf4980 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2ce2040 0x5607f2ceda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4affff640 0 client.25949 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2cdced0 0x5607f2ce6a20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1519534384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdc4e0 0x5607f2ce64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 >> 172.21.15.28:0/1519534384 conn(0x5607f2cd7a50 msgr2=0x5607f2cd6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 shutdown_connections 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1519534384 wait complete. 2024-03-22T02:15:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.744+0000 7ff4d9a40740 20 client.25949 trim_cache size 0 max 16384 2024-03-22T02:15:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 Processor -- start 2024-03-22T02:15:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 -- start start 2024-03-22T02:15:43.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cd63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01ad250 con 0x5607f2cdced0 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01aab20 con 0x5607f2ce2040 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01a8970 con 0x5607f2cdc4e0 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.759+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44598/0 (socket says 172.21.15.28:44598) 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 -- 172.21.15.28:0/1019349319 learned_addr learned my addr 172.21.15.28:0/1019349319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cd63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 -- 172.21.15.28:0/1019349319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 msgr2=0x5607f2cdb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 -- 172.21.15.28:0/1019349319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 msgr2=0x5607f2cd63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cd63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 -- 172.21.15.28:0/1019349319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607f2ce5220 con 0x5607f2cdced0 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc91a35b88b4e028 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4ba7fc640 1 -- 172.21.15.28:0/1019349319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc02e070 con 0x5607f2cdced0 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4ba7fc640 1 -- 172.21.15.28:0/1019349319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4bc005560 con 0x5607f2cdced0 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.760+0000 7ff4ba7fc640 1 -- 172.21.15.28:0/1019349319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc01c760 con 0x5607f2cdced0 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1019349319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 msgr2=0x5607f2ceda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc000c00 tx=0x7ff4bc00fc90 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1019349319 shutdown_connections 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cd63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceda20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/1019349319 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2cdb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1019349319 >> 172.21.15.28:0/1019349319 conn(0x5607f2cd7a50 msgr2=0x5607f2cdacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1019349319 shutdown_connections 2024-03-22T02:15:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- 172.21.15.28:0/1019349319 wait complete. 2024-03-22T02:15:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- start start 2024-03-22T02:15:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 Processor -- start 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2ceaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cebfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.761+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01a8fe0 con 0x5607f2cdced0 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01aa450 con 0x5607f2ce2040 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d01a96b0 con 0x5607f2cdc4e0 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 10 client.0 _ll_register_callbacks cb 0x7ff4bc012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44600/0 (socket says 172.21.15.28:44600) 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 -- 172.21.15.28:0/2220086195 learned_addr learned my addr 172.21.15.28:0/2220086195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:43.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2ceaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cebfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 -- 172.21.15.28:0/2220086195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 msgr2=0x5607f2ceaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2ceaab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 -- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 msgr2=0x5607f2cebfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cebfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 -- 172.21.15.28:0/2220086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607f2cec5a0 con 0x5607f2cdced0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2ceaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d53f9640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cebfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4cffff640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc012a40 tx=0x7ff4bc01c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7ba00452a265f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4affff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc037060 con 0x5607f2cdced0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff4bc009cf0 con 0x5607f2cdced0 2024-03-22T02:15:43.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4bc005560 con 0x5607f2cdced0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff4bc03e050 con 0x5607f2cdced0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 10 client.16116 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.762+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5607f2ced8b0 con 0x5607f2cdced0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4bc005560 con 0x5607f2cdced0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff4bc032530 con 0x5607f2cdced0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 10 client.16116 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 10 client.16116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 20 client.16116 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2d58c40 0x5607f2d5b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 4 client.16116 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5607f2c488d0 con 0x5607f2d58c40 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2d58c40 0x5607f2d5b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4affff640 10 client.16116 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.763+0000 7ff4d4bf8640 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2d58c40 0x5607f2d5b060 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb57f02c8755f5e7 server_cookie=5bd2d8feeffe9b7f in_seq=0 out_seq=0 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4affff640 1 -- 172.21.15.28:0/2220086195 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5607f2c488d0 con 0x5607f2d58c40 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4affff640 10 client.16116 handle_command_reply: tid=0 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 client.16116 shutdown 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 msgr2=0x5607f2ceaff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc012a40 tx=0x7ff4bc01c810 comp rx=0 tx=0).stop 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 shutdown_connections 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5607f2d58c40 0x5607f2d5b060 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5607f2ce2040 0x5607f2cebfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4affff640 0 client.16116 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5607f2cdced0 0x5607f2ceaff0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 --2- 172.21.15.28:0/2220086195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5607f2cdc4e0 0x5607f2ceaab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 >> 172.21.15.28:0/2220086195 conn(0x5607f2cd7a50 msgr2=0x5607f2cdb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 shutdown_connections 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 1 -- 172.21.15.28:0/2220086195 wait complete. 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.835+0000 7ff4d9a40740 20 client.16116 trim_cache size 0 max 16384 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:15:43.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:15:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 msgr2=0x7ff4c00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 0x7ff4c00a5480 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff4c402c170 tx=0x7ff4c402df90 comp rx=0 tx=0).stop 2024-03-22T02:15:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 msgr2=0x7ff4d019f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d019f460 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009fd0 tx=0x7ff4c800ef10 comp rx=0 tx=0).stop 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 shutdown_connections 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4c00a2fc0 0x7ff4c00a5480 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d0103770 0x7ff4d019f460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d0102db0 0x7ff4d01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 --2- 172.21.15.28:0/2885324536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d0101bc0 0x7ff4d0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 >> 172.21.15.28:0/2885324536 conn(0x7ff4d00fd150 msgr2=0x7ff4d0105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 shutdown_connections 2024-03-22T02:15:43.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:43.836+0000 7ff4d6e5c640 1 -- 172.21.15.28:0/2885324536 wait complete. 2024-03-22T02:15:44.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 17544 4.30 MB/sec warmup 26 sec latency 2568.804 ms 2024-03-22T02:15:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:44 smithi028 ceph-mon[31916]: pgmap v422: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-03-22T02:15:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:44 smithi120 ceph-mon[36803]: pgmap v422: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-03-22T02:15:45.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:44 smithi083 ceph-mon[36340]: pgmap v422: 129 pgs: 129 active+clean; 26 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 133 op/s 2024-03-22T02:15:45.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 17566 4.15 MB/sec warmup 27 sec latency 3537.275 ms 2024-03-22T02:15:45.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:45 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:15:45.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:45 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:15:45.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:45 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:15:45.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:45 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:15:46.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:45 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:15:46.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:45 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:15:46.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 19403 4.35 MB/sec warmup 28 sec latency 43.991 ms 2024-03-22T02:15:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:46 smithi028 ceph-mon[31916]: pgmap v423: 129 pgs: 129 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-22T02:15:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:46 smithi120 ceph-mon[36803]: pgmap v423: 129 pgs: 129 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-22T02:15:47.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:46 smithi083 ceph-mon[36340]: pgmap v423: 129 pgs: 129 active+clean; 28 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.0 MiB/s wr, 127 op/s 2024-03-22T02:15:47.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 21131 4.52 MB/sec warmup 29 sec latency 559.704 ms 2024-03-22T02:15:48.406 INFO:tasks.workunit.client.0.smithi028.stdout: 1 21131 4.37 MB/sec warmup 30 sec latency 1559.754 ms 2024-03-22T02:15:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:48 smithi028 ceph-mon[31916]: pgmap v424: 129 pgs: 129 active+clean; 24 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 395 KiB/s wr, 97 op/s 2024-03-22T02:15:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:48 smithi120 ceph-mon[36803]: pgmap v424: 129 pgs: 129 active+clean; 24 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 395 KiB/s wr, 97 op/s 2024-03-22T02:15:49.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:48 smithi083 ceph-mon[36340]: pgmap v424: 129 pgs: 129 active+clean; 24 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 998 KiB/s rd, 395 KiB/s wr, 97 op/s 2024-03-22T02:15:49.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 21131 4.23 MB/sec warmup 31 sec latency 2559.856 ms 2024-03-22T02:15:50.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 21157 4.10 MB/sec warmup 32 sec latency 3525.085 ms 2024-03-22T02:15:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:50 smithi028 ceph-mon[31916]: pgmap v425: 129 pgs: 129 active+clean; 25 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 490 KiB/s wr, 130 op/s 2024-03-22T02:15:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:50 smithi120 ceph-mon[36803]: pgmap v425: 129 pgs: 129 active+clean; 25 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 490 KiB/s wr, 130 op/s 2024-03-22T02:15:51.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:50 smithi083 ceph-mon[36340]: pgmap v425: 129 pgs: 129 active+clean; 25 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 490 KiB/s wr, 130 op/s 2024-03-22T02:15:51.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 23170 4.28 MB/sec warmup 33 sec latency 47.575 ms 2024-03-22T02:15:52.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 24675 4.42 MB/sec warmup 34 sec latency 615.556 ms 2024-03-22T02:15:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:52 smithi028 ceph-mon[31916]: pgmap v426: 129 pgs: 129 active+clean; 26 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 474 KiB/s wr, 119 op/s 2024-03-22T02:15:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:52 smithi120 ceph-mon[36803]: pgmap v426: 129 pgs: 129 active+clean; 26 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 474 KiB/s wr, 119 op/s 2024-03-22T02:15:53.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:52 smithi083 ceph-mon[36340]: pgmap v426: 129 pgs: 129 active+clean; 26 MiB data, 415 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 474 KiB/s wr, 119 op/s 2024-03-22T02:15:53.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 24675 4.29 MB/sec warmup 35 sec latency 1615.700 ms 2024-03-22T02:15:54.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 24675 4.17 MB/sec warmup 36 sec latency 2615.819 ms 2024-03-22T02:15:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:54 smithi028 ceph-mon[31916]: pgmap v427: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 439 KiB/s wr, 114 op/s 2024-03-22T02:15:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:54 smithi120 ceph-mon[36803]: pgmap v427: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 439 KiB/s wr, 114 op/s 2024-03-22T02:15:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:54 smithi083 ceph-mon[36340]: pgmap v427: 129 pgs: 129 active+clean; 26 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 439 KiB/s wr, 114 op/s 2024-03-22T02:15:55.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 24700 4.06 MB/sec warmup 37 sec latency 3580.828 ms 2024-03-22T02:15:56.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 26769 4.22 MB/sec warmup 38 sec latency 29.677 ms 2024-03-22T02:15:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:56 smithi028 ceph-mon[31916]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 501 KiB/s wr, 124 op/s 2024-03-22T02:15:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.925+0000 7f3c64735640 1 -- 172.21.15.28:0/1110381217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 msgr2=0x7f3c5c104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.925+0000 7f3c64735640 1 --2- 172.21.15.28:0/1110381217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c104a60 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c5402bc30 comp rx=0 tx=0).stop 2024-03-22T02:15:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- 172.21.15.28:0/1110381217 shutdown_connections 2024-03-22T02:15:56.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- 172.21.15.28:0/1110381217 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- 172.21.15.28:0/1110381217 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 0x7f3c5c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- 172.21.15.28:0/1110381217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c104a60 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- 172.21.15.28:0/1110381217 >> 172.21.15.28:0/1110381217 conn(0x7f3c5c0fc060 msgr2=0x7f3c5c0fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- 172.21.15.28:0/1110381217 shutdown_connections 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- 172.21.15.28:0/1110381217 wait complete. 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 Processor -- start 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- start start 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 0x7f3c5c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10d8a0 con 0x7f3c5c102670 2024-03-22T02:15:56.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10d720 con 0x7f3c5c1078d0 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c64735640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c10da20 con 0x7f3c5c104fa0 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c624d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c624d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46154/0 (socket says 172.21.15.28:46154) 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c624d1640 1 -- 172.21.15.28:0/3644207118 learned_addr learned my addr 172.21.15.28:0/3644207118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 0x7f3c5c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.926+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 -- 172.21.15.28:0/3644207118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 msgr2=0x7f3c5c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 0x7f3c5c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 -- 172.21.15.28:0/3644207118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 msgr2=0x7f3c5c19f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c19f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 -- 172.21.15.28:0/3644207118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5c19fab0 con 0x7f3c5c102670 2024-03-22T02:15:56.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c624d1640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009230 tx=0x7f3c5402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c537fe640 1 -- 172.21.15.28:0/3644207118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54041070 con 0x7f3c5c102670 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5c19fd40 con 0x7f3c5c102670 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c537fe640 1 -- 172.21.15.28:0/3644207118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c54004450 con 0x7f3c5c102670 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c537fe640 1 -- 172.21.15.28:0/3644207118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54003e20 con 0x7f3c5c102670 2024-03-22T02:15:56.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.927+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5c1a0180 con 0x7f3c5c102670 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c537fe640 1 -- 172.21.15.28:0/3644207118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3c54044070 con 0x7f3c5c102670 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 Processor -- start 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 -- start start 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b6c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b744d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d44ad1040 con 0x561d44b6c4e0 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561d44a799e0 con 0x561d44b72040 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c673f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561d44accff0 con 0x561d44b6ced0 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b6c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b6c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46170/0 (socket says 172.21.15.28:46170) 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c61cd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b744d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c62cd2640 1 -- 172.21.15.28:0/3839300336 learned_addr learned my addr 172.21.15.28:0/3839300336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c624d1640 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.930+0000 7f3c537fe640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 0x7f3c480a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 0x7f3c480a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 -- 172.21.15.28:0/3839300336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 msgr2=0x561d44b719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b719d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 -- 172.21.15.28:0/3839300336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 msgr2=0x561d44b6c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b6c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 -- 172.21.15.28:0/3839300336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d447e5210 con 0x561d44b72040 2024-03-22T02:15:56.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c537fe640 1 -- 172.21.15.28:0/3644207118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3c540f1410 con 0x7f3c5c102670 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b744d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c06a900 tx=0x7f3c4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0c4e32ff92e5073 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 0x7f3c480a5480 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x561d44b6d730 tx=0x7f3c58004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c50ff9640 1 -- 172.21.15.28:0/3839300336 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c004230 con 0x561d44b72040 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c50ff9640 1 -- 172.21.15.28:0/3839300336 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c4c009e10 con 0x561d44b72040 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.931+0000 7f3c50ff9640 1 -- 172.21.15.28:0/3839300336 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c004d70 con 0x561d44b72040 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- 172.21.15.28:0/3839300336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 msgr2=0x561d44b744d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b744d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c06a900 tx=0x7f3c4c011810 comp rx=0 tx=0).stop 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- 172.21.15.28:0/3839300336 shutdown_connections 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b744d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 --2- 172.21.15.28:0/3839300336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b6c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- 172.21.15.28:0/3839300336 >> 172.21.15.28:0/3839300336 conn(0x561d44b67a50 msgr2=0x561d44b67e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:56.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- 172.21.15.28:0/3839300336 shutdown_connections 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- 172.21.15.28:0/3839300336 wait complete. 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 -- start start 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.932+0000 7f3c673f5740 1 Processor -- start 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b76990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b7d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561d44b74ff0 con 0x561d44b6c4e0 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561d44a8cdb0 con 0x561d44b72040 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561d44b75170 con 0x561d44b6ced0 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 10 client.0 _ll_register_callbacks cb 0x561d447e4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46178/0 (socket says 172.21.15.28:46178) 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 -- 172.21.15.28:0/2405103005 learned_addr learned my addr 172.21.15.28:0/2405103005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b76990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b7d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 -- 172.21.15.28:0/2405103005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 msgr2=0x561d44b76990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b76990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 -- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 msgr2=0x561d44b7d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b7d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 -- 172.21.15.28:0/2405103005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d448b1620 con 0x561d44b6c4e0 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3c540ba100 tx=0x7f3c540b9310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d4fb71d83c6b5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:56.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c5404b030 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3c540b9da0 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c4c009e10 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c54004c60 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 10 client.16128 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.933+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d44b80d40 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c4c009e10 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c540c5cb0 con 0x561d44b6c4e0 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 10 client.16128 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 10 client.16128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 20 client.16128 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44b82450 0x561d44b848f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 4 client.16128 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:15:56.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d44ade580 con 0x561d44b82450 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44b82450 0x561d44b848f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44b82450 0x561d44b848f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5329fe8787644592 server_cookie=184e9725be107d92 in_seq=0 out_seq=0 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.934+0000 7f3c45ffb640 10 client.16128 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2405103005 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561d44ade580 con 0x561d44b82450 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c45ffb640 10 client.16128 handle_command_reply: tid=0 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 client.16128 shutdown 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 msgr2=0x561d44b76450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f3c540ba100 tx=0x7f3c540b9310 comp rx=0 tx=0).stop 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 shutdown_connections 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44b82450 0x561d44b848f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b72040 0x561d44b7d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c45ffb640 0 client.16128 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:56.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b76990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2405103005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b6c4e0 0x561d44b76450 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 >> 172.21.15.28:0/2405103005 conn(0x561d44b67a50 msgr2=0x561d44b66c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 shutdown_connections 2024-03-22T02:15:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 1 -- 172.21.15.28:0/2405103005 wait complete. 2024-03-22T02:15:56.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.935+0000 7f3c673f5740 20 client.16128 trim_cache size 0 max 16384 2024-03-22T02:15:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 Processor -- start 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 -- start start 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 0x561d44b768f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b7d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1ad190 con 0x561d44b72040 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1aaa60 con 0x561d44b6c4e0 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.950+0000 7f3c673f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1a88b0 con 0x561d44b6ced0 2024-03-22T02:15:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46192/0 (socket says 172.21.15.28:46192) 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 -- 172.21.15.28:0/4258130063 learned_addr learned my addr 172.21.15.28:0/4258130063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c624d1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 0x561d44b768f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c61cd0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b7d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 -- 172.21.15.28:0/4258130063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 msgr2=0x561d44b7d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b7d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 -- 172.21.15.28:0/4258130063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 msgr2=0x561d44b768f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 0x561d44b768f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 -- 172.21.15.28:0/4258130063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d44b6d2b0 con 0x561d44b72040 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c624d1640 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 0x561d44b768f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3c580126f0 tx=0x7f3c58009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8511baf109cbb0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c47fff640 1 -- 172.21.15.28:0/4258130063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580121d0 con 0x561d44b72040 2024-03-22T02:15:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c47fff640 1 -- 172.21.15.28:0/4258130063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c580109a0 con 0x561d44b72040 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.951+0000 7f3c47fff640 1 -- 172.21.15.28:0/4258130063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58005520 con 0x561d44b72040 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- 172.21.15.28:0/4258130063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 msgr2=0x561d44b66ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f3c580126f0 tx=0x7f3c58009be0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- 172.21.15.28:0/4258130063 shutdown_connections 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b66ad0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6ced0 0x561d44b7d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- 172.21.15.28:0/4258130063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6c4e0 0x561d44b768f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- 172.21.15.28:0/4258130063 >> 172.21.15.28:0/4258130063 conn(0x561d44b67a50 msgr2=0x561d44b663c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- 172.21.15.28:0/4258130063 shutdown_connections 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- 172.21.15.28:0/4258130063 wait complete. 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- start start 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 Processor -- start 2024-03-22T02:15:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6c4e0 0x561d44b7b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6ced0 0x561d44b7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1a8f20 con 0x561d44b72040 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1aa390 con 0x561d44b6ced0 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5c1a95f0 con 0x561d44b6c4e0 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c673f5740 10 client.0 _ll_register_callbacks cb 0x7f3c54009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:15:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c62cd2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46208/0 (socket says 172.21.15.28:46208) 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c62cd2640 1 -- 172.21.15.28:0/2435693710 learned_addr learned my addr 172.21.15.28:0/2435693710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c61cd0640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6ced0 0x561d44b7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.952+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6c4e0 0x561d44b7b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 -- 172.21.15.28:0/2435693710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6c4e0 msgr2=0x561d44b7b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6c4e0 0x561d44b7b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 -- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6ced0 msgr2=0x561d44b7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:15:56.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6ced0 0x561d44b7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 -- 172.21.15.28:0/2435693710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d44b7d2d0 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c62cd2640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3c58004a60 tx=0x7f3c58009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec452fd9b3ddc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58037060 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3c58010d60 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c580109a0 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c58004b20 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c673f5740 10 client.16134 fetch_fsmap learned FSMap version 27 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d44be8f00 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c580109a0 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.953+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c58031030 con 0x561d44b72040 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 10 client.16134 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 10 client.16134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:15:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 20 client.16134 populate_metadata read hostname 'smithi028' 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44be9930 0x561d44bebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 4 client.16134 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561d44ad88d0 con 0x561d44be9930 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44be9930 0x561d44bebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c624d1640 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44be9930 0x561d44bebd50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37ee80f736f0aab7 server_cookie=7b46f844f390d9a4 in_seq=0 out_seq=0 2024-03-22T02:15:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:56.954+0000 7f3c45ffb640 10 client.16134 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:56 smithi120 ceph-mon[36803]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 501 KiB/s wr, 124 op/s 2024-03-22T02:15:57.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:56 smithi083 ceph-mon[36340]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 501 KiB/s wr, 124 op/s 2024-03-22T02:15:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c45ffb640 1 -- 172.21.15.28:0/2435693710 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561d44ad88d0 con 0x561d44be9930 2024-03-22T02:15:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c45ffb640 10 client.16134 handle_command_reply: tid=0 2024-03-22T02:15:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 client.16134 shutdown 2024-03-22T02:15:57.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 msgr2=0x561d44b7cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3c58004a60 tx=0x7f3c58009f20 comp rx=0 tx=0).stop 2024-03-22T02:15:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 shutdown_connections 2024-03-22T02:15:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561d44be9930 0x561d44bebd50 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d44b72040 0x561d44b7cce0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c45ffb640 0 client.16134 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561d44b6ced0 0x561d44b7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 --2- 172.21.15.28:0/2435693710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561d44b6c4e0 0x561d44b7b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 >> 172.21.15.28:0/2435693710 conn(0x561d44b67a50 msgr2=0x561d44b76770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 shutdown_connections 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 1 -- 172.21.15.28:0/2435693710 wait complete. 2024-03-22T02:15:57.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.045+0000 7f3c673f5740 20 client.16134 trim_cache size 0 max 16384 2024-03-22T02:15:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:15:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:15:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:15:57.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:15:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 msgr2=0x7f3c480a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 0x7f3c480a5480 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x561d44b6d730 tx=0x7f3c58004ab0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 msgr2=0x7f3c5c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:15:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009230 tx=0x7f3c5402c650 comp rx=0 tx=0).stop 2024-03-22T02:15:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 shutdown_connections 2024-03-22T02:15:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3c480a2fc0 0x7f3c480a5480 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3c5c1078d0 0x7f3c5c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c5c104fa0 0x7f3c5c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 --2- 172.21.15.28:0/3644207118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c5c102670 0x7f3c5c197e90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:15:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 >> 172.21.15.28:0/3644207118 conn(0x7f3c5c0fc060 msgr2=0x7f3c5c102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:15:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 shutdown_connections 2024-03-22T02:15:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:15:57.046+0000 7f3c64735640 1 -- 172.21.15.28:0/3644207118 wait complete. 2024-03-22T02:15:57.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 28222 4.34 MB/sec warmup 39 sec latency 622.247 ms 2024-03-22T02:15:58.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 28222 4.23 MB/sec warmup 40 sec latency 1622.364 ms 2024-03-22T02:15:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:58 smithi028 ceph-mon[31916]: pgmap v429: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 379 KiB/s wr, 98 op/s 2024-03-22T02:15:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:58 smithi120 ceph-mon[36803]: pgmap v429: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 379 KiB/s wr, 98 op/s 2024-03-22T02:15:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:58 smithi083 ceph-mon[36340]: pgmap v429: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 379 KiB/s wr, 98 op/s 2024-03-22T02:15:59.407 INFO:tasks.workunit.client.0.smithi028.stdout: 1 28222 4.13 MB/sec warmup 41 sec latency 2622.456 ms 2024-03-22T02:15:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:15:59 smithi028 ceph-mon[31916]: pgmap v430: 129 pgs: 129 active+clean; 25 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 502 KiB/s wr, 135 op/s 2024-03-22T02:15:59.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:15:59 smithi120 ceph-mon[36803]: pgmap v430: 129 pgs: 129 active+clean; 25 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 502 KiB/s wr, 135 op/s 2024-03-22T02:16:00.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:15:59 smithi083 ceph-mon[36340]: pgmap v430: 129 pgs: 129 active+clean; 25 MiB data, 432 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 502 KiB/s wr, 135 op/s 2024-03-22T02:16:00.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 28242 4.03 MB/sec warmup 42 sec latency 3601.154 ms 2024-03-22T02:16:01.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 30181 4.17 MB/sec warmup 43 sec latency 26.892 ms 2024-03-22T02:16:01.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:01 smithi120 ceph-mon[36803]: pgmap v431: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 410 KiB/s wr, 120 op/s 2024-03-22T02:16:02.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:01 smithi083 ceph-mon[36340]: pgmap v431: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 410 KiB/s wr, 120 op/s 2024-03-22T02:16:02.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:01 smithi028 ceph-mon[31916]: pgmap v431: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 410 KiB/s wr, 120 op/s 2024-03-22T02:16:02.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 31767 4.28 MB/sec warmup 44 sec latency 579.529 ms 2024-03-22T02:16:03.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 31767 4.19 MB/sec warmup 45 sec latency 1579.623 ms 2024-03-22T02:16:03.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:03 smithi120 ceph-mon[36803]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 408 KiB/s wr, 128 op/s 2024-03-22T02:16:04.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:03 smithi083 ceph-mon[36340]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 408 KiB/s wr, 128 op/s 2024-03-22T02:16:04.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:03 smithi028 ceph-mon[31916]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 408 KiB/s wr, 128 op/s 2024-03-22T02:16:04.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 31767 4.10 MB/sec warmup 46 sec latency 2579.714 ms 2024-03-22T02:16:05.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 31776 4.01 MB/sec warmup 47 sec latency 3569.265 ms 2024-03-22T02:16:05.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:16:05.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 -- 172.21.15.28:0/349957541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0103800 msgr2=0x7efcc010a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 --2- 172.21.15.28:0/349957541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0103800 0x7efcc010a090 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009cd0 tx=0x7efcb002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 -- 172.21.15.28:0/349957541 shutdown_connections 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 --2- 172.21.15.28:0/349957541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0103800 0x7efcc010a090 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 --2- 172.21.15.28:0/349957541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0102e40 0x7efcc01032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 --2- 172.21.15.28:0/349957541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 0x7efcc0100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 -- 172.21.15.28:0/349957541 >> 172.21.15.28:0/349957541 conn(0x7efcc00fb7a0 msgr2=0x7efcc00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 -- 172.21.15.28:0/349957541 shutdown_connections 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 -- 172.21.15.28:0/349957541 wait complete. 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.872+0000 7efcc5a39640 1 Processor -- start 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 -- start start 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 0x7efcc01027b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0103800 0x7efcc0101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcc010d820 con 0x7efcc0102e40 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efcc010d6a0 con 0x7efcc00ffe70 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efcc010d9a0 con 0x7efcc0103800 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42158/0 (socket says 172.21.15.28:42158) 2024-03-22T02:16:05.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 -- 172.21.15.28:0/22926337 learned_addr learned my addr 172.21.15.28:0/22926337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbeffd640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 0x7efcc01027b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0103800 0x7efcc0101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 -- 172.21.15.28:0/22926337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0103800 msgr2=0x7efcc0101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0103800 0x7efcc0101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 -- 172.21.15.28:0/22926337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 msgr2=0x7efcc01027b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 0x7efcc01027b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 -- 172.21.15.28:0/22926337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc0101bc0 con 0x7efcc0102e40 2024-03-22T02:16:05.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efca800e9f0 tx=0x7efca800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc4a37640 1 -- 172.21.15.28:0/22926337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca800cd80 con 0x7efcc0102e40 2024-03-22T02:16:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc4a37640 1 -- 172.21.15.28:0/22926337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca8010910 con 0x7efcc0102e40 2024-03-22T02:16:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.873+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc01a1f50 con 0x7efcc0102e40 2024-03-22T02:16:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.874+0000 7efcc4a37640 1 -- 172.21.15.28:0/22926337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8018990 con 0x7efcc0102e40 2024-03-22T02:16:05.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.874+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc01a23b0 con 0x7efcc0102e40 2024-03-22T02:16:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.876+0000 7efcc4a37640 1 -- 172.21.15.28:0/22926337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7efca8010ac0 con 0x7efcc0102e40 2024-03-22T02:16:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.876+0000 7efcc8627740 1 Processor -- start 2024-03-22T02:16:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.876+0000 7efcc8627740 1 -- start start 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221430 0x5591d2221850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc4a37640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 0x7efcac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591d21868c0 con 0x5591d2226f90 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591d212ec20 con 0x5591d2221430 2024-03-22T02:16:05.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc8627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5591d2182870 con 0x5591d2221e20 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbeffd640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 0x7efcac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42172/0 (socket says 172.21.15.28:42172) 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55268/0 (socket says 172.21.15.28:55268) 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbf7fe640 1 -- 172.21.15.28:0/641175642 learned_addr learned my addr 172.21.15.28:0/641175642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221430 0x5591d2221850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcc4a37640 1 -- 172.21.15.28:0/22926337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7efca80cfae0 con 0x7efcc0102e40 2024-03-22T02:16:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 -- 172.21.15.28:0/641175642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221430 msgr2=0x5591d2221850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221430 0x5591d2221850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 -- 172.21.15.28:0/641175642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 msgr2=0x5591d2229420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 -- 172.21.15.28:0/641175642 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d2153150 con 0x5591d2221e20 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbeffd640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 0x7efcac0a53e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efcc0069f80 tx=0x7efcb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efca809bb50 tx=0x7efca806ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a928a63ab2d89d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.877+0000 7efcb5ffb640 1 -- 172.21.15.28:0/641175642 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8026040 con 0x5591d2221e20 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcb5ffb640 1 -- 172.21.15.28:0/641175642 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca809be50 con 0x5591d2221e20 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcb5ffb640 1 -- 172.21.15.28:0/641175642 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca80a1860 con 0x5591d2221e20 2024-03-22T02:16:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 -- 172.21.15.28:0/641175642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 msgr2=0x5591d2226920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efca809bb50 tx=0x7efca806ec40 comp rx=0 tx=0).stop 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 -- 172.21.15.28:0/641175642 shutdown_connections 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2229420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2226920 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 --2- 172.21.15.28:0/641175642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221430 0x5591d2221850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 -- 172.21.15.28:0/641175642 >> 172.21.15.28:0/641175642 conn(0x5591d221c9a0 msgr2=0x5591d221cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 -- 172.21.15.28:0/641175642 shutdown_connections 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.878+0000 7efcc8627740 1 -- 172.21.15.28:0/641175642 wait complete. 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 -- start start 2024-03-22T02:16:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 Processor -- start 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221430 0x5591d222b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d222beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591d2229f40 con 0x5591d2226f90 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5591d2142130 con 0x5591d2221e20 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5591d222a0c0 con 0x5591d2221430 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcc8627740 10 client.0 _ll_register_callbacks cb 0x5591d2141b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42184/0 (socket says 172.21.15.28:42184) 2024-03-22T02:16:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 -- 172.21.15.28:0/3772204031 learned_addr learned my addr 172.21.15.28:0/3772204031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221430 0x5591d222b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d222beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 -- 172.21.15.28:0/3772204031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221430 msgr2=0x5591d222b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221430 0x5591d222b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 -- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 msgr2=0x5591d222beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d222beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.879+0000 7efcbf7fe640 1 -- 172.21.15.28:0/3772204031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d1f647d0 con 0x5591d2226f90 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ca0 tx=0x7efcb00328c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8bb5014a02843a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0039040 con 0x5591d2226f90 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7efcb0003970 con 0x5591d2226f90 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efca809be50 con 0x5591d2226f90 2024-03-22T02:16:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efcb0051050 con 0x5591d2226f90 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 10 client.16146 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5591d1e9a3f0 con 0x5591d2226f90 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efca809be50 con 0x5591d2226f90 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efcb003f030 con 0x5591d2226f90 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 10 client.16146 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 20 client.16146 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d2239ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 4 client.16146 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.880+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591d2193e00 con 0x5591d2237840 2024-03-22T02:16:05.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.881+0000 7efcbeffd640 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d2239ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.881+0000 7efc9effd640 10 client.16146 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.881+0000 7efcbeffd640 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d2239ca0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0245739632612ee server_cookie=1ae2469c8f8adbf7 in_seq=0 out_seq=0 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efc9effd640 1 -- 172.21.15.28:0/3772204031 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5591d2193e00 con 0x5591d2237840 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efc9effd640 10 client.16146 handle_command_reply: tid=0 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 client.16146 shutdown 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 msgr2=0x5591d2232ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009ca0 tx=0x7efcb00328c0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 shutdown_connections 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d2239ca0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2226f90 0x5591d2232ef0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efc9effd640 0 client.16146 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d222beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 --2- 172.21.15.28:0/3772204031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221430 0x5591d222b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 >> 172.21.15.28:0/3772204031 conn(0x5591d221c9a0 msgr2=0x5591d221bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 shutdown_connections 2024-03-22T02:16:05.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 1 -- 172.21.15.28:0/3772204031 wait complete. 2024-03-22T02:16:05.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.882+0000 7efcc8627740 20 client.16146 trim_cache size 0 max 16384 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 Processor -- start 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 -- start start 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d2232eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2226f90 0x5591d223e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01af500 con 0x5591d2221430 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01acdd0 con 0x5591d2221e20 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcc8627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01aac20 con 0x5591d2226f90 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42186/0 (socket says 172.21.15.28:42186) 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 -- 172.21.15.28:0/1922165972 learned_addr learned my addr 172.21.15.28:0/1922165972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2226f90 0x5591d223e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d2232eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 -- 172.21.15.28:0/1922165972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2226f90 msgr2=0x5591d223e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2226f90 0x5591d223e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 -- 172.21.15.28:0/1922165972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 msgr2=0x5591d2232eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d2232eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 -- 172.21.15.28:0/1922165972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591d1e5a0f0 con 0x5591d2221430 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d2232eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcbeffd640 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efcb80118d0 tx=0x7efcb8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292ca028bcdbb2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcb57fa640 1 -- 172.21.15.28:0/1922165972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8004020 con 0x5591d2221430 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcb57fa640 1 -- 172.21.15.28:0/1922165972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb80041c0 con 0x5591d2221430 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.900+0000 7efcb57fa640 1 -- 172.21.15.28:0/1922165972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb8003a50 con 0x5591d2221430 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- 172.21.15.28:0/1922165972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 msgr2=0x5591d2220270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7efcb80118d0 tx=0x7efcb8014000 comp rx=0 tx=0).stop 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- 172.21.15.28:0/1922165972 shutdown_connections 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2226f90 0x5591d223e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2221e20 0x5591d2232eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 --2- 172.21.15.28:0/1922165972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2220270 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- 172.21.15.28:0/1922165972 >> 172.21.15.28:0/1922165972 conn(0x5591d221c9a0 msgr2=0x5591d222acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- 172.21.15.28:0/1922165972 shutdown_connections 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- 172.21.15.28:0/1922165972 wait complete. 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.901+0000 7efcc8627740 1 -- start start 2024-03-22T02:16:05.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 Processor -- start 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2230f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2226f90 0x5591d2232480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01ab290 con 0x5591d2221430 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01ac700 con 0x5591d2226f90 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efcc01ab960 con 0x5591d2221e20 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcc8627740 10 client.0 _ll_register_callbacks cb 0x7efcb002dde0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2230f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2230f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42196/0 (socket says 172.21.15.28:42196) 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55292/0 (socket says 172.21.15.28:55292) 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbeffd640 1 -- 172.21.15.28:0/14849220 learned_addr learned my addr 172.21.15.28:0/14849220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbf7fe640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2226f90 0x5591d2232480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 -- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2226f90 msgr2=0x5591d2232480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2226f90 0x5591d2232480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 -- 172.21.15.28:0/14849220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 msgr2=0x5591d2230f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2230f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 -- 172.21.15.28:0/14849220 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591d2232a70 con 0x5591d2221e20 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efcbe7fc640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5591d2229c10 tx=0x7efca80a1170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3481a182831f3061 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efc9effd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.902+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca809f030 con 0x5591d2221e20 2024-03-22T02:16:05.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efcb80041c0 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7efca80a1680 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efca80a18a0 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 10 client.25984 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5591d229e500 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcb80041c0 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efca809b2a0 con 0x5591d2221e20 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 10 client.25984 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 10 client.25984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 20 client.25984 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:05.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d22a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 4 client.25984 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591d218e150 con 0x5591d2237840 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.903+0000 7efcbeffd640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d22a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcbeffd640 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d22a0b30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f3d6ee9378556c2 server_cookie=ffca9fd1a05e8138 in_seq=0 out_seq=0 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efc9effd640 10 client.25984 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efc9effd640 1 -- 172.21.15.28:0/14849220 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591d218e150 con 0x5591d2237840 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efc9effd640 10 client.25984 handle_command_reply: tid=0 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 client.25984 shutdown 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 msgr2=0x5591d2231470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5591d2229c10 tx=0x7efca80a1170 comp rx=0 tx=0).stop 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 shutdown_connections 2024-03-22T02:16:05.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5591d2237840 0x5591d22a0b30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5591d2226f90 0x5591d2232480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5591d2221e20 0x5591d2231470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efc9effd640 0 client.25984 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 --2- 172.21.15.28:0/14849220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591d2221430 0x5591d2230f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 >> 172.21.15.28:0/14849220 conn(0x5591d221c9a0 msgr2=0x5591d222a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 shutdown_connections 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 1 -- 172.21.15.28:0/14849220 wait complete. 2024-03-22T02:16:05.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.904+0000 7efcc8627740 20 client.25984 trim_cache size 0 max 16384 2024-03-22T02:16:05.915 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:16:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:16:05.915 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 msgr2=0x7efcac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 0x7efcac0a53e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efcc0069f80 tx=0x7efcb8005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:05.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 msgr2=0x7efcc0100df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7efca800e9f0 tx=0x7efca800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 shutdown_connections 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efcac0a2f20 0x7efcac0a53e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcc0103800 0x7efcc0101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcc0102e40 0x7efcc0100df0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 --2- 172.21.15.28:0/22926337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efcc00ffe70 0x7efcc01027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 >> 172.21.15.28:0/22926337 conn(0x7efcc00fb7a0 msgr2=0x7efcc0106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 shutdown_connections 2024-03-22T02:16:05.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:05.905+0000 7efcc5a39640 1 -- 172.21.15.28:0/22926337 wait complete. 2024-03-22T02:16:05.917 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:16:05.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:16:05.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:05 smithi028 ceph-mon[31916]: pgmap v433: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 650 KiB/s wr, 157 op/s 2024-03-22T02:16:05.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:05 smithi120 ceph-mon[36803]: pgmap v433: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 650 KiB/s wr, 157 op/s 2024-03-22T02:16:06.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:05 smithi083 ceph-mon[36340]: pgmap v433: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 650 KiB/s wr, 157 op/s 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 -- 172.21.15.28:0/895216194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe56810d7b0 msgr2=0x7fe56810dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 --2- 172.21.15.28:0/895216194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe56810d7b0 0x7fe56810dc30 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe550009900 tx=0x7fe55002bc40 comp rx=0 tx=0).stop 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 -- 172.21.15.28:0/895216194 shutdown_connections 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 --2- 172.21.15.28:0/895216194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe568114a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 --2- 172.21.15.28:0/895216194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe56810d7b0 0x7fe56810dc30 secure :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe550009900 tx=0x7fe55002bc40 comp rx=0 tx=0).stop 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 --2- 172.21.15.28:0/895216194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 0x7fe56810c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.032+0000 7fe56cc64640 1 -- 172.21.15.28:0/895216194 >> 172.21.15.28:0/895216194 conn(0x7fe56806c5f0 msgr2=0x7fe56806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.033+0000 7fe56cc64640 1 -- 172.21.15.28:0/895216194 shutdown_connections 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.033+0000 7fe56cc64640 1 -- 172.21.15.28:0/895216194 wait complete. 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 Processor -- start 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 -- start start 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 0x7fe5681a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe5681a74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 0x7fe5681ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe56806e440 con 0x7fe5681ac540 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe56806e2c0 con 0x7fe56810c5c0 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56cc64640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe56806e5c0 con 0x7fe56810e170 2024-03-22T02:16:06.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe5681a74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe55ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe5681a74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55312/0 (socket says 172.21.15.28:55312) 2024-03-22T02:16:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe55ffff640 1 -- 172.21.15.28:0/4080606235 learned_addr learned my addr 172.21.15.28:0/4080606235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe566d9d640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 0x7fe5681ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.034+0000 7fe56659c640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 0x7fe5681a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 -- 172.21.15.28:0/4080606235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 msgr2=0x7fe5681a74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe5681a74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 -- 172.21.15.28:0/4080606235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 msgr2=0x7fe5681a6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 0x7fe5681a6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 -- 172.21.15.28:0/4080606235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5681aefa0 con 0x7fe5681ac540 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe566d9d640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 0x7fe5681ae930 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe56000e940 tx=0x7fe56000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.035+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe560004d20 con 0x7fe5681ac540 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.036+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5681af290 con 0x7fe5681ac540 2024-03-22T02:16:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.036+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe560005020 con 0x7fe5681ac540 2024-03-22T02:16:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.036+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5681af7b0 con 0x7fe5681ac540 2024-03-22T02:16:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.036+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5600116d0 con 0x7fe5681ac540 2024-03-22T02:16:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.037+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe568068980 con 0x7fe5681ac540 2024-03-22T02:16:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.039+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe560011870 con 0x7fe5681ac540 2024-03-22T02:16:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.040+0000 7fe55f7fe640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 0x7fe5540a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.040+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe5600cfce0 con 0x7fe5681ac540 2024-03-22T02:16:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.041+0000 7fe56659c640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 0x7fe5540a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.043+0000 7fe56659c640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 0x7fe5540a54b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe56810d610 tx=0x7fe558009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.048+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe560016030 con 0x7fe5681ac540 2024-03-22T02:16:06.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.290+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5681110f0 con 0x7fe5681ac540 2024-03-22T02:16:06.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.291+0000 7fe55f7fe640 1 -- 172.21.15.28:0/4080606235 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11869 (secure 0 0 0) 0x7fe56009c6a0 con 0x7fe5681ac540 2024-03-22T02:16:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:16:06.292 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:15:24.687791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:16:06.292 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 msgr2=0x7fe5540a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 0x7fe5540a54b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe56810d610 tx=0x7fe558009110 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 msgr2=0x7fe5681ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 0x7fe5681ae930 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe56000e940 tx=0x7fe56000ed40 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 shutdown_connections 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5540a2ff0 0x7fe5540a54b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5681ac540 0x7fe5681ae930 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe56810e170 0x7fe5681a74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 --2- 172.21.15.28:0/4080606235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe56810c5c0 0x7fe5681a6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 >> 172.21.15.28:0/4080606235 conn(0x7fe56806c5f0 msgr2=0x7fe56810ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 shutdown_connections 2024-03-22T02:16:06.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.293+0000 7fe56cc64640 1 -- 172.21.15.28:0/4080606235 wait complete. 2024-03-22T02:16:06.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:16:06.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 33659 4.13 MB/sec warmup 48 sec latency 32.590 ms 2024-03-22T02:16:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.453+0000 7fd65884e640 1 -- 172.21.15.28:0/1677158991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 msgr2=0x7fd654109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.453+0000 7fd65884e640 1 --2- 172.21.15.28:0/1677158991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd654109cc0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009ca0 tx=0x7fd64c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 -- 172.21.15.28:0/1677158991 shutdown_connections 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 --2- 172.21.15.28:0/1677158991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd654109cc0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 --2- 172.21.15.28:0/1677158991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 0x7fd654107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 --2- 172.21.15.28:0/1677158991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 0x7fd654104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 -- 172.21.15.28:0/1677158991 >> 172.21.15.28:0/1677158991 conn(0x7fd6540fc020 msgr2=0x7fd6540fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 -- 172.21.15.28:0/1677158991 shutdown_connections 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 -- 172.21.15.28:0/1677158991 wait complete. 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 Processor -- start 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 -- start start 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.454+0000 7fd65884e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 0x7fd654197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 0x7fd6541983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd65410d830 con 0x7fd654107870 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd65410d6b0 con 0x7fd654104f40 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd65410d9b0 con 0x7fd654102610 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd651d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 0x7fd6541983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 0x7fd654197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42238/0 (socket says 172.21.15.28:42238) 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 -- 172.21.15.28:0/2552020868 learned_addr learned my addr 172.21.15.28:0/2552020868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 -- 172.21.15.28:0/2552020868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 msgr2=0x7fd654197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 0x7fd654197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 -- 172.21.15.28:0/2552020868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 msgr2=0x7fd6541983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 0x7fd6541983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 -- 172.21.15.28:0/2552020868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65419fad0 con 0x7fd654107870 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd652d9d640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd64c037040 tx=0x7fd64c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd6437fe640 1 -- 172.21.15.28:0/2552020868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c041020 con 0x7fd654107870 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65419fd60 con 0x7fd654107870 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd6437fe640 1 -- 172.21.15.28:0/2552020868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd64c034cd0 con 0x7fd654107870 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd6437fe640 1 -- 172.21.15.28:0/2552020868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c039040 con 0x7fd654107870 2024-03-22T02:16:06.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.455+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6541a0240 con 0x7fd654107870 2024-03-22T02:16:06.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.458+0000 7fd6437fe640 1 -- 172.21.15.28:0/2552020868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd64c044020 con 0x7fd654107870 2024-03-22T02:16:06.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.458+0000 7fd65b432740 1 Processor -- start 2024-03-22T02:16:06.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.458+0000 7fd65b432740 1 -- start start 2024-03-22T02:16:06.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.458+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 0x7fd63c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff7676d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff7679810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bff75d68d0 con 0x55bff7671820 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bff757ee00 con 0x55bff7672210 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65b432740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bff75d2880 con 0x55bff7677380 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42240/0 (socket says 172.21.15.28:42240) 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd651d9b640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 0x7fd63c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd6437fe640 1 -- 172.21.15.28:0/2552020868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd64c0f1640 con 0x7fd654107870 2024-03-22T02:16:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 -- 172.21.15.28:0/335083147 learned_addr learned my addr 172.21.15.28:0/335083147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd652d9d640 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff7679810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd651d9b640 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff7676d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 -- 172.21.15.28:0/335083147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 msgr2=0x55bff7679810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff7679810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 -- 172.21.15.28:0/335083147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 msgr2=0x55bff7676d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff7676d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 -- 172.21.15.28:0/335083147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bff72ea3f0 con 0x55bff7671820 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd65259c640 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd644009d20 tx=0x7fd644011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0081fca4ab7021 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd651d9b640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 0x7fd63c0a53e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6541993c0 tx=0x7fd648011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.459+0000 7fd640ff9640 1 -- 172.21.15.28:0/335083147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64401e070 con 0x55bff7671820 2024-03-22T02:16:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd640ff9640 1 -- 172.21.15.28:0/335083147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd644019b10 con 0x55bff7671820 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd640ff9640 1 -- 172.21.15.28:0/335083147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64400eb40 con 0x55bff7671820 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 -- 172.21.15.28:0/335083147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 msgr2=0x55bff7671c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd644009d20 tx=0x7fd644011810 comp rx=0 tx=0).stop 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 -- 172.21.15.28:0/335083147 shutdown_connections 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff7679810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff7676d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 --2- 172.21.15.28:0/335083147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff7671c40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 -- 172.21.15.28:0/335083147 >> 172.21.15.28:0/335083147 conn(0x55bff766cd90 msgr2=0x55bff766d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 -- 172.21.15.28:0/335083147 shutdown_connections 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.460+0000 7fd65b432740 1 -- 172.21.15.28:0/335083147 wait complete. 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 -- start start 2024-03-22T02:16:06.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 Processor -- start 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 0x55bff767c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76832e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bff767a330 con 0x55bff7671820 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bff7592300 con 0x55bff7677380 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bff767a4b0 con 0x55bff7672210 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65b432740 10 client.0 _ll_register_callbacks cb 0x55bff75cde30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42254/0 (socket says 172.21.15.28:42254) 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd65259c640 1 -- 172.21.15.28:0/4008771330 learned_addr learned my addr 172.21.15.28:0/4008771330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd651d9b640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 0x55bff767c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.461+0000 7fd652d9d640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76832e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 -- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 msgr2=0x55bff767c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 0x55bff767c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 -- 172.21.15.28:0/4008771330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 msgr2=0x55bff76832e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76832e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 -- 172.21.15.28:0/4008771330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bff73b47d0 con 0x55bff7671820 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd651d9b640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 0x55bff767c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65259c640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd644009cf0 tx=0x7fd644009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6fcc9d24092f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644026070 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd644019840 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.462+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd644019b10 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd644032050 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 10 client.16170 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bff7686650 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd644019b10 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd6440046c0 con 0x55bff7671820 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 10 client.16170 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 10 client.16170 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 20 client.16170 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff7687a10 0x55bff7689e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 4 client.16170 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bff75e3e10 con 0x55bff7687a10 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.463+0000 7fd651d9b640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff7687a10 0x55bff7689e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.464+0000 7fd639ffb640 10 client.16170 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.464+0000 7fd651d9b640 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff7687a10 0x55bff7689e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43b20f46af4ac9ea server_cookie=48e02e7f4899bf0f in_seq=0 out_seq=0 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd639ffb640 1 -- 172.21.15.28:0/4008771330 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bff75e3e10 con 0x55bff7687a10 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd639ffb640 10 client.16170 handle_command_reply: tid=0 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 client.16170 shutdown 2024-03-22T02:16:06.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 msgr2=0x55bff767bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd644009cf0 tx=0x7fd644009f60 comp rx=0 tx=0).stop 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 shutdown_connections 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff7687a10 0x55bff7689e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76832e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7672210 0x55bff767c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd639ffb640 0 client.16170 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 --2- 172.21.15.28:0/4008771330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767bd60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 >> 172.21.15.28:0/4008771330 conn(0x55bff766cd90 msgr2=0x55bff766bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 shutdown_connections 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 1 -- 172.21.15.28:0/4008771330 wait complete. 2024-03-22T02:16:06.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.465+0000 7fd65b432740 20 client.16170 trim_cache size 0 max 16384 2024-03-22T02:16:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 Processor -- start 2024-03-22T02:16:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 -- start start 2024-03-22T02:16:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff767b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff767be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541ad280 con 0x55bff7671820 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541aab50 con 0x55bff7672210 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65b432740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541a89a0 con 0x55bff7677380 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd651d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff767b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42270/0 (socket says 172.21.15.28:42270) 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 -- 172.21.15.28:0/689721748 learned_addr learned my addr 172.21.15.28:0/689721748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd652d9d640 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff767be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 -- 172.21.15.28:0/689721748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 msgr2=0x55bff767be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff767be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 -- 172.21.15.28:0/689721748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 msgr2=0x55bff767b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff767b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.479+0000 7fd65259c640 1 -- 172.21.15.28:0/689721748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bff767a210 con 0x55bff7671820 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd65259c640 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd644005e00 tx=0x7fd644005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2073045dbc7bc01c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd63bfff640 1 -- 172.21.15.28:0/689721748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64402f020 con 0x55bff7671820 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd63bfff640 1 -- 172.21.15.28:0/689721748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd644019d20 con 0x55bff7671820 2024-03-22T02:16:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd63bfff640 1 -- 172.21.15.28:0/689721748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644026040 con 0x55bff7671820 2024-03-22T02:16:06.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd65b432740 1 -- 172.21.15.28:0/689721748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 msgr2=0x55bff767b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.480+0000 7fd65b432740 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd644005e00 tx=0x7fd644005850 comp rx=0 tx=0).stop 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- 172.21.15.28:0/689721748 shutdown_connections 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7677380 0x55bff767be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7672210 0x55bff767b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- 172.21.15.28:0/689721748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7671820 0x55bff767b380 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- 172.21.15.28:0/689721748 >> 172.21.15.28:0/689721748 conn(0x55bff766cd90 msgr2=0x55bff7670880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- 172.21.15.28:0/689721748 shutdown_connections 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- 172.21.15.28:0/689721748 wait complete. 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- start start 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 Processor -- start 2024-03-22T02:16:06.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 0x55bff76e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76f0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541a9010 con 0x55bff7672210 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541aa480 con 0x55bff7677380 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541a96e0 con 0x55bff7671820 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65b432740 10 client.0 _ll_register_callbacks cb 0x7fd64401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd651d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd65259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 0x55bff76e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd651d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42272/0 (socket says 172.21.15.28:42272) 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd651d9b640 1 -- 172.21.15.28:0/190760082 learned_addr learned my addr 172.21.15.28:0/190760082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.481+0000 7fd652d9d640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 -- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 msgr2=0x55bff76e90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 0x55bff76e90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 -- 172.21.15.28:0/190760082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 msgr2=0x55bff76f0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76f0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 -- 172.21.15.28:0/190760082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bff76f0c30 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd65259c640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 0x55bff76e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd651d9b640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd648005fd0 tx=0x7fd648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57d3c2f0c0239f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd639ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64802d070 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd648029b10 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd644019d20 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd648039050 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd65b432740 10 client.16176 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bff76f2030 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.482+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd644019d20 con 0x55bff7672210 2024-03-22T02:16:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd648029040 con 0x55bff7672210 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 10 client.16176 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 10 client.16176 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 20 client.16176 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff76f2a10 0x55bff76f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 4 client.16176 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bff75de160 con 0x55bff76f2a10 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65259c640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff76f2a10 0x55bff76f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd639ffb640 10 client.16176 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.483+0000 7fd65259c640 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff76f2a10 0x55bff76f4e30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67084cee07c5b8ba server_cookie=ba8dcd1ffcf777cc in_seq=0 out_seq=0 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd639ffb640 1 -- 172.21.15.28:0/190760082 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bff75de160 con 0x55bff76f2a10 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd639ffb640 10 client.16176 handle_command_reply: tid=0 2024-03-22T02:16:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 client.16176 shutdown 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 msgr2=0x55bff76e9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fd648005fd0 tx=0x7fd648004730 comp rx=0 tx=0).stop 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 shutdown_connections 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55bff76f2a10 0x55bff76f4e30 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bff7677380 0x55bff76f0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd639ffb640 0 client.16176 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bff7672210 0x55bff76e9600 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 --2- 172.21.15.28:0/190760082 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bff7671820 0x55bff76e90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 >> 172.21.15.28:0/190760082 conn(0x55bff766cd90 msgr2=0x55bff766b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 shutdown_connections 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 1 -- 172.21.15.28:0/190760082 wait complete. 2024-03-22T02:16:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.484+0000 7fd65b432740 20 client.16176 trim_cache size 0 max 16384 2024-03-22T02:16:06.492 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 msgr2=0x7fd63c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 0x7fd63c0a53e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd6541993c0 tx=0x7fd648011810 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 msgr2=0x7fd65419f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd64c037040 tx=0x7fd64c032f40 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 shutdown_connections 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd63c0a2f20 0x7fd63c0a53e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd654107870 0x7fd65419f460 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd654104f40 0x7fd6541983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 --2- 172.21.15.28:0/2552020868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654102610 0x7fd654197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 >> 172.21.15.28:0/2552020868 conn(0x7fd6540fc020 msgr2=0x7fd654102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 shutdown_connections 2024-03-22T02:16:06.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.485+0000 7fd65884e640 1 -- 172.21.15.28:0/2552020868 wait complete. 2024-03-22T02:16:06.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:16:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 -- 172.21.15.28:0/713409461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 msgr2=0x7ffac4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 --2- 172.21.15.28:0/713409461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4100eb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ffab8009900 tx=0x7ffab802bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 -- 172.21.15.28:0/713409461 shutdown_connections 2024-03-22T02:16:06.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 --2- 172.21.15.28:0/713409461 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4102640 0x7ffac4108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 --2- 172.21.15.28:0/713409461 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4101c80 0x7ffac4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 --2- 172.21.15.28:0/713409461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4100eb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 -- 172.21.15.28:0/713409461 >> 172.21.15.28:0/713409461 conn(0x7ffac40fc020 msgr2=0x7ffac40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 -- 172.21.15.28:0/713409461 shutdown_connections 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.605+0000 7ffacad77640 1 -- 172.21.15.28:0/713409461 wait complete. 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 Processor -- start 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 -- start start 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4101c80 0x7ffac4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4102640 0x7ffac419f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffac410c700 con 0x7ffac4100a90 2024-03-22T02:16:06.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffac410c580 con 0x7ffac4101c80 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffacad77640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffac410c880 con 0x7ffac4102640 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42290/0 (socket says 172.21.15.28:42290) 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 -- 172.21.15.28:0/2367081169 learned_addr learned my addr 172.21.15.28:0/2367081169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4101c80 0x7ffac4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac9314640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4102640 0x7ffac419f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 -- 172.21.15.28:0/2367081169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4102640 msgr2=0x7ffac419f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4102640 0x7ffac419f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 -- 172.21.15.28:0/2367081169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4101c80 msgr2=0x7ffac4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4101c80 0x7ffac4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 -- 172.21.15.28:0/2367081169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac419fb40 con 0x7ffac4100a90 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ffab8000c00 tx=0x7ffab802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.606+0000 7ffac1ffb640 1 -- 172.21.15.28:0/2367081169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8041070 con 0x7ffac4100a90 2024-03-22T02:16:06.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.607+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac419fdd0 con 0x7ffac4100a90 2024-03-22T02:16:06.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.607+0000 7ffac1ffb640 1 -- 172.21.15.28:0/2367081169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab8004450 con 0x7ffac4100a90 2024-03-22T02:16:06.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.607+0000 7ffac1ffb640 1 -- 172.21.15.28:0/2367081169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8034440 con 0x7ffac4100a90 2024-03-22T02:16:06.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.607+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac41a0230 con 0x7ffac4100a90 2024-03-22T02:16:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.609+0000 7ffac1ffb640 1 -- 172.21.15.28:0/2367081169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ffab8044070 con 0x7ffac4100a90 2024-03-22T02:16:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.609+0000 7ffacda36740 1 Processor -- start 2024-03-22T02:16:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.609+0000 7ffacda36740 1 -- start start 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105533210 0x560105537d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 0x56010553a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601054978d0 con 0x560105538380 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56010543fe00 con 0x560105532820 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffacda36740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560105493880 con 0x560105533210 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac1ffb640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 0x7ffab40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 0x56010553a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105533210 0x560105537d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56716/0 (socket says 172.21.15.28:56716) 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 -- 172.21.15.28:0/2834443884 learned_addr learned my addr 172.21.15.28:0/2834443884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 0x7ffab40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac1ffb640 1 -- 172.21.15.28:0/2367081169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffab80f12e0 con 0x7ffac4100a90 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 -- 172.21.15.28:0/2834443884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105533210 msgr2=0x560105537d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105533210 0x560105537d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 -- 172.21.15.28:0/2834443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 msgr2=0x56010553a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 0x56010553a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac9314640 1 -- 172.21.15.28:0/2834443884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601051ab3f0 con 0x560105532820 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.610+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 0x56010553a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffac9314640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 0x7ffab40a53e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00fc90 tx=0x7ffabc01e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffac9314640 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffabc011e20 tx=0x7ffabc00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac964fdcce68f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffaaeffd640 1 -- 172.21.15.28:0/2834443884 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc01dbf0 con 0x560105532820 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffaaeffd640 1 -- 172.21.15.28:0/2834443884 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffabc01dd90 con 0x560105532820 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffaaeffd640 1 -- 172.21.15.28:0/2834443884 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc004280 con 0x560105532820 2024-03-22T02:16:06.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffacda36740 1 -- 172.21.15.28:0/2834443884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 msgr2=0x560105532c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.611+0000 7ffacda36740 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffabc011e20 tx=0x7ffabc00f790 comp rx=0 tx=0).stop 2024-03-22T02:16:06.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- 172.21.15.28:0/2834443884 shutdown_connections 2024-03-22T02:16:06.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105538380 0x56010553a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105533210 0x560105537d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- 172.21.15.28:0/2834443884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x560105532c40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- 172.21.15.28:0/2834443884 >> 172.21.15.28:0/2834443884 conn(0x56010552dd90 msgr2=0x56010552e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- 172.21.15.28:0/2834443884 shutdown_connections 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- 172.21.15.28:0/2834443884 wait complete. 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- start start 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 Processor -- start 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x56010553d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56010553b270 con 0x560105533210 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560105453300 con 0x560105532820 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56010553b3f0 con 0x560105538380 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.612+0000 7ffacda36740 10 client.0 _ll_register_callbacks cb 0x56010548ee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac9314640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x56010553d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56732/0 (socket says 172.21.15.28:56732) 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac9314640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55426/0 (socket says 172.21.15.28:55426) 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 -- 172.21.15.28:0/3182346603 learned_addr learned my addr 172.21.15.28:0/3182346603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 -- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 msgr2=0x560105544250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 -- 172.21.15.28:0/3182346603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 msgr2=0x56010553d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x56010553d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 -- 172.21.15.28:0/3182346603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601052757d0 con 0x560105532820 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac9314640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac3fff640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x56010553d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffac8b13640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffab802cfd0 tx=0x7ffab8041000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c781429850746511 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffaa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab804b040 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffabc01dd90 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffab80c14b0 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffab80bd510 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffacda36740 10 client.26021 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.613+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5601055475c0 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffabc01dd90 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffab80a8080 con 0x560105532820 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 10 client.26021 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 10 client.26021 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 20 client.26021 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x560105548c00 0x56010554b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 4 client.26021 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601054a4e10 con 0x560105548c00 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffac3fff640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x560105548c00 0x56010554b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffaa3fff640 10 client.26021 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.614+0000 7ffac3fff640 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x560105548c00 0x56010554b050 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7cec7b051f888c9 server_cookie=330e73c2a76ab564 in_seq=0 out_seq=0 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffaa3fff640 1 -- 172.21.15.28:0/3182346603 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601054a4e10 con 0x560105548c00 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffaa3fff640 10 client.26021 handle_command_reply: tid=0 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 client.26021 shutdown 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 msgr2=0x56010553ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffab802cfd0 tx=0x7ffab8041000 comp rx=0 tx=0).stop 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 shutdown_connections 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x560105548c00 0x56010554b050 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105544250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x56010553d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffaa3fff640 0 client.26021 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 --2- 172.21.15.28:0/3182346603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x56010553ccd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.615+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 >> 172.21.15.28:0/3182346603 conn(0x56010552dd90 msgr2=0x56010552cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.616+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 shutdown_connections 2024-03-22T02:16:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.616+0000 7ffacda36740 1 -- 172.21.15.28:0/3182346603 wait complete. 2024-03-22T02:16:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.616+0000 7ffacda36740 20 client.26021 trim_cache size 0 max 16384 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 Processor -- start 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 -- start start 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105534cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41ad190 con 0x560105533210 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41aaa60 con 0x560105532820 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.629+0000 7ffacda36740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41a88b0 con 0x560105538380 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac8b13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42314/0 (socket says 172.21.15.28:42314) 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 -- 172.21.15.28:0/2020747787 learned_addr learned my addr 172.21.15.28:0/2020747787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac9314640 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105534cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 -- 172.21.15.28:0/2020747787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 msgr2=0x560105534cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105534cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 -- 172.21.15.28:0/2020747787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 msgr2=0x5601055347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055347b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 -- 172.21.15.28:0/2020747787 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56010554b6a0 con 0x560105533210 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x56010553e1b0 tx=0x7ffab0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc22dfce3aa2042 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffaae7fc640 1 -- 172.21.15.28:0/2020747787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab000f870 con 0x560105533210 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffaae7fc640 1 -- 172.21.15.28:0/2020747787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab0022170 con 0x560105533210 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.630+0000 7ffaae7fc640 1 -- 172.21.15.28:0/2020747787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0010690 con 0x560105533210 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- 172.21.15.28:0/2020747787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 msgr2=0x560105544210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x56010553e1b0 tx=0x7ffab0005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:06.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- 172.21.15.28:0/2020747787 shutdown_connections 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x560105534cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105544210 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 --2- 172.21.15.28:0/2020747787 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- 172.21.15.28:0/2020747787 >> 172.21.15.28:0/2020747787 conn(0x56010552dd90 msgr2=0x56010552e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- 172.21.15.28:0/2020747787 shutdown_connections 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- 172.21.15.28:0/2020747787 wait complete. 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 -- start start 2024-03-22T02:16:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.631+0000 7ffacda36740 1 Processor -- start 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055412f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x5601055427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41a8f20 con 0x560105533210 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41aa390 con 0x560105532820 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffac41a95f0 con 0x560105538380 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffacda36740 10 client.0 _ll_register_callbacks cb 0x7ffab80b9320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42316/0 (socket says 172.21.15.28:42316) 2024-03-22T02:16:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 -- 172.21.15.28:0/2490704652 learned_addr learned my addr 172.21.15.28:0/2490704652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac9314640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x5601055427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 -- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 msgr2=0x5601055427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x5601055427d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 -- 172.21.15.28:0/2490704652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 msgr2=0x5601055412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 -- 172.21.15.28:0/2490704652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560105542dc0 con 0x560105533210 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffac3fff640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffab0014010 tx=0x7ffab0012190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee2c3a2f7ab64a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.632+0000 7ffaa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab002f020 con 0x560105533210 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffab000fc80 con 0x560105533210 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffab0022170 con 0x560105533210 2024-03-22T02:16:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffab003a050 con 0x560105533210 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 10 client.16194 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560105544130 con 0x560105533210 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffab0022170 con 0x560105533210 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffab0004700 con 0x560105533210 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 10 client.16194 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 10 client.16194 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 20 client.16194 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5601055af320 0x5601055b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 4 client.16194 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56010549f160 con 0x5601055af320 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.633+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5601055af320 0x5601055b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffac8b13640 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5601055af320 0x5601055b1740 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c74d78c04132846c server_cookie=b45c152041708317 in_seq=0 out_seq=0 2024-03-22T02:16:06.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffaa3fff640 10 client.16194 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffaa3fff640 1 -- 172.21.15.28:0/2490704652 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56010549f160 con 0x5601055af320 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffaa3fff640 10 client.16194 handle_command_reply: tid=0 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 client.16194 shutdown 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 msgr2=0x560105541830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ffab0014010 tx=0x7ffab0012190 comp rx=0 tx=0).stop 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 shutdown_connections 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5601055af320 0x5601055b1740 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560105538380 0x5601055427d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffaa3fff640 0 client.16194 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560105533210 0x560105541830 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 --2- 172.21.15.28:0/2490704652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560105532820 0x5601055412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 >> 172.21.15.28:0/2490704652 conn(0x56010552dd90 msgr2=0x5601055347b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 shutdown_connections 2024-03-22T02:16:06.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 1 -- 172.21.15.28:0/2490704652 wait complete. 2024-03-22T02:16:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.634+0000 7ffacda36740 20 client.16194 trim_cache size 0 max 16384 2024-03-22T02:16:06.642 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 msgr2=0x7ffab40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 0x7ffab40a53e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00fc90 tx=0x7ffabc01e000 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 msgr2=0x7ffac4197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7ffab8000c00 tx=0x7ffab802c640 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 shutdown_connections 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffab40a2f20 0x7ffab40a53e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffac4102640 0x7ffac419f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffac4101c80 0x7ffac4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 --2- 172.21.15.28:0/2367081169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffac4100a90 0x7ffac4197f20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 >> 172.21.15.28:0/2367081169 conn(0x7ffac40fc020 msgr2=0x7ffac40fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 shutdown_connections 2024-03-22T02:16:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.635+0000 7ffacad77640 1 -- 172.21.15.28:0/2367081169 wait complete. 2024-03-22T02:16:06.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:16:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 -- 172.21.15.28:0/783777531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 msgr2=0x7fd0f0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 --2- 172.21.15.28:0/783777531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0107350 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009900 tx=0x7fd0e402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 -- 172.21.15.28:0/783777531 shutdown_connections 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 --2- 172.21.15.28:0/783777531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0107890 0x7fd0f0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 --2- 172.21.15.28:0/783777531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0107350 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 --2- 172.21.15.28:0/783777531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0102630 0x7fd0f0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 -- 172.21.15.28:0/783777531 >> 172.21.15.28:0/783777531 conn(0x7fd0f00fc040 msgr2=0x7fd0f00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 -- 172.21.15.28:0/783777531 shutdown_connections 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.758+0000 7fd0f604e640 1 -- 172.21.15.28:0/783777531 wait complete. 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 Processor -- start 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 -- start start 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102630 0x7fd0f0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0107890 0x7fd0f019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f010d840 con 0x7fd0f0104f60 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f010d6c0 con 0x7fd0f0102630 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0f604e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f010d9c0 con 0x7fd0f0107890 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42324/0 (socket says 172.21.15.28:42324) 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102630 0x7fd0f0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0eeffd640 1 -- 172.21.15.28:0/610807536 learned_addr learned my addr 172.21.15.28:0/610807536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.759+0000 7fd0effff640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0107890 0x7fd0f019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 -- 172.21.15.28:0/610807536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0107890 msgr2=0x7fd0f019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0107890 0x7fd0f019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 -- 172.21.15.28:0/610807536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102630 msgr2=0x7fd0f0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102630 0x7fd0f0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 -- 172.21.15.28:0/610807536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f019fb30 con 0x7fd0f0104f60 2024-03-22T02:16:06.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4000c00 tx=0x7fd0e402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0ecff9640 1 -- 172.21.15.28:0/610807536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4041070 con 0x7fd0f0104f60 2024-03-22T02:16:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f019fdc0 con 0x7fd0f0104f60 2024-03-22T02:16:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0ecff9640 1 -- 172.21.15.28:0/610807536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e4004450 con 0x7fd0f0104f60 2024-03-22T02:16:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0ecff9640 1 -- 172.21.15.28:0/610807536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4034440 con 0x7fd0f0104f60 2024-03-22T02:16:06.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.760+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f01a0250 con 0x7fd0f0104f60 2024-03-22T02:16:06.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.762+0000 7fd0ecff9640 1 -- 172.21.15.28:0/610807536 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd0e4044070 con 0x7fd0f0104f60 2024-03-22T02:16:06.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 Processor -- start 2024-03-22T02:16:06.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 -- start start 2024-03-22T02:16:06.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ecff9640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 0x7fd0d80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 0x5641fc9e6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9e8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ecff9640 1 -- 172.21.15.28:0/610807536 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0e40f1300 con 0x7fd0f0104f60 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0effff640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 0x7fd0d80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641fc9418d0 con 0x5641fc9e0c30 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5641fc8e9e00 con 0x5641fc9e1620 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5641fc93d880 con 0x5641fc9e6790 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42334/0 (socket says 172.21.15.28:42334) 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 -- 172.21.15.28:0/3619050980 learned_addr learned my addr 172.21.15.28:0/3619050980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 0x5641fc9e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0effff640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9e8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 -- 172.21.15.28:0/3619050980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 msgr2=0x5641fc9e8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9e8c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 -- 172.21.15.28:0/3619050980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 msgr2=0x5641fc9e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 0x5641fc9e6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0ef7fe640 1 -- 172.21.15.28:0/3619050980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641fc6553f0 con 0x5641fc9e0c30 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.763+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 0x5641fc9e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.764+0000 7fd0effff640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 0x7fd0d80a53e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8005fd0 tx=0x7fd0e8009e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.764+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009d20 tx=0x7fd0e0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec14a96ed0e9bb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.764+0000 7fd0d5ffb640 1 -- 172.21.15.28:0/3619050980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0019620 con 0x5641fc9e0c30 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.764+0000 7fd0d5ffb640 1 -- 172.21.15.28:0/3619050980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009e10 con 0x5641fc9e0c30 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.764+0000 7fd0d5ffb640 1 -- 172.21.15.28:0/3619050980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0018b20 con 0x5641fc9e0c30 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- 172.21.15.28:0/3619050980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 msgr2=0x5641fc9e1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009d20 tx=0x7fd0e0011810 comp rx=0 tx=0).stop 2024-03-22T02:16:06.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- 172.21.15.28:0/3619050980 shutdown_connections 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9e8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e1620 0x5641fc9e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/3619050980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9e1050 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- 172.21.15.28:0/3619050980 >> 172.21.15.28:0/3619050980 conn(0x5641fc9d7d90 msgr2=0x5641fc9d8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- 172.21.15.28:0/3619050980 shutdown_connections 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- 172.21.15.28:0/3619050980 wait complete. 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- start start 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 Processor -- start 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e0c30 0x5641fc9eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9eb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5641fc9e9740 con 0x5641fc9e1620 2024-03-22T02:16:06.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5641fc8fd300 con 0x5641fc9e6790 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5641fc9e98c0 con 0x5641fc9e0c30 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.765+0000 7fd0f8c34740 10 client.0 _ll_register_callbacks cb 0x5641fc938e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e0c30 0x5641fc9eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56802/0 (socket says 172.21.15.28:56802) 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0effff640 1 -- 172.21.15.28:0/1489206030 learned_addr learned my addr 172.21.15.28:0/1489206030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 -- 172.21.15.28:0/1489206030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e0c30 msgr2=0x5641fc9eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e0c30 0x5641fc9eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 -- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 msgr2=0x5641fc9f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 -- 172.21.15.28:0/1489206030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641fc71f7d0 con 0x5641fc9e1620 2024-03-22T02:16:06.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9eb600 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd0e404c000 tx=0x7fd0e40bd4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6946c428a4d72170 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e404b040 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0e40035f0 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e0009e10 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd0e4004c80 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0f8c34740 10 client.16206 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.766+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641fc9f59b0 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e0009e10 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd0e40c5b40 con 0x5641fc9e1620 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 10 client.16206 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 20 client.16206 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fc9f7090 0x5641fc9f9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 4 client.16206 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:06.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641fc94ee10 con 0x5641fc9f7090 2024-03-22T02:16:06.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fc9f7090 0x5641fc9f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0ceffd640 10 client.16206 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:06.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.767+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fc9f7090 0x5641fc9f9510 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=327da0bae190de25 server_cookie=ca79cca25aefaea7 in_seq=0 out_seq=0 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0ceffd640 1 -- 172.21.15.28:0/1489206030 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5641fc94ee10 con 0x5641fc9f7090 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0ceffd640 10 client.16206 handle_command_reply: tid=0 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 client.16206 shutdown 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 msgr2=0x5641fc9eb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9eb600 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd0e404c000 tx=0x7fd0e40bd4d0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 shutdown_connections 2024-03-22T02:16:06.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fc9f7090 0x5641fc9f9510 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0ceffd640 0 client.16206 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9eb600 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/1489206030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e0c30 0x5641fc9eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 >> 172.21.15.28:0/1489206030 conn(0x5641fc9d7d90 msgr2=0x5641fc9d6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 shutdown_connections 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 1 -- 172.21.15.28:0/1489206030 wait complete. 2024-03-22T02:16:06.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.769+0000 7fd0f8c34740 20 client.16206 trim_cache size 0 max 16384 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.783+0000 7fd0f8c34740 1 Processor -- start 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.783+0000 7fd0f8c34740 1 -- start start 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.783+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e0c30 0x5641fc9fde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.783+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01ab090 con 0x5641fc9e1620 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a8960 con 0x5641fc9e0c30 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a67b0 con 0x5641fc9e6790 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42348/0 (socket says 172.21.15.28:42348) 2024-03-22T02:16:06.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 -- 172.21.15.28:0/2914846132 learned_addr learned my addr 172.21.15.28:0/2914846132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e0c30 0x5641fc9fde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0effff640 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 -- 172.21.15.28:0/2914846132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 msgr2=0x5641fc9fe350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9fe350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 -- 172.21.15.28:0/2914846132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e0c30 msgr2=0x5641fc9fde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e0c30 0x5641fc9fde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 -- 172.21.15.28:0/2914846132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641fc9d8b00 con 0x5641fc9e1620 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40bb9b0 tx=0x7fd0e40a8280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50b12435f8f4d71 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0d57fa640 1 -- 172.21.15.28:0/2914846132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40c5fd0 con 0x5641fc9e1620 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0d57fa640 1 -- 172.21.15.28:0/2914846132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e40eee50 con 0x5641fc9e1620 2024-03-22T02:16:06.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.784+0000 7fd0d57fa640 1 -- 172.21.15.28:0/2914846132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40bcb50 con 0x5641fc9e1620 2024-03-22T02:16:06.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 -- 172.21.15.28:0/2914846132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 msgr2=0x5641fc9f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40bb9b0 tx=0x7fd0e40a8280 comp rx=0 tx=0).stop 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 -- 172.21.15.28:0/2914846132 shutdown_connections 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e6790 0x5641fc9fe350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e1620 0x5641fc9f2600 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/2914846132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e0c30 0x5641fc9fde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 -- 172.21.15.28:0/2914846132 >> 172.21.15.28:0/2914846132 conn(0x5641fc9d7d90 msgr2=0x5641fc9db880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 -- 172.21.15.28:0/2914846132 shutdown_connections 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.785+0000 7fd0f8c34740 1 -- 172.21.15.28:0/2914846132 wait complete. 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0f8c34740 1 -- start start 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0f8c34740 1 Processor -- start 2024-03-22T02:16:06.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 0x5641fc9f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0f8c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.786+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a6e20 con 0x5641fc9e0c30 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a8290 con 0x5641fc9e6790 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0f8c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f01a74f0 con 0x5641fc9e1620 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9f1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42354/0 (socket says 172.21.15.28:42354) 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0ef7fe640 1 -- 172.21.15.28:0/826193914 learned_addr learned my addr 172.21.15.28:0/826193914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 0x5641fc9f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0f8c34740 10 client.0 _ll_register_callbacks cb 0x7fd0e40b99a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0effff640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 -- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 msgr2=0x5641fc9f04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 -- 172.21.15.28:0/826193914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 msgr2=0x5641fc9f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:06.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 -- 172.21.15.28:0/826193914 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641fc9f0b20 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0effff640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0eeffd640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 0x5641fc9f23e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4040000 tx=0x7fd0e402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10a56ab075697b3f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40bd700 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd0e40b9450 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.787+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0e40eee50 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd0e40c5fd0 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 10 client.26044 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641fc9f0b20 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0e40eee50 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd0e40dd460 con 0x5641fc9e1620 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 10 client.26044 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:06.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 10 client.26044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 20 client.26044 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fca5e390 0x5641fca607b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 4 client.26044 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5641fc9eaeb0 con 0x5641fca5e390 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.788+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fca5e390 0x5641fca607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.789+0000 7fd0ceffd640 10 client.26044 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.789+0000 7fd0ef7fe640 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fca5e390 0x5641fca607b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6423ea53534be3f4 server_cookie=ab796902b1d75f4d in_seq=0 out_seq=0 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0ceffd640 1 -- 172.21.15.28:0/826193914 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5641fc9eaeb0 con 0x5641fca5e390 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0ceffd640 10 client.26044 handle_command_reply: tid=0 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 client.26044 shutdown 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 msgr2=0x5641fc9f23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 0x5641fc9f23e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4040000 tx=0x7fd0e402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:06.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 shutdown_connections 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5641fca5e390 0x5641fca607b0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5641fc9e6790 0x5641fc9f04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0ceffd640 0 client.26044 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5641fc9e1620 0x5641fc9f23e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 --2- 172.21.15.28:0/826193914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5641fc9e0c30 0x5641fc9f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 >> 172.21.15.28:0/826193914 conn(0x5641fc9d7d90 msgr2=0x5641fc9d6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 shutdown_connections 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 1 -- 172.21.15.28:0/826193914 wait complete. 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.790+0000 7fd0f8c34740 20 client.26044 trim_cache size 0 max 16384 2024-03-22T02:16:06.795 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 msgr2=0x7fd0d80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 0x7fd0d80a53e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8005fd0 tx=0x7fd0e8009e60 comp rx=0 tx=0).stop 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 msgr2=0x7fd0f0198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4000c00 tx=0x7fd0e402c640 comp rx=0 tx=0).stop 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 shutdown_connections 2024-03-22T02:16:06.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0d80a2f20 0x7fd0d80a53e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f0107890 0x7fd0f019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0f0104f60 0x7fd0f0198450 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 --2- 172.21.15.28:0/610807536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0f0102630 0x7fd0f0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 >> 172.21.15.28:0/610807536 conn(0x7fd0f00fc040 msgr2=0x7fd0f01030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 shutdown_connections 2024-03-22T02:16:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:06.791+0000 7fd0f604e640 1 -- 172.21.15.28:0/610807536 wait complete. 2024-03-22T02:16:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:06 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4080606235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:06 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:16:07.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:06 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4080606235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:06 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:16:07.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:06 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4080606235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:07.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:06 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:16:07.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 35969 4.33 MB/sec warmup 49 sec latency 16.528 ms 2024-03-22T02:16:07.803 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:16:07.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:16:07.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:16:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 -- 172.21.15.28:0/1308737484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 msgr2=0x7f1448108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 --2- 172.21.15.28:0/1308737484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f1448108ed0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f1440009ca0 tx=0x7f144002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 -- 172.21.15.28:0/1308737484 shutdown_connections 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 --2- 172.21.15.28:0/1308737484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f1448108ed0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 --2- 172.21.15.28:0/1308737484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 0x7f1448102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 --2- 172.21.15.28:0/1308737484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 0x7f1448100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.910+0000 7f144ca71640 1 -- 172.21.15.28:0/1308737484 >> 172.21.15.28:0/1308737484 conn(0x7f14480fc020 msgr2=0x7f14480fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- 172.21.15.28:0/1308737484 shutdown_connections 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- 172.21.15.28:0/1308737484 wait complete. 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 Processor -- start 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- start start 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 0x7f1448069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 0x7f144819cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c710 con 0x7f1448102640 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c590 con 0x7f1448100a90 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f144ca71640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f144810c890 con 0x7f1448101c80 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.911+0000 7f1446d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42362/0 (socket says 172.21.15.28:42362) 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f144659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 0x7f1448069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 -- 172.21.15.28:0/1477603253 learned_addr learned my addr 172.21.15.28:0/1477603253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f143ffff640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 0x7f144819cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 -- 172.21.15.28:0/1477603253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 msgr2=0x7f144819cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 0x7f144819cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 -- 172.21.15.28:0/1477603253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 msgr2=0x7f1448069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 0x7f1448069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 -- 172.21.15.28:0/1477603253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144819fb00 con 0x7f1448102640 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f1446d9d640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1440037040 tx=0x7f1440032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f143f7fe640 1 -- 172.21.15.28:0/1477603253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440041020 con 0x7f1448102640 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f143f7fe640 1 -- 172.21.15.28:0/1477603253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1440034cd0 con 0x7f1448102640 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144819fd30 con 0x7f1448102640 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f143f7fe640 1 -- 172.21.15.28:0/1477603253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440039040 con 0x7f1448102640 2024-03-22T02:16:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.912+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14481a0210 con 0x7f1448102640 2024-03-22T02:16:07.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f143f7fe640 1 -- 172.21.15.28:0/1477603253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1440044020 con 0x7f1448102640 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 Processor -- start 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 -- start start 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f03370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0339bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654f0296570 con 0x5654f0331bc0 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5654f023ee10 con 0x5654f0337720 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144f657740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654f0292520 con 0x5654f03325b0 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42376/0 (socket says 172.21.15.28:42376) 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f144659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f03370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f143ffff640 1 -- 172.21.15.28:0/2586874596 learned_addr learned my addr 172.21.15.28:0/2586874596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:07.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.915+0000 7f143ffff640 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0339bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143f7fe640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 0x7f14340a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 -- 172.21.15.28:0/2586874596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 msgr2=0x5654f0339bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0339bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 -- 172.21.15.28:0/2586874596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 msgr2=0x5654f0331fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f1446d9d640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 0x7f14340a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 -- 172.21.15.28:0/2586874596 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f00727e0 con 0x5654f03325b0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143ffff640 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143f7fe640 1 -- 172.21.15.28:0/1477603253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f14400f1580 con 0x7f1448102640 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f144659c640 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f03370b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1438009bf0 tx=0x7f1438011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d13a48b9ce139adf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f1446d9d640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 0x7f14340a5460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f14400bd630 tx=0x7f14400a8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143cff9640 1 -- 172.21.15.28:0/2586874596 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438009da0 con 0x5654f03325b0 2024-03-22T02:16:07.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143cff9640 1 -- 172.21.15.28:0/2586874596 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1438005020 con 0x5654f03325b0 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.916+0000 7f143cff9640 1 -- 172.21.15.28:0/2586874596 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438004a50 con 0x5654f03325b0 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- 172.21.15.28:0/2586874596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 msgr2=0x5654f03370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f03370b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1438009bf0 tx=0x7f1438011810 comp rx=0 tx=0).stop 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- 172.21.15.28:0/2586874596 shutdown_connections 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0339bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f03370b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 --2- 172.21.15.28:0/2586874596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0331fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- 172.21.15.28:0/2586874596 >> 172.21.15.28:0/2586874596 conn(0x5654f032d130 msgr2=0x5654f032d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- 172.21.15.28:0/2586874596 shutdown_connections 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- 172.21.15.28:0/2586874596 wait complete. 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.917+0000 7f144f657740 1 -- start start 2024-03-22T02:16:07.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 Processor -- start 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f033c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0343300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5654f033a6d0 con 0x5654f0331bc0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5654f0252110 con 0x5654f0337720 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5654f033a850 con 0x5654f03325b0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 10 client.0 _ll_register_callbacks cb 0x5654f028ebe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42390/0 (socket says 172.21.15.28:42390) 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 -- 172.21.15.28:0/3682867817 learned_addr learned my addr 172.21.15.28:0/3682867817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144659c640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f033c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0343300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 -- 172.21.15.28:0/3682867817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 msgr2=0x5654f033c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f033c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 -- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 msgr2=0x5654f0343300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0343300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 -- 172.21.15.28:0/3682867817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f00d2f70 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f143ffff640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1430011230 tx=0x7f1430004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ebc68906c34e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f142dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430010b40 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f1430009040 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.918+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1438005020 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1430029050 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 10 client.16224 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5654eff6a170 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1438005020 con 0x5654f0331bc0 2024-03-22T02:16:07.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1430011be0 con 0x5654f0331bc0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 10 client.16224 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 10 client.16224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 20 client.16224 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f0347be0 0x5654f034a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 4 client.16224 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654f02a3ab0 con 0x5654f0347be0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.919+0000 7f144659c640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f0347be0 0x5654f034a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.920+0000 7f144659c640 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f0347be0 0x5654f034a080 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dc56f7aa31e8abf server_cookie=8d5494e4a2a65410 in_seq=0 out_seq=0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.920+0000 7f142dffb640 10 client.16224 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f142dffb640 1 -- 172.21.15.28:0/3682867817 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5654f02a3ab0 con 0x5654f0347be0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f142dffb640 10 client.16224 handle_command_reply: tid=0 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 client.16224 shutdown 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 msgr2=0x5654f033bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1430011230 tx=0x7f1430004730 comp rx=0 tx=0).stop 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 shutdown_connections 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f0347be0 0x5654f034a080 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0343300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f033c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 --2- 172.21.15.28:0/3682867817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f033bd80 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f142dffb640 0 client.16224 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 >> 172.21.15.28:0/3682867817 conn(0x5654f032d130 msgr2=0x5654f032c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 shutdown_connections 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 1 -- 172.21.15.28:0/3682867817 wait complete. 2024-03-22T02:16:07.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.921+0000 7f144f657740 20 client.16224 trim_cache size 0 max 16384 2024-03-22T02:16:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 Processor -- start 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 -- start start 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0331bc0 0x5654f0330aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f0337720 0x5654f0333050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14481ad270 con 0x5654f03325b0 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14481aab40 con 0x5654f0331bc0 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144f657740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14481a8990 con 0x5654f0337720 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42392/0 (socket says 172.21.15.28:42392) 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f144659c640 1 -- 172.21.15.28:0/976915932 learned_addr learned my addr 172.21.15.28:0/976915932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f143ffff640 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0331bc0 0x5654f0330aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.940+0000 7f143ffff640 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f0337720 0x5654f0333050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 -- 172.21.15.28:0/976915932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f0337720 msgr2=0x5654f0333050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f0337720 0x5654f0333050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 -- 172.21.15.28:0/976915932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0331bc0 msgr2=0x5654f0330aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0331bc0 0x5654f0330aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 -- 172.21.15.28:0/976915932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f0332ca0 con 0x5654f03325b0 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f144659c640 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1438009380 tx=0x7f1438009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99229b9ef9e0fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f142ffff640 1 -- 172.21.15.28:0/976915932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14380055d0 con 0x5654f03325b0 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f142ffff640 1 -- 172.21.15.28:0/976915932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1438028030 con 0x5654f03325b0 2024-03-22T02:16:07.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.941+0000 7f142ffff640 1 -- 172.21.15.28:0/976915932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143802b6d0 con 0x5654f03325b0 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- 172.21.15.28:0/976915932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 msgr2=0x5654f03432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f1438009380 tx=0x7f1438009310 comp rx=0 tx=0).stop 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- 172.21.15.28:0/976915932 shutdown_connections 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f0337720 0x5654f0333050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f03325b0 0x5654f03432c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- 172.21.15.28:0/976915932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0331bc0 0x5654f0330aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- 172.21.15.28:0/976915932 >> 172.21.15.28:0/976915932 conn(0x5654f032d130 msgr2=0x5654f0330390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- 172.21.15.28:0/976915932 shutdown_connections 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- 172.21.15.28:0/976915932 wait complete. 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- start start 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 Processor -- start 2024-03-22T02:16:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f0340a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0341a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14481a9000 con 0x5654f0331bc0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14481aa470 con 0x5654f0337720 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14481a96d0 con 0x5654f03325b0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144f657740 10 client.0 _ll_register_callbacks cb 0x7f1430013a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f143ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42396/0 (socket says 172.21.15.28:42396) 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f143ffff640 1 -- 172.21.15.28:0/2196208416 learned_addr learned my addr 172.21.15.28:0/2196208416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.942+0000 7f144659c640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f0340a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0341a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 -- 172.21.15.28:0/2196208416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 msgr2=0x5654f0340a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f0340a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 -- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 msgr2=0x5654f0341a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0341a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 -- 172.21.15.28:0/2196208416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f0342050 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f143ffff640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1430013bb0 tx=0x7f1430009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69198e7a754d92f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f142dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1430013190 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f1430026070 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1438028030 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f143003d050 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f144f657740 10 client.16230 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5654f0343400 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.943+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1438028030 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1430004b80 con 0x5654f0331bc0 2024-03-22T02:16:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 10 client.16230 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 20 client.16230 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f03ae610 0x5654f03b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 4 client.16230 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5654f033b7c0 con 0x5654f03ae610 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144659c640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f03ae610 0x5654f03b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f142dffb640 10 client.16230 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.944+0000 7f144659c640 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f03ae610 0x5654f03b0a30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54e00728ac6d28dd server_cookie=3d60561b89e9b8b4 in_seq=0 out_seq=0 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.945+0000 7f142dffb640 1 -- 172.21.15.28:0/2196208416 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5654f033b7c0 con 0x5654f03ae610 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.945+0000 7f142dffb640 10 client.16230 handle_command_reply: tid=0 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.945+0000 7f144f657740 1 client.16230 shutdown 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 msgr2=0x5654f0340510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1430013bb0 tx=0x7f1430009ba0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 shutdown_connections 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5654f03ae610 0x5654f03b0a30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5654f0337720 0x5654f0341a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5654f03325b0 0x5654f0340a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 --2- 172.21.15.28:0/2196208416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5654f0331bc0 0x5654f0340510 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 >> 172.21.15.28:0/2196208416 conn(0x5654f032d130 msgr2=0x5654f0330c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f142dffb640 0 client.16230 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 shutdown_connections 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 1 -- 172.21.15.28:0/2196208416 wait complete. 2024-03-22T02:16:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144f657740 20 client.16230 trim_cache size 0 max 16384 2024-03-22T02:16:07.954 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:07.954 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:16:07.954 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "63dc58a2-6f78-4c12-a103-274f34bc0ab9", 2024-03-22T02:16:07.954 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:16:07.954 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 msgr2=0x7f14340a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 0x7f14340a5460 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f14400bd630 tx=0x7f14400a8040 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 msgr2=0x7f144819f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f1440037040 tx=0x7f1440032f40 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 shutdown_connections 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14340a2fa0 0x7f14340a5460 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1448102640 0x7f144819f1f0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1448101c80 0x7f144819cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 --2- 172.21.15.28:0/1477603253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1448100a90 0x7f1448069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 >> 172.21.15.28:0/1477603253 conn(0x7f14480fc020 msgr2=0x7f1448104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 shutdown_connections 2024-03-22T02:16:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:07.946+0000 7f144ca71640 1 -- 172.21.15.28:0/1477603253 wait complete. 2024-03-22T02:16:07.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:07 smithi028 ceph-mon[31916]: pgmap v434: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 542 KiB/s wr, 140 op/s 2024-03-22T02:16:07.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:16:07.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:16:07.958 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:16:07.959 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:16:07.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:16:07.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:07 smithi120 ceph-mon[36803]: pgmap v434: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 542 KiB/s wr, 140 op/s 2024-03-22T02:16:07.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:16:07.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:16:07.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:16:08.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:07 smithi083 ceph-mon[36340]: pgmap v434: 129 pgs: 129 active+clean; 26 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 542 KiB/s wr, 140 op/s 2024-03-22T02:16:08.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:16:08.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:16:08.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:16:08.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 -- 172.21.15.28:0/412633421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c101c80 msgr2=0x7fb42c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 --2- 172.21.15.28:0/412633421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c101c80 0x7fb42c102100 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb420009900 tx=0x7fb42002bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 -- 172.21.15.28:0/412633421 shutdown_connections 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 --2- 172.21.15.28:0/412633421 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c102640 0x7fb42c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 --2- 172.21.15.28:0/412633421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c101c80 0x7fb42c102100 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 --2- 172.21.15.28:0/412633421 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c100a90 0x7fb42c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.073+0000 7fb431f5e640 1 -- 172.21.15.28:0/412633421 >> 172.21.15.28:0/412633421 conn(0x7fb42c0fc020 msgr2=0x7fb42c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.074+0000 7fb431f5e640 1 -- 172.21.15.28:0/412633421 shutdown_connections 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.074+0000 7fb431f5e640 1 -- 172.21.15.28:0/412633421 wait complete. 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.074+0000 7fb431f5e640 1 Processor -- start 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.074+0000 7fb431f5e640 1 -- start start 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 0x7fb42c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 0x7fb42c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 0x7fb42c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c10c760 con 0x7fb42c102640 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c10c5e0 con 0x7fb42c100a90 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb431f5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c10c8e0 con 0x7fb42c101c80 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 0x7fb42c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 0x7fb42c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 0x7fb42c19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56904/0 (socket says 172.21.15.28:56904) 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42b7fe640 1 -- 172.21.15.28:0/3379063700 learned_addr learned my addr 172.21.15.28:0/3379063700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 0x7fb42c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 -- 172.21.15.28:0/3379063700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 msgr2=0x7fb42c19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 0x7fb42c19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 -- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 msgr2=0x7fb42c1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 0x7fb42c1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 -- 172.21.15.28:0/3379063700 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb42c1a1cf0 con 0x7fb42c101c80 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42bfff640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 0x7fb42c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.075+0000 7fb42affd640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 0x7fb42c19a610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb420036040 tx=0x7fb42002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.076+0000 7fb428ff9640 1 -- 172.21.15.28:0/3379063700 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420043020 con 0x7fb42c101c80 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.076+0000 7fb428ff9640 1 -- 172.21.15.28:0/3379063700 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4200054e0 con 0x7fb42c101c80 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.076+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb42c1a1f80 con 0x7fb42c101c80 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.076+0000 7fb428ff9640 1 -- 172.21.15.28:0/3379063700 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb420003c30 con 0x7fb42c101c80 2024-03-22T02:16:08.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.076+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb42c1a2490 con 0x7fb42c101c80 2024-03-22T02:16:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.078+0000 7fb428ff9640 1 -- 172.21.15.28:0/3379063700 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb420046020 con 0x7fb42c101c80 2024-03-22T02:16:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.078+0000 7fb434c25740 1 Processor -- start 2024-03-22T02:16:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.078+0000 7fb434c25740 1 -- start start 2024-03-22T02:16:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.078+0000 7fb434c25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb434c25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b64920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb434c25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b67420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb434c25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fe1ac48c0 con 0x562fe1b5f430 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb434c25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562fe1a6cc20 con 0x562fe1b5fe20 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb434c25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fe1ac0870 con 0x562fe1b64f90 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42420/0 (socket says 172.21.15.28:42420) 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 -- 172.21.15.28:0/2404749447 learned_addr learned my addr 172.21.15.28:0/2404749447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b64920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42affd640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b67420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb428ff9640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 0x7fb4140a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 -- 172.21.15.28:0/2404749447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 msgr2=0x562fe1b67420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b67420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 -- 172.21.15.28:0/2404749447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 msgr2=0x562fe1b64920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b64920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 -- 172.21.15.28:0/2404749447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fe1a91150 con 0x562fe1b5f430 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b64920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb428ff9640 1 -- 172.21.15.28:0/3379063700 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb4200f1770 con 0x7fb42c101c80 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 0x7fb4140a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.079+0000 7fb42bfff640 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb424011940 tx=0x7fb424011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765bd807b61a4bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb411ffb640 1 -- 172.21.15.28:0/2404749447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424017070 con 0x562fe1b5f430 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb411ffb640 1 -- 172.21.15.28:0/2404749447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb424011250 con 0x562fe1b5f430 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb411ffb640 1 -- 172.21.15.28:0/2404749447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb424010b70 con 0x562fe1b5f430 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb42bfff640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 0x7fb4140a5460 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb424023000 tx=0x7fb424024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 -- 172.21.15.28:0/2404749447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 msgr2=0x562fe1b5f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb424011940 tx=0x7fb424011e00 comp rx=0 tx=0).stop 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 -- 172.21.15.28:0/2404749447 shutdown_connections 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b67420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b64920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 --2- 172.21.15.28:0/2404749447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b5f850 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 -- 172.21.15.28:0/2404749447 >> 172.21.15.28:0/2404749447 conn(0x562fe1b5a9a0 msgr2=0x562fe1b5ada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 -- 172.21.15.28:0/2404749447 shutdown_connections 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.080+0000 7fb434c25740 1 -- 172.21.15.28:0/2404749447 wait complete. 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 -- start start 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 Processor -- start 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b64f90 0x562fe1b70ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fe1b67f40 con 0x562fe1b5fe20 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562fe1a80130 con 0x562fe1b64f90 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fe1b680c0 con 0x562fe1b5f430 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb434c25740 10 client.0 _ll_register_callbacks cb 0x562fe1a7fb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb42affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42436/0 (socket says 172.21.15.28:42436) 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb42affd640 1 -- 172.21.15.28:0/2507161648 learned_addr learned my addr 172.21.15.28:0/2507161648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.081+0000 7fb42bfff640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b64f90 0x562fe1b70ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 -- 172.21.15.28:0/2507161648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 msgr2=0x562fe1b69960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 -- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b64f90 msgr2=0x562fe1b70ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b64f90 0x562fe1b70ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 -- 172.21.15.28:0/2507161648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fe18a27d0 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb42affd640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb4200bda20 tx=0x7fb4200092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fec50a0e7937d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4200bd860 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb4200bbd90 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb424011250 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb4200bd3f0 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 10 client.16242 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562fe17d83f0 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb424011250 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb4200a1120 con 0x562fe1b5fe20 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 10 client.16242 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 10 client.16242 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 20 client.16242 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1b757a0 0x562fe1b77bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 4 client.16242 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.082+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fe1ad1e00 con 0x562fe1b757a0 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.083+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1b757a0 0x562fe1b77bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.083+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1b757a0 0x562fe1b77bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab261c4ff986efb server_cookie=ed43f7535d38fe26 in_seq=0 out_seq=0 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.083+0000 7fb40affd640 10 client.16242 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb40affd640 1 -- 172.21.15.28:0/2507161648 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fe1ad1e00 con 0x562fe1b757a0 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb40affd640 10 client.16242 handle_command_reply: tid=0 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 client.16242 shutdown 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 msgr2=0x562fe1b69ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb4200bda20 tx=0x7fb4200092d0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 shutdown_connections 2024-03-22T02:16:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1b757a0 0x562fe1b77bc0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b64f90 0x562fe1b70ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb40affd640 0 client.16242 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 --2- 172.21.15.28:0/2507161648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 >> 172.21.15.28:0/2507161648 conn(0x562fe1b5a9a0 msgr2=0x562fe1b59c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 shutdown_connections 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.084+0000 7fb434c25740 1 -- 172.21.15.28:0/2507161648 wait complete. 2024-03-22T02:16:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.085+0000 7fb434c25740 20 client.16242 trim_cache size 0 max 16384 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 Processor -- start 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 -- start start 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b70ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1af3b0 con 0x562fe1b5f430 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1acc80 con 0x562fe1b5fe20 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb434c25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1aaad0 con 0x562fe1b64f90 2024-03-22T02:16:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42442/0 (socket says 172.21.15.28:42442) 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb42b7fe640 1 -- 172.21.15.28:0/3313209139 learned_addr learned my addr 172.21.15.28:0/3313209139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb42bfff640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b70ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.102+0000 7fb42affd640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 -- 172.21.15.28:0/3313209139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 msgr2=0x562fe1b70ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b70ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 -- 172.21.15.28:0/3313209139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 msgr2=0x562fe1b69ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 -- 172.21.15.28:0/3313209139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fe17980f0 con 0x562fe1b5f430 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42bfff640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b70ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42affd640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009fd0 tx=0x7fb41c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1804a9d3560f4a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb4117fa640 1 -- 172.21.15.28:0/3313209139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c026070 con 0x562fe1b5f430 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb4117fa640 1 -- 172.21.15.28:0/3313209139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41c004310 con 0x562fe1b5f430 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.103+0000 7fb4117fa640 1 -- 172.21.15.28:0/3313209139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c003b30 con 0x562fe1b5f430 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- 172.21.15.28:0/3313209139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 msgr2=0x562fe1b69960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009fd0 tx=0x7fb41c005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- 172.21.15.28:0/3313209139 shutdown_connections 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b64f90 0x562fe1b70ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b69ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- 172.21.15.28:0/3313209139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b5f430 0x562fe1b69960 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- 172.21.15.28:0/3313209139 >> 172.21.15.28:0/3313209139 conn(0x562fe1b5a9a0 msgr2=0x562fe1b5ad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- 172.21.15.28:0/3313209139 shutdown_connections 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- 172.21.15.28:0/3313209139 wait complete. 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- start start 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 Processor -- start 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b6e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 0x562fe1b6fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1ab140 con 0x562fe1b64f90 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1ac5b0 con 0x562fe1b5fe20 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb42c1ab810 con 0x562fe1b5f430 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.104+0000 7fb434c25740 10 client.0 _ll_register_callbacks cb 0x562fe1a7fb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 0x562fe1b6fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b6e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56918/0 (socket says 172.21.15.28:56918) 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42affd640 1 -- 172.21.15.28:0/1946392778 learned_addr learned my addr 172.21.15.28:0/1946392778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 -- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 msgr2=0x562fe1b6eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 -- 172.21.15.28:0/1946392778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 msgr2=0x562fe1b6fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 0x562fe1b6fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 -- 172.21.15.28:0/1946392778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fe1b70110 con 0x562fe1b5f430 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42affd640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42bfff640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 0x562fe1b6fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb42b7fe640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b6e5d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x562fe1b64e10 tx=0x7fb41c003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aca4610a056962 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb40affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb41c026070 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb41c004310 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb41c020110 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb41c036050 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb434c25740 10 client.26080 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.105+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562fe1bdbc20 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb41c004310 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb41c01e860 con 0x562fe1b5f430 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 10 client.26080 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 10 client.26080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:08.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 20 client.26080 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1bdc540 0x562fe1bde960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 4 client.26080 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562fe1acc150 con 0x562fe1bdc540 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb42affd640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1bdc540 0x562fe1bde960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb42affd640 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1bdc540 0x562fe1bde960 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f21c619a822a10b server_cookie=4cc57dc4c0733da5 in_seq=0 out_seq=0 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb40affd640 10 client.26080 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb40affd640 1 -- 172.21.15.28:0/1946392778 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562fe1acc150 con 0x562fe1bdc540 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb40affd640 10 client.26080 handle_command_reply: tid=0 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.106+0000 7fb434c25740 1 client.26080 shutdown 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 msgr2=0x562fe1b6e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b6e5d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x562fe1b64e10 tx=0x7fb41c003ae0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 shutdown_connections 2024-03-22T02:16:08.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fe1bdc540 0x562fe1bde960 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fe1b64f90 0x562fe1b6fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb40affd640 0 client.26080 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fe1b5fe20 0x562fe1b6eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 --2- 172.21.15.28:0/1946392778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fe1b5f430 0x562fe1b6e5d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 >> 172.21.15.28:0/1946392778 conn(0x562fe1b5a9a0 msgr2=0x562fe1b69900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 shutdown_connections 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 1 -- 172.21.15.28:0/1946392778 wait complete. 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb434c25740 20 client.26080 trim_cache size 0 max 16384 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:08.116 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: "63dc58a2-6f78-4c12-a103-274f34bc0ab9": { 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "63dc58a2-6f78-4c12-a103-274f34bc0ab9", 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:16:08.117 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 msgr2=0x7fb4140a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 0x7fb4140a5460 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb424023000 tx=0x7fb424024000 comp rx=0 tx=0).stop 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 msgr2=0x7fb42c19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 0x7fb42c19a610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb420036040 tx=0x7fb42002c650 comp rx=0 tx=0).stop 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 shutdown_connections 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb4140a2fa0 0x7fb4140a5460 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb42c102640 0x7fb42c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb42c101c80 0x7fb42c19a610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 --2- 172.21.15.28:0/3379063700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb42c100a90 0x7fb42c19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 >> 172.21.15.28:0/3379063700 conn(0x7fb42c0fc020 msgr2=0x7fb42c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.107+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 shutdown_connections 2024-03-22T02:16:08.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:08.108+0000 7fb431f5e640 1 -- 172.21.15.28:0/3379063700 wait complete. 2024-03-22T02:16:08.120 INFO:tasks.cephfs.filesystem:scrub status for tag:63dc58a2-6f78-4c12-a103-274f34bc0ab9 - {'path': '/', 'tag': '63dc58a2-6f78-4c12-a103-274f34bc0ab9', 'options': 'recursive,force'} 2024-03-22T02:16:08.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 38801 4.53 MB/sec warmup 50 sec latency 41.201 ms 2024-03-22T02:16:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:08 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:16:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:08 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:09.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:08 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:16:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:08 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:09.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:08 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:16:09.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:08 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:09.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 40615 4.64 MB/sec warmup 51 sec latency 34.018 ms 2024-03-22T02:16:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:09 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:16:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:09 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:09 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:16:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:09 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:09.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:09 smithi120 ceph-mon[36803]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 752 KiB/s wr, 155 op/s 2024-03-22T02:16:10.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:09 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:16:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:09 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:09 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:16:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:09 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:09 smithi083 ceph-mon[36340]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 752 KiB/s wr, 155 op/s 2024-03-22T02:16:10.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:09 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:16:10.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:09 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:10.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:09 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:16:10.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:09 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:10.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:09 smithi028 ceph-mon[31916]: pgmap v435: 129 pgs: 129 active+clean; 27 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 752 KiB/s wr, 155 op/s 2024-03-22T02:16:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 -- 172.21.15.28:0/598812342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100440 msgr2=0x7f69e4100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/598812342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100440 0x7f69e4100860 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009900 tx=0x7f69d402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 -- 172.21.15.28:0/598812342 shutdown_connections 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/598812342 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4101740 0x7f69e4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/598812342 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4100da0 0x7f69e4101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/598812342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100440 0x7f69e4100860 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 -- 172.21.15.28:0/598812342 >> 172.21.15.28:0/598812342 conn(0x7f69e40fc040 msgr2=0x7f69e40fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 -- 172.21.15.28:0/598812342 shutdown_connections 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.137+0000 7f69e8f7a640 1 -- 172.21.15.28:0/598812342 wait complete. 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 Processor -- start 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 -- start start 2024-03-22T02:16:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 0x7f69e4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 0x7f69e41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69e410fa10 con 0x7f69e4100da0 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69e410f890 con 0x7f69e4100440 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e8f7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69e410fb90 con 0x7f69e4101740 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 0x7f69e4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42458/0 (socket says 172.21.15.28:42458) 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69dbfff640 1 -- 172.21.15.28:0/166731698 learned_addr learned my addr 172.21.15.28:0/166731698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.138+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 0x7f69e41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 -- 172.21.15.28:0/166731698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 msgr2=0x7f69e41a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 0x7f69e41a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 -- 172.21.15.28:0/166731698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 msgr2=0x7f69e4199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 0x7f69e4199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 -- 172.21.15.28:0/166731698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e41a1c10 con 0x7f69e4100da0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 0x7f69e41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69e259c640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 0x7f69e4199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69dbfff640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00e9f0 tx=0x7f69cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69db7fe640 1 -- 172.21.15.28:0/166731698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc00cd80 con 0x7f69e4100da0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69db7fe640 1 -- 172.21.15.28:0/166731698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc010910 con 0x7f69e4100da0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e41a1f00 con 0x7f69e4100da0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69db7fe640 1 -- 172.21.15.28:0/166731698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc018990 con 0x7f69e4100da0 2024-03-22T02:16:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.139+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e41a23c0 con 0x7f69e4100da0 2024-03-22T02:16:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.141+0000 7f69db7fe640 1 -- 172.21.15.28:0/166731698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f69cc010ac0 con 0x7f69e4100da0 2024-03-22T02:16:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 Processor -- start 2024-03-22T02:16:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69db7fe640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 0x7f69d00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 0x7f69d00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69db7fe640 1 -- 172.21.15.28:0/166731698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f69cc0cfae0 con 0x7f69e4100da0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 -- start start 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a43906040 con 0x560a439a7040 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560a438ae9e0 con 0x560a439a14e0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69ebb65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560a43901ff0 con 0x560a439a1ed0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69e259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55678/0 (socket says 172.21.15.28:55678) 2024-03-22T02:16:10.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69e259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56950/0 (socket says 172.21.15.28:56950) 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69dbfff640 1 -- 172.21.15.28:0/1328899265 learned_addr learned my addr 172.21.15.28:0/1328899265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.142+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 0x7f69d00a5480 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f69dc005fd0 tx=0x7f69dc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 -- 172.21.15.28:0/1328899265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 msgr2=0x560a439a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 -- 172.21.15.28:0/1328899265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 msgr2=0x560a439a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 -- 172.21.15.28:0/1328899265 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a4361a210 con 0x560a439a1ed0 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69dbfff640 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09bbb0 tx=0x7f69cc09bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f47223a91148834 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69d8ff9640 1 -- 172.21.15.28:0/1328899265 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc099f50 con 0x560a439a1ed0 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69d8ff9640 1 -- 172.21.15.28:0/1328899265 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc09a1b0 con 0x560a439a1ed0 2024-03-22T02:16:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.143+0000 7f69d8ff9640 1 -- 172.21.15.28:0/1328899265 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc018b30 con 0x560a439a1ed0 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- 172.21.15.28:0/1328899265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 msgr2=0x560a439a69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09bbb0 tx=0x7f69cc09bf70 comp rx=0 tx=0).stop 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- 172.21.15.28:0/1328899265 shutdown_connections 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439a69d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1328899265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- 172.21.15.28:0/1328899265 >> 172.21.15.28:0/1328899265 conn(0x560a4399ca50 msgr2=0x560a4399ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- 172.21.15.28:0/1328899265 shutdown_connections 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- 172.21.15.28:0/1328899265 wait complete. 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 -- start start 2024-03-22T02:16:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 Processor -- start 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439ab480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.144+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69ebb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560a439a9f90 con 0x560a439a7040 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69ebb65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560a438c1db0 con 0x560a439a1ed0 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69ebb65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560a439aa110 con 0x560a439a14e0 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69ebb65740 10 client.0 _ll_register_callbacks cb 0x560a43619a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42476/0 (socket says 172.21.15.28:42476) 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 -- 172.21.15.28:0/1635171165 learned_addr learned my addr 172.21.15.28:0/1635171165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 -- 172.21.15.28:0/1635171165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 msgr2=0x560a439ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 -- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 msgr2=0x560a439ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 -- 172.21.15.28:0/1635171165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a436e6620 con 0x560a439a7040 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69dbfff640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e259c640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f69dc003dc0 tx=0x7f69dc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656a2d5b25e85d93 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69dc01e800 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f69dc01e9a0 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.145+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69cc09a1b0 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f69dc031050 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 10 client.16260 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a439b5d70 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc09a1b0 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f69dc026020 con 0x560a439a7040 2024-03-22T02:16:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 10 client.16260 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 20 client.16260 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a439b7520 0x560a439b9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 4 client.16260 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a43913580 con 0x560a439b7520 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69e259c640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a439b7520 0x560a439b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69c9ffb640 10 client.16260 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.146+0000 7f69e259c640 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a439b7520 0x560a439b9910 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4af96b2b4df2d8e server_cookie=96d0c3986c71f9d3 in_seq=0 out_seq=0 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.147+0000 7f69c9ffb640 1 -- 172.21.15.28:0/1635171165 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560a43913580 con 0x560a439b7520 2024-03-22T02:16:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.147+0000 7f69c9ffb640 10 client.16260 handle_command_reply: tid=0 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.147+0000 7f69ebb65740 1 client.16260 shutdown 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.147+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 msgr2=0x560a439b2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.147+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f69dc003dc0 tx=0x7f69dc004ab0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 shutdown_connections 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a439b7520 0x560a439b9910 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a439b2990 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 --2- 172.21.15.28:0/1635171165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 >> 172.21.15.28:0/1635171165 conn(0x560a4399ca50 msgr2=0x560a4399bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69c9ffb640 0 client.16260 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 shutdown_connections 2024-03-22T02:16:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 1 -- 172.21.15.28:0/1635171165 wait complete. 2024-03-22T02:16:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.148+0000 7f69ebb65740 20 client.16260 trim_cache size 0 max 16384 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 Processor -- start 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 -- start start 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a4399b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41af4b0 con 0x560a439a7040 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41acd80 con 0x560a439a14e0 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69ebb65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41aabd0 con 0x560a439a1ed0 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a4399b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55686/0 (socket says 172.21.15.28:55686) 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 -- 172.21.15.28:0/3141000202 learned_addr learned my addr 172.21.15.28:0/3141000202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69e259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 -- 172.21.15.28:0/3141000202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 msgr2=0x560a439a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 -- 172.21.15.28:0/3141000202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 msgr2=0x560a4399b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a4399b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.164+0000 7f69dbfff640 1 -- 172.21.15.28:0/3141000202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a4399d4c0 con 0x560a439a1ed0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69dbfff640 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f69cc098830 tx=0x7f69cc018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754e3b01f0d0d526 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69cbfff640 1 -- 172.21.15.28:0/3141000202 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc09a1b0 con 0x560a439a1ed0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69cbfff640 1 -- 172.21.15.28:0/3141000202 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc0babe0 con 0x560a439a1ed0 2024-03-22T02:16:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69cbfff640 1 -- 172.21.15.28:0/3141000202 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc09b300 con 0x560a439a1ed0 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69ebb65740 1 -- 172.21.15.28:0/3141000202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 msgr2=0x560a439b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.165+0000 7f69ebb65740 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f69cc098830 tx=0x7f69cc018420 comp rx=0 tx=0).stop 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- 172.21.15.28:0/3141000202 shutdown_connections 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a4399b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a1ed0 0x560a439b2950 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- 172.21.15.28:0/3141000202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a14e0 0x560a439a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- 172.21.15.28:0/3141000202 >> 172.21.15.28:0/3141000202 conn(0x560a4399ca50 msgr2=0x560a4399fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- 172.21.15.28:0/3141000202 shutdown_connections 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- 172.21.15.28:0/3141000202 wait complete. 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- start start 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 Processor -- start 2024-03-22T02:16:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41ab240 con 0x560a439a7040 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41ac6b0 con 0x560a439a1ed0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69e41ab910 con 0x560a439a14e0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69ebb65740 10 client.0 _ll_register_callbacks cb 0x7f69dc00fb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69e259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69e2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42496/0 (socket says 172.21.15.28:42496) 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69e2d9d640 1 -- 172.21.15.28:0/2915503677 learned_addr learned my addr 172.21.15.28:0/2915503677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.166+0000 7f69dbfff640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 -- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 msgr2=0x560a439afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 -- 172.21.15.28:0/2915503677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 msgr2=0x560a43a1b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 -- 172.21.15.28:0/2915503677 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a439b01f0 con 0x560a439a14e0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e2d9d640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69dbfff640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69e259c640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439af6a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009230 tx=0x7f69d4004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27b72e2b37940577 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d40463f0 con 0x560a439a14e0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69cc0babe0 con 0x560a439a14e0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f69d4046590 con 0x560a439a14e0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f69d4057050 con 0x560a439a14e0 2024-03-22T02:16:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69ebb65740 10 client.26098 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.167+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560a43a1ce70 con 0x560a439a14e0 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69cc0babe0 con 0x560a439a14e0 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f69d4004aa0 con 0x560a439a14e0 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 10 client.26098 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 10 client.26098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 20 client.26098 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a43a1d850 0x560a43a1fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 4 client.26098 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560a439982b0 con 0x560a43a1d850 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69dbfff640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a43a1d850 0x560a43a1fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69c9ffb640 10 client.26098 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.168+0000 7f69dbfff640 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a43a1d850 0x560a43a1fc70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815aadb84b1daf6a server_cookie=a0db9c0ce058334d in_seq=0 out_seq=0 2024-03-22T02:16:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.292+0000 7f69c9ffb640 1 -- 172.21.15.28:0/2915503677 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560a439982b0 con 0x560a43a1d850 2024-03-22T02:16:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.292+0000 7f69c9ffb640 10 client.26098 handle_command_reply: tid=0 2024-03-22T02:16:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 client.26098 shutdown 2024-03-22T02:16:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 msgr2=0x560a439af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439af6a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009230 tx=0x7f69d4004580 comp rx=0 tx=0).stop 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 shutdown_connections 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560a43a1d850 0x560a43a1fc70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560a439a7040 0x560a43a1b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560a439a1ed0 0x560a439afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 --2- 172.21.15.28:0/2915503677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560a439a14e0 0x560a439af6a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69c9ffb640 0 client.26098 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 >> 172.21.15.28:0/2915503677 conn(0x560a4399ca50 msgr2=0x560a439a0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 shutdown_connections 2024-03-22T02:16:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 1 -- 172.21.15.28:0/2915503677 wait complete. 2024-03-22T02:16:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69ebb65740 20 client.26098 trim_cache size 0 max 16384 2024-03-22T02:16:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:16:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:16:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:16:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:16:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 msgr2=0x7f69d00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 0x7f69d00a5480 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f69dc005fd0 tx=0x7f69dc009500 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 msgr2=0x7f69e419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.293+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00e9f0 tx=0x7f69cc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 shutdown_connections 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69d00a2fc0 0x7f69d00a5480 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69e4101740 0x7f69e41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f69e4100da0 0x7f69e419a530 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 --2- 172.21.15.28:0/166731698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f69e4100440 0x7f69e4199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 >> 172.21.15.28:0/166731698 conn(0x7f69e40fc040 msgr2=0x7f69e40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 shutdown_connections 2024-03-22T02:16:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:10.294+0000 7f69e8f7a640 1 -- 172.21.15.28:0/166731698 wait complete. 2024-03-22T02:16:10.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 42986 4.82 MB/sec warmup 52 sec latency 75.751 ms 2024-03-22T02:16:11.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 45199 4.88 MB/sec warmup 53 sec latency 30.691 ms 2024-03-22T02:16:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:11 smithi120 ceph-mon[36803]: pgmap v436: 129 pgs: 129 active+clean; 30 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1003 KiB/s wr, 161 op/s 2024-03-22T02:16:12.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:11 smithi083 ceph-mon[36340]: pgmap v436: 129 pgs: 129 active+clean; 30 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1003 KiB/s wr, 161 op/s 2024-03-22T02:16:12.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:11 smithi028 ceph-mon[31916]: pgmap v436: 129 pgs: 129 active+clean; 30 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1003 KiB/s wr, 161 op/s 2024-03-22T02:16:12.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 47022 5.00 MB/sec warmup 54 sec latency 50.743 ms 2024-03-22T02:16:13.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 49928 5.24 MB/sec warmup 55 sec latency 24.071 ms 2024-03-22T02:16:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:13 smithi120 ceph-mon[36803]: pgmap v437: 129 pgs: 129 active+clean; 31 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-22T02:16:14.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:13 smithi083 ceph-mon[36340]: pgmap v437: 129 pgs: 129 active+clean; 31 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-22T02:16:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:13 smithi028 ceph-mon[31916]: pgmap v437: 129 pgs: 129 active+clean; 31 MiB data, 449 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 156 op/s 2024-03-22T02:16:14.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 52298 5.30 MB/sec warmup 56 sec latency 29.148 ms 2024-03-22T02:16:15.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 54304 5.43 MB/sec warmup 57 sec latency 34.511 ms 2024-03-22T02:16:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:15 smithi120 ceph-mon[36803]: pgmap v438: 129 pgs: 129 active+clean; 28 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-03-22T02:16:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:15 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:16:15.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:15 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:16:16.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:15 smithi083 ceph-mon[36340]: pgmap v438: 129 pgs: 129 active+clean; 28 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-03-22T02:16:16.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:15 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:16:16.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:15 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:16:16.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:15 smithi028 ceph-mon[31916]: pgmap v438: 129 pgs: 129 active+clean; 28 MiB data, 477 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.4 MiB/s wr, 232 op/s 2024-03-22T02:16:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:15 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:16:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:15 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:16:16.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 57044 5.63 MB/sec warmup 58 sec latency 27.183 ms 2024-03-22T02:16:17.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 59495 5.73 MB/sec warmup 59 sec latency 32.776 ms 2024-03-22T02:16:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:17 smithi120 ceph-mon[36803]: pgmap v439: 129 pgs: 129 active+clean; 32 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 284 op/s 2024-03-22T02:16:18.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:17 smithi083 ceph-mon[36340]: pgmap v439: 129 pgs: 129 active+clean; 32 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 284 op/s 2024-03-22T02:16:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:17 smithi028 ceph-mon[31916]: pgmap v439: 129 pgs: 129 active+clean; 32 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.6 MiB/s wr, 284 op/s 2024-03-22T02:16:18.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 61452 5.80 MB/sec warmup 60 sec latency 34.715 ms 2024-03-22T02:16:19.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 64256 5.99 MB/sec warmup 61 sec latency 20.124 ms 2024-03-22T02:16:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:19 smithi120 ceph-mon[36803]: pgmap v440: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:20.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:19 smithi083 ceph-mon[36340]: pgmap v440: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:19 smithi028 ceph-mon[31916]: pgmap v440: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:20.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 66841 6.09 MB/sec warmup 62 sec latency 42.460 ms 2024-03-22T02:16:21.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 68413 6.11 MB/sec warmup 63 sec latency 31.306 ms 2024-03-22T02:16:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:21 smithi120 ceph-mon[36803]: pgmap v441: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:21 smithi083 ceph-mon[36340]: pgmap v441: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:21 smithi028 ceph-mon[31916]: pgmap v441: 129 pgs: 129 active+clean; 33 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 1.8 MiB/s wr, 303 op/s 2024-03-22T02:16:22.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 71265 6.30 MB/sec warmup 64 sec latency 17.229 ms 2024-03-22T02:16:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- 172.21.15.28:0/1457415216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c100ac0 msgr2=0x7fb70c100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 --2- 172.21.15.28:0/1457415216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c100ac0 0x7fb70c100ee0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009ca0 tx=0x7fb6fc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- 172.21.15.28:0/1457415216 shutdown_connections 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 --2- 172.21.15.28:0/1457415216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 --2- 172.21.15.28:0/1457415216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c101cb0 0x7fb70c102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 --2- 172.21.15.28:0/1457415216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c100ac0 0x7fb70c100ee0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- 172.21.15.28:0/1457415216 >> 172.21.15.28:0/1457415216 conn(0x7fb70c0fc050 msgr2=0x7fb70c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- 172.21.15.28:0/1457415216 shutdown_connections 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- 172.21.15.28:0/1457415216 wait complete. 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 Processor -- start 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.387+0000 7fb711520640 1 -- start start 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c103640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10c730 con 0x7fb70c101cb0 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10c5b0 con 0x7fb70c102670 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb711520640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c10c8b0 con 0x7fb70c100ac0 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59734/0 (socket says 172.21.15.28:59734) 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:54946/0 (socket says 172.21.15.28:54946) 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c103640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 -- 172.21.15.28:0/743916901 learned_addr learned my addr 172.21.15.28:0/743916901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 -- 172.21.15.28:0/743916901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 msgr2=0x7fb70c104ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 -- 172.21.15.28:0/743916901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 msgr2=0x7fb70c103640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c103640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 -- 172.21.15.28:0/743916901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb70c103ed0 con 0x7fb70c101cb0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70affd640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.388+0000 7fb703fff640 1 -- 172.21.15.28:0/743916901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400cd80 con 0x7fb70c101cb0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.389+0000 7fb703fff640 1 -- 172.21.15.28:0/743916901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f4005500 con 0x7fb70c101cb0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.389+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb70c1a1f40 con 0x7fb70c101cb0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.389+0000 7fb703fff640 1 -- 172.21.15.28:0/743916901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4010690 con 0x7fb70c101cb0 2024-03-22T02:16:23.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.389+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb70c1a2350 con 0x7fb70c101cb0 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.391+0000 7fb703fff640 1 -- 172.21.15.28:0/743916901 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb6f40046c0 con 0x7fb70c101cb0 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 Processor -- start 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb703fff640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 0x7fb6f80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 -- start start 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 0x7fb6f80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb703fff640 1 -- 172.21.15.28:0/743916901 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb6f40cfc50 con 0x7fb70c101cb0 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 0x560cd54a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd54a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cd5403040 con 0x560cd549e4e0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560cd53ab9e0 con 0x560cd549eed0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb7141df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560cd53feff0 con 0x560cd54a4040 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59750/0 (socket says 172.21.15.28:59750) 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70affd640 1 -- 172.21.15.28:0/2485680654 learned_addr learned my addr 172.21.15.28:0/2485680654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 0x560cd54a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 0x7fb6f80a5480 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fc0 tx=0x7fb704005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.392+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd54a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 -- 172.21.15.28:0/2485680654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 msgr2=0x560cd54a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd54a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 -- 172.21.15.28:0/2485680654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 msgr2=0x560cd549e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 -- 172.21.15.28:0/2485680654 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd5117210 con 0x560cd549eed0 2024-03-22T02:16:23.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70affd640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 0x560cd54a39d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40cd390 tx=0x7fb6f409b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237ce425aad9cb0a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb7017fa640 1 -- 172.21.15.28:0/2485680654 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40b4020 con 0x560cd549eed0 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb7017fa640 1 -- 172.21.15.28:0/2485680654 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f40a00c0 con 0x560cd549eed0 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.393+0000 7fb7017fa640 1 -- 172.21.15.28:0/2485680654 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f409b620 con 0x560cd549eed0 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- 172.21.15.28:0/2485680654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 msgr2=0x560cd54a39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 0x560cd54a39d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40cd390 tx=0x7fb6f409b8d0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- 172.21.15.28:0/2485680654 shutdown_connections 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd54a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549eed0 0x560cd54a39d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 --2- 172.21.15.28:0/2485680654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd549e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- 172.21.15.28:0/2485680654 >> 172.21.15.28:0/2485680654 conn(0x560cd5499a50 msgr2=0x560cd5499e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- 172.21.15.28:0/2485680654 shutdown_connections 2024-03-22T02:16:23.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- 172.21.15.28:0/2485680654 wait complete. 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 -- start start 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.394+0000 7fb7141df740 1 Processor -- start 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 0x560cd54a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54afa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cd54a6ff0 con 0x560cd549eed0 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560cd53bedb0 con 0x560cd54a4040 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560cd54a7170 con 0x560cd549e4e0 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb7141df740 10 client.0 _ll_register_callbacks cb 0x560cd5116a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59760/0 (socket says 172.21.15.28:59760) 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 -- 172.21.15.28:0/3990703568 learned_addr learned my addr 172.21.15.28:0/3990703568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:23.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70affd640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 0x560cd54a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 -- 172.21.15.28:0/3990703568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 msgr2=0x560cd54a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 0x560cd54a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 -- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 msgr2=0x560cd54afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 -- 172.21.15.28:0/3990703568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd51e3620 con 0x560cd549eed0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70affd640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 0x560cd54a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40983d0 tx=0x7fb6f409ba00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a873f7e00968358 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb6f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.395+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f40a8020 con 0x560cd549eed0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb6f40a4bd0 con 0x560cd549eed0 2024-03-22T02:16:23.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6f40a00c0 con 0x560cd549eed0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6f40b4070 con 0x560cd549eed0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 10 client.16278 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cd54b2de0 con 0x560cd549eed0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6f40a00c0 con 0x560cd549eed0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6f40bace0 con 0x560cd549eed0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 10 client.16278 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 10 client.16278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 20 client.16278 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd54b4130 0x560cd54b6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 4 client.16278 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cd5410580 con 0x560cd54b4130 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.396+0000 7fb70affd640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd54b4130 0x560cd54b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.397+0000 7fb70affd640 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd54b4130 0x560cd54b6550 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52a12a7c5b5558d8 server_cookie=205968399a2e319a in_seq=0 out_seq=0 2024-03-22T02:16:23.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.397+0000 7fb6f27fc640 10 client.16278 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb6f27fc640 1 -- 172.21.15.28:0/3990703568 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560cd5410580 con 0x560cd54b4130 2024-03-22T02:16:23.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb6f27fc640 10 client.16278 handle_command_reply: tid=0 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 client.16278 shutdown 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 msgr2=0x560cd54a8a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40983d0 tx=0x7fb6f409ba00 comp rx=0 tx=0).stop 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 shutdown_connections 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd54b4130 0x560cd54b6550 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54a8a30 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb6f27fc640 0 client.16278 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 --2- 172.21.15.28:0/3990703568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549e4e0 0x560cd54a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 >> 172.21.15.28:0/3990703568 conn(0x560cd5499a50 msgr2=0x560cd5498c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 shutdown_connections 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 1 -- 172.21.15.28:0/3990703568 wait complete. 2024-03-22T02:16:23.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.398+0000 7fb7141df740 20 client.16278 trim_cache size 0 max 16384 2024-03-22T02:16:23.408 INFO:tasks.workunit.client.0.smithi028.stdout: 1 73604 6.33 MB/sec warmup 65 sec latency 27.893 ms 2024-03-22T02:16:23.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 Processor -- start 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 -- start start 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 0x560cd549f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54afa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd549fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1af530 con 0x560cd549eed0 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1ace00 con 0x560cd549e4e0 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb7141df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1aac50 con 0x560cd54a4040 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.414+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 0x560cd549f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd549fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 0x560cd549f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43888/0 (socket says 172.21.15.28:43888) 2024-03-22T02:16:23.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59776/0 (socket says 172.21.15.28:59776) 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70affd640 1 -- 172.21.15.28:0/2087032301 learned_addr learned my addr 172.21.15.28:0/2087032301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 -- 172.21.15.28:0/2087032301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 msgr2=0x560cd549f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 0x560cd549f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 -- 172.21.15.28:0/2087032301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 msgr2=0x560cd54afa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54afa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 -- 172.21.15.28:0/2087032301 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd54b6bb0 con 0x560cd54a4040 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd549fe80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb704011ca0 tx=0x7fb704014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb5c76307967be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb700ff9640 1 -- 172.21.15.28:0/2087032301 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70402d070 con 0x560cd54a4040 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb700ff9640 1 -- 172.21.15.28:0/2087032301 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb70401fb70 con 0x560cd54a4040 2024-03-22T02:16:23.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.415+0000 7fb700ff9640 1 -- 172.21.15.28:0/2087032301 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb704028c10 con 0x560cd54a4040 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- 172.21.15.28:0/2087032301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 msgr2=0x560cd549fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd549fe80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb704011ca0 tx=0x7fb704014000 comp rx=0 tx=0).stop 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- 172.21.15.28:0/2087032301 shutdown_connections 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd54a4040 0x560cd549fe80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549eed0 0x560cd54afa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 --2- 172.21.15.28:0/2087032301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd549e4e0 0x560cd549f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- 172.21.15.28:0/2087032301 >> 172.21.15.28:0/2087032301 conn(0x560cd5499a50 msgr2=0x560cd54a84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- 172.21.15.28:0/2087032301 shutdown_connections 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- 172.21.15.28:0/2087032301 wait complete. 2024-03-22T02:16:23.421 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 -- start start 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.416+0000 7fb7141df740 1 Processor -- start 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549eed0 0x560cd54ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1ab2c0 con 0x560cd549e4e0 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1ac730 con 0x560cd54a4040 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb70c1ab990 con 0x560cd549eed0 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 10 client.0 _ll_register_callbacks cb 0x7fb704005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59792/0 (socket says 172.21.15.28:59792) 2024-03-22T02:16:23.422 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 -- 172.21.15.28:0/1540467166 learned_addr learned my addr 172.21.15.28:0/1540467166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549eed0 0x560cd54ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70b7fe640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 -- 172.21.15.28:0/1540467166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549eed0 msgr2=0x560cd54ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549eed0 0x560cd54ad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 -- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 msgr2=0x560cd54ae310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54ae310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 -- 172.21.15.28:0/1540467166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd54ae900 con 0x560cd549e4e0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb70affd640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005e30 tx=0x7fb6fc032e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84301d9dd5127b29 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb6f27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc039040 con 0x560cd549e4e0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fb6fc032890 con 0x560cd549e4e0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.417+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb70401fb70 con 0x560cd549e4e0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.418+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb6fc057050 con 0x560cd549e4e0 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.418+0000 7fb7141df740 10 client.16284 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:23.423 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.418+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560cd54afc80 con 0x560cd549e4e0 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.418+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb70401fb70 con 0x560cd549e4e0 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.418+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb6fc004040 con 0x560cd549e4e0 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 10 client.16284 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 10 client.16284 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 20 client.16284 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd551ab30 0x560cd551cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 4 client.16284 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560cd540a8d0 con 0x560cd551ab30 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd551ab30 0x560cd551cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb6f27fc640 10 client.16284 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:23.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.419+0000 7fb70a7fc640 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd551ab30 0x560cd551cf50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=223a63a7c5e02b53 server_cookie=813a143e132b398 in_seq=0 out_seq=0 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb6f27fc640 1 -- 172.21.15.28:0/1540467166 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560cd540a8d0 con 0x560cd551ab30 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb6f27fc640 10 client.16284 handle_command_reply: tid=0 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 client.16284 shutdown 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 msgr2=0x560cd54aceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc005e30 tx=0x7fb6fc032e50 comp rx=0 tx=0).stop 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 shutdown_connections 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd551ab30 0x560cd551cf50 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd54a4040 0x560cd54ae310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd549eed0 0x560cd54ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 --2- 172.21.15.28:0/1540467166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd549e4e0 0x560cd54aceb0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 >> 172.21.15.28:0/1540467166 conn(0x560cd5499a50 msgr2=0x560cd549ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb6f27fc640 0 client.16284 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 shutdown_connections 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 1 -- 172.21.15.28:0/1540467166 wait complete. 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.615+0000 7fb7141df740 20 client.16284 trim_cache size 0 max 16384 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:16:23.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:16:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:16:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 msgr2=0x7fb6f80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 0x7fb6f80a5480 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb704009fc0 tx=0x7fb704005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:23.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 msgr2=0x7fb70c103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400e9f0 tx=0x7fb6f400edf0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 shutdown_connections 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6f80a2fc0 0x7fb6f80a5480 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb70c102670 0x7fb70c103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb70c101cb0 0x7fb70c103100 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 --2- 172.21.15.28:0/743916901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb70c100ac0 0x7fb70c104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 >> 172.21.15.28:0/743916901 conn(0x7fb70c0fc050 msgr2=0x7fb70c06a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 shutdown_connections 2024-03-22T02:16:23.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:23.616+0000 7fb711520640 1 -- 172.21.15.28:0/743916901 wait complete. 2024-03-22T02:16:23.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:23 smithi120 ceph-mon[36803]: pgmap v442: 129 pgs: 129 active+clean; 37 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 313 op/s 2024-03-22T02:16:24.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:23 smithi083 ceph-mon[36340]: pgmap v442: 129 pgs: 129 active+clean; 37 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 313 op/s 2024-03-22T02:16:24.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:23 smithi028 ceph-mon[31916]: pgmap v442: 129 pgs: 129 active+clean; 37 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 313 op/s 2024-03-22T02:16:24.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 75145 6.40 MB/sec warmup 66 sec latency 152.477 ms 2024-03-22T02:16:25.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 78205 6.57 MB/sec warmup 67 sec latency 29.085 ms 2024-03-22T02:16:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:25 smithi120 ceph-mon[36803]: pgmap v443: 129 pgs: 129 active+clean; 40 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.3 MiB/s wr, 368 op/s 2024-03-22T02:16:26.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:25 smithi083 ceph-mon[36340]: pgmap v443: 129 pgs: 129 active+clean; 40 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.3 MiB/s wr, 368 op/s 2024-03-22T02:16:26.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:25 smithi028 ceph-mon[31916]: pgmap v443: 129 pgs: 129 active+clean; 40 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 2.3 MiB/s wr, 368 op/s 2024-03-22T02:16:26.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 80455 6.60 MB/sec warmup 68 sec latency 34.716 ms 2024-03-22T02:16:27.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 82357 6.68 MB/sec warmup 69 sec latency 31.812 ms 2024-03-22T02:16:27.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:27 smithi120 ceph-mon[36803]: pgmap v444: 129 pgs: 129 active+clean; 32 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 325 op/s 2024-03-22T02:16:28.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:27 smithi083 ceph-mon[36340]: pgmap v444: 129 pgs: 129 active+clean; 32 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 325 op/s 2024-03-22T02:16:28.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:27 smithi028 ceph-mon[31916]: pgmap v444: 129 pgs: 129 active+clean; 32 MiB data, 490 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 325 op/s 2024-03-22T02:16:28.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 85394 6.85 MB/sec warmup 70 sec latency 9.761 ms 2024-03-22T02:16:29.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 87618 6.86 MB/sec warmup 71 sec latency 29.614 ms 2024-03-22T02:16:29.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:29 smithi120 ceph-mon[36803]: pgmap v445: 129 pgs: 129 active+clean; 35 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-03-22T02:16:30.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:29 smithi083 ceph-mon[36340]: pgmap v445: 129 pgs: 129 active+clean; 35 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-03-22T02:16:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:29 smithi028 ceph-mon[31916]: pgmap v445: 129 pgs: 129 active+clean; 35 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 2.0 MiB/s wr, 367 op/s 2024-03-22T02:16:30.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 89597 6.94 MB/sec warmup 72 sec latency 37.992 ms 2024-03-22T02:16:31.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 92407 7.08 MB/sec warmup 73 sec latency 32.896 ms 2024-03-22T02:16:31.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:31 smithi120 ceph-mon[36803]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 387 op/s 2024-03-22T02:16:32.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:31 smithi083 ceph-mon[36340]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 387 op/s 2024-03-22T02:16:32.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:31 smithi028 ceph-mon[31916]: pgmap v446: 129 pgs: 129 active+clean; 37 MiB data, 469 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 387 op/s 2024-03-22T02:16:32.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 94739 7.11 MB/sec warmup 74 sec latency 27.893 ms 2024-03-22T02:16:33.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 96683 7.17 MB/sec warmup 75 sec latency 28.404 ms 2024-03-22T02:16:33.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:33 smithi120 ceph-mon[36803]: pgmap v447: 129 pgs: 129 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 381 op/s 2024-03-22T02:16:34.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:33 smithi083 ceph-mon[36340]: pgmap v447: 129 pgs: 129 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 381 op/s 2024-03-22T02:16:34.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:33 smithi028 ceph-mon[31916]: pgmap v447: 129 pgs: 129 active+clean; 38 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 381 op/s 2024-03-22T02:16:34.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 99578 7.32 MB/sec warmup 76 sec latency 19.258 ms 2024-03-22T02:16:35.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 101729 7.32 MB/sec warmup 77 sec latency 28.971 ms 2024-03-22T02:16:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:35 smithi120 ceph-mon[36803]: pgmap v448: 129 pgs: 129 active+clean; 42 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 396 op/s 2024-03-22T02:16:36.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:35 smithi083 ceph-mon[36340]: pgmap v448: 129 pgs: 129 active+clean; 42 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 396 op/s 2024-03-22T02:16:36.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:35 smithi028 ceph-mon[31916]: pgmap v448: 129 pgs: 129 active+clean; 42 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.3 MiB/s wr, 396 op/s 2024-03-22T02:16:36.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 103517 7.38 MB/sec warmup 78 sec latency 30.602 ms 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 -- 172.21.15.28:0/471519853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e41026a0 msgr2=0x7f25e4104a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/471519853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e41026a0 0x7f25e4104a90 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009900 tx=0x7f25d402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 -- 172.21.15.28:0/471519853 shutdown_connections 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/471519853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e4107900 0x7f25e4109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/471519853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4104fd0 0x7f25e41073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/471519853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e41026a0 0x7f25e4104a90 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 -- 172.21.15.28:0/471519853 >> 172.21.15.28:0/471519853 conn(0x7f25e40fc030 msgr2=0x7f25e40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 -- 172.21.15.28:0/471519853 shutdown_connections 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.706+0000 7f25ea7d7640 1 -- 172.21.15.28:0/471519853 wait complete. 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 Processor -- start 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 -- start start 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e41026a0 0x7f25e4197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4107900 0x7f25e419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25e410d860 con 0x7f25e4104fd0 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25e410d6e0 con 0x7f25e4107900 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25ea7d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25e410d9e0 con 0x7f25e41026a0 2024-03-22T02:16:36.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37754/0 (socket says 172.21.15.28:37754) 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e41026a0 0x7f25e4197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 -- 172.21.15.28:0/4061678600 learned_addr learned my addr 172.21.15.28:0/4061678600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e8d74640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4107900 0x7f25e419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 -- 172.21.15.28:0/4061678600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e41026a0 msgr2=0x7f25e4197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e41026a0 0x7f25e4197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 -- 172.21.15.28:0/4061678600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4107900 msgr2=0x7f25e419f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4107900 0x7f25e419f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.707+0000 7f25e37fe640 1 -- 172.21.15.28:0/4061678600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25e419fa30 con 0x7f25e4104fd0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25e37fe640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f25d800e9c0 tx=0x7f25d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25e17fa640 1 -- 172.21.15.28:0/4061678600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d800cd80 con 0x7f25e4104fd0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25e419fd20 con 0x7f25e4104fd0 2024-03-22T02:16:36.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25e17fa640 1 -- 172.21.15.28:0/4061678600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d8005500 con 0x7f25e4104fd0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25e17fa640 1 -- 172.21.15.28:0/4061678600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d8010690 con 0x7f25e4104fd0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.708+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25e41a0210 con 0x7f25e4104fd0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25e17fa640 1 -- 172.21.15.28:0/4061678600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f25d80046c0 con 0x7f25e4104fd0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 Processor -- start 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 -- start start 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 0x555cdeb599d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb5c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555cdeab9040 con 0x555cdeb544e0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555cdea619e0 con 0x555cdeb54ed0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.710+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555cdeab4ff0 con 0x555cdeb5a040 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 0x555cdeb599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37762/0 (socket says 172.21.15.28:37762) 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 -- 172.21.15.28:0/852427260 learned_addr learned my addr 172.21.15.28:0/852427260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e37fe640 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb5c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e17fa640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 0x7f25cc0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 -- 172.21.15.28:0/852427260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 msgr2=0x555cdeb5c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb5c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 -- 172.21.15.28:0/852427260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 msgr2=0x555cdeb54900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 -- 172.21.15.28:0/852427260 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cde7cd210 con 0x555cdeb54ed0 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 0x7f25cc0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e17fa640 1 -- 172.21.15.28:0/4061678600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f25d80cfb60 con 0x7f25e4104fd0 2024-03-22T02:16:36.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e3fff640 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 0x555cdeb599d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d52ebc0b09bf6757 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25e8d74640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 0x7f25cc0a5480 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x555cdeb55730 tx=0x7f25dc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.711+0000 7f25ca7fc640 1 -- 172.21.15.28:0/852427260 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d402cb80 con 0x555cdeb54ed0 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ca7fc640 1 -- 172.21.15.28:0/852427260 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d4005040 con 0x555cdeb54ed0 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ca7fc640 1 -- 172.21.15.28:0/852427260 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4004920 con 0x555cdeb54ed0 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 -- 172.21.15.28:0/852427260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 msgr2=0x555cdeb599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 0x555cdeb599d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f25d4009230 tx=0x7f25d4009560 comp rx=0 tx=0).stop 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 -- 172.21.15.28:0/852427260 shutdown_connections 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb5c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb54ed0 0x555cdeb599d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/852427260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdeb54900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 -- 172.21.15.28:0/852427260 >> 172.21.15.28:0/852427260 conn(0x555cdeb4fa50 msgr2=0x555cdeb4fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 -- 172.21.15.28:0/852427260 shutdown_connections 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.712+0000 7f25ed3bf740 1 -- 172.21.15.28:0/852427260 wait complete. 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 -- start start 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 Processor -- start 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 0x555cdeb5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdeb659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555cdeb5cf90 con 0x555cdeb54ed0 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555cdea74db0 con 0x555cdeb5a040 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555cdeb5d110 con 0x555cdeb544e0 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25ed3bf740 10 client.0 _ll_register_callbacks cb 0x555cde7cca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37768/0 (socket says 172.21.15.28:37768) 2024-03-22T02:16:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 -- 172.21.15.28:0/3176237444 learned_addr learned my addr 172.21.15.28:0/3176237444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 0x555cdeb5e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e8d74640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdeb659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 -- 172.21.15.28:0/3176237444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 msgr2=0x555cdeb5e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 0x555cdeb5e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 -- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 msgr2=0x555cdeb659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdeb659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e37fe640 1 -- 172.21.15.28:0/3176237444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cde899620 con 0x555cdeb54ed0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.713+0000 7f25e3fff640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 0x555cdeb5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25e37fe640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f25d80987f0 tx=0x7f25d8004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d57bd31c141d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d809b960 con 0x555cdeb54ed0 2024-03-22T02:16:36.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f25d80a45a0 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d4005040 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f25d809b780 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 10 client.16296 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555cdeb68e20 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d4005040 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f25d80bab90 con 0x555cdeb54ed0 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 10 client.16296 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:36.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 20 client.16296 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdeb6a4a0 0x555cdeb6c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 4 client.16296 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.714+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cdeac6580 con 0x555cdeb6a4a0 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.715+0000 7f25e3fff640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdeb6a4a0 0x555cdeb6c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.715+0000 7f25e3fff640 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdeb6a4a0 0x555cdeb6c910 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afb6303005553f4d server_cookie=870f88f7fd96d5c5 in_seq=0 out_seq=0 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.715+0000 7f25c37fe640 10 client.16296 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25c37fe640 1 -- 172.21.15.28:0/3176237444 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555cdeac6580 con 0x555cdeb6a4a0 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25c37fe640 10 client.16296 handle_command_reply: tid=0 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 client.16296 shutdown 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 msgr2=0x555cdeb5e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f25d80987f0 tx=0x7f25d8004b30 comp rx=0 tx=0).stop 2024-03-22T02:16:36.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 shutdown_connections 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdeb6a4a0 0x555cdeb6c910 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdeb659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb5e9a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25c37fe640 0 client.16296 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/3176237444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb544e0 0x555cdeb5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 >> 172.21.15.28:0/3176237444 conn(0x555cdeb4fa50 msgr2=0x555cdeb4ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 shutdown_connections 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.716+0000 7f25ed3bf740 1 -- 172.21.15.28:0/3176237444 wait complete. 2024-03-22T02:16:36.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.717+0000 7f25ed3bf740 20 client.16296 trim_cache size 0 max 16384 2024-03-22T02:16:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 Processor -- start 2024-03-22T02:16:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 -- start start 2024-03-22T02:16:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb544e0 0x555cdeb533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb4e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41ad230 con 0x555cdeb54ed0 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41aab00 con 0x555cdeb544e0 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41a8950 con 0x555cdeb5a040 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37778/0 (socket says 172.21.15.28:37778) 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 -- 172.21.15.28:0/2337781221 learned_addr learned my addr 172.21.15.28:0/2337781221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb544e0 0x555cdeb533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e8d74640 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb4e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 -- 172.21.15.28:0/2337781221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 msgr2=0x555cdeb4e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb4e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 -- 172.21.15.28:0/2337781221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb544e0 msgr2=0x555cdeb533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb544e0 0x555cdeb533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.732+0000 7f25e37fe640 1 -- 172.21.15.28:0/2337781221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cdeb5c920 con 0x555cdeb54ed0 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f25d80058c0 tx=0x7f25d80d3190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ed27a4bbd9ad86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25c9ffb640 1 -- 172.21.15.28:0/2337781221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d809b960 con 0x555cdeb54ed0 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25c9ffb640 1 -- 172.21.15.28:0/2337781221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d809c520 con 0x555cdeb54ed0 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25c9ffb640 1 -- 172.21.15.28:0/2337781221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d80a43a0 con 0x555cdeb54ed0 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2337781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 msgr2=0x555cdeb4e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f25d80058c0 tx=0x7f25d80d3190 comp rx=0 tx=0).stop 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2337781221 shutdown_connections 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb5a040 0x555cdeb4e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb54ed0 0x555cdeb4e3c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.733+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2337781221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb544e0 0x555cdeb533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2337781221 >> 172.21.15.28:0/2337781221 conn(0x555cdeb4fa50 msgr2=0x555cdeb52cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2337781221 shutdown_connections 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2337781221 wait complete. 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- start start 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 Processor -- start 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb54ed0 0x555cdebcc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdebd3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41a8fc0 con 0x555cdeb544e0 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41aa430 con 0x555cdeb5a040 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25e41a9690 con 0x555cdeb54ed0 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25ed3bf740 10 client.0 _ll_register_callbacks cb 0x7f25d809bce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25e3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37782/0 (socket says 172.21.15.28:37782) 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25e3fff640 1 -- 172.21.15.28:0/2965939801 learned_addr learned my addr 172.21.15.28:0/2965939801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.734+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb54ed0 0x555cdebcc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e8d74640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdebd3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 -- 172.21.15.28:0/2965939801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb54ed0 msgr2=0x555cdebcc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb54ed0 0x555cdebcc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 -- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 msgr2=0x555cdebd3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdebd3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 -- 172.21.15.28:0/2965939801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cdebd3920 con 0x555cdeb544e0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25e3fff640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f25d4000c00 tx=0x7f25d4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204066bf47321497 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25d4005d50 con 0x555cdeb544e0 2024-03-22T02:16:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f25d4004210 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25d809c520 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f25d405a050 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25ed3bf740 10 client.16302 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555cdebd4ca0 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.735+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25d809c520 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f25d40050c0 con 0x555cdeb544e0 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 10 client.16302 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 10 client.16302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 20 client.16302 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdebd52d0 0x555cdebd76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:36.742 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 4 client.16302 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:16:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555cdeac08d0 con 0x555cdebd52d0 2024-03-22T02:16:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdebd52d0 0x555cdebd76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25c37fe640 10 client.16302 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:36.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.736+0000 7f25e37fe640 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdebd52d0 0x555cdebd76c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d20df3de5bd6c1c5 server_cookie=6b5b2cf1305c0707 in_seq=0 out_seq=0 2024-03-22T02:16:36.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25c37fe640 1 -- 172.21.15.28:0/2965939801 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555cdeac08d0 con 0x555cdebd52d0 2024-03-22T02:16:36.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25c37fe640 10 client.16302 handle_command_reply: tid=0 2024-03-22T02:16:36.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 client.16302 shutdown 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 msgr2=0x555cdebcbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f25d4000c00 tx=0x7f25d4005b50 comp rx=0 tx=0).stop 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 shutdown_connections 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555cdebd52d0 0x555cdebd76c0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555cdeb5a040 0x555cdebd3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555cdeb54ed0 0x555cdebcc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25c37fe640 0 client.16302 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 --2- 172.21.15.28:0/2965939801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555cdeb544e0 0x555cdebcbdb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 >> 172.21.15.28:0/2965939801 conn(0x555cdeb4fa50 msgr2=0x555cdeb53540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 shutdown_connections 2024-03-22T02:16:36.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 1 -- 172.21.15.28:0/2965939801 wait complete. 2024-03-22T02:16:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.934+0000 7f25ed3bf740 20 client.16302 trim_cache size 0 max 16384 2024-03-22T02:16:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:16:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:16:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:16:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:16:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 msgr2=0x7f25cc0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 0x7f25cc0a5480 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x555cdeb55730 tx=0x7f25dc004a40 comp rx=0 tx=0).stop 2024-03-22T02:16:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 msgr2=0x7f25e4198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f25d800e9c0 tx=0x7f25d800ee80 comp rx=0 tx=0).stop 2024-03-22T02:16:36.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 shutdown_connections 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25cc0a2fc0 0x7f25cc0a5480 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f25e4107900 0x7f25e419f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25e4104fd0 0x7f25e4198350 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 --2- 172.21.15.28:0/4061678600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25e41026a0 0x7f25e4197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 >> 172.21.15.28:0/4061678600 conn(0x7f25e40fc030 msgr2=0x7f25e4102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 shutdown_connections 2024-03-22T02:16:36.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:36.935+0000 7f25ea7d7640 1 -- 172.21.15.28:0/4061678600 wait complete. 2024-03-22T02:16:37.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 105676 7.42 MB/sec warmup 79 sec latency 80.647 ms 2024-03-22T02:16:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:37 smithi120 ceph-mon[36803]: pgmap v449: 129 pgs: 129 active+clean; 42 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-22T02:16:38.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:37 smithi083 ceph-mon[36340]: pgmap v449: 129 pgs: 129 active+clean; 42 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-22T02:16:38.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:37 smithi028 ceph-mon[31916]: pgmap v449: 129 pgs: 129 active+clean; 42 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 318 op/s 2024-03-22T02:16:38.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:16:38.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/786341382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 msgr2=0x7fe8b4109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/786341382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b4109fe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009900 tx=0x7fe8ac02bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/786341382 shutdown_connections 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/786341382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b4109fe0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/786341382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 0x7fe8b4103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/786341382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 0x7fe8b4101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/786341382 >> 172.21.15.28:0/786341382 conn(0x7fe8b40fd130 msgr2=0x7fe8b40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.239+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/786341382 shutdown_connections 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/786341382 wait complete. 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 Processor -- start 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 -- start start 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 0x7fe8b4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 0x7fe8b41983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410d820 con 0x7fe8b4103750 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410d6a0 con 0x7fe8b4102d90 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8ba1f7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b410d9a0 con 0x7fe8b4101ba0 2024-03-22T02:16:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37808/0 (socket says 172.21.15.28:37808) 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8b3fff640 1 -- 172.21.15.28:0/4272932314 learned_addr learned my addr 172.21.15.28:0/4272932314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.240+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 0x7fe8b4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 0x7fe8b41983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 -- 172.21.15.28:0/4272932314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 msgr2=0x7fe8b4197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 0x7fe8b4197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 -- 172.21.15.28:0/4272932314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 msgr2=0x7fe8b41983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 0x7fe8b41983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 -- 172.21.15.28:0/4272932314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b419fa80 con 0x7fe8b4103750 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b3fff640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac005ab0 tx=0x7fe8ac02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b0ff9640 1 -- 172.21.15.28:0/4272932314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac041070 con 0x7fe8b4103750 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b419fd10 con 0x7fe8b4103750 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b0ff9640 1 -- 172.21.15.28:0/4272932314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8ac004450 con 0x7fe8b4103750 2024-03-22T02:16:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8b0ff9640 1 -- 172.21.15.28:0/4272932314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac003e60 con 0x7fe8b4103750 2024-03-22T02:16:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.241+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b41a01b0 con 0x7fe8b4103750 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.243+0000 7fe8b0ff9640 1 -- 172.21.15.28:0/4272932314 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe8ac044070 con 0x7fe8b4103750 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 Processor -- start 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 -- start start 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 0x562ab1f52850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f57920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f57f90 0x562ab1f5a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562ab1eb78c0 con 0x562ab1f52430 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562ab1e5fc20 con 0x562ab1f57f90 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562ab1eb3870 con 0x562ab1f52e20 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b0ff9640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 0x7fe89c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 0x562ab1f52850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f57920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 0x562ab1f52850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37816/0 (socket says 172.21.15.28:37816) 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/2239805649 learned_addr learned my addr 172.21.15.28:0/2239805649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f57f90 0x562ab1f5a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b3fff640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 0x7fe89c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.244+0000 7fe8b0ff9640 1 -- 172.21.15.28:0/4272932314 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe8ac0f1450 con 0x7fe8b4103750 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 -- 172.21.15.28:0/2239805649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f57f90 msgr2=0x562ab1f5a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f57f90 0x562ab1f5a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 -- 172.21.15.28:0/2239805649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 msgr2=0x562ab1f52850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 0x562ab1f52850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 -- 172.21.15.28:0/2239805649 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ab1e84150 con 0x562ab1f52e20 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f57920 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005fd0 tx=0x7fe8a40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f5df77446cbdfa8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe8b3fff640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 0x7fe89c0a53e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0b9c60 tx=0x7fe8ac0887f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe899ffb640 1 -- 172.21.15.28:0/2239805649 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4004b20 con 0x562ab1f52e20 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe899ffb640 1 -- 172.21.15.28:0/2239805649 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a4003e50 con 0x562ab1f52e20 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.245+0000 7fe899ffb640 1 -- 172.21.15.28:0/2239805649 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4005020 con 0x562ab1f52e20 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- 172.21.15.28:0/2239805649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 msgr2=0x562ab1f57920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f57920 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005fd0 tx=0x7fe8a40118d0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- 172.21.15.28:0/2239805649 shutdown_connections 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f57f90 0x562ab1f5a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f57920 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/2239805649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52430 0x562ab1f52850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- 172.21.15.28:0/2239805649 >> 172.21.15.28:0/2239805649 conn(0x562ab1f4d9a0 msgr2=0x562ab1f4dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- 172.21.15.28:0/2239805649 shutdown_connections 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- 172.21.15.28:0/2239805649 wait complete. 2024-03-22T02:16:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 -- start start 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.246+0000 7fe8bcde3740 1 Processor -- start 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f5ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f63e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562ab1f5af40 con 0x562ab1f52e20 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562ab1e73130 con 0x562ab1f52430 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562ab1f5b0c0 con 0x562ab1f57f90 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8bcde3740 10 client.0 _ll_register_callbacks cb 0x562ab1e72b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40510/0 (socket says 172.21.15.28:40510) 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/3782357235 learned_addr learned my addr 172.21.15.28:0/3782357235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f63e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f5ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/3782357235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 msgr2=0x562ab1f63e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f63e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/3782357235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 msgr2=0x562ab1f5ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f5ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/3782357235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ab1c957d0 con 0x562ab1f52430 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.247+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f5ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8013420 tx=0x7fe8a8009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326acd81ba09a498 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a8004310 con 0x562ab1f52430 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe8a8005020 con 0x562ab1f52430 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a4003e50 con 0x562ab1f52430 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe8a802b050 con 0x562ab1f52430 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe8bcde3740 10 client.26147 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ab1bcb3f0 con 0x562ab1f52430 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a4003e50 con 0x562ab1f52430 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe8a800f7c0 con 0x562ab1f52430 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.248+0000 7fe8bcde3740 10 client.26147 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8bcde3740 10 client.26147 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8bcde3740 20 client.26147 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1f6caf0 0x562ab1f6ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8bcde3740 4 client.26147 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ab1ec4e00 con 0x562ab1f6caf0 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1f6caf0 0x562ab1f6ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1f6caf0 0x562ab1f6ef70 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eb9e884a3babd9f server_cookie=45aba36ef8405926 in_seq=0 out_seq=0 2024-03-22T02:16:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.249+0000 7fe892ffd640 10 client.26147 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe892ffd640 1 -- 172.21.15.28:0/3782357235 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ab1ec4e00 con 0x562ab1f6caf0 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe892ffd640 10 client.26147 handle_command_reply: tid=0 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 client.26147 shutdown 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 msgr2=0x562ab1f5c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe8a8013420 tx=0x7fe8a8009c40 comp rx=0 tx=0).stop 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 shutdown_connections 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1f6caf0 0x562ab1f6ef70 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f63e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe892ffd640 0 client.26147 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f5ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/3782357235 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f5c8d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 >> 172.21.15.28:0/3782357235 conn(0x562ab1f4d9a0 msgr2=0x562ab1f4cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 shutdown_connections 2024-03-22T02:16:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 1 -- 172.21.15.28:0/3782357235 wait complete. 2024-03-22T02:16:38.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.250+0000 7fe8bcde3740 20 client.26147 trim_cache size 0 max 16384 2024-03-22T02:16:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 Processor -- start 2024-03-22T02:16:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 -- start start 2024-03-22T02:16:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f51310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f5bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41aafc0 con 0x562ab1f52e20 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41a8890 con 0x562ab1f52430 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41a66e0 con 0x562ab1f57f90 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f5bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37828/0 (socket says 172.21.15.28:37828) 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 -- 172.21.15.28:0/1450105674 learned_addr learned my addr 172.21.15.28:0/1450105674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f51310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 -- 172.21.15.28:0/1450105674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 msgr2=0x562ab1f5bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f5bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 -- 172.21.15.28:0/1450105674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 msgr2=0x562ab1f51310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f51310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.274+0000 7fe8b37fe640 1 -- 172.21.15.28:0/1450105674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ab1b8b0f0 con 0x562ab1f52e20 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005fd0 tx=0x7fe8a40092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f7e6bc52bdb15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8997fa640 1 -- 172.21.15.28:0/1450105674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a402d020 con 0x562ab1f52e20 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8997fa640 1 -- 172.21.15.28:0/1450105674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a4004e20 con 0x562ab1f52e20 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8997fa640 1 -- 172.21.15.28:0/1450105674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4003f00 con 0x562ab1f52e20 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8bcde3740 1 -- 172.21.15.28:0/1450105674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 msgr2=0x562ab1f63e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.275+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4005fd0 tx=0x7fe8a40092f0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- 172.21.15.28:0/1450105674 shutdown_connections 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f57f90 0x562ab1f5bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f52e20 0x562ab1f63e10 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/1450105674 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f51310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- 172.21.15.28:0/1450105674 >> 172.21.15.28:0/1450105674 conn(0x562ab1f4d9a0 msgr2=0x562ab1f50c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- 172.21.15.28:0/1450105674 shutdown_connections 2024-03-22T02:16:38.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- 172.21.15.28:0/1450105674 wait complete. 2024-03-22T02:16:38.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- start start 2024-03-22T02:16:38.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 Processor -- start 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f61500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f61a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41a6d50 con 0x562ab1f57f90 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41a81c0 con 0x562ab1f52430 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41a7420 con 0x562ab1f52e20 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8bcde3740 10 client.0 _ll_register_callbacks cb 0x7fe8a8013860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37834/0 (socket says 172.21.15.28:37834) 2024-03-22T02:16:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/566958231 learned_addr learned my addr 172.21.15.28:0/566958231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f61a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.276+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f61500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/566958231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 msgr2=0x562ab1f61a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f61a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 msgr2=0x562ab1f61500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f61500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 -- 172.21.15.28:0/566958231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ab1f63040 con 0x562ab1f57f90 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8b2ffd640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80095f0 tx=0x7fe8a8009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0953362c46ef3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a800f9a0 con 0x562ab1f57f90 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe8a80035f0 con 0x562ab1f57f90 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8a4004e20 con 0x562ab1f57f90 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe8a80136a0 con 0x562ab1f57f90 2024-03-22T02:16:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 10 client.16320 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562ab1fd2eb0 con 0x562ab1f57f90 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a4004e20 con 0x562ab1f57f90 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe8a8010980 con 0x562ab1f57f90 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 10 client.16320 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.277+0000 7fe8bcde3740 20 client.16320 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1fd38e0 0x562ab1fd5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 4 client.16320 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562ab1ebf150 con 0x562ab1fd38e0 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1fd38e0 0x562ab1fd5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8b37fe640 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1fd38e0 0x562ab1fd5d00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d2be7fa868a117 server_cookie=4be04cc9b624f21 in_seq=0 out_seq=0 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe892ffd640 10 client.16320 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe892ffd640 1 -- 172.21.15.28:0/566958231 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562ab1ebf150 con 0x562ab1fd38e0 2024-03-22T02:16:38.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe892ffd640 10 client.16320 handle_command_reply: tid=0 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 client.16320 shutdown 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 msgr2=0x562ab1f62a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80095f0 tx=0x7fe8a8009770 comp rx=0 tx=0).stop 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 shutdown_connections 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562ab1fd38e0 0x562ab1fd5d00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ab1f57f90 0x562ab1f62a50 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe892ffd640 0 client.16320 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562ab1f52e20 0x562ab1f61a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 --2- 172.21.15.28:0/566958231 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562ab1f52430 0x562ab1f61500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.278+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 >> 172.21.15.28:0/566958231 conn(0x562ab1f4d9a0 msgr2=0x562ab1f51490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 shutdown_connections 2024-03-22T02:16:38.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8bcde3740 1 -- 172.21.15.28:0/566958231 wait complete. 2024-03-22T02:16:38.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8bcde3740 20 client.16320 trim_cache size 0 max 16384 2024-03-22T02:16:38.287 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:38.287 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:16:38.287 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:16:38.287 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 msgr2=0x7fe89c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 0x7fe89c0a53e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac0b9c60 tx=0x7fe8ac0887f0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 msgr2=0x7fe8b419f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac005ab0 tx=0x7fe8ac02c6c0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 shutdown_connections 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe89c0a2f20 0x7fe89c0a53e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8b4103750 0x7fe8b419f410 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe8b4102d90 0x7fe8b41983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 --2- 172.21.15.28:0/4272932314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8b4101ba0 0x7fe8b4197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 >> 172.21.15.28:0/4272932314 conn(0x7fe8b40fd130 msgr2=0x7fe8b4106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 shutdown_connections 2024-03-22T02:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.279+0000 7fe8ba1f7640 1 -- 172.21.15.28:0/4272932314 wait complete. 2024-03-22T02:16:38.290 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:16:38.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:16:38.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 -- 172.21.15.28:0/2932682216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 msgr2=0x7f1e40102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/2932682216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e40102120 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f1e34009900 tx=0x7f1e3402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 -- 172.21.15.28:0/2932682216 shutdown_connections 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/2932682216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 0x7f1e40108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/2932682216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e40102120 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/2932682216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 0x7f1e40100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 -- 172.21.15.28:0/2932682216 >> 172.21.15.28:0/2932682216 conn(0x7f1e400fc020 msgr2=0x7f1e400fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 -- 172.21.15.28:0/2932682216 shutdown_connections 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 -- 172.21.15.28:0/2932682216 wait complete. 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.402+0000 7f1e45ca4640 1 Processor -- start 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 -- start start 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 0x7f1e40104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 0x7f1e40103710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4010c730 con 0x7f1e40101ca0 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4010c5b0 con 0x7f1e40102660 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e45ca4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4010c8b0 con 0x7f1e40100ab0 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 0x7f1e40104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37860/0 (socket says 172.21.15.28:37860) 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 -- 172.21.15.28:0/3112794874 learned_addr learned my addr 172.21.15.28:0/3112794874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3ffff640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 0x7f1e40103710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 -- 172.21.15.28:0/3112794874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 msgr2=0x7f1e40104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 0x7f1e40104b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 -- 172.21.15.28:0/3112794874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 msgr2=0x7f1e40103710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 0x7f1e40103710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 -- 172.21.15.28:0/3112794874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40103fd0 con 0x7f1e40101ca0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.403+0000 7f1e3effd640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1e34009230 tx=0x7f1e3402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.404+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34041070 con 0x7f1e40101ca0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.404+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e401a1fd0 con 0x7f1e40101ca0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.404+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e34004450 con 0x7f1e40101ca0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.404+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34003e20 con 0x7f1e40101ca0 2024-03-22T02:16:38.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.404+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e401a2440 con 0x7f1e40101ca0 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.405+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e401a26d0 con 0x7f1e40101ca0 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.406+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1e34044070 con 0x7f1e40101ca0 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.406+0000 7f1e3cff9640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 0x7f1e280a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.406+0000 7f1e3f7fe640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 0x7f1e280a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.406+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f1e340f1690 con 0x7f1e40101ca0 2024-03-22T02:16:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.407+0000 7f1e3f7fe640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 0x7f1e280a53e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1e40101b00 tx=0x7f1e30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 107395 7.44 MB/sec warmup 80 sec latency 32.084 ms 2024-03-22T02:16:38.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.410+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e340bda60 con 0x7f1e40101ca0 2024-03-22T02:16:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.639+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e401a2a60 con 0x7f1e40101ca0 2024-03-22T02:16:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.640+0000 7f1e3cff9640 1 -- 172.21.15.28:0/3112794874 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11869 (secure 0 0 0) 0x7f1e340c1b00 con 0x7f1e40101ca0 2024-03-22T02:16:38.641 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:16:38.641 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:15:24.687791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:16:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 msgr2=0x7f1e280a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 0x7f1e280a53e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1e40101b00 tx=0x7f1e30005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 msgr2=0x7f1e401031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1e34009230 tx=0x7f1e3402c650 comp rx=0 tx=0).stop 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 shutdown_connections 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1e280a2f20 0x7f1e280a53e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1e40102660 0x7f1e40103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e40101ca0 0x7f1e401031d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 --2- 172.21.15.28:0/3112794874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1e40100ab0 0x7f1e40104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 >> 172.21.15.28:0/3112794874 conn(0x7f1e400fc020 msgr2=0x7f1e40068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.642+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 shutdown_connections 2024-03-22T02:16:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.643+0000 7f1e45ca4640 1 -- 172.21.15.28:0/3112794874 wait complete. 2024-03-22T02:16:38.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 -- 172.21.15.28:0/2078564899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488102680 msgr2=0x7f9488108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 --2- 172.21.15.28:0/2078564899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488102680 0x7f9488108f10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9480009900 tx=0x7f948002bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 -- 172.21.15.28:0/2078564899 shutdown_connections 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 --2- 172.21.15.28:0/2078564899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488102680 0x7f9488108f10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 --2- 172.21.15.28:0/2078564899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488101cc0 0x7f9488102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 --2- 172.21.15.28:0/2078564899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488100ad0 0x7f9488100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 -- 172.21.15.28:0/2078564899 >> 172.21.15.28:0/2078564899 conn(0x7f94880fc040 msgr2=0x7f94880fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 -- 172.21.15.28:0/2078564899 shutdown_connections 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.765+0000 7f948d288640 1 -- 172.21.15.28:0/2078564899 wait complete. 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 Processor -- start 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 -- start start 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488101cc0 0x7f9488198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488102680 0x7f948819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f948810c730 con 0x7f9488100ad0 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f948810c5b0 con 0x7f9488102680 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f948d288640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f948810c8b0 con 0x7f9488101cc0 2024-03-22T02:16:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37868/0 (socket says 172.21.15.28:37868) 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f9486ffd640 1 -- 172.21.15.28:0/1457174531 learned_addr learned my addr 172.21.15.28:0/1457174531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.766+0000 7f94867fc640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488101cc0 0x7f9488198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f94877fe640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488102680 0x7f948819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 -- 172.21.15.28:0/1457174531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488101cc0 msgr2=0x7f9488198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488101cc0 0x7f9488198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 -- 172.21.15.28:0/1457174531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488102680 msgr2=0x7f948819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488102680 0x7f948819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 -- 172.21.15.28:0/1457174531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948819fb00 con 0x7f9488100ad0 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9478009d20 tx=0x7f947800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f947ffff640 1 -- 172.21.15.28:0/1457174531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478010630 con 0x7f9488100ad0 2024-03-22T02:16:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f947ffff640 1 -- 172.21.15.28:0/1457174531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9478010cb0 con 0x7f9488100ad0 2024-03-22T02:16:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948819fdf0 con 0x7f9488100ad0 2024-03-22T02:16:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f947ffff640 1 -- 172.21.15.28:0/1457174531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478018de0 con 0x7f9488100ad0 2024-03-22T02:16:38.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.767+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94881a0250 con 0x7f9488100ad0 2024-03-22T02:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f947ffff640 1 -- 172.21.15.28:0/1457174531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f947801f020 con 0x7f9488100ad0 2024-03-22T02:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 Processor -- start 2024-03-22T02:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 -- start start 2024-03-22T02:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f90050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f95120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2ef08d0 con 0x55f1b2f90620 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2e98e00 con 0x55f1b2f95790 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f948fe7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2eec880 con 0x55f1b2f8fc30 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f947ffff640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 0x7f94740a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f95120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f94867fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f94867fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40580/0 (socket says 172.21.15.28:40580) 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f94867fc640 1 -- 172.21.15.28:0/3304023094 learned_addr learned my addr 172.21.15.28:0/3304023094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.770+0000 7f94877fe640 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f90050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94877fe640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 0x7f94740a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 -- 172.21.15.28:0/3304023094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 msgr2=0x55f1b2f90050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f947ffff640 1 -- 172.21.15.28:0/1457174531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94780cfc40 con 0x7f9488100ad0 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f90050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 -- 172.21.15.28:0/3304023094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 msgr2=0x55f1b2f95120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f95120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 -- 172.21.15.28:0/3304023094 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1b2c043f0 con 0x55f1b2f95790 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94877fe640 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f90050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94867fc640 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9470009c90 tx=0x7f9470011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a15b0c423bb094e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f94877fe640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 0x7f94740a53e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9480006010 tx=0x7f948002c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f947d7fa640 1 -- 172.21.15.28:0/3304023094 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947001e070 con 0x55f1b2f95790 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f947d7fa640 1 -- 172.21.15.28:0/3304023094 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9470019b10 con 0x55f1b2f95790 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.771+0000 7f947d7fa640 1 -- 172.21.15.28:0/3304023094 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947000e9c0 con 0x55f1b2f95790 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- 172.21.15.28:0/3304023094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 msgr2=0x55f1b2f97c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9470009c90 tx=0x7f9470011810 comp rx=0 tx=0).stop 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- 172.21.15.28:0/3304023094 shutdown_connections 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2f97c20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f95120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 --2- 172.21.15.28:0/3304023094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f90050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- 172.21.15.28:0/3304023094 >> 172.21.15.28:0/3304023094 conn(0x55f1b2f86d90 msgr2=0x55f1b2f87190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- 172.21.15.28:0/3304023094 shutdown_connections 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- 172.21.15.28:0/3304023094 wait complete. 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 -- start start 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.772+0000 7f948fe7c740 1 Processor -- start 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2fa1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2f98740 con 0x55f1b2f90620 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2eac300 con 0x55f1b2f95790 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f1b2f988c0 con 0x55f1b2f8fc30 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f948fe7c740 10 client.0 _ll_register_callbacks cb 0x55f1b2ee7e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37894/0 (socket says 172.21.15.28:37894) 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 -- 172.21.15.28:0/2608944801 learned_addr learned my addr 172.21.15.28:0/2608944801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94877fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2fa1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 -- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 msgr2=0x55f1b2f9a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 -- 172.21.15.28:0/2608944801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 msgr2=0x55f1b2fa1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2fa1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 -- 172.21.15.28:0/2608944801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1b2cce7d0 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f94867fc640 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9470000c00 tx=0x7f9470009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbb5bd33828958 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f946e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.773+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947001e050 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9470005450 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9470019b10 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9470033050 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 10 client.16338 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1b2fa4a00 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9470019b10 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9470004990 con 0x55f1b2f90620 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 10 client.16338 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 10 client.16338 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:16:38.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 20 client.16338 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b2fa5f10 0x55f1b2fa8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 4 client.16338 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1b2efde10 con 0x55f1b2fa5f10 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.774+0000 7f9486ffd640 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b2fa5f10 0x55f1b2fa8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.775+0000 7f946e7fc640 10 client.16338 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.775+0000 7f9486ffd640 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b2fa5f10 0x55f1b2fa8340 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d2f84972096ee6b server_cookie=59b920032ae66e25 in_seq=0 out_seq=0 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.775+0000 7f946e7fc640 1 -- 172.21.15.28:0/2608944801 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1b2efde10 con 0x55f1b2fa5f10 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f946e7fc640 10 client.16338 handle_command_reply: tid=0 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 client.16338 shutdown 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 msgr2=0x55f1b2f9a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9470000c00 tx=0x7f9470009f60 comp rx=0 tx=0).stop 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 shutdown_connections 2024-03-22T02:16:38.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b2fa5f10 0x55f1b2fa8340 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b2fa1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9a650 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 --2- 172.21.15.28:0/2608944801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f946e7fc640 0 client.16338 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 >> 172.21.15.28:0/2608944801 conn(0x55f1b2f86d90 msgr2=0x55f1b2f85fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 shutdown_connections 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 1 -- 172.21.15.28:0/2608944801 wait complete. 2024-03-22T02:16:38.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.776+0000 7f948fe7c740 20 client.16338 trim_cache size 0 max 16384 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 Processor -- start 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 -- start start 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f85740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f95790 0x55f1b2f85c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94881ab210 con 0x55f1b2f90620 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94881a8ae0 con 0x55f1b2f8fc30 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f948fe7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94881a6930 con 0x55f1b2f95790 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37908/0 (socket says 172.21.15.28:37908) 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.790+0000 7f94867fc640 1 -- 172.21.15.28:0/1319804828 learned_addr learned my addr 172.21.15.28:0/1319804828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94877fe640 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f95790 0x55f1b2f85c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f85740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 -- 172.21.15.28:0/1319804828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f95790 msgr2=0x55f1b2f85c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f95790 0x55f1b2f85c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 -- 172.21.15.28:0/1319804828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f8fc30 msgr2=0x55f1b2f85740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f85740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 -- 172.21.15.28:0/1319804828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1b2f90a00 con 0x55f1b2f90620 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f94867fc640 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9470004fd0 tx=0x7f9470009b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa92d6811239d773 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f947cff9640 1 -- 172.21.15.28:0/1319804828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947001e070 con 0x55f1b2f90620 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f947cff9640 1 -- 172.21.15.28:0/1319804828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947002a070 con 0x55f1b2f90620 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.791+0000 7f947cff9640 1 -- 172.21.15.28:0/1319804828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9470025c30 con 0x55f1b2f90620 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- 172.21.15.28:0/1319804828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 msgr2=0x55f1b2fa1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9470004fd0 tx=0x7f9470009b10 comp rx=0 tx=0).stop 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- 172.21.15.28:0/1319804828 shutdown_connections 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f95790 0x55f1b2f85c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2fa1650 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1319804828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f85740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- 172.21.15.28:0/1319804828 >> 172.21.15.28:0/1319804828 conn(0x55f1b2f86d90 msgr2=0x55f1b2f9a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- 172.21.15.28:0/1319804828 shutdown_connections 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- 172.21.15.28:0/1319804828 wait complete. 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 -- start start 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 Processor -- start 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.792+0000 7f948fe7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b300a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94881a6fa0 con 0x55f1b2f90620 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94881a8410 con 0x55f1b2f95790 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94881a7670 con 0x55f1b2f8fc30 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 10 client.0 _ll_register_callbacks cb 0x7f947001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37916/0 (socket says 172.21.15.28:37916) 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55720/0 (socket says 172.21.15.28:55720) 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 -- 172.21.15.28:0/1111266525 learned_addr learned my addr 172.21.15.28:0/1111266525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94877fe640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b300a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 -- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 msgr2=0x55f1b2f9e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 -- 172.21.15.28:0/1111266525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 msgr2=0x55f1b300a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b300a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 -- 172.21.15.28:0/1111266525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1b300a830 con 0x55f1b2f90620 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f94867fc640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9470011850 tx=0x7f9470009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16aac2cfef10f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f946e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:38.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947003b030 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9470005250 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947002a070 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9470048050 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 10 client.16344 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.793+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f1b300bb30 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947002a070 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9470003680 con 0x55f1b2f90620 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 10 client.16344 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 10 client.16344 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 20 client.16344 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b300c3c0 0x55f1b300e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 4 client.16344 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1b2ef8160 con 0x55f1b300c3c0 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b300c3c0 0x55f1b300e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f9486ffd640 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b300c3c0 0x55f1b300e7e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56b163c6d5570c0c server_cookie=6729ee7fc72be16b in_seq=0 out_seq=0 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f946e7fc640 10 client.16344 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f946e7fc640 1 -- 172.21.15.28:0/1111266525 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1b2ef8160 con 0x55f1b300c3c0 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f946e7fc640 10 client.16344 handle_command_reply: tid=0 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.794+0000 7f948fe7c740 1 client.16344 shutdown 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 msgr2=0x55f1b2f9ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f9470011850 tx=0x7f9470009c30 comp rx=0 tx=0).stop 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 shutdown_connections 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f1b300c3c0 0x55f1b300e7e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f1b2f95790 0x55f1b300a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f1b2f90620 0x55f1b2f9ec90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 --2- 172.21.15.28:0/1111266525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f1b2f8fc30 0x55f1b2f9e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f946e7fc640 0 client.16344 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 >> 172.21.15.28:0/1111266525 conn(0x55f1b2f86d90 msgr2=0x55f1b2f89ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 shutdown_connections 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 1 -- 172.21.15.28:0/1111266525 wait complete. 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948fe7c740 20 client.16344 trim_cache size 0 max 16384 2024-03-22T02:16:38.803 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 msgr2=0x7f94740a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 0x7f94740a53e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9480006010 tx=0x7f948002c880 comp rx=0 tx=0).stop 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 msgr2=0x7f9488197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9478009d20 tx=0x7f947800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 shutdown_connections 2024-03-22T02:16:38.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94740a2f20 0x7f94740a53e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9488102680 0x7f948819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9488101cc0 0x7f9488198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 --2- 172.21.15.28:0/1457174531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9488100ad0 0x7f9488197f30 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 >> 172.21.15.28:0/1457174531 conn(0x7f94880fc040 msgr2=0x7f94880fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 shutdown_connections 2024-03-22T02:16:38.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.795+0000 7f948d288640 1 -- 172.21.15.28:0/1457174531 wait complete. 2024-03-22T02:16:38.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:16:38.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- 172.21.15.28:0/831525682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c101c80 msgr2=0x7fc46c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 --2- 172.21.15.28:0/831525682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c101c80 0x7fc46c102100 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fc454009900 tx=0x7fc45402bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:38.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- 172.21.15.28:0/831525682 shutdown_connections 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 --2- 172.21.15.28:0/831525682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c102640 0x7fc46c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 --2- 172.21.15.28:0/831525682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c101c80 0x7fc46c102100 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 --2- 172.21.15.28:0/831525682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c100a90 0x7fc46c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- 172.21.15.28:0/831525682 >> 172.21.15.28:0/831525682 conn(0x7fc46c0fc020 msgr2=0x7fc46c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- 172.21.15.28:0/831525682 shutdown_connections 2024-03-22T02:16:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- 172.21.15.28:0/831525682 wait complete. 2024-03-22T02:16:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 Processor -- start 2024-03-22T02:16:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.916+0000 7fc470e9a640 1 -- start start 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c101c80 0x7fc46c109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c102640 0x7fc46c109f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c10d830 con 0x7fc46c100a90 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c10d6b0 con 0x7fc46c101c80 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c10d9b0 con 0x7fc46c102640 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37924/0 (socket says 172.21.15.28:37924) 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 -- 172.21.15.28:0/2373934552 learned_addr learned my addr 172.21.15.28:0/2373934552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46ad9d640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c102640 0x7fc46c109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc463fff640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c101c80 0x7fc46c109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 -- 172.21.15.28:0/2373934552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c102640 msgr2=0x7fc46c109f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c102640 0x7fc46c109f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 -- 172.21.15.28:0/2373934552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c101c80 msgr2=0x7fc46c109a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c101c80 0x7fc46c109a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 -- 172.21.15.28:0/2373934552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46c10a820 con 0x7fc46c100a90 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e9c0 tx=0x7fc45c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc4637fe640 1 -- 172.21.15.28:0/2373934552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00cd80 con 0x7fc46c100a90 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc4637fe640 1 -- 172.21.15.28:0/2373934552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc45c0035f0 con 0x7fc46c100a90 2024-03-22T02:16:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46c1a1fb0 con 0x7fc46c100a90 2024-03-22T02:16:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc4637fe640 1 -- 172.21.15.28:0/2373934552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c010690 con 0x7fc46c100a90 2024-03-22T02:16:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.917+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc46c1a2410 con 0x7fc46c100a90 2024-03-22T02:16:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.919+0000 7fc4637fe640 1 -- 172.21.15.28:0/2373934552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc45c0037a0 con 0x7fc46c100a90 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc4637fe640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 0x7fc4580a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc473722740 1 Processor -- start 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc4637fe640 1 -- 172.21.15.28:0/2373934552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc45c0cfaf0 con 0x7fc46c100a90 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc463fff640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 0x7fc4580a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc473722740 1 -- start start 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc473722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc473722740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57d210 0x5647cc581d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.920+0000 7fc473722740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc584810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc473722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647cc4e18d0 con 0x5647cc57c820 2024-03-22T02:16:38.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc473722740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5647cc489e00 con 0x5647cc57d210 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc473722740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5647cc4dd880 con 0x5647cc582380 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37940/0 (socket says 172.21.15.28:37940) 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 -- 172.21.15.28:0/2909561203 learned_addr learned my addr 172.21.15.28:0/2909561203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46ad9d640 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc584810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc463fff640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 0x7fc4580a5460 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc45402c7b0 tx=0x7fc454005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc463fff640 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57d210 0x5647cc581d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 -- 172.21.15.28:0/2909561203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 msgr2=0x5647cc584810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc584810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 -- 172.21.15.28:0/2909561203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57d210 msgr2=0x5647cc581d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57d210 0x5647cc581d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 -- 172.21.15.28:0/2909561203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cc1f53f0 con 0x5647cc57c820 2024-03-22T02:16:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0a42f0 tx=0x7fc45c09ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fb3fd77650d29 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc460ff9640 1 -- 172.21.15.28:0/2909561203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c0b4020 con 0x5647cc57c820 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc460ff9640 1 -- 172.21.15.28:0/2909561203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc45c09be10 con 0x5647cc57c820 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.921+0000 7fc460ff9640 1 -- 172.21.15.28:0/2909561203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c09c070 con 0x5647cc57c820 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- 172.21.15.28:0/2909561203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 msgr2=0x5647cc57cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0a42f0 tx=0x7fc45c09ba10 comp rx=0 tx=0).stop 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- 172.21.15.28:0/2909561203 shutdown_connections 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc584810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57d210 0x5647cc581d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 --2- 172.21.15.28:0/2909561203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57c820 0x5647cc57cc40 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- 172.21.15.28:0/2909561203 >> 172.21.15.28:0/2909561203 conn(0x5647cc577d90 msgr2=0x5647cc578190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- 172.21.15.28:0/2909561203 shutdown_connections 2024-03-22T02:16:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- 172.21.15.28:0/2909561203 wait complete. 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 -- start start 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.922+0000 7fc473722740 1 Processor -- start 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc586d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc58e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5647cc585330 con 0x5647cc57d210 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5647cc49d300 con 0x5647cc57c820 2024-03-22T02:16:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5647cc5854b0 con 0x5647cc582380 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37952/0 (socket says 172.21.15.28:37952) 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc586d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc473722740 10 client.0 _ll_register_callbacks cb 0x5647cc4d8e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 -- 172.21.15.28:0/2060542984 learned_addr learned my addr 172.21.15.28:0/2060542984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc46ad9d640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc58e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 -- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 msgr2=0x5647cc58e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc58e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 -- 172.21.15.28:0/2060542984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 msgr2=0x5647cc586d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc586d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 -- 172.21.15.28:0/2060542984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cc2bf7d0 con 0x5647cc57d210 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc586d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc463fff640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc454005ec0 tx=0x7fc454009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66295aa4b158957 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc451ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.923+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4540046c0 con 0x5647cc57d210 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc454004860 con 0x5647cc57d210 2024-03-22T02:16:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc45c09be10 con 0x5647cc57d210 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc454004a60 con 0x5647cc57d210 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 10 client.16356 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5647cc5916c0 con 0x5647cc57d210 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc45c09be10 con 0x5647cc57d210 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc454034440 con 0x5647cc57d210 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 10 client.16356 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 10 client.16356 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 20 client.16356 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc592dc0 0x5647cc595260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 4 client.16356 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647cc4eee10 con 0x5647cc592dc0 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.924+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc592dc0 0x5647cc595260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.925+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc592dc0 0x5647cc595260 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90fff693652d216a server_cookie=c7f34d11ccc79925 in_seq=0 out_seq=0 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.925+0000 7fc451ffb640 10 client.16356 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc451ffb640 1 -- 172.21.15.28:0/2060542984 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5647cc4eee10 con 0x5647cc592dc0 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc451ffb640 10 client.16356 handle_command_reply: tid=0 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 client.16356 shutdown 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 msgr2=0x5647cc5872a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc454005ec0 tx=0x7fc454009260 comp rx=0 tx=0).stop 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 shutdown_connections 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc592dc0 0x5647cc595260 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc58e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc5872a0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc451ffb640 0 client.16356 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 --2- 172.21.15.28:0/2060542984 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc586d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 >> 172.21.15.28:0/2060542984 conn(0x5647cc577d90 msgr2=0x5647cc576fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 shutdown_connections 2024-03-22T02:16:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 1 -- 172.21.15.28:0/2060542984 wait complete. 2024-03-22T02:16:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.926+0000 7fc473722740 20 client.16356 trim_cache size 0 max 16384 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.939+0000 7fc473722740 1 Processor -- start 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.939+0000 7fc473722740 1 -- start start 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc57b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc599b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1af4a0 con 0x5647cc57d210 2024-03-22T02:16:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1acd70 con 0x5647cc57c820 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc473722740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1aabc0 con 0x5647cc582380 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37962/0 (socket says 172.21.15.28:37962) 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc57b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 -- 172.21.15.28:0/2779419363 learned_addr learned my addr 172.21.15.28:0/2779419363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc46ad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc599b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 -- 172.21.15.28:0/2779419363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 msgr2=0x5647cc599b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc599b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 -- 172.21.15.28:0/2779419363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 msgr2=0x5647cc57b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc57b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 -- 172.21.15.28:0/2779419363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cc57c460 con 0x5647cc57d210 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc46a59c640 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc57b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:38.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc463fff640 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc45402f020 tx=0x7fc454009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9dd0673943467d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc453fff640 1 -- 172.21.15.28:0/2779419363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454065070 con 0x5647cc57d210 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc453fff640 1 -- 172.21.15.28:0/2779419363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc454009520 con 0x5647cc57d210 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.940+0000 7fc453fff640 1 -- 172.21.15.28:0/2779419363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454005410 con 0x5647cc57d210 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- 172.21.15.28:0/2779419363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 msgr2=0x5647cc58e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc45402f020 tx=0x7fc454009800 comp rx=0 tx=0).stop 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- 172.21.15.28:0/2779419363 shutdown_connections 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc582380 0x5647cc599b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58e2a0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 --2- 172.21.15.28:0/2779419363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc57c820 0x5647cc57b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- 172.21.15.28:0/2779419363 >> 172.21.15.28:0/2779419363 conn(0x5647cc577d90 msgr2=0x5647cc57aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- 172.21.15.28:0/2779419363 shutdown_connections 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- 172.21.15.28:0/2779419363 wait complete. 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 -- start start 2024-03-22T02:16:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.941+0000 7fc473722740 1 Processor -- start 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 0x5647cc58d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1ab230 con 0x5647cc57d210 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1ac6a0 con 0x5647cc582380 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1ab900 con 0x5647cc57c820 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc473722740 10 client.0 _ll_register_callbacks cb 0x7fc45c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55794/0 (socket says 172.21.15.28:55794) 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46a59c640 1 -- 172.21.15.28:0/4210462070 learned_addr learned my addr 172.21.15.28:0/4210462070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46ad9d640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 0x5647cc58d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 -- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 msgr2=0x5647cc58bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 -- 172.21.15.28:0/4210462070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 msgr2=0x5647cc58d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 0x5647cc58d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 -- 172.21.15.28:0/4210462070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cc58d710 con 0x5647cc57d210 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46a59c640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc46ad9d640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 0x5647cc58d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc463fff640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58c110 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc454005ec0 tx=0x7fc454000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7377ed1fbf37b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc451ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.942+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454065040 con 0x5647cc57d210 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc454034ac0 con 0x5647cc57d210 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc454009520 con 0x5647cc57d210 2024-03-22T02:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc454078050 con 0x5647cc57d210 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 10 client.16362 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5647cc58d710 con 0x5647cc57d210 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc454009520 con 0x5647cc57d210 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc45406a450 con 0x5647cc57d210 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 10 client.16362 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 10 client.16362 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 20 client.16362 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc5f98c0 0x5647cc5fbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 4 client.16362 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5647cc4e9160 con 0x5647cc5f98c0 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.943+0000 7fc46a59c640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc5f98c0 0x5647cc5fbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc451ffb640 10 client.16362 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:38.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc46a59c640 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc5f98c0 0x5647cc5fbce0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ea9dfc2ca9cdf11 server_cookie=57325bbbf9811862 in_seq=0 out_seq=0 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc451ffb640 1 -- 172.21.15.28:0/4210462070 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5647cc4e9160 con 0x5647cc5f98c0 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc451ffb640 10 client.16362 handle_command_reply: tid=0 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 client.16362 shutdown 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 msgr2=0x5647cc58c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58c110 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc454005ec0 tx=0x7fc454000bc0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 shutdown_connections 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5647cc5f98c0 0x5647cc5fbce0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5647cc582380 0x5647cc58d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5647cc57d210 0x5647cc58c110 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc451ffb640 0 client.16362 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 --2- 172.21.15.28:0/4210462070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647cc57c820 0x5647cc58bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 >> 172.21.15.28:0/4210462070 conn(0x5647cc577d90 msgr2=0x5647cc57b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 shutdown_connections 2024-03-22T02:16:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 1 -- 172.21.15.28:0/4210462070 wait complete. 2024-03-22T02:16:38.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.944+0000 7fc473722740 20 client.16362 trim_cache size 0 max 16384 2024-03-22T02:16:38.953 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 msgr2=0x7fc4580a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 0x7fc4580a5460 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc45402c7b0 tx=0x7fc454005fd0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 msgr2=0x7fc46c10b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e9c0 tx=0x7fc45c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 shutdown_connections 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4580a2fa0 0x7fc4580a5460 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc46c102640 0x7fc46c109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc46c101c80 0x7fc46c109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 --2- 172.21.15.28:0/2373934552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc46c100a90 0x7fc46c10b410 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 >> 172.21.15.28:0/2373934552 conn(0x7fc46c0fc020 msgr2=0x7fc46c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 shutdown_connections 2024-03-22T02:16:38.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:38.945+0000 7fc470e9a640 1 -- 172.21.15.28:0/2373934552 wait complete. 2024-03-22T02:16:38.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:38 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3112794874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:38.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:16:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:38 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3112794874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:39.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:38 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3112794874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 -- 172.21.15.28:0/4199971770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4101ba0 msgr2=0x7f11c4101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/4199971770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4101ba0 0x7f11c4101fc0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009900 tx=0x7f11b802bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 -- 172.21.15.28:0/4199971770 shutdown_connections 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/4199971770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4103750 0x7f11c4109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/4199971770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4102d90 0x7f11c4103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/4199971770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4101ba0 0x7f11c4101fc0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 -- 172.21.15.28:0/4199971770 >> 172.21.15.28:0/4199971770 conn(0x7f11c40fd130 msgr2=0x7f11c40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 -- 172.21.15.28:0/4199971770 shutdown_connections 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.062+0000 7f11cb1d7640 1 -- 172.21.15.28:0/4199971770 wait complete. 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 Processor -- start 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 -- start start 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 0x7f11c4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 0x7f11c4198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410d810 con 0x7f11c4103750 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410d690 con 0x7f11c4102d90 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11cb1d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11c410d990 con 0x7f11c4101ba0 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 0x7f11c4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 0x7f11c4198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 0x7f11c4197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55800/0 (socket says 172.21.15.28:55800) 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 0x7f11c4198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40698/0 (socket says 172.21.15.28:40698) 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c8f73640 1 -- 172.21.15.28:0/459729065 learned_addr learned my addr 172.21.15.28:0/459729065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37988/0 (socket says 172.21.15.28:37988) 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 -- 172.21.15.28:0/459729065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 msgr2=0x7f11c4197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 0x7f11c4197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 -- 172.21.15.28:0/459729065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 msgr2=0x7f11c4198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 0x7f11c4198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.063+0000 7f11c9774640 1 -- 172.21.15.28:0/459729065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c419fb20 con 0x7f11c4103750 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11c9774640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00e940 tx=0x7f11bc00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11c1ffb640 1 -- 172.21.15.28:0/459729065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc004d20 con 0x7f11c4103750 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11c1ffb640 1 -- 172.21.15.28:0/459729065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11bc005020 con 0x7f11c4103750 2024-03-22T02:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c419fe10 con 0x7f11c4103750 2024-03-22T02:16:39.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11c1ffb640 1 -- 172.21.15.28:0/459729065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0116b0 con 0x7f11c4103750 2024-03-22T02:16:39.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.064+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c41a0280 con 0x7f11c4103750 2024-03-22T02:16:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.066+0000 7f11c1ffb640 1 -- 172.21.15.28:0/459729065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f11bc0054d0 con 0x7f11c4103750 2024-03-22T02:16:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.066+0000 7f11c1ffb640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 0x7f11b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.066+0000 7f11cddc3740 1 Processor -- start 2024-03-22T02:16:39.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.066+0000 7f11c1ffb640 1 -- 172.21.15.28:0/459729065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f11bc0cf850 con 0x7f11c4103750 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c8f73640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 0x7f11b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 -- start start 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 0x557852db6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852dbbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dbe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557852d1b8d0 con 0x557852db6820 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557852cc3e00 con 0x557852dbc380 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11cddc3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557852d17880 con 0x557852db7210 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 0x557852db6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852dbbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852dbbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55806/0 (socket says 172.21.15.28:55806) 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 0x557852db6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37994/0 (socket says 172.21.15.28:37994) 2024-03-22T02:16:39.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c3fff640 1 -- 172.21.15.28:0/288396526 learned_addr learned my addr 172.21.15.28:0/288396526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dbe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c8f73640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 0x7f11b40a53e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f11b802c8e0 tx=0x7f11b8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 -- 172.21.15.28:0/288396526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 msgr2=0x557852dbbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852dbbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 -- 172.21.15.28:0/288396526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 msgr2=0x557852db6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 0x557852db6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 -- 172.21.15.28:0/288396526 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557852a2f3f0 con 0x557852dbc380 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.067+0000 7f11c9774640 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dbe810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f11bc005c40 tx=0x7f11bc09bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca8767e3d550d27e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11aeffd640 1 -- 172.21.15.28:0/288396526 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc01f070 con 0x557852dbc380 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11aeffd640 1 -- 172.21.15.28:0/288396526 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11bc0051d0 con 0x557852dbc380 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11aeffd640 1 -- 172.21.15.28:0/288396526 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc097910 con 0x557852dbc380 2024-03-22T02:16:39.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 -- 172.21.15.28:0/288396526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 msgr2=0x557852dbe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dbe810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f11bc005c40 tx=0x7f11bc09bbc0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 -- 172.21.15.28:0/288396526 shutdown_connections 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dbe810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852dbbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 --2- 172.21.15.28:0/288396526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db6820 0x557852db6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 -- 172.21.15.28:0/288396526 >> 172.21.15.28:0/288396526 conn(0x557852db1d90 msgr2=0x557852db2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 -- 172.21.15.28:0/288396526 shutdown_connections 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.068+0000 7f11cddc3740 1 -- 172.21.15.28:0/288396526 wait complete. 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 -- start start 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 Processor -- start 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852dc0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dc8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557852dbf330 con 0x557852db7210 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557852cd7300 con 0x557852dbc380 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557852dbf4b0 con 0x557852db6820 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11cddc3740 10 client.0 _ll_register_callbacks cb 0x557852d12e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37998/0 (socket says 172.21.15.28:37998) 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 -- 172.21.15.28:0/317205621 learned_addr learned my addr 172.21.15.28:0/317205621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c8f73640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852dc0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c9774640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dc8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 -- 172.21.15.28:0/317205621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 msgr2=0x557852dc0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852dc0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 -- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 msgr2=0x557852dc8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dc8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.069+0000 7f11c3fff640 1 -- 172.21.15.28:0/317205621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557852af97d0 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11c3fff640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f11b00126e0 tx=0x7f11b0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bbd562de7d4d167 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b0012110 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f11b0004050 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11bc0051d0 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11b0025050 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 10 client.16374 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557852dcb750 con 0x557852db7210 2024-03-22T02:16:39.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11bc0051d0 con 0x557852db7210 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f11b0003850 con 0x557852db7210 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 10 client.16374 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 10 client.16374 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 20 client.16374 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852dccf40 0x557852dcf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 4 client.16374 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.070+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557852d28e10 con 0x557852dccf40 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.071+0000 7f11c8f73640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852dccf40 0x557852dcf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.071+0000 7f11c8f73640 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852dccf40 0x557852dcf3a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd61c45065c2c3c server_cookie=426c1f6e8757ec2c in_seq=0 out_seq=0 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.071+0000 7f11a3fff640 10 client.16374 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11a3fff640 1 -- 172.21.15.28:0/317205621 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557852d28e10 con 0x557852dccf40 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11a3fff640 10 client.16374 handle_command_reply: tid=0 2024-03-22T02:16:39.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 client.16374 shutdown 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 msgr2=0x557852dc12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f11b00126e0 tx=0x7f11b0012ba0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 shutdown_connections 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852dccf40 0x557852dcf3a0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852dc8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852dc12d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11a3fff640 0 client.16374 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 --2- 172.21.15.28:0/317205621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852dc0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 >> 172.21.15.28:0/317205621 conn(0x557852db1d90 msgr2=0x557852db0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 shutdown_connections 2024-03-22T02:16:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 1 -- 172.21.15.28:0/317205621 wait complete. 2024-03-22T02:16:39.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.072+0000 7f11cddc3740 20 client.16374 trim_cache size 0 max 16384 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 Processor -- start 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 -- start start 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852db6820 0x557852db0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852db0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41ad330 con 0x557852dbc380 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41aac00 con 0x557852db6820 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11cddc3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41a8a50 con 0x557852db7210 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11c9774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852db0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11c9774640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38012/0 (socket says 172.21.15.28:38012) 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11c9774640 1 -- 172.21.15.28:0/1507909925 learned_addr learned my addr 172.21.15.28:0/1507909925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.086+0000 7f11c8f73640 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852db6820 0x557852db0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 -- 172.21.15.28:0/1507909925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852db6820 msgr2=0x557852db0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852db6820 0x557852db0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 -- 172.21.15.28:0/1507909925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 msgr2=0x557852db11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 -- 172.21.15.28:0/1507909925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557852db75f0 con 0x557852db7210 2024-03-22T02:16:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c9774640 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11c3fff640 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852db0c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f11b0003e60 tx=0x7f11b0010430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a83379f8f6571ec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11ae7fc640 1 -- 172.21.15.28:0/1507909925 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b002d020 con 0x557852db7210 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11ae7fc640 1 -- 172.21.15.28:0/1507909925 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b00121d0 con 0x557852db7210 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.087+0000 7f11ae7fc640 1 -- 172.21.15.28:0/1507909925 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b000f790 con 0x557852db7210 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- 172.21.15.28:0/1507909925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 msgr2=0x557852db0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852db0c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f11b0003e60 tx=0x7f11b0010430 comp rx=0 tx=0).stop 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- 172.21.15.28:0/1507909925 shutdown_connections 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852dbc380 0x557852db11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db7210 0x557852db0c80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- 172.21.15.28:0/1507909925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852db6820 0x557852db0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- 172.21.15.28:0/1507909925 >> 172.21.15.28:0/1507909925 conn(0x557852db1d90 msgr2=0x557852db8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- 172.21.15.28:0/1507909925 shutdown_connections 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- 172.21.15.28:0/1507909925 wait complete. 2024-03-22T02:16:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 -- start start 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 Processor -- start 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852e2ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.088+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41a90c0 con 0x557852db7210 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41aa530 con 0x557852dbc380 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f11c41a9790 con 0x557852db6820 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 10 client.0 _ll_register_callbacks cb 0x7f11b0009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852e2ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c8f73640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852e2ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55838/0 (socket says 172.21.15.28:55838) 2024-03-22T02:16:39.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40742/0 (socket says 172.21.15.28:40742) 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c8f73640 1 -- 172.21.15.28:0/974737305 learned_addr learned my addr 172.21.15.28:0/974737305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c3fff640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38026/0 (socket says 172.21.15.28:38026) 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 -- 172.21.15.28:0/974737305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 msgr2=0x557852e2ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852e2ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 -- 172.21.15.28:0/974737305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 msgr2=0x557852e2f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 -- 172.21.15.28:0/974737305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557852e367e0 con 0x557852dbc380 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c3fff640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11c9774640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0053f0 tx=0x7f11bc004f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cd6d9b187818383 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0b8070 con 0x557852dbc380 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11b00121d0 con 0x557852dbc380 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f11bc09b210 con 0x557852dbc380 2024-03-22T02:16:39.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.089+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f11bc01f040 con 0x557852dbc380 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 10 client.26213 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557852e37ae0 con 0x557852dbc380 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b00121d0 con 0x557852dbc380 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f11bc0b4040 con 0x557852dbc380 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 10 client.26213 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 20 client.26213 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852e38100 0x557852e3a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 4 client.26213 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557852d23160 con 0x557852e38100 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11c8f73640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852e38100 0x557852e3a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11a3fff640 10 client.26213 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:39.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.090+0000 7f11c8f73640 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852e38100 0x557852e3a4f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4214a52baab0d04 server_cookie=8e62243360e72efb in_seq=0 out_seq=0 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11a3fff640 1 -- 172.21.15.28:0/974737305 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557852d23160 con 0x557852e38100 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11a3fff640 10 client.26213 handle_command_reply: tid=0 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 client.26213 shutdown 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 msgr2=0x557852e361f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0053f0 tx=0x7f11bc004f00 comp rx=0 tx=0).stop 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 shutdown_connections 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557852e38100 0x557852e3a4f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557852dbc380 0x557852e361f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11a3fff640 0 client.26213 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557852db7210 0x557852e2f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 --2- 172.21.15.28:0/974737305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557852db6820 0x557852e2ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 >> 172.21.15.28:0/974737305 conn(0x557852db1d90 msgr2=0x557852db4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 shutdown_connections 2024-03-22T02:16:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 1 -- 172.21.15.28:0/974737305 wait complete. 2024-03-22T02:16:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.091+0000 7f11cddc3740 20 client.26213 trim_cache size 0 max 16384 2024-03-22T02:16:39.099 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 msgr2=0x7f11b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 0x7f11b40a53e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f11b802c8e0 tx=0x7f11b8005e30 comp rx=0 tx=0).stop 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 msgr2=0x7f11c419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00e940 tx=0x7f11bc00ee00 comp rx=0 tx=0).stop 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 shutdown_connections 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f11b40a2f20 0x7f11b40a53e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11c4103750 0x7f11c419f4b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f11c4102d90 0x7f11c4198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 --2- 172.21.15.28:0/459729065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f11c4101ba0 0x7f11c4197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 >> 172.21.15.28:0/459729065 conn(0x7f11c40fd130 msgr2=0x7f11c41061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 shutdown_connections 2024-03-22T02:16:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:39.092+0000 7f11cb1d7640 1 -- 172.21.15.28:0/459729065 wait complete. 2024-03-22T02:16:39.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 110277 7.57 MB/sec warmup 81 sec latency 15.258 ms 2024-03-22T02:16:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:39 smithi120 ceph-mon[36803]: pgmap v450: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-22T02:16:40.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:39 smithi083 ceph-mon[36340]: pgmap v450: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-22T02:16:40.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:39 smithi028 ceph-mon[31916]: pgmap v450: 129 pgs: 129 active+clean; 40 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.2 MiB/s wr, 372 op/s 2024-03-22T02:16:40.107 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:16:40.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:16:40.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:16:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 -- 172.21.15.28:0/224846236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660102550 msgr2=0x7f1660104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 --2- 172.21.15.28:0/224846236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660102550 0x7f1660104940 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f165c009900 tx=0x7f165c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 -- 172.21.15.28:0/224846236 shutdown_connections 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 --2- 172.21.15.28:0/224846236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16601077b0 0x7f1660109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 --2- 172.21.15.28:0/224846236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1660104e80 0x7f1660107270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 --2- 172.21.15.28:0/224846236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660102550 0x7f1660104940 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 -- 172.21.15.28:0/224846236 >> 172.21.15.28:0/224846236 conn(0x7f16600fbf00 msgr2=0x7f16600fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 -- 172.21.15.28:0/224846236 shutdown_connections 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.229+0000 7f16671a0640 1 -- 172.21.15.28:0/224846236 wait complete. 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 Processor -- start 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 -- start start 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 0x7f1660197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 0x7f1660198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16601077b0 0x7f166019f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f166010d710 con 0x7f1660104e80 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f166010d590 con 0x7f1660102550 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f16671a0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f166010d890 con 0x7f16601077b0 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f166619e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 0x7f1660197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 0x7f1660198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f166619e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 0x7f1660197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40758/0 (socket says 172.21.15.28:40758) 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f166619e640 1 -- 172.21.15.28:0/184489967 learned_addr learned my addr 172.21.15.28:0/184489967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.230+0000 7f166699f640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16601077b0 0x7f166019f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 -- 172.21.15.28:0/184489967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16601077b0 msgr2=0x7f166019f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16601077b0 0x7f166019f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 -- 172.21.15.28:0/184489967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 msgr2=0x7f1660197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 0x7f1660197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 -- 172.21.15.28:0/184489967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f166019f9e0 con 0x7f1660104e80 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f166599d640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 0x7f1660198300 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f165000e9c0 tx=0x7f165000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f16577fe640 1 -- 172.21.15.28:0/184489967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165000ccc0 con 0x7f1660104e80 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f166019fcd0 con 0x7f1660104e80 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f16577fe640 1 -- 172.21.15.28:0/184489967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1650005020 con 0x7f1660104e80 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f16577fe640 1 -- 172.21.15.28:0/184489967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1650004a00 con 0x7f1660104e80 2024-03-22T02:16:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.231+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16601a01d0 con 0x7f1660104e80 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.233+0000 7f16577fe640 1 -- 172.21.15.28:0/184489967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f16500051d0 con 0x7f1660104e80 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 Processor -- start 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 -- start start 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f16577fe640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 0x7f164c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2feb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fedd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577b2f4a570 con 0x5577b2fe6770 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5577b2ef2e10 con 0x5577b2fe5d80 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166afe2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5577b2f46520 con 0x5577b2feb8e0 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166699f640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 0x7f164c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2feb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.234+0000 7f166619e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40760/0 (socket says 172.21.15.28:40760) 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 -- 172.21.15.28:0/966202607 learned_addr learned my addr 172.21.15.28:0/966202607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166699f640 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fedd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f16577fe640 1 -- 172.21.15.28:0/184489967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f16500cf800 con 0x7f1660104e80 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 -- 172.21.15.28:0/966202607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 msgr2=0x5577b2fedd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fedd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 -- 172.21.15.28:0/966202607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 msgr2=0x5577b2feb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2feb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 -- 172.21.15.28:0/966202607 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b2d267e0 con 0x5577b2fe5d80 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166599d640 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2feb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166699f640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 0x7f164c0a53e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1658005fd0 tx=0x7f1658009e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f166619e640 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f165c009230 tx=0x7f165c0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4439f99b637dc462 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f1654bfe640 1 -- 172.21.15.28:0/966202607 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c042070 con 0x5577b2fe5d80 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f1654bfe640 1 -- 172.21.15.28:0/966202607 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165c02cb80 con 0x5577b2fe5d80 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.235+0000 7f1654bfe640 1 -- 172.21.15.28:0/966202607 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c004ed0 con 0x5577b2fe5d80 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 -- 172.21.15.28:0/966202607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 msgr2=0x5577b2fe61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f165c009230 tx=0x7f165c0095e0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 -- 172.21.15.28:0/966202607 shutdown_connections 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fedd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2feb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 --2- 172.21.15.28:0/966202607 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fe61a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 -- 172.21.15.28:0/966202607 >> 172.21.15.28:0/966202607 conn(0x5577b2fe12f0 msgr2=0x5577b2fe16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 -- 172.21.15.28:0/966202607 shutdown_connections 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.236+0000 7f166afe2740 1 -- 172.21.15.28:0/966202607 wait complete. 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 -- start start 2024-03-22T02:16:40.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 Processor -- start 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 0x5577b2fefed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2feb8e0 0x5577b2ff7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577b2fee830 con 0x5577b2fe6770 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5577b2f06110 con 0x5577b2feb8e0 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5577b2fee9b0 con 0x5577b2fe5d80 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166afe2740 10 client.0 _ll_register_callbacks cb 0x5577b2f42be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166619e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 0x5577b2fefed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166619e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 0x5577b2fefed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55868/0 (socket says 172.21.15.28:55868) 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166699f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2feb8e0 0x5577b2ff7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166619e640 1 -- 172.21.15.28:0/3509598466 learned_addr learned my addr 172.21.15.28:0/3509598466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38064/0 (socket says 172.21.15.28:38064) 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 -- 172.21.15.28:0/3509598466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 msgr2=0x5577b2fefed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 0x5577b2fefed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 -- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2feb8e0 msgr2=0x5577b2ff7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2feb8e0 0x5577b2ff7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 -- 172.21.15.28:0/3509598466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b2d86f70 con 0x5577b2fe6770 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f166599d640 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1650097f20 tx=0x7f165000af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87aac50c1a34702c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.237+0000 7f1649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165009c1c0 con 0x5577b2fe6770 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f1650099dc0 con 0x5577b2fe6770 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f165c02cb80 con 0x5577b2fe6770 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f165009bac0 con 0x5577b2fe6770 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 10 client.16392 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5577b2c1e170 con 0x5577b2fe6770 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165c02cb80 con 0x5577b2fe6770 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f16500baa90 con 0x5577b2fe6770 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 10 client.16392 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 20 client.16392 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b2ffbe50 0x5577b2ffe2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 4 client.16392 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577b2f57ab0 con 0x5577b2ffbe50 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.238+0000 7f166619e640 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b2ffbe50 0x5577b2ffe2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.239+0000 7f166619e640 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b2ffbe50 0x5577b2ffe2d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=310e3786748695ff server_cookie=506c925b4e487a0b in_seq=0 out_seq=0 2024-03-22T02:16:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.239+0000 7f1649ffb640 10 client.16392 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f1649ffb640 1 -- 172.21.15.28:0/3509598466 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577b2f57ab0 con 0x5577b2ffbe50 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f1649ffb640 10 client.16392 handle_command_reply: tid=0 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 client.16392 shutdown 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 msgr2=0x5577b2ff0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1650097f20 tx=0x7f165000af00 comp rx=0 tx=0).stop 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 shutdown_connections 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b2ffbe50 0x5577b2ffe2d0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2feb8e0 0x5577b2ff7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff0410 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 --2- 172.21.15.28:0/3509598466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2fe5d80 0x5577b2fefed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 >> 172.21.15.28:0/3509598466 conn(0x5577b2fe12f0 msgr2=0x5577b2fe0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f1649ffb640 0 client.16392 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 shutdown_connections 2024-03-22T02:16:40.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 1 -- 172.21.15.28:0/3509598466 wait complete. 2024-03-22T02:16:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.240+0000 7f166afe2740 20 client.16392 trim_cache size 0 max 16384 2024-03-22T02:16:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 Processor -- start 2024-03-22T02:16:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 -- start start 2024-03-22T02:16:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fdfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fe0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16601acfa0 con 0x5577b2fe6770 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16601aa870 con 0x5577b2fe5d80 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166afe2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16601a86c0 con 0x5577b2feb8e0 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38070/0 (socket says 172.21.15.28:38070) 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 -- 172.21.15.28:0/2502459154 learned_addr learned my addr 172.21.15.28:0/2502459154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166619e640 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fdfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166699f640 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fe0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 -- 172.21.15.28:0/2502459154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 msgr2=0x5577b2fe0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fe0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 -- 172.21.15.28:0/2502459154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 msgr2=0x5577b2fdfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fdfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.259+0000 7f166599d640 1 -- 172.21.15.28:0/2502459154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577b2fe6b50 con 0x5577b2fe6770 2024-03-22T02:16:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f166599d640 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1650098490 tx=0x7f165000efb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574ac618fefd6e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f164bfff640 1 -- 172.21.15.28:0/2502459154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16500b2070 con 0x5577b2fe6770 2024-03-22T02:16:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f164bfff640 1 -- 172.21.15.28:0/2502459154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165009c1c0 con 0x5577b2fe6770 2024-03-22T02:16:40.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f164bfff640 1 -- 172.21.15.28:0/2502459154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16500a4140 con 0x5577b2fe6770 2024-03-22T02:16:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f166afe2740 1 -- 172.21.15.28:0/2502459154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 msgr2=0x5577b2ff7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.260+0000 7f166afe2740 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1650098490 tx=0x7f165000efb0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- 172.21.15.28:0/2502459154 shutdown_connections 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2fe0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe6770 0x5577b2ff7410 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- 172.21.15.28:0/2502459154 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe5d80 0x5577b2fdfcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- 172.21.15.28:0/2502459154 >> 172.21.15.28:0/2502459154 conn(0x5577b2fe12f0 msgr2=0x5577b2fefed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- 172.21.15.28:0/2502459154 shutdown_connections 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- 172.21.15.28:0/2502459154 wait complete. 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- start start 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 Processor -- start 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe6770 0x5577b2ff4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2ff5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16601a8d30 con 0x5577b2fe5d80 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16601aa1a0 con 0x5577b2fe6770 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16601a9400 con 0x5577b2feb8e0 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166afe2740 10 client.0 _ll_register_callbacks cb 0x7f165009b890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166619e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.261+0000 7f166619e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38080/0 (socket says 172.21.15.28:38080) 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 -- 172.21.15.28:0/3778724614 learned_addr learned my addr 172.21.15.28:0/3778724614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166699f640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2ff5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166599d640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe6770 0x5577b2ff4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 -- 172.21.15.28:0/3778724614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 msgr2=0x5577b2ff5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2ff5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 -- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe6770 msgr2=0x5577b2ff4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe6770 0x5577b2ff4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 -- 172.21.15.28:0/3778724614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577b3060270 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166619e640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f165c005ec0 tx=0x7f165c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43aad683fb28da2a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f1649ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f165c052070 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f165c04eca0 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f165009c1c0 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f165c05e050 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166afe2740 10 client.16398 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.262+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5577b2ff7470 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f165009c1c0 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f165c0047d0 con 0x5577b2fe5d80 2024-03-22T02:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 10 client.16398 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 20 client.16398 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b306add0 0x5577b306d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 4 client.16398 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5577b2f51e00 con 0x5577b306add0 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.263+0000 7f166599d640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b306add0 0x5577b306d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.264+0000 7f166599d640 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b306add0 0x5577b306d1c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8090f177557800d1 server_cookie=bfa38db016cce3e in_seq=0 out_seq=0 2024-03-22T02:16:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.264+0000 7f1649ffb640 10 client.16398 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f1649ffb640 1 -- 172.21.15.28:0/3778724614 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5577b2f51e00 con 0x5577b306add0 2024-03-22T02:16:40.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f1649ffb640 10 client.16398 handle_command_reply: tid=0 2024-03-22T02:16:40.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 client.16398 shutdown 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 msgr2=0x5577b2ff4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f165c005ec0 tx=0x7f165c004250 comp rx=0 tx=0).stop 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 shutdown_connections 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5577b306add0 0x5577b306d1c0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5577b2feb8e0 0x5577b2ff5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5577b2fe6770 0x5577b2ff4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 --2- 172.21.15.28:0/3778724614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577b2fe5d80 0x5577b2ff4960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 >> 172.21.15.28:0/3778724614 conn(0x5577b2fe12f0 msgr2=0x5577b2fe4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f1649ffb640 0 client.16398 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 shutdown_connections 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 1 -- 172.21.15.28:0/3778724614 wait complete. 2024-03-22T02:16:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.266+0000 7f166afe2740 20 client.16398 trim_cache size 0 max 16384 2024-03-22T02:16:40.272 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:40.272 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:16:40.272 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0d34129b-ca5a-4709-bf5b-05786f22dd51", 2024-03-22T02:16:40.272 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:16:40.272 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 msgr2=0x7f164c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 0x7f164c0a53e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1658005fd0 tx=0x7f1658009e50 comp rx=0 tx=0).stop 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 msgr2=0x7f1660198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 0x7f1660198300 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f165000e9c0 tx=0x7f165000ee80 comp rx=0 tx=0).stop 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 shutdown_connections 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f164c0a2f20 0x7f164c0a53e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16601077b0 0x7f166019f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1660104e80 0x7f1660198300 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 --2- 172.21.15.28:0/184489967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1660102550 0x7f1660197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 >> 172.21.15.28:0/184489967 conn(0x7f16600fbf00 msgr2=0x7f1660102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 shutdown_connections 2024-03-22T02:16:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.267+0000 7f16671a0640 1 -- 172.21.15.28:0/184489967 wait complete. 2024-03-22T02:16:40.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:16:40.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:16:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 -- 172.21.15.28:0/787411002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c103750 msgr2=0x7fd00c109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 --2- 172.21.15.28:0/787411002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c103750 0x7fd00c109fe0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009ca0 tx=0x7fcffc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 -- 172.21.15.28:0/787411002 shutdown_connections 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 --2- 172.21.15.28:0/787411002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c103750 0x7fd00c109fe0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 --2- 172.21.15.28:0/787411002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c102d90 0x7fd00c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 --2- 172.21.15.28:0/787411002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 -- 172.21.15.28:0/787411002 >> 172.21.15.28:0/787411002 conn(0x7fd00c0fd110 msgr2=0x7fd00c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 -- 172.21.15.28:0/787411002 shutdown_connections 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.385+0000 7fd012b0d640 1 -- 172.21.15.28:0/787411002 wait complete. 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 Processor -- start 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 -- start start 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c19a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c103750 0x7fd00c1a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d820 con 0x7fd00c102d90 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d6a0 con 0x7fd00c101ba0 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd012b0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c10d9a0 con 0x7fd00c103750 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c19a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38098/0 (socket says 172.21.15.28:38098) 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd00bfff640 1 -- 172.21.15.28:0/3290039698 learned_addr learned my addr 172.21.15.28:0/3290039698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.386+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c103750 0x7fd00c1a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 -- 172.21.15.28:0/3290039698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c103750 msgr2=0x7fd00c1a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c103750 0x7fd00c1a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 -- 172.21.15.28:0/3290039698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 msgr2=0x7fd00c19a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c19a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 -- 172.21.15.28:0/3290039698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00c1a1cc0 con 0x7fd00c102d90 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd0108a9640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd00bfff640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd009ffb640 1 -- 172.21.15.28:0/3290039698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff400cd80 con 0x7fd00c102d90 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd009ffb640 1 -- 172.21.15.28:0/3290039698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcff4005500 con 0x7fd00c102d90 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd00c1a1fb0 con 0x7fd00c102d90 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd009ffb640 1 -- 172.21.15.28:0/3290039698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4010690 con 0x7fd00c102d90 2024-03-22T02:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.387+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd00c1a2420 con 0x7fd00c102d90 2024-03-22T02:16:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd009ffb640 1 -- 172.21.15.28:0/3290039698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fcff40046c0 con 0x7fd00c102d90 2024-03-22T02:16:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 Processor -- start 2024-03-22T02:16:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 -- start start 2024-03-22T02:16:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5e20 0x5586c2ada920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2add420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5586c2a3a8c0 con 0x5586c2ad5430 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5586c29e2c20 con 0x5586c2adaf90 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0157cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586c2a36870 con 0x5586c2ad5e20 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0110aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0110aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38108/0 (socket says 172.21.15.28:38108) 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0110aa640 1 -- 172.21.15.28:0/3761892079 learned_addr learned my addr 172.21.15.28:0/3761892079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd00bfff640 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2add420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd0108a9640 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5e20 0x5586c2ada920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.390+0000 7fd009ffb640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 0x7fcff80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 -- 172.21.15.28:0/3761892079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5e20 msgr2=0x5586c2ada920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5e20 0x5586c2ada920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 -- 172.21.15.28:0/3761892079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 msgr2=0x5586c2add420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2add420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 -- 172.21.15.28:0/3761892079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586c2a07150 con 0x5586c2ad5430 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd009ffb640 1 -- 172.21.15.28:0/3290039698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fcff40cfc20 con 0x7fd00c102d90 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 0x7fcff80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009ca0 tx=0x7fcffc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e3b5ff81f7ca58 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd002ffd640 1 -- 172.21.15.28:0/3761892079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc039040 con 0x5586c2ad5430 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd002ffd640 1 -- 172.21.15.28:0/3761892079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcffc003e50 con 0x5586c2ad5430 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.391+0000 7fd002ffd640 1 -- 172.21.15.28:0/3761892079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc034440 con 0x5586c2ad5430 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- 172.21.15.28:0/3761892079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 msgr2=0x5586c2ad5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0110aa640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 0x7fcff80a53e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcffc04c000 tx=0x7fcffc045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009ca0 tx=0x7fcffc005f70 comp rx=0 tx=0).stop 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- 172.21.15.28:0/3761892079 shutdown_connections 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2add420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5e20 0x5586c2ada920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 --2- 172.21.15.28:0/3761892079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2ad5850 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- 172.21.15.28:0/3761892079 >> 172.21.15.28:0/3761892079 conn(0x5586c2ad09a0 msgr2=0x5586c2ad0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- 172.21.15.28:0/3761892079 shutdown_connections 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- 172.21.15.28:0/3761892079 wait complete. 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 -- start start 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.392+0000 7fd0157cd740 1 Processor -- start 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 0x5586c2adfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2ae6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5586c2addf40 con 0x5586c2ad5e20 2024-03-22T02:16:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5586c29f6130 con 0x5586c2adaf90 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5586c2ade0c0 con 0x5586c2ad5430 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0157cd740 10 client.0 _ll_register_callbacks cb 0x5586c29f5b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 0x5586c2adfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55928/0 (socket says 172.21.15.28:55928) 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0108a9640 1 -- 172.21.15.28:0/175360431 learned_addr learned my addr 172.21.15.28:0/175360431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0110aa640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2ae6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 -- 172.21.15.28:0/175360431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 msgr2=0x5586c2adf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 -- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 msgr2=0x5586c2ae6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2ae6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 -- 172.21.15.28:0/175360431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586c28187d0 con 0x5586c2ad5e20 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0108a9640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd0110aa640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2ae6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fd00bfff640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 0x5586c2adfea0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcff409bbe0 tx=0x7fcff4004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a53eb442da67c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fcfebfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff409c5e0 con 0x5586c2ad5e20 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.393+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fcff40a4460 con 0x5586c2ad5e20 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcffc003e50 con 0x5586c2ad5e20 2024-03-22T02:16:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcff409bee0 con 0x5586c2ad5e20 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 10 client.16410 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5586c274e3f0 con 0x5586c2ad5e20 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcffc003e50 con 0x5586c2ad5e20 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcff409ada0 con 0x5586c2ad5e20 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 10 client.16410 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 20 client.16410 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2aeb860 0x5586c2aedc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 4 client.16410 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586c2a47e00 con 0x5586c2aeb860 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.394+0000 7fd0108a9640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2aeb860 0x5586c2aedc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.395+0000 7fcfebfff640 10 client.16410 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.395+0000 7fd0108a9640 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2aeb860 0x5586c2aedc70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dce65f8fbd8a7ac server_cookie=aa1add1beaebf9c5 in_seq=0 out_seq=0 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fcfebfff640 1 -- 172.21.15.28:0/175360431 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5586c2a47e00 con 0x5586c2aeb860 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fcfebfff640 10 client.16410 handle_command_reply: tid=0 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 client.16410 shutdown 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 msgr2=0x5586c2adfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 0x5586c2adfea0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcff409bbe0 tx=0x7fcff4004ae0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 shutdown_connections 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2aeb860 0x5586c2aedc70 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2adaf90 0x5586c2ae6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fcfebfff640 0 client.16410 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5e20 0x5586c2adfea0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 --2- 172.21.15.28:0/175360431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ad5430 0x5586c2adf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.396+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 >> 172.21.15.28:0/175360431 conn(0x5586c2ad09a0 msgr2=0x5586c2acfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.397+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 shutdown_connections 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.397+0000 7fd0157cd740 1 -- 172.21.15.28:0/175360431 wait complete. 2024-03-22T02:16:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.397+0000 7fd0157cd740 20 client.16410 trim_cache size 0 max 16384 2024-03-22T02:16:40.409 INFO:tasks.workunit.client.0.smithi028.stdout: 1 112607 7.58 MB/sec warmup 82 sec latency 27.588 ms 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.414+0000 7fd0157cd740 1 Processor -- start 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.414+0000 7fd0157cd740 1 -- start start 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2adaf90 0x5586c2acf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1af490 con 0x5586c2ad5430 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1acd60 con 0x5586c2ad5e20 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0157cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1aabb0 con 0x5586c2adaf90 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38130/0 (socket says 172.21.15.28:38130) 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 -- 172.21.15.28:0/2979720295 learned_addr learned my addr 172.21.15.28:0/2979720295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd00bfff640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0110aa640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2adaf90 0x5586c2acf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 -- 172.21.15.28:0/2979720295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2adaf90 msgr2=0x5586c2acf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2adaf90 0x5586c2acf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 -- 172.21.15.28:0/2979720295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 msgr2=0x5586c2ae6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 -- 172.21.15.28:0/2979720295 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586c270e0f0 con 0x5586c2ad5430 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd00bfff640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd004000c00 tx=0x7fd004005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1499a986962b2b29 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0027fc640 1 -- 172.21.15.28:0/2979720295 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00400fca0 con 0x5586c2ad5430 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0027fc640 1 -- 172.21.15.28:0/2979720295 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd00400fe40 con 0x5586c2ad5430 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.415+0000 7fd0027fc640 1 -- 172.21.15.28:0/2979720295 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd004022640 con 0x5586c2ad5430 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- 172.21.15.28:0/2979720295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 msgr2=0x5586c2acf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd004000c00 tx=0x7fd004005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- 172.21.15.28:0/2979720295 shutdown_connections 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2adaf90 0x5586c2acf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2979720295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2ad5430 0x5586c2acf370 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- 172.21.15.28:0/2979720295 >> 172.21.15.28:0/2979720295 conn(0x5586c2ad09a0 msgr2=0x5586c2adf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- 172.21.15.28:0/2979720295 shutdown_connections 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- 172.21.15.28:0/2979720295 wait complete. 2024-03-22T02:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 -- start start 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.416+0000 7fd0157cd740 1 Processor -- start 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ae5aa0 0x5586c2ae5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1ab220 con 0x5586c2adaf90 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1ac690 con 0x5586c2ad5e20 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd00c1ab8f0 con 0x5586c2ae5aa0 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0157cd740 10 client.0 _ll_register_callbacks cb 0x7fcff400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0110aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ae5aa0 0x5586c2ae5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38142/0 (socket says 172.21.15.28:38142) 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd00bfff640 1 -- 172.21.15.28:0/2620949303 learned_addr learned my addr 172.21.15.28:0/2620949303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 -- 172.21.15.28:0/2620949303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ae5aa0 msgr2=0x5586c2ae5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ae5aa0 0x5586c2ae5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 -- 172.21.15.28:0/2620949303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 msgr2=0x5586c2ae4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd0108a9640 1 -- 172.21.15.28:0/2620949303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586c2ae6460 con 0x5586c2ad5e20 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.417+0000 7fd00bfff640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0108a9640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae4550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5586c2add6c0 tx=0x7fd00400fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4161eefe03d8de2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:16:40.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0040225c0 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd00400fe40 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fd004022760 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd00403c050 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 10 client.26249 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5586c2b51e50 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd00400fe40 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd004004b50 con 0x5586c2ad5e20 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 10 client.26249 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 10 client.26249 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 20 client.26249 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:40.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2b527b0 0x5586c2b54bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 4 client.26249 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.418+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5586c2a42150 con 0x5586c2b527b0 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd00bfff640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2b527b0 0x5586c2b54bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fcfebfff640 10 client.26249 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd00bfff640 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2b527b0 0x5586c2b54bd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0cb2fa0741e5cf7 server_cookie=cfbf2e0ee1d862ec in_seq=0 out_seq=0 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fcfebfff640 1 -- 172.21.15.28:0/2620949303 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5586c2a42150 con 0x5586c2b527b0 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fcfebfff640 10 client.26249 handle_command_reply: tid=0 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 client.26249 shutdown 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 msgr2=0x5586c2ae4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae4550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x5586c2add6c0 tx=0x7fd00400fca0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 shutdown_connections 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5586c2b527b0 0x5586c2b54bd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5586c2ae5aa0 0x5586c2ae5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fcfebfff640 0 client.26249 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5586c2adaf90 0x5586c2ae4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 --2- 172.21.15.28:0/2620949303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5586c2ad5e20 0x5586c2ae4550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.419+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 >> 172.21.15.28:0/2620949303 conn(0x5586c2ad09a0 msgr2=0x5586c2ad3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 shutdown_connections 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd0157cd740 1 -- 172.21.15.28:0/2620949303 wait complete. 2024-03-22T02:16:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd0157cd740 20 client.26249 trim_cache size 0 max 16384 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "0d34129b-ca5a-4709-bf5b-05786f22dd51": { 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0d34129b-ca5a-4709-bf5b-05786f22dd51", 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:16:40.429 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:16:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 msgr2=0x7fcff80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 0x7fcff80a53e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcffc04c000 tx=0x7fcffc045000 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 msgr2=0x7fd00c19a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcff400e9f0 tx=0x7fcff400edf0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 shutdown_connections 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcff80a2f20 0x7fcff80a53e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd00c103750 0x7fd00c1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00c102d90 0x7fd00c19a5e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 --2- 172.21.15.28:0/3290039698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd00c101ba0 0x7fd00c19a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 >> 172.21.15.28:0/3290039698 conn(0x7fd00c0fd110 msgr2=0x7fd00c073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 shutdown_connections 2024-03-22T02:16:40.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:16:40.420+0000 7fd012b0d640 1 -- 172.21.15.28:0/3290039698 wait complete. 2024-03-22T02:16:40.432 INFO:tasks.cephfs.filesystem:scrub status for tag:0d34129b-ca5a-4709-bf5b-05786f22dd51 - {'path': '/', 'tag': '0d34129b-ca5a-4709-bf5b-05786f22dd51', 'options': 'recursive,force'} 2024-03-22T02:16:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:16:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:40.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:16:40.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:16:40.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:40.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:40 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:40 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:40 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:16:41.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 114333 7.63 MB/sec warmup 83 sec latency 63.897 ms 2024-03-22T02:16:41.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:41 smithi120 ceph-mon[36803]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 341 op/s 2024-03-22T02:16:42.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:41 smithi083 ceph-mon[36340]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 341 op/s 2024-03-22T02:16:42.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:41 smithi028 ceph-mon[31916]: pgmap v451: 129 pgs: 129 active+clean; 39 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.1 MiB/s wr, 341 op/s 2024-03-22T02:16:42.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 117227 7.75 MB/sec warmup 84 sec latency 29.445 ms 2024-03-22T02:16:43.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 119480 7.75 MB/sec warmup 85 sec latency 43.348 ms 2024-03-22T02:16:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:43 smithi120 ceph-mon[36803]: pgmap v452: 129 pgs: 129 active+clean; 40 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-03-22T02:16:44.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:43 smithi083 ceph-mon[36340]: pgmap v452: 129 pgs: 129 active+clean; 40 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-03-22T02:16:44.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:43 smithi028 ceph-mon[31916]: pgmap v452: 129 pgs: 129 active+clean; 40 MiB data, 555 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 321 op/s 2024-03-22T02:16:44.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 121393 7.81 MB/sec warmup 86 sec latency 28.357 ms 2024-03-22T02:16:45.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 124394 7.93 MB/sec warmup 87 sec latency 17.346 ms 2024-03-22T02:16:45.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:45 smithi120 ceph-mon[36803]: pgmap v453: 129 pgs: 129 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 415 op/s 2024-03-22T02:16:46.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:45 smithi083 ceph-mon[36340]: pgmap v453: 129 pgs: 129 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 415 op/s 2024-03-22T02:16:46.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:45 smithi028 ceph-mon[31916]: pgmap v453: 129 pgs: 129 active+clean; 43 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 415 op/s 2024-03-22T02:16:46.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 126610 7.93 MB/sec warmup 88 sec latency 62.284 ms 2024-03-22T02:16:47.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 128329 7.97 MB/sec warmup 89 sec latency 81.980 ms 2024-03-22T02:16:47.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:47 smithi120 ceph-mon[36803]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 351 op/s 2024-03-22T02:16:48.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:47 smithi083 ceph-mon[36340]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 351 op/s 2024-03-22T02:16:48.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:47 smithi028 ceph-mon[31916]: pgmap v454: 129 pgs: 129 active+clean; 43 MiB data, 574 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.2 MiB/s wr, 351 op/s 2024-03-22T02:16:48.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 131286 8.08 MB/sec warmup 90 sec latency 28.540 ms 2024-03-22T02:16:49.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 132914 8.06 MB/sec warmup 91 sec latency 27.116 ms 2024-03-22T02:16:49.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:49 smithi120 ceph-mon[36803]: pgmap v455: 129 pgs: 129 active+clean; 49 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:16:49.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:49 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:16:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:49 smithi083 ceph-mon[36340]: pgmap v455: 129 pgs: 129 active+clean; 49 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:16:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:49 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:16:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2803446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88102630 msgr2=0x7f5e88104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2803446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88102630 0x7f5e88104a20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009900 tx=0x7f5e8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:50.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2803446669 shutdown_connections 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2803446669 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88107890 0x7f5e88109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2803446669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 0x7f5e88107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2803446669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88102630 0x7f5e88104a20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.024+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2803446669 >> 172.21.15.28:0/2803446669 conn(0x7f5e880fc020 msgr2=0x7f5e880fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2803446669 shutdown_connections 2024-03-22T02:16:50.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2803446669 wait complete. 2024-03-22T02:16:50.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 Processor -- start 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 -- start start 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 0x7f5e88197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.025+0000 7f5e8ce50640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 0x7f5e881983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810d8b0 con 0x7f5e88107890 2024-03-22T02:16:50.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810d730 con 0x7f5e88102630 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810da30 con 0x7f5e88104f60 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 0x7f5e88197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e86d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49316/0 (socket says 172.21.15.28:49316) 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e86d9d640 1 -- 172.21.15.28:0/2164369661 learned_addr learned my addr 172.21.15.28:0/2164369661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 0x7f5e881983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 -- 172.21.15.28:0/2164369661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 msgr2=0x7f5e881983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 0x7f5e881983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 -- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 msgr2=0x7f5e8819f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 -- 172.21.15.28:0/2164369661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8819fa90 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 0x7f5e88197e70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e80036040 tx=0x7f5e8002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e7f7fe640 1 -- 172.21.15.28:0/2164369661 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80043020 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e8819fd20 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e7f7fe640 1 -- 172.21.15.28:0/2164369661 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800054e0 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e881a01d0 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.026+0000 7f5e7f7fe640 1 -- 172.21.15.28:0/2164369661 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80003bd0 con 0x7f5e88102630 2024-03-22T02:16:50.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.028+0000 7f5e7f7fe640 1 -- 172.21.15.28:0/2164369661 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5e80046020 con 0x7f5e88102630 2024-03-22T02:16:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 Processor -- start 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 -- start start 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e7f7fe640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 0x7f5e740a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb1ed0 0x562232eb69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 0x7f5e740a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e7f7fe640 1 -- 172.21.15.28:0/2164369661 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5e800f1760 con 0x7f5e88102630 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232eb94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562232e16040 con 0x562232eb14e0 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562232dbe9e0 con 0x562232eb7040 2024-03-22T02:16:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.029+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562232e11ff0 con 0x562232eb1ed0 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb1ed0 0x562232eb69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49328/0 (socket says 172.21.15.28:49328) 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e86d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232eb94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 -- 172.21.15.28:0/2364461275 learned_addr learned my addr 172.21.15.28:0/2364461275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 -- 172.21.15.28:0/2364461275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb1ed0 msgr2=0x562232eb69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb1ed0 0x562232eb69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 -- 172.21.15.28:0/2364461275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 msgr2=0x562232eb94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232eb94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 -- 172.21.15.28:0/2364461275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562232b2a210 con 0x562232eb14e0 2024-03-22T02:16:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 0x7f5e740a5480 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5e780049d0 tx=0x7f5e780136a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5e800ba450 tx=0x7f5e80003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c491ac57dd4ff1e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e7cff9640 1 -- 172.21.15.28:0/2364461275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800be180 con 0x562232eb14e0 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e7cff9640 1 -- 172.21.15.28:0/2364461275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800d5070 con 0x562232eb14e0 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.030+0000 7f5e7cff9640 1 -- 172.21.15.28:0/2364461275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800bbd80 con 0x562232eb14e0 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2364461275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 msgr2=0x562232eb1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5e800ba450 tx=0x7f5e80003760 comp rx=0 tx=0).stop 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2364461275 shutdown_connections 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232eb94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb1ed0 0x562232eb69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2364461275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb14e0 0x562232eb1900 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2364461275 >> 172.21.15.28:0/2364461275 conn(0x562232eaca50 msgr2=0x562232eace50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2364461275 shutdown_connections 2024-03-22T02:16:50.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2364461275 wait complete. 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 -- start start 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.031+0000 7f5e8fa31740 1 Processor -- start 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232ebb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232ec2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562232eb9db0 con 0x562232eb1ed0 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562232dd1db0 con 0x562232eb14e0 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562232eb9f30 con 0x562232eb7040 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8fa31740 10 client.0 _ll_register_callbacks cb 0x562232b29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232ebb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49332/0 (socket says 172.21.15.28:49332) 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e85d9b640 1 -- 172.21.15.28:0/2009141795 learned_addr learned my addr 172.21.15.28:0/2009141795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232ec2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 -- 172.21.15.28:0/2009141795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 msgr2=0x562232ec2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232ec2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 -- 172.21.15.28:0/2009141795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 msgr2=0x562232ebb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 -- 172.21.15.28:0/2009141795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562232bf6620 con 0x562232eb14e0 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e8659c640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232ebb200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x562232eb9bc0 tx=0x7f5e8002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1cad1c0adf9ed4c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.032+0000 7f5e6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:16:50.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e800a1060 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5e80003d70 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e800d5070 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e800bda80 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 10 client.26261 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562232eb9db0 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e800d5070 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5e800bced0 con 0x562232eb14e0 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 10 client.26261 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 10 client.26261 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 20 client.26261 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232ec6fe0 0x562232ec9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 4 client.26261 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562232e23580 con 0x562232ec6fe0 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.033+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232ec6fe0 0x562232ec9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.034+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232ec6fe0 0x562232ec9420 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b50ad231deb31f59 server_cookie=1948fd6028c1fafc in_seq=0 out_seq=0 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.034+0000 7f5e6dffb640 10 client.26261 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e6dffb640 1 -- 172.21.15.28:0/2009141795 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562232e23580 con 0x562232ec6fe0 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e6dffb640 10 client.26261 handle_command_reply: tid=0 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 client.26261 shutdown 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 msgr2=0x562232ebb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232ebb200 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x562232eb9bc0 tx=0x7f5e8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 shutdown_connections 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232ec6fe0 0x562232ec9420 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232ec2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232ebb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/2009141795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232ebb200 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e6dffb640 0 client.26261 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 >> 172.21.15.28:0/2009141795 conn(0x562232eaca50 msgr2=0x562232eabc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 shutdown_connections 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 1 -- 172.21.15.28:0/2009141795 wait complete. 2024-03-22T02:16:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.035+0000 7f5e8fa31740 20 client.26261 trim_cache size 0 max 16384 2024-03-22T02:16:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 Processor -- start 2024-03-22T02:16:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 -- start start 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb14e0 0x562232ebb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232ecdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881ad270 con 0x562232eb1ed0 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881aab40 con 0x562232eb7040 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881a8990 con 0x562232eb14e0 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49334/0 (socket says 172.21.15.28:49334) 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e8659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb14e0 0x562232ebb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e85d9b640 1 -- 172.21.15.28:0/3893066743 learned_addr learned my addr 172.21.15.28:0/3893066743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.050+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232ecdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 -- 172.21.15.28:0/3893066743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb14e0 msgr2=0x562232ebb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb14e0 0x562232ebb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 -- 172.21.15.28:0/3893066743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 msgr2=0x562232ecdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232ecdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 -- 172.21.15.28:0/3893066743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562232ead7c0 con 0x562232eb1ed0 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x562232ebc6e0 tx=0x7f5e70005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45e461a4122474e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e6ffff640 1 -- 172.21.15.28:0/3893066743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70004030 con 0x562232eb1ed0 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e6ffff640 1 -- 172.21.15.28:0/3893066743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e70005500 con 0x562232eb1ed0 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.051+0000 7f5e6ffff640 1 -- 172.21.15.28:0/3893066743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70022710 con 0x562232eb1ed0 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- 172.21.15.28:0/3893066743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 msgr2=0x562232eb3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x562232ebc6e0 tx=0x7f5e70005f90 comp rx=0 tx=0).stop 2024-03-22T02:16:50.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- 172.21.15.28:0/3893066743 shutdown_connections 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb7040 0x562232ecdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232eb3050 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/3893066743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb14e0 0x562232ebb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- 172.21.15.28:0/3893066743 >> 172.21.15.28:0/3893066743 conn(0x562232eaca50 msgr2=0x562232eb2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- 172.21.15.28:0/3893066743 shutdown_connections 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- 172.21.15.28:0/3893066743 wait complete. 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- start start 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 Processor -- start 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232f28d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232f302b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881a9000 con 0x562232eb1ed0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881aa470 con 0x562232eb14e0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e881a96d0 con 0x562232eb7040 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8fa31740 10 client.0 _ll_register_callbacks cb 0x7f5e78013bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e85d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49338/0 (socket says 172.21.15.28:49338) 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e85d9b640 1 -- 172.21.15.28:0/724139626 learned_addr learned my addr 172.21.15.28:0/724139626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.052+0000 7f5e8659c640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232f28d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e86d9d640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232f302b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 -- 172.21.15.28:0/724139626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 msgr2=0x562232f302b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232f302b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 -- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 msgr2=0x562232f28d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232f28d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 -- 172.21.15.28:0/724139626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562232f308a0 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e8659c640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232f28d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e85d9b640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5e70025040 tx=0x7f5e70012630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420dc021214bb6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e6dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e70030020 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f5e70005220 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e70005500 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5e7003b050 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e8fa31740 10 client.16434 fetch_fsmap learned FSMap version 27 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562232f31c50 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.053+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e70005500 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5e700046c0 con 0x562232eb1ed0 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 10 client.16434 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:16:50.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 10 client.16434 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 20 client.16434 populate_metadata read hostname 'smithi028' 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232f32700 0x562232f34b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 4 client.16434 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562232e1d8d0 con 0x562232f32700 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8659c640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232f32700 0x562232f34b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e8659c640 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232f32700 0x562232f34b20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bc9e61339198ed4 server_cookie=fb3792560e08dc in_seq=0 out_seq=0 2024-03-22T02:16:50.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.054+0000 7f5e6dffb640 10 client.16434 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:50.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:49 smithi028 ceph-mon[31916]: pgmap v455: 129 pgs: 129 active+clean; 49 MiB data, 593 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:16:50.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:49 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e6dffb640 1 -- 172.21.15.28:0/724139626 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562232e1d8d0 con 0x562232f32700 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e6dffb640 10 client.16434 handle_command_reply: tid=0 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 client.16434 shutdown 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 msgr2=0x562232f29270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5e70025040 tx=0x7f5e70012630 comp rx=0 tx=0).stop 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 shutdown_connections 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562232f32700 0x562232f34b20 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e6dffb640 0 client.16434 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562232eb7040 0x562232f302b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562232eb1ed0 0x562232f29270 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 --2- 172.21.15.28:0/724139626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562232eb14e0 0x562232f28d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 >> 172.21.15.28:0/724139626 conn(0x562232eaca50 msgr2=0x562232ebb490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 shutdown_connections 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.240+0000 7f5e8fa31740 1 -- 172.21.15.28:0/724139626 wait complete. 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8fa31740 20 client.16434 trim_cache size 0 max 16384 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:16:50.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:16:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 msgr2=0x7f5e740a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 0x7f5e740a5480 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5e780049d0 tx=0x7f5e780136a0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 msgr2=0x7f5e88197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:16:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 0x7f5e88197e70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e80036040 tx=0x7f5e8002c650 comp rx=0 tx=0).stop 2024-03-22T02:16:50.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 shutdown_connections 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5e740a2fc0 0x7f5e740a5480 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88107890 0x7f5e8819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e88104f60 0x7f5e881983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 --2- 172.21.15.28:0/2164369661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e88102630 0x7f5e88197e70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 >> 172.21.15.28:0/2164369661 conn(0x7f5e880fc020 msgr2=0x7f5e88102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 shutdown_connections 2024-03-22T02:16:50.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:16:50.241+0000 7f5e8ce50640 1 -- 172.21.15.28:0/2164369661 wait complete. 2024-03-22T02:16:50.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 135210 8.13 MB/sec warmup 92 sec latency 138.536 ms 2024-03-22T02:16:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:50 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:16:51.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:50 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:16:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:50 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:16:51.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 137743 8.17 MB/sec warmup 93 sec latency 39.933 ms 2024-03-22T02:16:51.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:51 smithi120 ceph-mon[36803]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-03-22T02:16:52.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:51 smithi083 ceph-mon[36340]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-03-22T02:16:52.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:51 smithi028 ceph-mon[31916]: pgmap v456: 129 pgs: 129 active+clean; 49 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-03-22T02:16:52.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 139066 8.16 MB/sec warmup 94 sec latency 108.989 ms 2024-03-22T02:16:53.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 142006 8.26 MB/sec warmup 95 sec latency 66.268 ms 2024-03-22T02:16:53.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:53 smithi120 ceph-mon[36803]: pgmap v457: 129 pgs: 129 active+clean; 47 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:16:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:53 smithi083 ceph-mon[36340]: pgmap v457: 129 pgs: 129 active+clean; 47 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:16:54.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:53 smithi028 ceph-mon[31916]: pgmap v457: 129 pgs: 129 active+clean; 47 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:16:54.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 143974 8.25 MB/sec warmup 96 sec latency 43.253 ms 2024-03-22T02:16:55.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 146136 8.30 MB/sec warmup 97 sec latency 27.732 ms 2024-03-22T02:16:55.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:55 smithi120 ceph-mon[36803]: pgmap v458: 129 pgs: 129 active+clean; 39 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 474 op/s 2024-03-22T02:16:56.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:55 smithi083 ceph-mon[36340]: pgmap v458: 129 pgs: 129 active+clean; 39 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 474 op/s 2024-03-22T02:16:56.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:55 smithi028 ceph-mon[31916]: pgmap v458: 129 pgs: 129 active+clean; 39 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 474 op/s 2024-03-22T02:16:56.410 INFO:tasks.workunit.client.0.smithi028.stdout: 1 149003 8.39 MB/sec warmup 98 sec latency 10.134 ms 2024-03-22T02:16:57.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 149875 8.33 MB/sec warmup 99 sec latency 118.429 ms 2024-03-22T02:16:58.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:57 smithi083 ceph-mon[36340]: pgmap v459: 129 pgs: 129 active+clean; 39 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:16:58.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:57 smithi028 ceph-mon[31916]: pgmap v459: 129 pgs: 129 active+clean; 39 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:16:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:57 smithi120 ceph-mon[36803]: pgmap v459: 129 pgs: 129 active+clean; 39 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:16:58.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 152786 8.43 MB/sec warmup 100 sec latency 12.164 ms 2024-03-22T02:16:59.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 155017 8.42 MB/sec warmup 101 sec latency 36.737 ms 2024-03-22T02:17:00.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:16:59 smithi083 ceph-mon[36340]: pgmap v460: 129 pgs: 129 active+clean; 41 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-03-22T02:17:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:16:59 smithi028 ceph-mon[31916]: pgmap v460: 129 pgs: 129 active+clean; 41 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-03-22T02:17:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:16:59 smithi120 ceph-mon[36803]: pgmap v460: 129 pgs: 129 active+clean; 41 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 433 op/s 2024-03-22T02:17:00.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 156960 8.46 MB/sec warmup 102 sec latency 43.383 ms 2024-03-22T02:17:01.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 159901 8.56 MB/sec warmup 103 sec latency 23.827 ms 2024-03-22T02:17:02.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:01 smithi083 ceph-mon[36340]: pgmap v461: 129 pgs: 129 active+clean; 44 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-03-22T02:17:02.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:01 smithi028 ceph-mon[31916]: pgmap v461: 129 pgs: 129 active+clean; 44 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-03-22T02:17:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:01 smithi120 ceph-mon[36803]: pgmap v461: 129 pgs: 129 active+clean; 44 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 408 op/s 2024-03-22T02:17:02.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 161500 8.53 MB/sec warmup 104 sec latency 103.741 ms 2024-03-22T02:17:03.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.330+0000 7f01061bc640 1 -- 172.21.15.28:0/2700484378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100101ca0 msgr2=0x7f0100102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 --2- 172.21.15.28:0/2700484378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100101ca0 0x7f0100102120 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009900 tx=0x7f00f402bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 -- 172.21.15.28:0/2700484378 shutdown_connections 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 --2- 172.21.15.28:0/2700484378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 0x7f0100108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 --2- 172.21.15.28:0/2700484378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100101ca0 0x7f0100102120 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 --2- 172.21.15.28:0/2700484378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100100ab0 0x7f0100100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 -- 172.21.15.28:0/2700484378 >> 172.21.15.28:0/2700484378 conn(0x7f01000fc020 msgr2=0x7f01000fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 -- 172.21.15.28:0/2700484378 shutdown_connections 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 -- 172.21.15.28:0/2700484378 wait complete. 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 Processor -- start 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.331+0000 7f01061bc640 1 -- start start 2024-03-22T02:17:03.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 0x7f010019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 0x7f01001a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c730 con 0x7f0100100ab0 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c5b0 con 0x7f0100101ca0 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f010010c8b0 con 0x7f0100102660 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43116/0 (socket says 172.21.15.28:43116) 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2645928881 learned_addr learned my addr 172.21.15.28:0/2645928881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 0x7f010019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00fffff640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 0x7f01001a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2645928881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 msgr2=0x7f01001a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 0x7f01001a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2645928881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 msgr2=0x7f010019a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 0x7f010019a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2645928881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01001a1c50 con 0x7f0100100ab0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00feffd640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 0x7f010019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f00f000e9c0 tx=0x7f00f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00fcff9640 1 -- 172.21.15.28:0/2645928881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f000cd80 con 0x7f0100100ab0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00fcff9640 1 -- 172.21.15.28:0/2645928881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00f00035f0 con 0x7f0100100ab0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01001a1f40 con 0x7f0100100ab0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.332+0000 7f00fcff9640 1 -- 172.21.15.28:0/2645928881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0010690 con 0x7f0100100ab0 2024-03-22T02:17:03.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.333+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01001a23a0 con 0x7f0100100ab0 2024-03-22T02:17:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fcff9640 1 -- 172.21.15.28:0/2645928881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f00f00037a0 con 0x7f0100100ab0 2024-03-22T02:17:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 Processor -- start 2024-03-22T02:17:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 -- start start 2024-03-22T02:17:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060c4e0 0x5612e060c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e06119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fcff9640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 0x7f00e80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612e0571040 con 0x5612e0612040 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5612e05199e0 con 0x5612e060c4e0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f0108da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612e056cff0 con 0x5612e060ced0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e06119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43122/0 (socket says 172.21.15.28:43122) 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fffff640 1 -- 172.21.15.28:0/3190881367 learned_addr learned my addr 172.21.15.28:0/3190881367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00feffd640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 0x7f00e80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fffff640 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060c4e0 0x5612e060c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.335+0000 7f00fcff9640 1 -- 172.21.15.28:0/2645928881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f00f00cfbd0 con 0x7f0100100ab0 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 -- 172.21.15.28:0/3190881367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 msgr2=0x5612e06119d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e06119d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 -- 172.21.15.28:0/3190881367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060c4e0 msgr2=0x5612e060c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060c4e0 0x5612e060c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 -- 172.21.15.28:0/3190881367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e0285210 con 0x5612e0612040 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e06119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00fffff640 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009fd0 tx=0x7f00f8009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aaeda563046c41f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00feffd640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 0x7f00e80a5480 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f00f402c7b0 tx=0x7f00f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00e5ffb640 1 -- 172.21.15.28:0/3190881367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8011070 con 0x5612e0612040 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00e5ffb640 1 -- 172.21.15.28:0/3190881367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00f8003e50 con 0x5612e0612040 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.336+0000 7f00e5ffb640 1 -- 172.21.15.28:0/3190881367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f80042b0 con 0x5612e0612040 2024-03-22T02:17:03.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- 172.21.15.28:0/3190881367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 msgr2=0x5612e06144d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009fd0 tx=0x7f00f8009b80 comp rx=0 tx=0).stop 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- 172.21.15.28:0/3190881367 shutdown_connections 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e06144d0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e06119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- 172.21.15.28:0/3190881367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060c4e0 0x5612e060c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- 172.21.15.28:0/3190881367 >> 172.21.15.28:0/3190881367 conn(0x5612e0607a50 msgr2=0x5612e0607e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- 172.21.15.28:0/3190881367 shutdown_connections 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- 172.21.15.28:0/3190881367 wait complete. 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 -- start start 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 Processor -- start 2024-03-22T02:17:03.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e0616480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.337+0000 7f0108da4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5612e0614ff0 con 0x5612e0612040 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5612e052cdb0 con 0x5612e060ced0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5612e0615170 con 0x5612e060c4e0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 10 client.0 _ll_register_callbacks cb 0x5612e0284a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e0616480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49370/0 (socket says 172.21.15.28:49370) 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00feffd640 1 -- 172.21.15.28:0/2381400056 learned_addr learned my addr 172.21.15.28:0/2381400056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00fffff640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 msgr2=0x5612e06169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2381400056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 msgr2=0x5612e061da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2381400056 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e0351620 con 0x5612e060c4e0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00feffd640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e0616480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f00f00cd630 tx=0x7f00f0014d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3619f244eaa733e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00deffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f009c5f0 con 0x5612e060c4e0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f00f0014da0 con 0x5612e060c4e0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.338+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00f8003e50 con 0x5612e060c4e0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f00f009bef0 con 0x5612e060c4e0 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 10 client.26278 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:03.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612e0620e40 con 0x5612e060c4e0 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00f8003e50 con 0x5612e060c4e0 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f00f001d260 con 0x5612e060c4e0 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 10 client.26278 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 10 client.26278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 20 client.26278 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e0622540 0x5612e06249e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 4 client.26278 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612e057e580 con 0x5612e0622540 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.339+0000 7f00feffd640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e0622540 0x5612e06249e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.340+0000 7f00deffd640 10 client.26278 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.340+0000 7f00feffd640 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e0622540 0x5612e06249e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4cd5556d609fd64 server_cookie=e1dc48f703073636 in_seq=0 out_seq=0 2024-03-22T02:17:03.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f00deffd640 1 -- 172.21.15.28:0/2381400056 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5612e057e580 con 0x5612e0622540 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f00deffd640 10 client.26278 handle_command_reply: tid=0 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 client.26278 shutdown 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 msgr2=0x5612e0616480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e0616480 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f00f00cd630 tx=0x7f00f0014d30 comp rx=0 tx=0).stop 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 shutdown_connections 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e0622540 0x5612e06249e0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e06169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f00deffd640 0 client.26278 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 --2- 172.21.15.28:0/2381400056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e0616480 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 >> 172.21.15.28:0/2381400056 conn(0x5612e0607a50 msgr2=0x5612e0606c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 shutdown_connections 2024-03-22T02:17:03.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 1 -- 172.21.15.28:0/2381400056 wait complete. 2024-03-22T02:17:03.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.341+0000 7f0108da4740 20 client.26278 trim_cache size 0 max 16384 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 Processor -- start 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 -- start start 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e061d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 0x5612e0606900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01001af430 con 0x5612e060c4e0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01001acd00 con 0x5612e0612040 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.357+0000 7f0108da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f01001aab50 con 0x5612e060ced0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43140/0 (socket says 172.21.15.28:43140) 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2183553897 learned_addr learned my addr 172.21.15.28:0/2183553897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00feffd640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e061d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00fffff640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 0x5612e0606900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2183553897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 msgr2=0x5612e061d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e061d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2183553897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 msgr2=0x5612e0606900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 0x5612e0606900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 -- 172.21.15.28:0/2183553897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612e060d2b0 con 0x5612e060c4e0 2024-03-22T02:17:03.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00feffd640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e061d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00fffff640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 0x5612e0606900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f00f009bbf0 tx=0x7f00f00cff80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d756843984c561f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00e57fa640 1 -- 172.21.15.28:0/2183553897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00a0040 con 0x5612e060c4e0 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00e57fa640 1 -- 172.21.15.28:0/2183553897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00f009c5f0 con 0x5612e060c4e0 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.358+0000 7f00e57fa640 1 -- 172.21.15.28:0/2183553897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f009a280 con 0x5612e060c4e0 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- 172.21.15.28:0/2183553897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 msgr2=0x5612e06063c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f00f009bbf0 tx=0x7f00f00cff80 comp rx=0 tx=0).stop 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- 172.21.15.28:0/2183553897 shutdown_connections 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e0612040 0x5612e0606900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060ced0 0x5612e061d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 --2- 172.21.15.28:0/2183553897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e060c4e0 0x5612e06063c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- 172.21.15.28:0/2183553897 >> 172.21.15.28:0/2183553897 conn(0x5612e0607a50 msgr2=0x5612e0616090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- 172.21.15.28:0/2183553897 shutdown_connections 2024-03-22T02:17:03.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- 172.21.15.28:0/2183553897 wait complete. 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 -- start start 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.359+0000 7f0108da4740 1 Processor -- start 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e061b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e061b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f01001ab1c0 con 0x5612e0612040 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f01001ac630 con 0x5612e060ced0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f01001ab890 con 0x5612e060c4e0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 10 client.0 _ll_register_callbacks cb 0x7f00f0097a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e061b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e061b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e061b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51582/0 (socket says 172.21.15.28:51582) 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e061b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49392/0 (socket says 172.21.15.28:49392) 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00ff7fe640 1 -- 172.21.15.28:0/3400901444 learned_addr learned my addr 172.21.15.28:0/3400901444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43152/0 (socket says 172.21.15.28:43152) 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 -- 172.21.15.28:0/3400901444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 msgr2=0x5612e061b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e061b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 -- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 msgr2=0x5612e061b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e061b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 -- 172.21.15.28:0/3400901444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612e061cdd0 con 0x5612e0612040 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00fffff640 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x5612e0603590 tx=0x7f00f8004eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d074a8c4f418fe42 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00deffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8011250 con 0x5612e0612040 2024-03-22T02:17:03.366 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f00f80113f0 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.360+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00f009c5f0 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f00f8005040 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 10 client.16452 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5612e0688a30 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00f009c5f0 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f00f801d020 con 0x5612e0612040 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 10 client.16452 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 20 client.16452 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e06891b0 0x5612e068b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 4 client.16452 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5612e05788d0 con 0x5612e06891b0 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.361+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e06891b0 0x5612e068b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:03.367 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.362+0000 7f00deffd640 10 client.16452 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:03.368 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.362+0000 7f00ff7fe640 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e06891b0 0x5612e068b5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac9fc2b926d5b2bc server_cookie=fbec54aec1a9dc22 in_seq=0 out_seq=0 2024-03-22T02:17:03.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 163768 8.58 MB/sec warmup 105 sec latency 13.500 ms 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.656+0000 7f00deffd640 1 -- 172.21.15.28:0/3400901444 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5612e05788d0 con 0x5612e06891b0 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.656+0000 7f00deffd640 10 client.16452 handle_command_reply: tid=0 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.656+0000 7f0108da4740 1 client.16452 shutdown 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 msgr2=0x5612e061c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x5612e0603590 tx=0x7f00f8004eb0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 shutdown_connections 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5612e06891b0 0x5612e068b5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612e0612040 0x5612e061c7e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5612e060ced0 0x5612e061b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 --2- 172.21.15.28:0/3400901444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5612e060c4e0 0x5612e061b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 >> 172.21.15.28:0/3400901444 conn(0x5612e0607a50 msgr2=0x5612e060d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f00deffd640 0 client.16452 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 shutdown_connections 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 1 -- 172.21.15.28:0/3400901444 wait complete. 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.657+0000 7f0108da4740 20 client.16452 trim_cache size 0 max 16384 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:17:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:17:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:17:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:17:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 msgr2=0x7f00e80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 0x7f00e80a5480 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f00f402c7b0 tx=0x7f00f4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 msgr2=0x7f010019a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f00f000e9c0 tx=0x7f00f000ee80 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 shutdown_connections 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00e80a2fc0 0x7f00e80a5480 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0100102660 0x7f01001a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0100101ca0 0x7f010019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 --2- 172.21.15.28:0/2645928881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0100100ab0 0x7f010019a030 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 >> 172.21.15.28:0/2645928881 conn(0x7f01000fc020 msgr2=0x7f01000fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 shutdown_connections 2024-03-22T02:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:03.658+0000 7f01061bc640 1 -- 172.21.15.28:0/2645928881 wait complete. 2024-03-22T02:17:04.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:03 smithi083 ceph-mon[36340]: pgmap v462: 129 pgs: 129 active+clean; 42 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:17:04.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:03 smithi028 ceph-mon[31916]: pgmap v462: 129 pgs: 129 active+clean; 42 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:17:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:03 smithi120 ceph-mon[36803]: pgmap v462: 129 pgs: 129 active+clean; 42 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:17:04.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 166194 8.61 MB/sec warmup 106 sec latency 63.355 ms 2024-03-22T02:17:05.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 168157 8.64 MB/sec warmup 107 sec latency 42.134 ms 2024-03-22T02:17:06.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:05 smithi083 ceph-mon[36340]: pgmap v463: 129 pgs: 129 active+clean; 37 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 483 op/s 2024-03-22T02:17:06.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:05 smithi028 ceph-mon[31916]: pgmap v463: 129 pgs: 129 active+clean; 37 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 483 op/s 2024-03-22T02:17:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:05 smithi120 ceph-mon[36803]: pgmap v463: 129 pgs: 129 active+clean; 37 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.9 MiB/s wr, 483 op/s 2024-03-22T02:17:06.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 170661 8.69 MB/sec warmup 108 sec latency 21.506 ms 2024-03-22T02:17:07.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 172658 8.68 MB/sec warmup 109 sec latency 54.254 ms 2024-03-22T02:17:08.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:07 smithi083 ceph-mon[36340]: pgmap v464: 129 pgs: 129 active+clean; 38 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 333 op/s 2024-03-22T02:17:08.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:07 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:17:08.050 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:07 smithi120 ceph-mon[36803]: pgmap v464: 129 pgs: 129 active+clean; 38 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 333 op/s 2024-03-22T02:17:08.050 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:07 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:17:08.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:07 smithi028 ceph-mon[31916]: pgmap v464: 129 pgs: 129 active+clean; 38 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 333 op/s 2024-03-22T02:17:08.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:07 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:17:08.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 174299 8.71 MB/sec warmup 110 sec latency 82.875 ms 2024-03-22T02:17:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:17:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:17:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:08 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:17:09.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:17:09.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:17:09.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:08 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:17:09.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:17:09.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:17:09.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:08 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:17:09.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 177183 8.76 MB/sec warmup 111 sec latency 40.008 ms 2024-03-22T02:17:10.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 178934 8.77 MB/sec warmup 112 sec latency 29.694 ms 2024-03-22T02:17:10.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:17:10.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:17:10.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:10 smithi028 ceph-mon[31916]: pgmap v465: 129 pgs: 129 active+clean; 42 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 454 op/s 2024-03-22T02:17:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:10 smithi120 ceph-mon[36803]: pgmap v465: 129 pgs: 129 active+clean; 42 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 454 op/s 2024-03-22T02:17:10.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:10 smithi083 ceph-mon[36340]: pgmap v465: 129 pgs: 129 active+clean; 42 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.7 MiB/s wr, 454 op/s 2024-03-22T02:17:10.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 -- 172.21.15.28:0/2997068883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4102d90 msgr2=0x7f2ef4103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 --2- 172.21.15.28:0/2997068883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4102d90 0x7f2ef4103210 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009900 tx=0x7f2ee802bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 -- 172.21.15.28:0/2997068883 shutdown_connections 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 --2- 172.21.15.28:0/2997068883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4103750 0x7f2ef4109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 --2- 172.21.15.28:0/2997068883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4102d90 0x7f2ef4103210 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 --2- 172.21.15.28:0/2997068883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 0x7f2ef4101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 -- 172.21.15.28:0/2997068883 >> 172.21.15.28:0/2997068883 conn(0x7f2ef40fd130 msgr2=0x7f2ef40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 -- 172.21.15.28:0/2997068883 shutdown_connections 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.543+0000 7f2efa281640 1 -- 172.21.15.28:0/2997068883 wait complete. 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 Processor -- start 2024-03-22T02:17:10.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 -- start start 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 0x7f2ef4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4102d90 0x7f2ef4198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef410d820 con 0x7f2ef4103750 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef410d6a0 con 0x7f2ef4102d90 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2efa281640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef410d9a0 con 0x7f2ef4101ba0 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43172/0 (socket says 172.21.15.28:43172) 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 -- 172.21.15.28:0/1483237088 learned_addr learned my addr 172.21.15.28:0/1483237088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 0x7f2ef4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4102d90 0x7f2ef4198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 -- 172.21.15.28:0/1483237088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 msgr2=0x7f2ef4197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 0x7f2ef4197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 -- 172.21.15.28:0/1483237088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4102d90 msgr2=0x7f2ef4198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4102d90 0x7f2ef4198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.544+0000 7f2ef881e640 1 -- 172.21.15.28:0/1483237088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef419fae0 con 0x7f2ef4103750 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2ef881e640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00eb10 tx=0x7f2eec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2ef17fa640 1 -- 172.21.15.28:0/1483237088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec004b20 con 0x7f2ef4103750 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2ef17fa640 1 -- 172.21.15.28:0/1483237088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec004cc0 con 0x7f2ef4103750 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef419fdd0 con 0x7f2ef4103750 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2ef17fa640 1 -- 172.21.15.28:0/1483237088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec010760 con 0x7f2ef4103750 2024-03-22T02:17:10.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.545+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef41a0240 con 0x7f2ef4103750 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.547+0000 7f2ef17fa640 1 -- 172.21.15.28:0/1483237088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2eec005020 con 0x7f2ef4103750 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.547+0000 7f2efce64740 1 Processor -- start 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.547+0000 7f2efce64740 1 -- start start 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.547+0000 7f2ef17fa640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 0x7f2edc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 0x555f459b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459b9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f459bc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f459198c0 con 0x555f459b9f90 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555f458c1c20 con 0x555f459b4e20 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2efce64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555f45915870 con 0x555f459b4430 2024-03-22T02:17:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 0x7f2edc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef17fa640 1 -- 172.21.15.28:0/1483237088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2eec0cfd50 con 0x7f2ef4103750 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 0x555f459b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459b9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f459bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 0x555f459b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51610/0 (socket says 172.21.15.28:51610) 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef3fff640 1 -- 172.21.15.28:0/382481380 learned_addr learned my addr 172.21.15.28:0/382481380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 -- 172.21.15.28:0/382481380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 msgr2=0x555f459b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 0x555f459b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 -- 172.21.15.28:0/382481380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 msgr2=0x555f459b9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459b9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 -- 172.21.15.28:0/382481380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f458e6150 con 0x555f459b9f90 2024-03-22T02:17:10.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 0x7f2edc0a53e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2ee802c7b0 tx=0x7f2ee8005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2ef881e640 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f459bc420 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0cd490 tx=0x7f2eec09bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abccf99f0cff4260 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.548+0000 7f2eda7fc640 1 -- 172.21.15.28:0/382481380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec09c070 con 0x555f459b9f90 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2eda7fc640 1 -- 172.21.15.28:0/382481380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec0a01c0 con 0x555f459b9f90 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2eda7fc640 1 -- 172.21.15.28:0/382481380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec09c2d0 con 0x555f459b9f90 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 -- 172.21.15.28:0/382481380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 msgr2=0x555f459bc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f459bc420 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0cd490 tx=0x7f2eec09bf00 comp rx=0 tx=0).stop 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 -- 172.21.15.28:0/382481380 shutdown_connections 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f459bc420 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459b9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 --2- 172.21.15.28:0/382481380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4430 0x555f459b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 -- 172.21.15.28:0/382481380 >> 172.21.15.28:0/382481380 conn(0x555f459af9a0 msgr2=0x555f459afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 -- 172.21.15.28:0/382481380 shutdown_connections 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.549+0000 7f2efce64740 1 -- 172.21.15.28:0/382481380 wait complete. 2024-03-22T02:17:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 -- start start 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 Processor -- start 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459be970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555f459bcf40 con 0x555f459b4e20 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555f458d5130 con 0x555f459b4430 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555f459bd0c0 con 0x555f459b9f90 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2efce64740 10 client.0 _ll_register_callbacks cb 0x555f458d4b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2ef37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2ef881e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2ef37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43198/0 (socket says 172.21.15.28:43198) 2024-03-22T02:17:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.550+0000 7f2ef37fe640 1 -- 172.21.15.28:0/1839366433 learned_addr learned my addr 172.21.15.28:0/1839366433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 -- 172.21.15.28:0/1839366433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 msgr2=0x555f459c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 -- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 msgr2=0x555f459be970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459be970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 -- 172.21.15.28:0/1839366433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f456f77d0 con 0x555f459b4e20 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef881e640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8035000 tx=0x7f2ee802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8659833ca70ca server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ed37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8053020 con 0x555f459b4e20 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2ee8005450 con 0x555f459b4e20 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2eec0a01c0 con 0x555f459b4e20 2024-03-22T02:17:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2ee805e050 con 0x555f459b4e20 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2efce64740 10 client.16464 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.551+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f4562d3f0 con 0x555f459b4e20 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eec0a01c0 con 0x555f459b4e20 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2ee80038c0 con 0x555f459b4e20 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 10 client.16464 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 10 client.16464 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 20 client.16464 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f459ca8c0 0x555f459ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 4 client.16464 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f45926e00 con 0x555f459ca8c0 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f459ca8c0 0x555f459ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2ed37fe640 10 client.16464 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.552+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f459ca8c0 0x555f459ccd40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a1d1a94d1166f3 server_cookie=cf9f52948b5341e2 in_seq=0 out_seq=0 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2ed37fe640 1 -- 172.21.15.28:0/1839366433 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555f45926e00 con 0x555f459ca8c0 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2ed37fe640 10 client.16464 handle_command_reply: tid=0 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 client.16464 shutdown 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 msgr2=0x555f459beeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8035000 tx=0x7f2ee802bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 shutdown_connections 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f459ca8c0 0x555f459ccd40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4e20 0x555f459beeb0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2ed37fe640 0 client.16464 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 --2- 172.21.15.28:0/1839366433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459be970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 >> 172.21.15.28:0/1839366433 conn(0x555f459af9a0 msgr2=0x555f459aec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 shutdown_connections 2024-03-22T02:17:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 1 -- 172.21.15.28:0/1839366433 wait complete. 2024-03-22T02:17:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.554+0000 7f2efce64740 20 client.16464 trim_cache size 0 max 16384 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 Processor -- start 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 -- start start 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41ad1b0 con 0x555f459b4430 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41aaa80 con 0x555f459b4e20 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2efce64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41a88d0 con 0x555f459b9f90 2024-03-22T02:17:10.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43204/0 (socket says 172.21.15.28:43204) 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 -- 172.21.15.28:0/4192868941 learned_addr learned my addr 172.21.15.28:0/4192868941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef881e640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 -- 172.21.15.28:0/4192868941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 msgr2=0x555f459b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 -- 172.21.15.28:0/4192868941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 msgr2=0x555f459c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 -- 172.21.15.28:0/4192868941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f455ed0f0 con 0x555f459b4430 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef37fe640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x555f459b0910 tx=0x7f2ee4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e0b52eedac2a3a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.573+0000 7f2ed9ffb640 1 -- 172.21.15.28:0/4192868941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4005220 con 0x555f459b4430 2024-03-22T02:17:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2ed9ffb640 1 -- 172.21.15.28:0/4192868941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee40206b0 con 0x555f459b4430 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2ed9ffb640 1 -- 172.21.15.28:0/4192868941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4004d40 con 0x555f459b4430 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- 172.21.15.28:0/4192868941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 msgr2=0x555f459b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x555f459b0910 tx=0x7f2ee4005f90 comp rx=0 tx=0).stop 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- 172.21.15.28:0/4192868941 shutdown_connections 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b9f90 0x555f459b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4e20 0x555f459c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 --2- 172.21.15.28:0/4192868941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b4430 0x555f459b56b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- 172.21.15.28:0/4192868941 >> 172.21.15.28:0/4192868941 conn(0x555f459af9a0 msgr2=0x555f459bd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- 172.21.15.28:0/4192868941 shutdown_connections 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- 172.21.15.28:0/4192868941 wait complete. 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.574+0000 7f2efce64740 1 -- start start 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 Processor -- start 2024-03-22T02:17:10.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459c2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 0x555f459c3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f45a2ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41a8f40 con 0x555f459b9f90 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41aa3b0 con 0x555f459b4430 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef41a9610 con 0x555f459b4e20 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2efce64740 10 client.0 _ll_register_callbacks cb 0x7f2ee802caa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f45a2ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 0x555f459c3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459c2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 0x555f459c3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51632/0 (socket says 172.21.15.28:51632) 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef37fe640 1 -- 172.21.15.28:0/2139253657 learned_addr learned my addr 172.21.15.28:0/2139253657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 -- 172.21.15.28:0/2139253657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 msgr2=0x555f459c3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 0x555f459c3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 -- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 msgr2=0x555f459c2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459c2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.575+0000 7f2ef881e640 1 -- 172.21.15.28:0/2139253657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f45a2f260 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ef881e640 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f45a2ec70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x555f459bcb10 tx=0x7f2eec00cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73756c0a7664ddb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eec09c4d0 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2eec0a45f0 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ee40206b0 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2eec0b5170 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 10 client.16470 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555f45a30580 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ee40206b0 con 0x555f459b9f90 2024-03-22T02:17:10.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2eec005310 con 0x555f459b9f90 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 10 client.16470 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 10 client.16470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 20 client.16470 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f45a30a70 0x555f45a32e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 4 client.16470 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555f45921150 con 0x555f45a30a70 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.576+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f45a30a70 0x555f45a32e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.577+0000 7f2ef3fff640 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f45a30a70 0x555f45a32e90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=556ca0dfc7ba7ef9 server_cookie=42281235a0e7ee6c in_seq=0 out_seq=0 2024-03-22T02:17:10.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.577+0000 7f2ed37fe640 10 client.16470 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2ed37fe640 1 -- 172.21.15.28:0/2139253657 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555f45921150 con 0x555f45a30a70 2024-03-22T02:17:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2ed37fe640 10 client.16470 handle_command_reply: tid=0 2024-03-22T02:17:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 client.16470 shutdown 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 msgr2=0x555f45a2ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f45a2ec70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x555f459bcb10 tx=0x7f2eec00cee0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 shutdown_connections 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555f45a30a70 0x555f45a32e90 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f459b9f90 0x555f45a2ec70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555f459b4e20 0x555f459c3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2ed37fe640 0 client.16470 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 --2- 172.21.15.28:0/2139253657 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555f459b4430 0x555f459c2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.578+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 >> 172.21.15.28:0/2139253657 conn(0x555f459af9a0 msgr2=0x555f459b6360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 shutdown_connections 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efce64740 1 -- 172.21.15.28:0/2139253657 wait complete. 2024-03-22T02:17:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efce64740 20 client.16470 trim_cache size 0 max 16384 2024-03-22T02:17:10.586 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:10.586 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:17:10.586 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:17:10.586 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 msgr2=0x7f2edc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 0x7f2edc0a53e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2ee802c7b0 tx=0x7f2ee8005d50 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 msgr2=0x7f2ef419f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00eb10 tx=0x7f2eec00c2d0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 shutdown_connections 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2edc0a2f20 0x7f2edc0a53e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ef4103750 0x7f2ef419f470 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ef4102d90 0x7f2ef4198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 --2- 172.21.15.28:0/1483237088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ef4101ba0 0x7f2ef4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 >> 172.21.15.28:0/1483237088 conn(0x7f2ef40fd130 msgr2=0x7f2ef4106160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 shutdown_connections 2024-03-22T02:17:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.579+0000 7f2efa281640 1 -- 172.21.15.28:0/1483237088 wait complete. 2024-03-22T02:17:10.591 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:17:10.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:17:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 -- 172.21.15.28:0/2796764504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700100ab0 msgr2=0x7f0700100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 --2- 172.21.15.28:0/2796764504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700100ab0 0x7f0700100ed0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009900 tx=0x7f06f002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 -- 172.21.15.28:0/2796764504 shutdown_connections 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 --2- 172.21.15.28:0/2796764504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 0x7f0700108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 --2- 172.21.15.28:0/2796764504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700101ca0 0x7f0700102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 --2- 172.21.15.28:0/2796764504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700100ab0 0x7f0700100ed0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 -- 172.21.15.28:0/2796764504 >> 172.21.15.28:0/2796764504 conn(0x7f07000fc020 msgr2=0x7f07000fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 -- 172.21.15.28:0/2796764504 shutdown_connections 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.736+0000 7f070692b640 1 -- 172.21.15.28:0/2796764504 wait complete. 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 Processor -- start 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 -- start start 2024-03-22T02:17:10.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700100ab0 0x7f0700069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 0x7f07001a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f070010c730 con 0x7f0700101ca0 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f070010c5b0 con 0x7f0700100ab0 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f070692b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f070010c8b0 con 0x7f0700102660 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43234/0 (socket says 172.21.15.28:43234) 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 -- 172.21.15.28:0/2273899881 learned_addr learned my addr 172.21.15.28:0/2273899881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06fffff640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700100ab0 0x7f0700069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f0704ec8640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 0x7f07001a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 -- 172.21.15.28:0/2273899881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 msgr2=0x7f07001a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 0x7f07001a1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 -- 172.21.15.28:0/2273899881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700100ab0 msgr2=0x7f0700069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700100ab0 0x7f0700069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 -- 172.21.15.28:0/2273899881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07001a1d30 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.737+0000 7f06ff7fe640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f06f400e9f0 tx=0x7f06f400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.738+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f400cd80 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.738+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06f4010910 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.738+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07001a1fc0 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.738+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f4018990 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.738+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07001a24d0 con 0x7f0700101ca0 2024-03-22T02:17:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.739+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0700107dd0 con 0x7f0700101ca0 2024-03-22T02:17:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.740+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f06f4010ac0 con 0x7f0700101ca0 2024-03-22T02:17:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.740+0000 7f06fd7fa640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 0x7f06e80a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.741+0000 7f06fffff640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 0x7f06e80a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.741+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f06f40cf9f0 con 0x7f0700101ca0 2024-03-22T02:17:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.741+0000 7f06fffff640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 0x7f06e80a5490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000c00 tx=0x7f06f0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.745+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f06f409bdc0 con 0x7f0700101ca0 2024-03-22T02:17:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.980+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0700062830 con 0x7f0700101ca0 2024-03-22T02:17:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.981+0000 7f06fd7fa640 1 -- 172.21.15.28:0/2273899881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11869 (secure 0 0 0) 0x7f06f409fe60 con 0x7f0700101ca0 2024-03-22T02:17:10.982 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:17:10.982 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:15:24.687791+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:17:10.982 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.982+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 msgr2=0x7f06e80a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.982+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 0x7f06e80a5490 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f06f0000c00 tx=0x7f06f0005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.982+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 msgr2=0x7f070019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.982+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f06f400e9f0 tx=0x7f06f400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 shutdown_connections 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f06e80a2fd0 0x7f06e80a5490 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0700102660 0x7f07001a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0700101ca0 0x7f070019eff0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 --2- 172.21.15.28:0/2273899881 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0700100ab0 0x7f0700069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 >> 172.21.15.28:0/2273899881 conn(0x7f07000fc020 msgr2=0x7f0700104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 shutdown_connections 2024-03-22T02:17:10.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:10.983+0000 7f070692b640 1 -- 172.21.15.28:0/2273899881 wait complete. 2024-03-22T02:17:10.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:17:11.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 -- 172.21.15.28:0/4058269301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 msgr2=0x7fdda4108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 --2- 172.21.15.28:0/4058269301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4108fd0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009ca0 tx=0x7fdd9c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 -- 172.21.15.28:0/4058269301 shutdown_connections 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 --2- 172.21.15.28:0/4058269301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4108fd0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 --2- 172.21.15.28:0/4058269301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4100d70 0x7fdda41011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 --2- 172.21.15.28:0/4058269301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4102040 0x7fdda4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 -- 172.21.15.28:0/4058269301 >> 172.21.15.28:0/4058269301 conn(0x7fdda40fb7c0 msgr2=0x7fdda40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 -- 172.21.15.28:0/4058269301 shutdown_connections 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 -- 172.21.15.28:0/4058269301 wait complete. 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.106+0000 7fdda98b8640 1 Processor -- start 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 -- start start 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4100d70 0x7fdda410b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102040 0x7fdda4109f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410d860 con 0x7fdda4101730 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410d6e0 con 0x7fdda4100d70 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda410d9e0 con 0x7fdda4102040 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43258/0 (socket says 172.21.15.28:43258) 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 -- 172.21.15.28:0/1436781280 learned_addr learned my addr 172.21.15.28:0/1436781280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4100d70 0x7fdda410b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102040 0x7fdda4109f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 -- 172.21.15.28:0/1436781280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102040 msgr2=0x7fdda4109f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102040 0x7fdda4109f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 -- 172.21.15.28:0/1436781280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4100d70 msgr2=0x7fdda410b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4100d70 0x7fdda410b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 -- 172.21.15.28:0/1436781280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda410a820 con 0x7fdda4101730 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda27fc640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800e9f0 tx=0x7fdd9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda88b6640 1 -- 172.21.15.28:0/1436781280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800cd80 con 0x7fdda4101730 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda88b6640 1 -- 172.21.15.28:0/1436781280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd98005500 con 0x7fdda4101730 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda41a1f80 con 0x7fdda4101730 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.107+0000 7fdda88b6640 1 -- 172.21.15.28:0/1436781280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98010690 con 0x7fdda4101730 2024-03-22T02:17:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.108+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda41a23e0 con 0x7fdda4101730 2024-03-22T02:17:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda88b6640 1 -- 172.21.15.28:0/1436781280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdd980046c0 con 0x7fdda4101730 2024-03-22T02:17:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 Processor -- start 2024-03-22T02:17:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 -- start start 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810add820 0x556810addc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ae5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556810a428d0 con 0x556810ade210 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568109eae00 con 0x556810add820 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda88b6640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 0x7fdd8c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fddac576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556810a3e880 con 0x556810ae3380 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810add820 0x556810addc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43270/0 (socket says 172.21.15.28:43270) 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda27fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ae5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda2ffd640 1 -- 172.21.15.28:0/2589050136 learned_addr learned my addr 172.21.15.28:0/2589050136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 0x7fdd8c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.110+0000 7fdda88b6640 1 -- 172.21.15.28:0/1436781280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdd980cfb10 con 0x7fdda4101730 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 -- 172.21.15.28:0/2589050136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810add820 msgr2=0x556810addc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810add820 0x556810addc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 -- 172.21.15.28:0/2589050136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 msgr2=0x556810ae2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 -- 172.21.15.28:0/2589050136 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568107563f0 con 0x556810ae3380 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda27fc640 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ae5810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd9809bdc0 tx=0x7fdd9806db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=660572cfc7550f5f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 0x7fdd8c0a5460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c005e30 tx=0x7fdd9c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdd89ffb640 1 -- 172.21.15.28:0/2589050136 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd98097b80 con 0x556810ae3380 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdd89ffb640 1 -- 172.21.15.28:0/2589050136 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9809c530 con 0x556810ae3380 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.111+0000 7fdd89ffb640 1 -- 172.21.15.28:0/2589050136 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9800b2c0 con 0x556810ae3380 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- 172.21.15.28:0/2589050136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 msgr2=0x556810ae5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ae5810 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd9809bdc0 tx=0x7fdd9806db70 comp rx=0 tx=0).stop 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- 172.21.15.28:0/2589050136 shutdown_connections 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ae5810 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ade210 0x556810ae2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 --2- 172.21.15.28:0/2589050136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810add820 0x556810addc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- 172.21.15.28:0/2589050136 >> 172.21.15.28:0/2589050136 conn(0x556810ad8d90 msgr2=0x556810ad9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- 172.21.15.28:0/2589050136 shutdown_connections 2024-03-22T02:17:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- 172.21.15.28:0/2589050136 wait complete. 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 -- start start 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.112+0000 7fddac576740 1 Processor -- start 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810ae7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810ae8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556810ae6270 con 0x556810ae3380 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568109fe300 con 0x556810ade210 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556810ae63f0 con 0x556810add820 2024-03-22T02:17:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 10 client.0 _ll_register_callbacks cb 0x556810a39e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810ae7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43282/0 (socket says 172.21.15.28:43282) 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 -- 172.21.15.28:0/2611733350 learned_addr learned my addr 172.21.15.28:0/2611733350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda27fc640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810ae8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 -- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 msgr2=0x556810ae7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810ae7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 -- 172.21.15.28:0/2611733350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 msgr2=0x556810ae8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810ae8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 -- 172.21.15.28:0/2611733350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568108207d0 con 0x556810ae3380 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810ae7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda27fc640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810ae8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdda37fe640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009ca0 tx=0x7fdd9c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7856a2931e663e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdd82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c054060 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdd9c004040 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.113+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd9809c530 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdd9c039040 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 10 client.16488 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556810af2630 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd9809c530 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdd9c04f470 con 0x556810ae3380 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 10 client.16488 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 10 client.16488 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 20 client.16488 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810af3d50 0x556810af61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 4 client.16488 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556810a4fe10 con 0x556810af3d50 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.114+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810af3d50 0x556810af61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.115+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810af3d50 0x556810af61b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bb956c4c2f9f4138 server_cookie=ebac004f23e447b6 in_seq=0 out_seq=0 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.115+0000 7fdd82ffd640 10 client.16488 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.115+0000 7fdd82ffd640 1 -- 172.21.15.28:0/2611733350 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556810a4fe10 con 0x556810af3d50 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.115+0000 7fdd82ffd640 10 client.16488 handle_command_reply: tid=0 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 client.16488 shutdown 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 msgr2=0x556810aef250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c009ca0 tx=0x7fdd9c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 shutdown_connections 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aef250 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810af3d50 0x556810af61b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810ae8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 --2- 172.21.15.28:0/2611733350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810ae7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fdd82ffd640 0 client.16488 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 >> 172.21.15.28:0/2611733350 conn(0x556810ad8d90 msgr2=0x556810ad7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 shutdown_connections 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 1 -- 172.21.15.28:0/2611733350 wait complete. 2024-03-22T02:17:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.116+0000 7fddac576740 20 client.16488 trim_cache size 0 max 16384 2024-03-22T02:17:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 Processor -- start 2024-03-22T02:17:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 -- start start 2024-03-22T02:17:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ad7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41af500 con 0x556810add820 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41acdd0 con 0x556810ade210 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fddac576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41aac20 con 0x556810ae3380 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43288/0 (socket says 172.21.15.28:43288) 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 -- 172.21.15.28:0/220342440 learned_addr learned my addr 172.21.15.28:0/220342440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda27fc640 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda37fe640 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ad7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 -- 172.21.15.28:0/220342440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 msgr2=0x556810ad7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ad7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 -- 172.21.15.28:0/220342440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 msgr2=0x556810aef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 -- 172.21.15.28:0/220342440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556810adf210 con 0x556810add820 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.130+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdd94012600 tx=0x7fdd94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4beee734335fc584 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fdd897fa640 1 -- 172.21.15.28:0/220342440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94023030 con 0x556810add820 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fdd897fa640 1 -- 172.21.15.28:0/220342440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd94005500 con 0x556810add820 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fdd897fa640 1 -- 172.21.15.28:0/220342440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9401f370 con 0x556810add820 2024-03-22T02:17:11.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 -- 172.21.15.28:0/220342440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 msgr2=0x556810ae7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fdd94012600 tx=0x7fdd94005f90 comp rx=0 tx=0).stop 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 -- 172.21.15.28:0/220342440 shutdown_connections 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810ae3380 0x556810ad7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 --2- 172.21.15.28:0/220342440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810add820 0x556810ae7020 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 -- 172.21.15.28:0/220342440 >> 172.21.15.28:0/220342440 conn(0x556810ad8d90 msgr2=0x556810adf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 -- 172.21.15.28:0/220342440 shutdown_connections 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.131+0000 7fddac576740 1 -- 172.21.15.28:0/220342440 wait complete. 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 -- start start 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 Processor -- start 2024-03-22T02:17:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810aec580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aecac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ab290 con 0x556810ae3380 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ac700 con 0x556810ade210 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdda41ab960 con 0x556810add820 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fddac576740 10 client.0 _ll_register_callbacks cb 0x7fdd98005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810aec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43302/0 (socket says 172.21.15.28:43302) 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 -- 172.21.15.28:0/1746734332 learned_addr learned my addr 172.21.15.28:0/1746734332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda27fc640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aecac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 -- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 msgr2=0x556810aec580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810aec580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 -- 172.21.15.28:0/1746734332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 msgr2=0x556810aecac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aecac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.132+0000 7fdda37fe640 1 -- 172.21.15.28:0/1746734332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556810aee0c0 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdda37fe640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c033000 tx=0x7fdd9c050d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeebf538ceb4d4b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c048070 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdd9c004d50 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd94005500 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdd9c054040 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 10 client.16494 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556810aef3c0 con 0x556810ae3380 2024-03-22T02:17:11.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd94005500 con 0x556810ae3380 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdd9c0504d0 con 0x556810ae3380 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 10 client.16494 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 10 client.16494 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 20 client.16494 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810b5a6f0 0x556810b5cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 4 client.16494 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556810a4a160 con 0x556810b5a6f0 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.133+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810b5a6f0 0x556810b5cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fdd82ffd640 10 client.16494 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fdda2ffd640 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810b5a6f0 0x556810b5cb10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=676cf1bd253349f7 server_cookie=337fbc06c024a726 in_seq=0 out_seq=0 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fdd82ffd640 1 -- 172.21.15.28:0/1746734332 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556810a4a160 con 0x556810b5a6f0 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fdd82ffd640 10 client.16494 handle_command_reply: tid=0 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 client.16494 shutdown 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 msgr2=0x556810aedad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c033000 tx=0x7fdd9c050d70 comp rx=0 tx=0).stop 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 shutdown_connections 2024-03-22T02:17:11.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556810ae3380 0x556810aedad0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556810b5a6f0 0x556810b5cb10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556810ade210 0x556810aecac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fdd82ffd640 0 client.16494 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 --2- 172.21.15.28:0/1746734332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556810add820 0x556810aec580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 >> 172.21.15.28:0/1746734332 conn(0x556810ad8d90 msgr2=0x556810adc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 shutdown_connections 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 1 -- 172.21.15.28:0/1746734332 wait complete. 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.134+0000 7fddac576740 20 client.16494 trim_cache size 0 max 16384 2024-03-22T02:17:11.147 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 msgr2=0x7fdd8c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 0x7fdd8c0a5460 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c005e30 tx=0x7fdd9c032590 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 msgr2=0x7fdda4109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fdd9800e9f0 tx=0x7fdd9800edf0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 shutdown_connections 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdd8c0a2fa0 0x7fdd8c0a5460 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdda4102040 0x7fdda4109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda4101730 0x7fdda4109a50 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 --2- 172.21.15.28:0/1436781280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdda4100d70 0x7fdda410b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 >> 172.21.15.28:0/1436781280 conn(0x7fdda40fb7c0 msgr2=0x7fdda4105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 shutdown_connections 2024-03-22T02:17:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.135+0000 7fdda98b8640 1 -- 172.21.15.28:0/1436781280 wait complete. 2024-03-22T02:17:11.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 -- 172.21.15.28:0/24328798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041081b0 msgr2=0x7fe6041085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/24328798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041081b0 0x7fe6041085d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fe600009900 tx=0x7fe60002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 -- 172.21.15.28:0/24328798 shutdown_connections 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/24328798 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 0x7fe6041105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/24328798 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041093a0 0x7fe604109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/24328798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041081b0 0x7fe6041085d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.276+0000 7fe60bc6f640 1 -- 172.21.15.28:0/24328798 >> 172.21.15.28:0/24328798 conn(0x7fe604077140 msgr2=0x7fe604079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- 172.21.15.28:0/24328798 shutdown_connections 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- 172.21.15.28:0/24328798 wait complete. 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 Processor -- start 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- start start 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041093a0 0x7fe60419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 0x7fe6041a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe604113e40 con 0x7fe6041093a0 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe604113cc0 con 0x7fe604109d60 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60bc6f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe604113fc0 con 0x7fe6041081b0 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe609a0b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe609a0b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51736/0 (socket says 172.21.15.28:51736) 2024-03-22T02:17:11.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe609a0b640 1 -- 172.21.15.28:0/3164062192 learned_addr learned my addr 172.21.15.28:0/3164062192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60a20c640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 0x7fe6041a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.277+0000 7fe60920a640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041093a0 0x7fe60419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 -- 172.21.15.28:0/3164062192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 msgr2=0x7fe6041a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 0x7fe6041a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 -- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041093a0 msgr2=0x7fe60419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041093a0 0x7fe60419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 -- 172.21.15.28:0/3164062192 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6041a6120 con 0x7fe6041081b0 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe609a0b640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe600000c00 tx=0x7fe60002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe5faffd640 1 -- 172.21.15.28:0/3164062192 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600041070 con 0x7fe6041081b0 2024-03-22T02:17:11.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6041a63b0 con 0x7fe6041081b0 2024-03-22T02:17:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe5faffd640 1 -- 172.21.15.28:0/3164062192 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe600004450 con 0x7fe6041081b0 2024-03-22T02:17:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe5faffd640 1 -- 172.21.15.28:0/3164062192 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600034440 con 0x7fe6041081b0 2024-03-22T02:17:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.278+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6041a68f0 con 0x7fe6041081b0 2024-03-22T02:17:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe5faffd640 1 -- 172.21.15.28:0/3164062192 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe600044070 con 0x7fe6041081b0 2024-03-22T02:17:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 Processor -- start 2024-03-22T02:17:11.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 -- start start 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518dbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518de810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e65183b8d0 con 0x55e6518dc380 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e6517e3e00 con 0x55e6518d6820 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60e84c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e651837880 con 0x55e6518d7210 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60920a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518de810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe609a0b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518dbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49556/0 (socket says 172.21.15.28:49556) 2024-03-22T02:17:11.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe60a20c640 1 -- 172.21.15.28:0/332752391 learned_addr learned my addr 172.21.15.28:0/332752391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.281+0000 7fe5faffd640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 0x7fe5f00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 -- 172.21.15.28:0/332752391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 msgr2=0x55e6518dbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518dbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 -- 172.21.15.28:0/332752391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 msgr2=0x55e6518de810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518de810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 -- 172.21.15.28:0/332752391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e65154f3f0 con 0x55e6518d6820 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe5faffd640 1 -- 172.21.15.28:0/3164062192 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe6000f12d0 con 0x7fe6041081b0 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 0x7fe5f00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc011590 tx=0x7fe5fc011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd0e3495eb06c479 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe5effff640 1 -- 172.21.15.28:0/332752391 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc017050 con 0x55e6518d6820 2024-03-22T02:17:11.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe5effff640 1 -- 172.21.15.28:0/332752391 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc00fd20 con 0x55e6518d6820 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe5effff640 1 -- 172.21.15.28:0/332752391 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc01bad0 con 0x55e6518d6820 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.282+0000 7fe60a20c640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 0x7fe5f00a53e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc022000 tx=0x7fe5fc023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- 172.21.15.28:0/332752391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 msgr2=0x55e6518d6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc011590 tx=0x7fe5fc011a50 comp rx=0 tx=0).stop 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- 172.21.15.28:0/332752391 shutdown_connections 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518de810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518dbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 --2- 172.21.15.28:0/332752391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518d6c40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- 172.21.15.28:0/332752391 >> 172.21.15.28:0/332752391 conn(0x55e6518d1d90 msgr2=0x55e6518d2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- 172.21.15.28:0/332752391 shutdown_connections 2024-03-22T02:17:11.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- 172.21.15.28:0/332752391 wait complete. 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 -- start start 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.283+0000 7fe60e84c740 1 Processor -- start 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518e0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6518df330 con 0x55e6518dc380 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e6517f7300 con 0x55e6518d7210 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6518df4b0 con 0x55e6518d6820 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60e84c740 10 client.0 _ll_register_callbacks cb 0x55e651832e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60920a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe609a0b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518e0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60920a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49558/0 (socket says 172.21.15.28:49558) 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60920a640 1 -- 172.21.15.28:0/169846084 learned_addr learned my addr 172.21.15.28:0/169846084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43346/0 (socket says 172.21.15.28:43346) 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 -- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 msgr2=0x55e6518e0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518e0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 -- 172.21.15.28:0/169846084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 msgr2=0x55e6518e1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 -- 172.21.15.28:0/169846084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6516197d0 con 0x55e6518dc380 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe609a0b640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518e0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60920a640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe60a20c640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc011560 tx=0x7fe5fc009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e5779f7e8b7d62 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.284+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc035060 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe5fc004bf0 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5fc00fd20 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5fc017020 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 10 client.16506 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e6518eb610 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5fc00fd20 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe5fc01a070 con 0x55e6518dc380 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 10 client.16506 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 10 client.16506 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 20 client.16506 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e6518eef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 4 client.16506 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e651848e10 con 0x55e6518ecb00 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.285+0000 7fe609a0b640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e6518eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.286+0000 7fe609a0b640 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e6518eef50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c5897197eedef90d server_cookie=960850971fa6f616 in_seq=0 out_seq=0 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.286+0000 7fe5ed7fa640 10 client.16506 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.286+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/169846084 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e651848e10 con 0x55e6518ecb00 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.286+0000 7fe5ed7fa640 10 client.16506 handle_command_reply: tid=0 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.286+0000 7fe60e84c740 1 client.16506 shutdown 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 msgr2=0x55e6518e82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc011560 tx=0x7fe5fc009d70 comp rx=0 tx=0).stop 2024-03-22T02:17:11.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 shutdown_connections 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e6518eef50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e82a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe5ed7fa640 0 client.16506 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d7210 0x55e6518e1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 --2- 172.21.15.28:0/169846084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518e0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 >> 172.21.15.28:0/169846084 conn(0x55e6518d1d90 msgr2=0x55e6518d0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 shutdown_connections 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 1 -- 172.21.15.28:0/169846084 wait complete. 2024-03-22T02:17:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.287+0000 7fe60e84c740 20 client.16506 trim_cache size 0 max 16384 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 Processor -- start 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 -- start start 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 0x55e6518d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041b3910 con 0x55e6518d7210 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041b11e0 con 0x55e6518dc380 2024-03-22T02:17:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60e84c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041af030 con 0x55e6518d6820 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60920a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60920a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43354/0 (socket says 172.21.15.28:43354) 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe60920a640 1 -- 172.21.15.28:0/3501005801 learned_addr learned my addr 172.21.15.28:0/3501005801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.300+0000 7fe609a0b640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60a20c640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 0x55e6518d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 -- 172.21.15.28:0/3501005801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 msgr2=0x55e6518d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 -- 172.21.15.28:0/3501005801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 msgr2=0x55e6518d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 0x55e6518d0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 -- 172.21.15.28:0/3501005801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6518d6460 con 0x55e6518d7210 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60a20c640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 0x55e6518d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe60920a640 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4012610 tx=0x7fe5f4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2278046d6127f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe5ef7fe640 1 -- 172.21.15.28:0/3501005801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f4022070 con 0x55e6518d7210 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe5ef7fe640 1 -- 172.21.15.28:0/3501005801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5f40035f0 con 0x55e6518d7210 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.301+0000 7fe5ef7fe640 1 -- 172.21.15.28:0/3501005801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f400f840 con 0x55e6518d7210 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- 172.21.15.28:0/3501005801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 msgr2=0x55e6518e8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4012610 tx=0x7fe5f4005f90 comp rx=0 tx=0).stop 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- 172.21.15.28:0/3501005801 shutdown_connections 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518dc380 0x55e6518d0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518d7210 0x55e6518e8260 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- 172.21.15.28:0/3501005801 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d6820 0x55e6518d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- 172.21.15.28:0/3501005801 >> 172.21.15.28:0/3501005801 conn(0x55e6518d1d90 msgr2=0x55e6518e0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- 172.21.15.28:0/3501005801 shutdown_connections 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- 172.21.15.28:0/3501005801 wait complete. 2024-03-22T02:17:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- start start 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 Processor -- start 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518e5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518e5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041af6a0 con 0x55e6518dc380 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041b0b10 con 0x55e6518d6820 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe6041afd70 con 0x55e6518d7210 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60e84c740 10 client.0 _ll_register_callbacks cb 0x7fe5f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.302+0000 7fe609a0b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518e5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60920a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518e5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43358/0 (socket says 172.21.15.28:43358) 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 -- 172.21.15.28:0/4160117253 learned_addr learned my addr 172.21.15.28:0/4160117253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 -- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 msgr2=0x55e6518e5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518e5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 -- 172.21.15.28:0/4160117253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 msgr2=0x55e6518e5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518e5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 -- 172.21.15.28:0/4160117253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6518e72d0 con 0x55e6518dc380 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60a20c640 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc003a20 tx=0x7fe5fc011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0b6853dbb1e2f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe5ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc035070 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe5fc0058a0 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5f40035f0 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe5fc046050 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60e84c740 10 client.16512 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e6518e8640 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.303+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe5f40035f0 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe5fc03f060 con 0x55e6518dc380 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 10 client.16512 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 10 client.16512 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 20 client.16512 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e651955740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 4 client.16512 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e651843160 con 0x55e6518ecb00 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe609a0b640 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e651955740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe5ed7fa640 10 client.16512 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe609a0b640 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e651955740 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=15029d3e69394a2 server_cookie=2958910dc5d4841c in_seq=0 out_seq=0 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe5ed7fa640 1 -- 172.21.15.28:0/4160117253 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e651843160 con 0x55e6518ecb00 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe5ed7fa640 10 client.16512 handle_command_reply: tid=0 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 1 client.16512 shutdown 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 msgr2=0x55e6518e6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.304+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc003a20 tx=0x7fe5fc011440 comp rx=0 tx=0).stop 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 shutdown_connections 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55e6518ecb00 0x55e651955740 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6518dc380 0x55e6518e6ce0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6518d7210 0x55e6518e5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe5ed7fa640 0 client.16512 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 --2- 172.21.15.28:0/4160117253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6518d6820 0x55e6518e5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 >> 172.21.15.28:0/4160117253 conn(0x55e6518d1d90 msgr2=0x55e6518d4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 shutdown_connections 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 1 -- 172.21.15.28:0/4160117253 wait complete. 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60e84c740 20 client.16512 trim_cache size 0 max 16384 2024-03-22T02:17:11.314 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 msgr2=0x7fe5f00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 0x7fe5f00a53e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc022000 tx=0x7fe5fc023000 comp rx=0 tx=0).stop 2024-03-22T02:17:11.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 msgr2=0x7fe60419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe600000c00 tx=0x7fe60002c640 comp rx=0 tx=0).stop 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 shutdown_connections 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5f00a2f20 0x7fe5f00a53e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe604109d60 0x7fe6041a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6041093a0 0x7fe60419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 --2- 172.21.15.28:0/3164062192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6041081b0 0x7fe60419e500 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 >> 172.21.15.28:0/3164062192 conn(0x7fe604077140 msgr2=0x7fe60410c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 shutdown_connections 2024-03-22T02:17:11.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.305+0000 7fe60bc6f640 1 -- 172.21.15.28:0/3164062192 wait complete. 2024-03-22T02:17:11.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2273899881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:11.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:17:11.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 181456 8.82 MB/sec warmup 113 sec latency 17.301 ms 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- 172.21.15.28:0/3344409427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 msgr2=0x7fdca810a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/3344409427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca810a1c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009ca0 tx=0x7fdca002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- 172.21.15.28:0/3344409427 shutdown_connections 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/3344409427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca810a1c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/3344409427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 0x7fdca81000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/3344409427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 0x7fdca8103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- 172.21.15.28:0/3344409427 >> 172.21.15.28:0/3344409427 conn(0x7fdca80fb790 msgr2=0x7fdca80fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- 172.21.15.28:0/3344409427 shutdown_connections 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- 172.21.15.28:0/3344409427 wait complete. 2024-03-22T02:17:11.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 Processor -- start 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- start start 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 0x7fdca819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 0x7fdca81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdca8101a10 con 0x7fdca8100610 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdca8101890 con 0x7fdca80ffc50 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.422+0000 7fdcaff3a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdca8101b90 con 0x7fdca81031e0 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43378/0 (socket says 172.21.15.28:43378) 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcadcd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 0x7fdca819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 -- 172.21.15.28:0/706055937 learned_addr learned my addr 172.21.15.28:0/706055937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 0x7fdca81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 -- 172.21.15.28:0/706055937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 msgr2=0x7fdca81a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 0x7fdca81a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 -- 172.21.15.28:0/706055937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 msgr2=0x7fdca819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 0x7fdca819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 -- 172.21.15.28:0/706055937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca81a1c60 con 0x7fdca8100610 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800e9f0 tx=0x7fdc9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdc9effd640 1 -- 172.21.15.28:0/706055937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9800cd80 con 0x7fdca8100610 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdc9effd640 1 -- 172.21.15.28:0/706055937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc98010910 con 0x7fdca8100610 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca81a1f50 con 0x7fdca8100610 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdc9effd640 1 -- 172.21.15.28:0/706055937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98018990 con 0x7fdca8100610 2024-03-22T02:17:11.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.423+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca81a2460 con 0x7fdca8100610 2024-03-22T02:17:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdc9effd640 1 -- 172.21.15.28:0/706055937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdc98010ac0 con 0x7fdca8100610 2024-03-22T02:17:11.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 Processor -- start 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 -- start start 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddead10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dded810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55683dd4a8d0 con 0x55683dde6210 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55683dcf2e00 con 0x55683dde5820 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55683dd46880 con 0x55683ddeb380 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdc9effd640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 0x7fdc940a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcae4d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcadcd6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddead10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcae4d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49614/0 (socket says 172.21.15.28:49614) 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcae4d7640 1 -- 172.21.15.28:0/811986390 learned_addr learned my addr 172.21.15.28:0/811986390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.426+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dded810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 0x7fdc940a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdc9effd640 1 -- 172.21.15.28:0/706055937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdc980cfa30 con 0x7fdca8100610 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 -- 172.21.15.28:0/811986390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 msgr2=0x55683dded810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dded810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 -- 172.21.15.28:0/811986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 msgr2=0x55683ddead10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddead10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 -- 172.21.15.28:0/811986390 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683da5e3f0 con 0x55683dde5820 2024-03-22T02:17:11.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddead10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdca00324c0 tx=0x7fdca002ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d4e8dd84b86356 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 0x7fdc940a5460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdca0047000 tx=0x7fdca0048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdc93fff640 1 -- 172.21.15.28:0/811986390 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0050070 con 0x55683dde5820 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdc93fff640 1 -- 172.21.15.28:0/811986390 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdca0003b70 con 0x55683dde5820 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.427+0000 7fdc93fff640 1 -- 172.21.15.28:0/811986390 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca0039040 con 0x55683dde5820 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/811986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 msgr2=0x55683dde5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdca00324c0 tx=0x7fdca002ceb0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/811986390 shutdown_connections 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dded810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddead10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/811986390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dde5c40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/811986390 >> 172.21.15.28:0/811986390 conn(0x55683dde0d90 msgr2=0x55683dde1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/811986390 shutdown_connections 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/811986390 wait complete. 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.428+0000 7fdcb2bfb740 1 -- start start 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 Processor -- start 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 0x55683ddefd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683ddeb380 0x55683ddf72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55683ddee2d0 con 0x55683dde6210 2024-03-22T02:17:11.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55683dd06300 con 0x55683ddeb380 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55683ddee450 con 0x55683dde5820 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcb2bfb740 10 client.0 _ll_register_callbacks cb 0x55683dd41e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43386/0 (socket says 172.21.15.28:43386) 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 -- 172.21.15.28:0/3220907807 learned_addr learned my addr 172.21.15.28:0/3220907807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 0x55683ddefd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683ddeb380 0x55683ddf72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 -- 172.21.15.28:0/3220907807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 msgr2=0x55683ddefd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 0x55683ddefd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 -- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683ddeb380 msgr2=0x55683ddf72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683ddeb380 0x55683ddf72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcad4d5640 1 -- 172.21.15.28:0/3220907807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683db287d0 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.429+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 0x55683ddefd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdc98098260 tx=0x7fdc9806b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c16d116cbceb69 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98018d30 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdc9809b250 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdca0003b70 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdc9809be00 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 10 client.16524 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55683ddfa690 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdca0003b70 con 0x55683dde6210 2024-03-22T02:17:11.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdc98097b50 con 0x55683dde6210 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 10 client.16524 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 20 client.16524 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683ddfbd40 0x55683ddfe1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 4 client.16524 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:11.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55683dd57e10 con 0x55683ddfbd40 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.430+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683ddfbd40 0x55683ddfe1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.431+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683ddfbd40 0x55683ddfe1c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7571ae7121f985 server_cookie=47de3752fc8cc74f in_seq=0 out_seq=0 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.431+0000 7fdc917fa640 10 client.16524 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdc917fa640 1 -- 172.21.15.28:0/3220907807 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55683dd57e10 con 0x55683ddfbd40 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdc917fa640 10 client.16524 handle_command_reply: tid=0 2024-03-22T02:17:11.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 client.16524 shutdown 2024-03-22T02:17:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 msgr2=0x55683ddf0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fdc98098260 tx=0x7fdc9806b9e0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 shutdown_connections 2024-03-22T02:17:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683ddfbd40 0x55683ddfe1c0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683ddeb380 0x55683ddf72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdc917fa640 0 client.16524 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf0270 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/3220907807 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683dde5820 0x55683ddefd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.432+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 >> 172.21.15.28:0/3220907807 conn(0x55683dde0d90 msgr2=0x55683dddffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.433+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 shutdown_connections 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.433+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/3220907807 wait complete. 2024-03-22T02:17:11.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.433+0000 7fdcb2bfb740 20 client.16524 trim_cache size 0 max 16384 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 Processor -- start 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 -- start start 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dddf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dddfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81af470 con 0x55683dde6210 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81acd40 con 0x55683dde5820 2024-03-22T02:17:11.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81aab90 con 0x55683ddeb380 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcadcd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dddf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43392/0 (socket says 172.21.15.28:43392) 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 -- 172.21.15.28:0/2103450877 learned_addr learned my addr 172.21.15.28:0/2103450877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dddfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 -- 172.21.15.28:0/2103450877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 msgr2=0x55683dddfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dddfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 -- 172.21.15.28:0/2103450877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 msgr2=0x55683dddf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dddf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 -- 172.21.15.28:0/2103450877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55683ddfe840 con 0x55683dde6210 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dddf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdc980cd490 tx=0x7fdc980bc0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca375badc7eda890 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdc937fe640 1 -- 172.21.15.28:0/2103450877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9809fea0 con 0x55683dde6210 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdc937fe640 1 -- 172.21.15.28:0/2103450877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc980a4760 con 0x55683dde6210 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.447+0000 7fdc937fe640 1 -- 172.21.15.28:0/2103450877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc9809b250 con 0x55683dde6210 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2103450877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 msgr2=0x55683ddf7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fdc980cd490 tx=0x7fdc980bc0b0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2103450877 shutdown_connections 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683dddfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683dde6210 0x55683ddf7270 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2103450877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683dddf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2103450877 >> 172.21.15.28:0/2103450877 conn(0x55683dde0d90 msgr2=0x55683dde7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2103450877 shutdown_connections 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2103450877 wait complete. 2024-03-22T02:17:11.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 -- start start 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.448+0000 7fdcb2bfb740 1 Processor -- start 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683ddf4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683ddf5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81ab200 con 0x55683ddf61b0 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81ac670 con 0x55683dde5820 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdca81ab8d0 con 0x55683ddeb380 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcb2bfb740 10 client.0 _ll_register_callbacks cb 0x7fdc98009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcae4d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683ddf4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcae4d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43404/0 (socket says 172.21.15.28:43404) 2024-03-22T02:17:11.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcae4d7640 1 -- 172.21.15.28:0/2665836790 learned_addr learned my addr 172.21.15.28:0/2665836790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683ddf5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 -- 172.21.15.28:0/2665836790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 msgr2=0x55683ddf5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683ddf5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 -- 172.21.15.28:0/2665836790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 msgr2=0x55683ddf6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 -- 172.21.15.28:0/2665836790 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55683ddf6b70 con 0x55683dde5820 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcae4d7640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdcadcd6640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683ddf4cd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55683ddfea70 tx=0x7fdca4012ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=111aaf4319664590 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.449+0000 7fdc917fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4020600 con 0x55683dde5820 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc980a4760 con 0x55683dde5820 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdca40207a0 con 0x55683dde5820 2024-03-22T02:17:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdca40114a0 con 0x55683dde5820 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 10 client.26363 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55683de627a0 con 0x55683dde5820 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc980a4760 con 0x55683dde5820 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdca40298b0 con 0x55683dde5820 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 10 client.26363 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 20 client.26363 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683de63130 0x55683de65520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 4 client.26363 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55683dd52160 con 0x55683de63130 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.450+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683de63130 0x55683de65520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.451+0000 7fdc917fa640 10 client.26363 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:11.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.451+0000 7fdcad4d5640 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683de63130 0x55683de65520 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7cb94b84df6308d server_cookie=9c9acd47fbcd9afe in_seq=0 out_seq=0 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdc917fa640 1 -- 172.21.15.28:0/2665836790 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55683dd52160 con 0x55683de63130 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdc917fa640 10 client.26363 handle_command_reply: tid=0 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 client.26363 shutdown 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 msgr2=0x55683ddf4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683ddf4cd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55683ddfea70 tx=0x7fdca4012ae0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 shutdown_connections 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55683de63130 0x55683de65520 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55683ddf61b0 0x55683ddf6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55683ddeb380 0x55683ddf5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 --2- 172.21.15.28:0/2665836790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55683dde5820 0x55683ddf4cd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 >> 172.21.15.28:0/2665836790 conn(0x55683dde0d90 msgr2=0x55683dde3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdc917fa640 0 client.26363 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:11.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 shutdown_connections 2024-03-22T02:17:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 1 -- 172.21.15.28:0/2665836790 wait complete. 2024-03-22T02:17:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcb2bfb740 20 client.26363 trim_cache size 0 max 16384 2024-03-22T02:17:11.461 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 msgr2=0x7fdc940a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 0x7fdc940a5460 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdca0047000 tx=0x7fdca0048000 comp rx=0 tx=0).stop 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 msgr2=0x7fdca819a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.452+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdc9800e9f0 tx=0x7fdc9800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 shutdown_connections 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc940a2fa0 0x7fdc940a5460 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdca81031e0 0x7fdca81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdca8100610 0x7fdca819a580 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 --2- 172.21.15.28:0/706055937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdca80ffc50 0x7fdca819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 >> 172.21.15.28:0/706055937 conn(0x7fdca80fb790 msgr2=0x7fdca80fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 shutdown_connections 2024-03-22T02:17:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:11.453+0000 7fdcaff3a640 1 -- 172.21.15.28:0/706055937 wait complete. 2024-03-22T02:17:11.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2273899881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:11.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2273899881' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:12.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 183514 8.81 MB/sec warmup 114 sec latency 86.051 ms 2024-03-22T02:17:12.464 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:17:12.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:17:12.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:17:12.489 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:12 smithi028 ceph-mon[31916]: pgmap v466: 129 pgs: 129 active+clean; 42 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-03-22T02:17:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:12 smithi120 ceph-mon[36803]: pgmap v466: 129 pgs: 129 active+clean; 42 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-03-22T02:17:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:12 smithi083 ceph-mon[36340]: pgmap v466: 129 pgs: 129 active+clean; 42 MiB data, 672 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 425 op/s 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 -- 172.21.15.28:0/4243337708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330100a90 msgr2=0x7f8330100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 --2- 172.21.15.28:0/4243337708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330100a90 0x7f8330100eb0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8328009900 tx=0x7f832802bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 -- 172.21.15.28:0/4243337708 shutdown_connections 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 --2- 172.21.15.28:0/4243337708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102640 0x7f8330108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 --2- 172.21.15.28:0/4243337708 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330101c80 0x7f8330102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 --2- 172.21.15.28:0/4243337708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330100a90 0x7f8330100eb0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 -- 172.21.15.28:0/4243337708 >> 172.21.15.28:0/4243337708 conn(0x7f83300fc020 msgr2=0x7f83300fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 -- 172.21.15.28:0/4243337708 shutdown_connections 2024-03-22T02:17:12.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.576+0000 7f8334a2a640 1 -- 172.21.15.28:0/4243337708 wait complete. 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 Processor -- start 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 -- start start 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330100a90 0x7f833019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330102640 0x7f83301a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f833010d830 con 0x7f8330101c80 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f833010d6b0 con 0x7f8330102640 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f8334a2a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f833010d9b0 con 0x7f8330100a90 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53708/0 (socket says 172.21.15.28:53708) 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 -- 172.21.15.28:0/3750443974 learned_addr learned my addr 172.21.15.28:0/3750443974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832e59c640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330100a90 0x7f833019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330102640 0x7f83301a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 -- 172.21.15.28:0/3750443974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330100a90 msgr2=0x7f833019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330100a90 0x7f833019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 -- 172.21.15.28:0/3750443974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330102640 msgr2=0x7f83301a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330102640 0x7f83301a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 -- 172.21.15.28:0/3750443974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83301a1d40 con 0x7f8330101c80 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f832dd9b640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f831800d950 tx=0x7f831800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.577+0000 7f83277fe640 1 -- 172.21.15.28:0/3750443974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8318017030 con 0x7f8330101c80 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.578+0000 7f83277fe640 1 -- 172.21.15.28:0/3750443974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8318004020 con 0x7f8330101c80 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.578+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83301a1fd0 con 0x7f8330101c80 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.578+0000 7f83277fe640 1 -- 172.21.15.28:0/3750443974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83180047a0 con 0x7f8330101c80 2024-03-22T02:17:12.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.578+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83301a24f0 con 0x7f8330101c80 2024-03-22T02:17:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f83277fe640 1 -- 172.21.15.28:0/3750443974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f83180041d0 con 0x7f8330101c80 2024-03-22T02:17:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 Processor -- start 2024-03-22T02:17:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 -- start start 2024-03-22T02:17:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce7b029570 con 0x55ce7b0ca720 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ce7afd1e10 con 0x55ce7b0c55b0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f8337614740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ce7b025520 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f83277fe640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 0x7f831c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f832e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.580+0000 7f832e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:60400/0 (socket says 172.21.15.28:60400) 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 -- 172.21.15.28:0/3099635644 learned_addr learned my addr 172.21.15.28:0/3099635644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 0x7f831c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f83277fe640 1 -- 172.21.15.28:0/3750443974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f83180cf180 con 0x7f8330101c80 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 -- 172.21.15.28:0/3099635644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 msgr2=0x55ce7b0c4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 -- 172.21.15.28:0/3099635644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 msgr2=0x55ce7b0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 -- 172.21.15.28:0/3099635644 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce7ae057e0 con 0x55ce7b0c55b0 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 0x7f831c0a53e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55ce7b0c5e10 tx=0x7f8320009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f832e59c640 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8328009900 tx=0x7f83280095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8c31fbd21a8a486 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f8324ff9640 1 -- 172.21.15.28:0/3099635644 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832802cb80 con 0x55ce7b0c55b0 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f8324ff9640 1 -- 172.21.15.28:0/3099635644 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8328005040 con 0x55ce7b0c55b0 2024-03-22T02:17:12.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.581+0000 7f8324ff9640 1 -- 172.21.15.28:0/3099635644 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328004920 con 0x55ce7b0c55b0 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- 172.21.15.28:0/3099635644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 msgr2=0x55ce7b0ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8328009900 tx=0x7f83280095e0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- 172.21.15.28:0/3099635644 shutdown_connections 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0ca0b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 --2- 172.21.15.28:0/3099635644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- 172.21.15.28:0/3099635644 >> 172.21.15.28:0/3099635644 conn(0x55ce7b0c0130 msgr2=0x55ce7b0c0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- 172.21.15.28:0/3099635644 shutdown_connections 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- 172.21.15.28:0/3099635644 wait complete. 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.582+0000 7f8337614740 1 -- start start 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 Processor -- start 2024-03-22T02:17:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce7b0cd670 con 0x55ce7b0ca720 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ce7afe5110 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ce7b0cd7f0 con 0x55ce7b0c55b0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8337614740 10 client.0 _ll_register_callbacks cb 0x55ce7b021be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:60404/0 (socket says 172.21.15.28:60404) 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d6300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53730/0 (socket says 172.21.15.28:53730) 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 -- 172.21.15.28:0/100204759 learned_addr learned my addr 172.21.15.28:0/100204759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 -- 172.21.15.28:0/100204759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c55b0 msgr2=0x55ce7b0cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 -- 172.21.15.28:0/100204759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 msgr2=0x55ce7b0d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 -- 172.21.15.28:0/100204759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce7ae65f70 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f832e59c640 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f832802cfd0 tx=0x7f8328000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f90296dccdf68471 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.583+0000 7f8315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328004ad0 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8328004c70 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8328005040 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8328056050 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 10 client.26375 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:12.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce7acfd170 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8328005040 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8328045420 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 10 client.26375 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 10 client.26375 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 20 client.26375 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b0dcbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 4 client.26375 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce7b036ab0 con 0x55ce7b0da7f0 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.584+0000 7f832dd9b640 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b0dcbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.585+0000 7f832dd9b640 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b0dcbe0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51490cbf60a507d5 server_cookie=c87411bdd6a47761 in_seq=0 out_seq=0 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.585+0000 7f8315ffb640 10 client.26375 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.585+0000 7f8315ffb640 1 -- 172.21.15.28:0/100204759 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce7b036ab0 con 0x55ce7b0da7f0 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.585+0000 7f8315ffb640 10 client.26375 handle_command_reply: tid=0 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 client.26375 shutdown 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 msgr2=0x55ce7b0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f832802cfd0 tx=0x7f8328000bc0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 shutdown_connections 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b0dcbe0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8315ffb640 0 client.26375 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 --2- 172.21.15.28:0/100204759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0ced80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 >> 172.21.15.28:0/100204759 conn(0x55ce7b0c0130 msgr2=0x55ce7b0bf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 shutdown_connections 2024-03-22T02:17:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 1 -- 172.21.15.28:0/100204759 wait complete. 2024-03-22T02:17:12.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.586+0000 7f8337614740 20 client.26375 trim_cache size 0 max 16384 2024-03-22T02:17:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 Processor -- start 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 -- start start 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 0x55ce7b0c6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83301af490 con 0x55ce7b0c55b0 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83301acd60 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f8337614740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83301aabb0 con 0x55ce7b0ca720 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53738/0 (socket says 172.21.15.28:53738) 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 0x55ce7b0c6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832e59c640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832dd9b640 1 -- 172.21.15.28:0/3599265752 learned_addr learned my addr 172.21.15.28:0/3599265752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 -- 172.21.15.28:0/3599265752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 msgr2=0x55ce7b0c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 -- 172.21.15.28:0/3599265752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 msgr2=0x55ce7b0c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832ed9d640 1 -- 172.21.15.28:0/3599265752 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce7b0cd6d0 con 0x55ce7b0ca720 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.605+0000 7f832dd9b640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f832ed9d640 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 0x55ce7b0c6aa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f832000fe80 tx=0x7f8320008000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe572a059fea588 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f8317fff640 1 -- 172.21.15.28:0/3599265752 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320011c70 con 0x55ce7b0ca720 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f8317fff640 1 -- 172.21.15.28:0/3599265752 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8320011e10 con 0x55ce7b0ca720 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f8317fff640 1 -- 172.21.15.28:0/3599265752 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320010430 con 0x55ce7b0ca720 2024-03-22T02:17:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f8337614740 1 -- 172.21.15.28:0/3599265752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 msgr2=0x55ce7b0c6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.606+0000 7f8337614740 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 0x55ce7b0c6aa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f832000fe80 tx=0x7f8320008000 comp rx=0 tx=0).stop 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- 172.21.15.28:0/3599265752 shutdown_connections 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0ca720 0x55ce7b0c6aa0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- 172.21.15.28:0/3599265752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- 172.21.15.28:0/3599265752 >> 172.21.15.28:0/3599265752 conn(0x55ce7b0c0130 msgr2=0x55ce7b0c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- 172.21.15.28:0/3599265752 shutdown_connections 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- 172.21.15.28:0/3599265752 wait complete. 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- start start 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 Processor -- start 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83301ab220 con 0x55ce7b0c55b0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83301ac690 con 0x55ce7b0ca720 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83301ab8f0 con 0x55ce7b0c4bc0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f8337614740 10 client.0 _ll_register_callbacks cb 0x7f8318097100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f832dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53740/0 (socket says 172.21.15.28:53740) 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f832dd9b640 1 -- 172.21.15.28:0/393176933 learned_addr learned my addr 172.21.15.28:0/393176933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f832ed9d640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.607+0000 7f832e59c640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 -- 172.21.15.28:0/393176933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 msgr2=0x55ce7b0d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0d33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 -- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0ca720 msgr2=0x55ce7b0d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 -- 172.21.15.28:0/393176933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce7b0d4e80 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832e59c640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f832dd9b640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f831809b1a0 tx=0x7f831806d120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4359e065f5d05b2b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831809b4a0 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8318097b10 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8320011e10 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f831809f5f0 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 10 client.16548 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ce7b141170 con 0x55ce7b0c55b0 2024-03-22T02:17:12.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8320011e10 con 0x55ce7b0c55b0 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f831809a8f0 con 0x55ce7b0c55b0 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 10 client.16548 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 10 client.16548 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 20 client.16548 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 4 client.16548 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.608+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ce7b030e00 con 0x55ce7b0da7f0 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.609+0000 7f832e59c640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.609+0000 7f8315ffb640 10 client.16548 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.609+0000 7f832e59c640 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b143bc0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8457a8ae1caa1f97 server_cookie=9fd1edcd9f619707 in_seq=0 out_seq=0 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8315ffb640 1 -- 172.21.15.28:0/393176933 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ce7b030e00 con 0x55ce7b0da7f0 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8315ffb640 10 client.16548 handle_command_reply: tid=0 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 client.16548 shutdown 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 msgr2=0x55ce7b0d38f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f831809b1a0 tx=0x7f831806d120 comp rx=0 tx=0).stop 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 shutdown_connections 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ce7b0da7f0 0x55ce7b143bc0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ce7b0ca720 0x55ce7b0d4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8315ffb640 0 client.16548 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce7b0c55b0 0x55ce7b0d38f0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 --2- 172.21.15.28:0/393176933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ce7b0c4bc0 0x55ce7b0d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 >> 172.21.15.28:0/393176933 conn(0x55ce7b0c0130 msgr2=0x55ce7b0beb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.610+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 shutdown_connections 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8337614740 1 -- 172.21.15.28:0/393176933 wait complete. 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8337614740 20 client.16548 trim_cache size 0 max 16384 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "5e1f42cf-11b8-4d95-8e9f-9594103864c5", 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:17:12.618 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:12.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 msgr2=0x7f831c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 0x7f831c0a53e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55ce7b0c5e10 tx=0x7f8320009500 comp rx=0 tx=0).stop 2024-03-22T02:17:12.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 msgr2=0x7f833019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f831800d950 tx=0x7f831800de10 comp rx=0 tx=0).stop 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 shutdown_connections 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f831c0a2f20 0x7f831c0a53e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8330102640 0x7f83301a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8330101c80 0x7f833019a660 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 --2- 172.21.15.28:0/3750443974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330100a90 0x7f833019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 >> 172.21.15.28:0/3750443974 conn(0x7f83300fc020 msgr2=0x7f8330107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 shutdown_connections 2024-03-22T02:17:12.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.611+0000 7f8334a2a640 1 -- 172.21.15.28:0/3750443974 wait complete. 2024-03-22T02:17:12.623 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:17:12.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:17:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 -- 172.21.15.28:0/1815122681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0102610 msgr2=0x7f3be0104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 --2- 172.21.15.28:0/1815122681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0102610 0x7f3be0104a00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0009900 tx=0x7f3bd002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 -- 172.21.15.28:0/1815122681 shutdown_connections 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 --2- 172.21.15.28:0/1815122681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0107870 0x7f3be0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 --2- 172.21.15.28:0/1815122681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 0x7f3be0107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 --2- 172.21.15.28:0/1815122681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0102610 0x7f3be0104a00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.772+0000 7f3be651e640 1 -- 172.21.15.28:0/1815122681 >> 172.21.15.28:0/1815122681 conn(0x7f3be00fc020 msgr2=0x7f3be00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.773+0000 7f3be651e640 1 -- 172.21.15.28:0/1815122681 shutdown_connections 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.773+0000 7f3be651e640 1 -- 172.21.15.28:0/1815122681 wait complete. 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.773+0000 7f3be651e640 1 Processor -- start 2024-03-22T02:17:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.773+0000 7f3be651e640 1 -- start start 2024-03-22T02:17:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.773+0000 7f3be651e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0102610 0x7f3be0197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 0x7f3be0198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d860 con 0x7f3be0107870 2024-03-22T02:17:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d6e0 con 0x7f3be0104f40 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3be010d9e0 con 0x7f3be0102610 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 0x7f3be0198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0102610 0x7f3be0197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53756/0 (socket says 172.21.15.28:53756) 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 -- 172.21.15.28:0/2695104054 learned_addr learned my addr 172.21.15.28:0/2695104054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 -- 172.21.15.28:0/2695104054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0102610 msgr2=0x7f3be0197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0102610 0x7f3be0197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 -- 172.21.15.28:0/2695104054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 msgr2=0x7f3be0198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 0x7f3be0198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 -- 172.21.15.28:0/2695104054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be019fb50 con 0x7f3be0107870 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be4abb640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800eb10 tx=0x7f3bd800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3bdd7fa640 1 -- 172.21.15.28:0/2695104054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8004b20 con 0x7f3be0107870 2024-03-22T02:17:12.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be019fe40 con 0x7f3be0107870 2024-03-22T02:17:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3bdd7fa640 1 -- 172.21.15.28:0/2695104054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd8004cc0 con 0x7f3be0107870 2024-03-22T02:17:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3bdd7fa640 1 -- 172.21.15.28:0/2695104054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8010760 con 0x7f3be0107870 2024-03-22T02:17:12.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.774+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be01a02b0 con 0x7f3be0107870 2024-03-22T02:17:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdd7fa640 1 -- 172.21.15.28:0/2695104054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3bd8005020 con 0x7f3be0107870 2024-03-22T02:17:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 Processor -- start 2024-03-22T02:17:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 -- start start 2024-03-22T02:17:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d3ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d403a0 0x55f458d42830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f458c9b8c0 con 0x55f458d3b230 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f458c43c20 con 0x55f458d3a840 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be91e3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f458c97870 con 0x55f458d403a0 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d3ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53760/0 (socket says 172.21.15.28:53760) 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 -- 172.21.15.28:0/111135331 learned_addr learned my addr 172.21.15.28:0/111135331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdd7fa640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 0x7f3bc80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d403a0 0x55f458d42830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3be4abb640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 0x7f3bc80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 -- 172.21.15.28:0/111135331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d403a0 msgr2=0x55f458d42830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d403a0 0x55f458d42830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 -- 172.21.15.28:0/111135331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 msgr2=0x55f458d3ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d3ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdffff640 1 -- 172.21.15.28:0/111135331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f458c68150 con 0x55f458d3b230 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.777+0000 7f3bdd7fa640 1 -- 172.21.15.28:0/2695104054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3bd80cfd50 con 0x7f3be0107870 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.778+0000 7f3bdffff640 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e2621333a164e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.778+0000 7f3be4abb640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 0x7f3bc80a53e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bd809bd70 tx=0x7f3bd80bc040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.778+0000 7f3bc67fc640 1 -- 172.21.15.28:0/111135331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0045070 con 0x55f458d3b230 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.778+0000 7f3bc67fc640 1 -- 172.21.15.28:0/111135331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0004040 con 0x55f458d3b230 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.778+0000 7f3bc67fc640 1 -- 172.21.15.28:0/111135331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0004c30 con 0x55f458d3b230 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- 172.21.15.28:0/111135331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 msgr2=0x55f458d3fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f3bd0000c00 tx=0x7f3bd0009560 comp rx=0 tx=0).stop 2024-03-22T02:17:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- 172.21.15.28:0/111135331 shutdown_connections 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d403a0 0x55f458d42830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3b230 0x55f458d3fd30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- 172.21.15.28:0/111135331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d3ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- 172.21.15.28:0/111135331 >> 172.21.15.28:0/111135331 conn(0x55f458d319a0 msgr2=0x55f458d31da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- 172.21.15.28:0/111135331 shutdown_connections 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- 172.21.15.28:0/111135331 wait complete. 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- start start 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 Processor -- start 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d44cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d403a0 0x55f458d4c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.779+0000 7f3be91e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f458d432f0 con 0x55f458d403a0 2024-03-22T02:17:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3be91e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f458c57130 con 0x55f458d3a840 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3be91e3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f458d43470 con 0x55f458d3b230 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3be91e3740 10 client.0 _ll_register_callbacks cb 0x55f458c56b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d403a0 0x55f458d4c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:50982/0 (socket says 172.21.15.28:50982) 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d44cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdffff640 1 -- 172.21.15.28:0/1786494534 learned_addr learned my addr 172.21.15.28:0/1786494534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/1786494534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 msgr2=0x55f458d45230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/1786494534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d403a0 msgr2=0x55f458d4c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d403a0 0x55f458d4c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/1786494534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f458a797d0 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdffff640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d44cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009fd0 tx=0x7f3bd40096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afad831d12427ec6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400fe30 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3bd4011070 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.780+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd0004040 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3bd402b050 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 10 client.26393 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f4589af3f0 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd0004040 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3bd40035f0 con 0x55f458d3a840 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 10 client.26393 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 10 client.26393 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 20 client.26393 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458d50ad0 0x55f458d52f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 4 client.26393 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f458ca8e00 con 0x55f458d50ad0 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.781+0000 7f3bdffff640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458d50ad0 0x55f458d52f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.782+0000 7f3bdffff640 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458d50ad0 0x55f458d52f90 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90dfcbd0662803b server_cookie=20a72576f87d1c8b in_seq=0 out_seq=0 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.782+0000 7f3bbf7fe640 10 client.26393 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/1786494534 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f458ca8e00 con 0x55f458d50ad0 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3bbf7fe640 10 client.26393 handle_command_reply: tid=0 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 client.26393 shutdown 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 msgr2=0x55f458d44cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d44cf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009fd0 tx=0x7f3bd40096c0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 shutdown_connections 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458d50ad0 0x55f458d52f90 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d403a0 0x55f458d4c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3bbf7fe640 0 client.26393 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 --2- 172.21.15.28:0/1786494534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d3a840 0x55f458d44cf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 >> 172.21.15.28:0/1786494534 conn(0x55f458d319a0 msgr2=0x55f458d30c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 shutdown_connections 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 1 -- 172.21.15.28:0/1786494534 wait complete. 2024-03-22T02:17:12.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.783+0000 7f3be91e3740 20 client.26393 trim_cache size 0 max 16384 2024-03-22T02:17:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.800+0000 7f3be91e3740 1 Processor -- start 2024-03-22T02:17:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 -- start start 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d4c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458d44e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01ab060 con 0x55f458d3a840 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01a8930 con 0x55f458d403a0 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3be91e3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01a6780 con 0x55f458d3b230 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53792/0 (socket says 172.21.15.28:53792) 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/921493424 learned_addr learned my addr 172.21.15.28:0/921493424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdffff640 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d4c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458d44e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/921493424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 msgr2=0x55f458d4c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d4c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/921493424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 msgr2=0x55f458d44e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458d44e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/921493424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f458d43db0 con 0x55f458d3a840 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006010 tx=0x7f3bd4013ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89956109b5d841ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bc5ffb640 1 -- 172.21.15.28:0/921493424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd400fe30 con 0x55f458d3a840 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.801+0000 7f3bc5ffb640 1 -- 172.21.15.28:0/921493424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd4003bb0 con 0x55f458d3a840 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3bc5ffb640 1 -- 172.21.15.28:0/921493424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4022bb0 con 0x55f458d3a840 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- 172.21.15.28:0/921493424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 msgr2=0x55f458d44900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4006010 tx=0x7f3bd4013ac0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- 172.21.15.28:0/921493424 shutdown_connections 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458d44e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d4c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 --2- 172.21.15.28:0/921493424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d44900 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- 172.21.15.28:0/921493424 >> 172.21.15.28:0/921493424 conn(0x55f458d319a0 msgr2=0x55f458d3c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- 172.21.15.28:0/921493424 shutdown_connections 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- 172.21.15.28:0/921493424 wait complete. 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.802+0000 7f3be91e3740 1 -- start start 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 Processor -- start 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d495b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458db4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01a6df0 con 0x55f458d3a840 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01a8260 con 0x55f458d403a0 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3be01a74c0 con 0x55f458d3b230 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3be91e3740 10 client.0 _ll_register_callbacks cb 0x7f3bd4013c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53804/0 (socket says 172.21.15.28:53804) 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdffff640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d495b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/494921582 learned_addr learned my addr 172.21.15.28:0/494921582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458db4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/494921582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 msgr2=0x55f458d495b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d495b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 msgr2=0x55f458db4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458db4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdf7fe640 1 -- 172.21.15.28:0/494921582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f45896f0f0 con 0x55f458d3a840 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.803+0000 7f3bdffff640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bdf7fe640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400a010 tx=0x7f3bd4013120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea74d11d8cfe1ab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4005500 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f3bd40056a0 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3bd4003bb0 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3bd4041050 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 10 client.16566 fetch_fsmap learned FSMap version 27 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f458db62f0 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bd4003bb0 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3bd402e040 con 0x55f458d3a840 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 10 client.16566 fetch_fsmap finished waiting for FSMap version 27 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 10 client.16566 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 20 client.16566 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458db6cb0 0x55f458db90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:12.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 4 client.16566 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:17:12.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.804+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f458ca3150 con 0x55f458db6cb0 2024-03-22T02:17:12.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.805+0000 7f3bdffff640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458db6cb0 0x55f458db90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:12.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.805+0000 7f3bdffff640 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458db6cb0 0x55f458db90d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841a2a4a9336ac15 server_cookie=ebf3c36d717b4a51 in_seq=0 out_seq=0 2024-03-22T02:17:12.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.805+0000 7f3bbf7fe640 10 client.16566 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3bbf7fe640 1 -- 172.21.15.28:0/494921582 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f458ca3150 con 0x55f458db6cb0 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3bbf7fe640 10 client.16566 handle_command_reply: tid=0 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 client.16566 shutdown 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 msgr2=0x55f458d49070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3bd400a010 tx=0x7f3bd4013120 comp rx=0 tx=0).stop 2024-03-22T02:17:12.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 shutdown_connections 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f458db6cb0 0x55f458db90d0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f458d403a0 0x55f458db4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f458d3b230 0x55f458d495b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3bbf7fe640 0 client.16566 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 --2- 172.21.15.28:0/494921582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f458d3a840 0x55f458d49070 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 >> 172.21.15.28:0/494921582 conn(0x55f458d319a0 msgr2=0x55f458d44900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 shutdown_connections 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 1 -- 172.21.15.28:0/494921582 wait complete. 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.807+0000 7f3be91e3740 20 client.16566 trim_cache size 0 max 16384 2024-03-22T02:17:12.813 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "5e1f42cf-11b8-4d95-8e9f-9594103864c5": { 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "5e1f42cf-11b8-4d95-8e9f-9594103864c5", 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:17:12.814 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 msgr2=0x7f3bc80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 0x7f3bc80a53e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3bd809bd70 tx=0x7f3bd80bc040 comp rx=0 tx=0).stop 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 msgr2=0x7f3be019f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f3bd800eb10 tx=0x7f3bd800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 shutdown_connections 2024-03-22T02:17:12.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3bc80a2f20 0x7f3bc80a53e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3be0107870 0x7f3be019f4e0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3be0104f40 0x7f3be0198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 --2- 172.21.15.28:0/2695104054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3be0102610 0x7f3be0197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 >> 172.21.15.28:0/2695104054 conn(0x7f3be00fc020 msgr2=0x7f3be0103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 shutdown_connections 2024-03-22T02:17:12.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:12.808+0000 7f3be651e640 1 -- 172.21.15.28:0/2695104054 wait complete. 2024-03-22T02:17:12.819 INFO:tasks.cephfs.filesystem:scrub status for tag:5e1f42cf-11b8-4d95-8e9f-9594103864c5 - {'path': '/', 'tag': '5e1f42cf-11b8-4d95-8e9f-9594103864c5', 'options': 'recursive,force'} 2024-03-22T02:17:13.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 185382 8.84 MB/sec warmup 115 sec latency 29.570 ms 2024-03-22T02:17:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:13 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:17:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:13 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:13.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:13 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:17:13.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:13 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:13 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:17:13.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:13 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:13.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:13 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:17:13.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:13 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:13.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:13 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:17:13.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:13 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:13.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:13 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:17:13.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:13 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:14.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 188301 8.91 MB/sec warmup 116 sec latency 15.807 ms 2024-03-22T02:17:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:14 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:17:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:14 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:14 smithi120 ceph-mon[36803]: pgmap v467: 129 pgs: 129 active+clean; 42 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 369 op/s 2024-03-22T02:17:14.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:14 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:17:14.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:14 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:14.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:14 smithi083 ceph-mon[36340]: pgmap v467: 129 pgs: 129 active+clean; 42 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 369 op/s 2024-03-22T02:17:14.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:14 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:17:14.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:14 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:14.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:14 smithi028 ceph-mon[31916]: pgmap v467: 129 pgs: 129 active+clean; 42 MiB data, 660 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.9 MiB/s wr, 369 op/s 2024-03-22T02:17:15.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 190883 8.94 MB/sec warmup 117 sec latency 26.219 ms 2024-03-22T02:17:16.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 192473 8.94 MB/sec warmup 118 sec latency 44.935 ms 2024-03-22T02:17:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:16 smithi120 ceph-mon[36803]: pgmap v468: 129 pgs: 129 active+clean; 44 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 449 op/s 2024-03-22T02:17:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:16 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:16.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:16 smithi083 ceph-mon[36340]: pgmap v468: 129 pgs: 129 active+clean; 44 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 449 op/s 2024-03-22T02:17:16.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:16 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:16 smithi028 ceph-mon[31916]: pgmap v468: 129 pgs: 129 active+clean; 44 MiB data, 681 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 449 op/s 2024-03-22T02:17:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:16 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 -- 172.21.15.28:0/2792309473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8102650 msgr2=0x7fe3a8104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/2792309473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8102650 0x7fe3a8104a40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009900 tx=0x7fe3a002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 -- 172.21.15.28:0/2792309473 shutdown_connections 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/2792309473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a81078b0 0x7fe3a8109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/2792309473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104f80 0x7fe3a8107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/2792309473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8102650 0x7fe3a8104a40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 -- 172.21.15.28:0/2792309473 >> 172.21.15.28:0/2792309473 conn(0x7fe3a80fc040 msgr2=0x7fe3a80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 -- 172.21.15.28:0/2792309473 shutdown_connections 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.750+0000 7fe3af7b6640 1 -- 172.21.15.28:0/2792309473 wait complete. 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 Processor -- start 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 -- start start 2024-03-22T02:17:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 0x7fe3a8197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81078b0 0x7fe3a819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a810d880 con 0x7fe3a8104f80 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a810d700 con 0x7fe3a8102650 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3af7b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a810da00 con 0x7fe3a81078b0 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3ad552640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 0x7fe3a8197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53824/0 (socket says 172.21.15.28:53824) 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 -- 172.21.15.28:0/871372246 learned_addr learned my addr 172.21.15.28:0/871372246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3add53640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81078b0 0x7fe3a819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 -- 172.21.15.28:0/871372246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81078b0 msgr2=0x7fe3a819f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81078b0 0x7fe3a819f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 -- 172.21.15.28:0/871372246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 msgr2=0x7fe3a8197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 0x7fe3a8197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 -- 172.21.15.28:0/871372246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a819fa80 con 0x7fe3a8104f80 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3ad552640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 0x7fe3a8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3acd51640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00e9f0 tx=0x7fe39c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.751+0000 7fe3a67fc640 1 -- 172.21.15.28:0/871372246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c00ccc0 con 0x7fe3a8104f80 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.752+0000 7fe3a67fc640 1 -- 172.21.15.28:0/871372246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe39c005500 con 0x7fe3a8104f80 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.752+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a81004a0 con 0x7fe3a8104f80 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.752+0000 7fe3a67fc640 1 -- 172.21.15.28:0/871372246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c010690 con 0x7fe3a8104f80 2024-03-22T02:17:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.752+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a81009a0 con 0x7fe3a8104f80 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3a67fc640 1 -- 172.21.15.28:0/871372246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe39c0046c0 con 0x7fe3a8104f80 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3a67fc640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 0x7fe3900a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3b2397740 1 Processor -- start 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3b2397740 1 -- start start 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3ad552640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 0x7fe3900a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.754+0000 7fe3a67fc640 1 -- 172.21.15.28:0/871372246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe39c0cfb40 con 0x7fe3a8104f80 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa8165900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa816d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557aa80ca040 con 0x557aa8165ed0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557aa80729e0 con 0x557aa816b040 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3b2397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557aa80c5ff0 con 0x557aa81654e0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa816d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53838/0 (socket says 172.21.15.28:53838) 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3acd51640 1 -- 172.21.15.28:0/3345894820 learned_addr learned my addr 172.21.15.28:0/3345894820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3ad552640 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa8165900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 -- 172.21.15.28:0/3345894820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 msgr2=0x557aa8165900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa8165900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3ad552640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 0x7fe3900a5480 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3a002c7b0 tx=0x7fe3a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 -- 172.21.15.28:0/3345894820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 msgr2=0x557aa816a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 -- 172.21.15.28:0/3345894820 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aa7dde210 con 0x557aa816b040 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3acd51640 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe3add53640 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa816d4d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe398009f00 tx=0x7fe398011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd69bbea5bbd275c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe38f7fe640 1 -- 172.21.15.28:0/3345894820 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398004d20 con 0x557aa816b040 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.755+0000 7fe38f7fe640 1 -- 172.21.15.28:0/3345894820 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe398004050 con 0x557aa816b040 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe38f7fe640 1 -- 172.21.15.28:0/3345894820 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398005350 con 0x557aa816b040 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 -- 172.21.15.28:0/3345894820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 msgr2=0x557aa816d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa816d4d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe398009f00 tx=0x7fe398011810 comp rx=0 tx=0).stop 2024-03-22T02:17:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 -- 172.21.15.28:0/3345894820 shutdown_connections 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa816d4d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa816a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 --2- 172.21.15.28:0/3345894820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa8165900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 -- 172.21.15.28:0/3345894820 >> 172.21.15.28:0/3345894820 conn(0x557aa8160a50 msgr2=0x557aa8160e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 -- 172.21.15.28:0/3345894820 shutdown_connections 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.756+0000 7fe3b2397740 1 -- 172.21.15.28:0/3345894820 wait complete. 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 -- start start 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 Processor -- start 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa816fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa816b040 0x557aa8176a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557aa816dff0 con 0x557aa81654e0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557aa8085db0 con 0x557aa8165ed0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557aa816e170 con 0x557aa816b040 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3b2397740 10 client.0 _ll_register_callbacks cb 0x557aa7ddda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3add53640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa816b040 0x557aa8176a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53854/0 (socket says 172.21.15.28:53854) 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa816fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 -- 172.21.15.28:0/4167706305 learned_addr learned my addr 172.21.15.28:0/4167706305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 -- 172.21.15.28:0/4167706305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa816b040 msgr2=0x557aa8176a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa816b040 0x557aa8176a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 -- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 msgr2=0x557aa816fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa816fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.757+0000 7fe3ad552640 1 -- 172.21.15.28:0/4167706305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aa7eaa620 con 0x557aa81654e0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3ad552640 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0036040 tx=0x7fe3a002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85001fbe8249fc89 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0054020 con 0x557aa81654e0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe3a00048c0 con 0x557aa81654e0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe398004050 con 0x557aa81654e0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe3a005f050 con 0x557aa81654e0 2024-03-22T02:17:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 10 client.16578 fetch_fsmap learned FSMap version 28 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557aa8179e70 con 0x557aa81654e0 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe398004050 con 0x557aa81654e0 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe3a004f030 con 0x557aa81654e0 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 10 client.16578 fetch_fsmap finished waiting for FSMap version 28 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 10 client.16578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 20 client.16578 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa817b600 0x557aa817da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 4 client.16578 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.758+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557aa80d7580 con 0x557aa817b600 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.759+0000 7fe3acd51640 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa817b600 0x557aa817da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.759+0000 7fe3acd51640 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa817b600 0x557aa817da90 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b195624cc67e949b server_cookie=12705832d19bec4f in_seq=0 out_seq=0 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.759+0000 7fe38cff9640 10 client.16578 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe38cff9640 1 -- 172.21.15.28:0/4167706305 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557aa80d7580 con 0x557aa817b600 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe38cff9640 10 client.16578 handle_command_reply: tid=0 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 client.16578 shutdown 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 msgr2=0x557aa816f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0036040 tx=0x7fe3a002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 shutdown_connections 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa817b600 0x557aa817da90 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa816b040 0x557aa8176a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe38cff9640 0 client.16578 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa816fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 --2- 172.21.15.28:0/4167706305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa81654e0 0x557aa816f510 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 >> 172.21.15.28:0/4167706305 conn(0x557aa8160a50 msgr2=0x557aa815fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 shutdown_connections 2024-03-22T02:17:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 1 -- 172.21.15.28:0/4167706305 wait complete. 2024-03-22T02:17:16.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.760+0000 7fe3b2397740 20 client.16578 trim_cache size 0 max 16384 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 Processor -- start 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 -- start start 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa816f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa8176a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81ab020 con 0x557aa816b040 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81a88f0 con 0x557aa8165ed0 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3b2397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81a6740 con 0x557aa81654e0 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3ad552640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa816f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53866/0 (socket says 172.21.15.28:53866) 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 -- 172.21.15.28:0/294163287 learned_addr learned my addr 172.21.15.28:0/294163287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa8176a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 -- 172.21.15.28:0/294163287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 msgr2=0x557aa816f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa816f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 -- 172.21.15.28:0/294163287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 msgr2=0x557aa8176a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa8176a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 -- 172.21.15.28:0/294163287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557aa816a1b0 con 0x557aa816b040 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3ad552640 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa816f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3acd51640 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa8176a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.777+0000 7fe3add53640 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe398019000 tx=0x7fe398009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d21be5f502d9e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.784 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe38effd640 1 -- 172.21.15.28:0/294163287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39801e040 con 0x557aa816b040 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe38effd640 1 -- 172.21.15.28:0/294163287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe398029070 con 0x557aa816b040 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe38effd640 1 -- 172.21.15.28:0/294163287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398024940 con 0x557aa816b040 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 -- 172.21.15.28:0/294163287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 msgr2=0x557aa8166810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe398019000 tx=0x7fe398009c00 comp rx=0 tx=0).stop 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 -- 172.21.15.28:0/294163287 shutdown_connections 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa816b040 0x557aa8166810 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa8165ed0 0x557aa8176a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 --2- 172.21.15.28:0/294163287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa816f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 -- 172.21.15.28:0/294163287 >> 172.21.15.28:0/294163287 conn(0x557aa8160a50 msgr2=0x557aa816f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 -- 172.21.15.28:0/294163287 shutdown_connections 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.778+0000 7fe3b2397740 1 -- 172.21.15.28:0/294163287 wait complete. 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 -- start start 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 Processor -- start 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa81743b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa8175900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81a6db0 con 0x557aa8165ed0 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81a8220 con 0x557aa816b040 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a81a7480 con 0x557aa81654e0 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3b2397740 10 client.0 _ll_register_callbacks cb 0x7fe3a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3acd51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53880/0 (socket says 172.21.15.28:53880) 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3acd51640 1 -- 172.21.15.28:0/1589941885 learned_addr learned my addr 172.21.15.28:0/1589941885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3ad552640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa81743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.779+0000 7fe3add53640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa8175900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 -- 172.21.15.28:0/1589941885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 msgr2=0x557aa81743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa81743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 -- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 msgr2=0x557aa8175900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa8175900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 -- 172.21.15.28:0/1589941885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557aa8175ef0 con 0x557aa8165ed0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3acd51640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c6a0 tx=0x7fe39c09bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68aa5b696e071c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0a4870 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fe39c09a160 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe398029070 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe39c0984d0 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 10 client.16584 fetch_fsmap learned FSMap version 28 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557aa81e1d10 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe398029070 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 28) ==== 8350+0+0 (secure 0 0 0) 0x7fe39c0a44e0 con 0x557aa8165ed0 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 10 client.16584 fetch_fsmap finished waiting for FSMap version 28 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 10 client.16584 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 20 client.16584 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa81e2720 0x557aa81e4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 4 client.16584 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:17:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.780+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557aa80d18d0 con 0x557aa81e2720 2024-03-22T02:17:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.781+0000 7fe3ad552640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa81e2720 0x557aa81e4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.781+0000 7fe38cff9640 10 client.16584 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.781+0000 7fe3ad552640 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa81e2720 0x557aa81e4b40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a4bd1bdc640f2ad server_cookie=73c955217ec68c7 in_seq=0 out_seq=0 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.987+0000 7fe38cff9640 1 -- 172.21.15.28:0/1589941885 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557aa80d18d0 con 0x557aa81e2720 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.987+0000 7fe38cff9640 10 client.16584 handle_command_reply: tid=0 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.987+0000 7fe3b2397740 1 client.16584 shutdown 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 msgr2=0x557aa81748f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00c6a0 tx=0x7fe39c09bd60 comp rx=0 tx=0).stop 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 shutdown_connections 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557aa81e2720 0x557aa81e4b40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557aa816b040 0x557aa8175900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe38cff9640 0 client.16584 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557aa8165ed0 0x557aa81748f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 --2- 172.21.15.28:0/1589941885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557aa81654e0 0x557aa81743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 >> 172.21.15.28:0/1589941885 conn(0x557aa8160a50 msgr2=0x557aa815f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 shutdown_connections 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 1 -- 172.21.15.28:0/1589941885 wait complete. 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3b2397740 20 client.16584 trim_cache size 0 max 16384 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:17:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:17:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:17:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 msgr2=0x7fe3900a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 0x7fe3900a5480 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe3a002c7b0 tx=0x7fe3a0005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 msgr2=0x7fe3a81983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00e9f0 tx=0x7fe39c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 shutdown_connections 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe3900a2fc0 0x7fe3900a5480 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81078b0 0x7fe3a819f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a8104f80 0x7fe3a81983a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 --2- 172.21.15.28:0/871372246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe3a8102650 0x7fe3a8197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.988+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 >> 172.21.15.28:0/871372246 conn(0x7fe3a80fc040 msgr2=0x7fe3a8102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.989+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 shutdown_connections 2024-03-22T02:17:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:16.989+0000 7fe3af7b6640 1 -- 172.21.15.28:0/871372246 wait complete. 2024-03-22T02:17:17.411 INFO:tasks.workunit.client.0.smithi028.stdout: 1 194545 8.96 MB/sec warmup 119 sec latency 129.615 ms 2024-03-22T02:17:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:18 smithi120 ceph-mon[36803]: pgmap v469: 129 pgs: 129 active+clean; 47 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-03-22T02:17:18.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:18 smithi083 ceph-mon[36340]: pgmap v469: 129 pgs: 129 active+clean; 47 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-03-22T02:17:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:18 smithi028 ceph-mon[31916]: pgmap v469: 129 pgs: 129 active+clean; 47 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-03-22T02:17:19.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 199060 16.98 MB/sec execute 1 sec latency 27.711 ms 2024-03-22T02:17:20.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:20 smithi028 ceph-mon[31916]: pgmap v470: 129 pgs: 129 active+clean; 39 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 467 op/s 2024-03-22T02:17:20.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:20 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:17:20.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 201611 14.77 MB/sec execute 2 sec latency 42.425 ms 2024-03-22T02:17:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:20 smithi120 ceph-mon[36803]: pgmap v470: 129 pgs: 129 active+clean; 39 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 467 op/s 2024-03-22T02:17:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:20 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:17:20.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:20 smithi083 ceph-mon[36340]: pgmap v470: 129 pgs: 129 active+clean; 39 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 467 op/s 2024-03-22T02:17:20.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:20 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:17:21.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 203406 12.74 MB/sec execute 3 sec latency 28.841 ms 2024-03-22T02:17:21.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:21 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:17:21.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:21 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:17:21.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:21 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:17:22.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 205941 13.73 MB/sec execute 4 sec latency 70.530 ms 2024-03-22T02:17:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:22 smithi120 ceph-mon[36803]: pgmap v471: 129 pgs: 129 active+clean; 41 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 392 op/s 2024-03-22T02:17:22.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:22 smithi083 ceph-mon[36340]: pgmap v471: 129 pgs: 129 active+clean; 41 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 392 op/s 2024-03-22T02:17:22.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:22 smithi028 ceph-mon[31916]: pgmap v471: 129 pgs: 129 active+clean; 41 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 392 op/s 2024-03-22T02:17:23.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 207619 12.13 MB/sec execute 5 sec latency 90.264 ms 2024-03-22T02:17:24.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 209877 12.39 MB/sec execute 6 sec latency 16.289 ms 2024-03-22T02:17:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:24 smithi120 ceph-mon[36803]: pgmap v472: 129 pgs: 129 active+clean; 40 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:17:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:24 smithi083 ceph-mon[36340]: pgmap v472: 129 pgs: 129 active+clean; 40 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:17:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:24 smithi028 ceph-mon[31916]: pgmap v472: 129 pgs: 129 active+clean; 40 MiB data, 715 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:17:25.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 212253 12.41 MB/sec execute 7 sec latency 44.663 ms 2024-03-22T02:17:26.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 213891 11.93 MB/sec execute 8 sec latency 29.056 ms 2024-03-22T02:17:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:26 smithi120 ceph-mon[36803]: pgmap v473: 129 pgs: 129 active+clean; 48 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 484 op/s 2024-03-22T02:17:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:26 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:26 smithi083 ceph-mon[36340]: pgmap v473: 129 pgs: 129 active+clean; 48 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 484 op/s 2024-03-22T02:17:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:26 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:26 smithi028 ceph-mon[31916]: pgmap v473: 129 pgs: 129 active+clean; 48 MiB data, 750 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 484 op/s 2024-03-22T02:17:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:26 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:17:27.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 216565 12.37 MB/sec execute 9 sec latency 58.297 ms 2024-03-22T02:17:28.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 218337 11.81 MB/sec execute 10 sec latency 68.490 ms 2024-03-22T02:17:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:28 smithi120 ceph-mon[36803]: pgmap v474: 129 pgs: 129 active+clean; 48 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 405 op/s 2024-03-22T02:17:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:28 smithi083 ceph-mon[36340]: pgmap v474: 129 pgs: 129 active+clean; 48 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 405 op/s 2024-03-22T02:17:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:28 smithi028 ceph-mon[31916]: pgmap v474: 129 pgs: 129 active+clean; 48 MiB data, 763 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 405 op/s 2024-03-22T02:17:29.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 220568 11.98 MB/sec execute 11 sec latency 12.112 ms 2024-03-22T02:17:30.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/1077167913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac101ca0 msgr2=0x7fc4ac102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/1077167913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac102120 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fc49c009900 tx=0x7fc49c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/1077167913 shutdown_connections 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/1077167913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac102660 0x7fc4ac108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/1077167913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac102120 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/1077167913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 0x7fc4ac100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/1077167913 >> 172.21.15.28:0/1077167913 conn(0x7fc4ac0fc020 msgr2=0x7fc4ac0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.079+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/1077167913 shutdown_connections 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/1077167913 wait complete. 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 Processor -- start 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 -- start start 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 0x7fc4ac197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac1983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10c730 con 0x7fc4ac102660 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10c5b0 con 0x7fc4ac101ca0 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b3fc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac10c8b0 con 0x7fc4ac100ab0 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55380/0 (socket says 172.21.15.28:55380) 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b2566640 1 -- 172.21.15.28:0/868578324 learned_addr learned my addr 172.21.15.28:0/868578324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 0x7fc4ac197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.080+0000 7fc4b1564640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac1983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 -- 172.21.15.28:0/868578324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 msgr2=0x7fc4ac197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 0x7fc4ac197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 -- 172.21.15.28:0/868578324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac101ca0 msgr2=0x7fc4ac1983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac1983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 -- 172.21.15.28:0/868578324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac19fa80 con 0x7fc4ac102660 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b2566640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400eb10 tx=0x7fc4a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4a2ffd640 1 -- 172.21.15.28:0/868578324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4004b20 con 0x7fc4ac102660 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4a2ffd640 1 -- 172.21.15.28:0/868578324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a4004cc0 con 0x7fc4ac102660 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ac19fd70 con 0x7fc4ac102660 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4a2ffd640 1 -- 172.21.15.28:0/868578324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4010760 con 0x7fc4ac102660 2024-03-22T02:17:30.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.081+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ac1a01e0 con 0x7fc4ac102660 2024-03-22T02:17:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.083+0000 7fc4a2ffd640 1 -- 172.21.15.28:0/868578324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc4a4005020 con 0x7fc4ac102660 2024-03-22T02:17:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.083+0000 7fc4b6ba8740 1 Processor -- start 2024-03-22T02:17:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 -- start start 2024-03-22T02:17:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4a2ffd640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 0x7fc4980a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014722e0 0x55e001476de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e001477450 0x55e0014798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e0013d2040 con 0x55e0014722e0 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e00137a9e0 con 0x55e001477450 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e0013cdff0 con 0x55e0014718f0 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4a2ffd640 1 -- 172.21.15.28:0/868578324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc4a40cfd50 con 0x7fc4ac102660 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1564640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014722e0 0x55e001476de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1564640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 0x7fc4980a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:50638/0 (socket says 172.21.15.28:50638) 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 -- 172.21.15.28:0/4278953702 learned_addr learned my addr 172.21.15.28:0/4278953702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:30.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b2566640 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e001477450 0x55e0014798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 -- 172.21.15.28:0/4278953702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e001477450 msgr2=0x55e0014798e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e001477450 0x55e0014798e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 -- 172.21.15.28:0/4278953702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014722e0 msgr2=0x55e001476de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014722e0 0x55e001476de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 -- 172.21.15.28:0/4278953702 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0010e6210 con 0x55e0014718f0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009ca0 tx=0x7fc4a8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=717704ebb3edab34 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.084+0000 7fc4b1564640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 0x7fc4980a5480 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc49c02c7b0 tx=0x7fc49c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc497fff640 1 -- 172.21.15.28:0/4278953702 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004220 con 0x55e0014718f0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc497fff640 1 -- 172.21.15.28:0/4278953702 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8009e50 con 0x55e0014718f0 2024-03-22T02:17:30.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc497fff640 1 -- 172.21.15.28:0/4278953702 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8019760 con 0x55e0014718f0 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/4278953702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 msgr2=0x55e001471d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009ca0 tx=0x7fc4a8011810 comp rx=0 tx=0).stop 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/4278953702 shutdown_connections 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e001477450 0x55e0014798e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014722e0 0x55e001476de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/4278953702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e001471d10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.085+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/4278953702 >> 172.21.15.28:0/4278953702 conn(0x55e001468a50 msgr2=0x55e001468e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/4278953702 shutdown_connections 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/4278953702 wait complete. 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- start start 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 Processor -- start 2024-03-22T02:17:30.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e00147bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e001477450 0x55e001482d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e00147a400 con 0x55e0014718f0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e00138ddb0 con 0x55e0014722e0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e00147a580 con 0x55e001477450 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b6ba8740 10 client.0 _ll_register_callbacks cb 0x55e0010e5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55390/0 (socket says 172.21.15.28:55390) 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b1d65640 1 -- 172.21.15.28:0/1315328619 learned_addr learned my addr 172.21.15.28:0/1315328619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b1564640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e00147bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.086+0000 7fc4b2566640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e001477450 0x55e001482d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 -- 172.21.15.28:0/1315328619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e001477450 msgr2=0x55e001482d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e001477450 0x55e001482d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 -- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 msgr2=0x55e00147bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e00147bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 -- 172.21.15.28:0/1315328619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0011b2620 con 0x55e0014718f0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1564640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e00147bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8011d10 tx=0x7fc4a80052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9aa284efded287 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4957fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8028070 con 0x55e0014718f0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc4a80035f0 con 0x55e0014718f0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a8009e50 con 0x55e0014718f0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc4a8034050 con 0x55e0014718f0 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b6ba8740 10 client.16596 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:30.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e001486100 con 0x55e0014718f0 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.087+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8009e50 con 0x55e0014718f0 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc4a80194a0 con 0x55e0014718f0 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 10 client.16596 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 10 client.16596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 20 client.16596 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e001487660 0x55e001489a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 4 client.16596 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0013df580 con 0x55e001487660 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b1564640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e001487660 0x55e001489a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4b1564640 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e001487660 0x55e001489a90 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4897946f8cc747ca server_cookie=a1f534ef4afcddf in_seq=0 out_seq=0 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.088+0000 7fc4957fa640 10 client.16596 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4957fa640 1 -- 172.21.15.28:0/1315328619 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e0013df580 con 0x55e001487660 2024-03-22T02:17:30.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4957fa640 10 client.16596 handle_command_reply: tid=0 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 client.16596 shutdown 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 msgr2=0x55e00147b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8011d10 tx=0x7fc4a80052f0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 shutdown_connections 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e001487660 0x55e001489a90 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e001477450 0x55e001482d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e00147bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/1315328619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e0014718f0 0x55e00147b810 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4957fa640 0 client.16596 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 >> 172.21.15.28:0/1315328619 conn(0x55e001468a50 msgr2=0x55e001467c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 shutdown_connections 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/1315328619 wait complete. 2024-03-22T02:17:30.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.089+0000 7fc4b6ba8740 20 client.16596 trim_cache size 0 max 16384 2024-03-22T02:17:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.104+0000 7fc4b6ba8740 1 Processor -- start 2024-03-22T02:17:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.104+0000 7fc4b6ba8740 1 -- start start 2024-03-22T02:17:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00146c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e001482d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1aaee0 con 0x55e001477450 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1a87b0 con 0x55e0014722e0 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1a6600 con 0x55e0014718f0 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00146c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55394/0 (socket says 172.21.15.28:55394) 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 -- 172.21.15.28:0/373268090 learned_addr learned my addr 172.21.15.28:0/373268090 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b1564640 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e001482d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 -- 172.21.15.28:0/373268090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 msgr2=0x55e00146c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00146c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 -- 172.21.15.28:0/373268090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 msgr2=0x55e001482d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e001482d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 -- 172.21.15.28:0/373268090 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e0014726c0 con 0x55e001477450 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4b2566640 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40049d0 tx=0x7fc4a40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ba5d59b9a67132 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4977fe640 1 -- 172.21.15.28:0/373268090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a409c770 con 0x55e001477450 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4977fe640 1 -- 172.21.15.28:0/373268090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a409c070 con 0x55e001477450 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.105+0000 7fc4977fe640 1 -- 172.21.15.28:0/373268090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40821b0 con 0x55e001477450 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/373268090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 msgr2=0x55e00147ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fc4a40049d0 tx=0x7fc4a40058d0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/373268090 shutdown_connections 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e00147ae30 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e001482d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/373268090 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00146c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/373268090 >> 172.21.15.28:0/373268090 conn(0x55e001468a50 msgr2=0x55e001467c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/373268090 shutdown_connections 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/373268090 wait complete. 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.106+0000 7fc4b6ba8740 1 -- start start 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 Processor -- start 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00147ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1a6c70 con 0x55e001477450 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1a80e0 con 0x55e0014722e0 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ac1a7340 con 0x55e0014718f0 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b6ba8740 10 client.0 _ll_register_callbacks cb 0x7fc4a8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:30.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00147ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:39332/0 (socket says 172.21.15.28:39332) 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1d65640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00147ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:50672/0 (socket says 172.21.15.28:50672) 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b2566640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55406/0 (socket says 172.21.15.28:55406) 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 -- 172.21.15.28:0/162123920 learned_addr learned my addr 172.21.15.28:0/162123920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 -- 172.21.15.28:0/162123920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 msgr2=0x55e00147ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00147ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 -- 172.21.15.28:0/162123920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 msgr2=0x55e0014814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b1564640 1 -- 172.21.15.28:0/162123920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e001481ab0 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.107+0000 7fc4b2566640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4b1564640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc49c02cfd0 tx=0x7fc49c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713b5ecc1cad875c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49c0542f0 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc49c054490 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4a409c070 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc49c065050 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4b6ba8740 10 client.26435 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e001482e60 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a409c070 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.108+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc49c0035f0 con 0x55e0014722e0 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 10 client.26435 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 10 client.26435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:30.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 20 client.26435 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e0014edfd0 0x55e0014f03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 4 client.26435 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e0013d98d0 con 0x55e0014edfd0 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e0014edfd0 0x55e0014f03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4b1d65640 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e0014edfd0 0x55e0014f03f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24407119f481a800 server_cookie=5e12359f82b585b2 in_seq=0 out_seq=0 2024-03-22T02:17:30.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.109+0000 7fc4957fa640 10 client.26435 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4957fa640 1 -- 172.21.15.28:0/162123920 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e0013d98d0 con 0x55e0014edfd0 2024-03-22T02:17:30.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4957fa640 10 client.26435 handle_command_reply: tid=0 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 client.26435 shutdown 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 msgr2=0x55e0014804b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc49c02cfd0 tx=0x7fc49c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 shutdown_connections 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e0014edfd0 0x55e0014f03f0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e001477450 0x55e0014814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e0014722e0 0x55e0014804b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4957fa640 0 client.26435 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 --2- 172.21.15.28:0/162123920 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e0014718f0 0x55e00147ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 >> 172.21.15.28:0/162123920 conn(0x55e001468a50 msgr2=0x55e001472a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 shutdown_connections 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.316+0000 7fc4b6ba8740 1 -- 172.21.15.28:0/162123920 wait complete. 2024-03-22T02:17:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.317+0000 7fc4b6ba8740 20 client.26435 trim_cache size 0 max 16384 2024-03-22T02:17:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:17:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:17:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:17:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 msgr2=0x7fc4980a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 0x7fc4980a5480 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc49c02c7b0 tx=0x7fc49c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 msgr2=0x7fc4ac19f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc4a400eb10 tx=0x7fc4a400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 shutdown_connections 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc4980a2fc0 0x7fc4980a5480 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4ac102660 0x7fc4ac19f410 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc4ac101ca0 0x7fc4ac1983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 --2- 172.21.15.28:0/868578324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4ac100ab0 0x7fc4ac197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 >> 172.21.15.28:0/868578324 conn(0x7fc4ac0fc020 msgr2=0x7fc4ac0fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 shutdown_connections 2024-03-22T02:17:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:30.318+0000 7fc4b3fc9640 1 -- 172.21.15.28:0/868578324 wait complete. 2024-03-22T02:17:30.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 222858 11.97 MB/sec execute 12 sec latency 134.795 ms 2024-03-22T02:17:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:30 smithi120 ceph-mon[36803]: pgmap v475: 129 pgs: 129 active+clean; 50 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:17:30.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:30 smithi028 ceph-mon[31916]: pgmap v475: 129 pgs: 129 active+clean; 50 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:17:30.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:30 smithi083 ceph-mon[36340]: pgmap v475: 129 pgs: 129 active+clean; 50 MiB data, 761 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:17:31.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 224683 11.77 MB/sec execute 13 sec latency 28.258 ms 2024-03-22T02:17:32.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 227426 12.14 MB/sec execute 14 sec latency 16.456 ms 2024-03-22T02:17:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:32 smithi120 ceph-mon[36803]: pgmap v476: 129 pgs: 129 active+clean; 53 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 335 op/s 2024-03-22T02:17:32.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:32 smithi083 ceph-mon[36340]: pgmap v476: 129 pgs: 129 active+clean; 53 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 335 op/s 2024-03-22T02:17:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:32 smithi028 ceph-mon[31916]: pgmap v476: 129 pgs: 129 active+clean; 53 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.9 MiB/s wr, 335 op/s 2024-03-22T02:17:33.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 229391 11.79 MB/sec execute 15 sec latency 54.692 ms 2024-03-22T02:17:34.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 231210 11.82 MB/sec execute 16 sec latency 52.482 ms 2024-03-22T02:17:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:34 smithi120 ceph-mon[36803]: pgmap v477: 129 pgs: 129 active+clean; 53 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-03-22T02:17:34.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:34 smithi083 ceph-mon[36340]: pgmap v477: 129 pgs: 129 active+clean; 53 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-03-22T02:17:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:34 smithi028 ceph-mon[31916]: pgmap v477: 129 pgs: 129 active+clean; 53 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-03-22T02:17:35.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 234160 12.15 MB/sec execute 17 sec latency 25.714 ms 2024-03-22T02:17:36.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 236447 11.95 MB/sec execute 18 sec latency 36.820 ms 2024-03-22T02:17:36.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:36 smithi120 ceph-mon[36803]: pgmap v478: 129 pgs: 129 active+clean; 46 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 435 op/s 2024-03-22T02:17:36.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:36 smithi083 ceph-mon[36340]: pgmap v478: 129 pgs: 129 active+clean; 46 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 435 op/s 2024-03-22T02:17:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:36 smithi028 ceph-mon[31916]: pgmap v478: 129 pgs: 129 active+clean; 46 MiB data, 801 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.6 MiB/s wr, 435 op/s 2024-03-22T02:17:37.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 238373 11.98 MB/sec execute 19 sec latency 35.368 ms 2024-03-22T02:17:38.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 240905 12.04 MB/sec execute 20 sec latency 49.728 ms 2024-03-22T02:17:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:38 smithi120 ceph-mon[36803]: pgmap v479: 129 pgs: 129 active+clean; 42 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 373 op/s 2024-03-22T02:17:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:38 smithi083 ceph-mon[36340]: pgmap v479: 129 pgs: 129 active+clean; 42 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 373 op/s 2024-03-22T02:17:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:38 smithi028 ceph-mon[31916]: pgmap v479: 129 pgs: 129 active+clean; 42 MiB data, 822 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.7 MiB/s wr, 373 op/s 2024-03-22T02:17:39.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 242814 11.98 MB/sec execute 21 sec latency 42.647 ms 2024-03-22T02:17:40.412 INFO:tasks.workunit.client.0.smithi028.stdout: 1 245263 12.07 MB/sec execute 22 sec latency 14.759 ms 2024-03-22T02:17:40.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:40 smithi120 ceph-mon[36803]: pgmap v480: 129 pgs: 129 active+clean; 46 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-03-22T02:17:40.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:40 smithi083 ceph-mon[36340]: pgmap v480: 129 pgs: 129 active+clean; 46 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-03-22T02:17:40.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:40 smithi028 ceph-mon[31916]: pgmap v480: 129 pgs: 129 active+clean; 46 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 3.4 MiB/s wr, 450 op/s 2024-03-22T02:17:41.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 248136 12.18 MB/sec execute 23 sec latency 26.525 ms 2024-03-22T02:17:42.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 249730 12.07 MB/sec execute 24 sec latency 37.131 ms 2024-03-22T02:17:42.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:42 smithi120 ceph-mon[36803]: pgmap v481: 129 pgs: 129 active+clean; 46 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 423 op/s 2024-03-22T02:17:42.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:42 smithi083 ceph-mon[36340]: pgmap v481: 129 pgs: 129 active+clean; 46 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 423 op/s 2024-03-22T02:17:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:42 smithi028 ceph-mon[31916]: pgmap v481: 129 pgs: 129 active+clean; 46 MiB data, 828 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 423 op/s 2024-03-22T02:17:42.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:17:42.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:17:42.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.928+0000 7fc59136b640 1 -- 172.21.15.28:0/3235421873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c100a90 msgr2=0x7fc58c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.928+0000 7fc59136b640 1 --2- 172.21.15.28:0/3235421873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c100a90 0x7fc58c100eb0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009900 tx=0x7fc57c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:42.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- 172.21.15.28:0/3235421873 shutdown_connections 2024-03-22T02:17:42.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- 172.21.15.28:0/3235421873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c102640 0x7fc58c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- 172.21.15.28:0/3235421873 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c101c80 0x7fc58c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- 172.21.15.28:0/3235421873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c100a90 0x7fc58c100eb0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- 172.21.15.28:0/3235421873 >> 172.21.15.28:0/3235421873 conn(0x7fc58c0fc020 msgr2=0x7fc58c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- 172.21.15.28:0/3235421873 shutdown_connections 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- 172.21.15.28:0/3235421873 wait complete. 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 Processor -- start 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- start start 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c100a90 0x7fc58c104a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c102640 0x7fc58c1035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10c710 con 0x7fc58c101c80 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10c590 con 0x7fc58c102640 2024-03-22T02:17:42.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc59136b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c10c890 con 0x7fc58c100a90 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53000/0 (socket says 172.21.15.28:53000) 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc583fff640 1 -- 172.21.15.28:0/321279994 learned_addr learned my addr 172.21.15.28:0/321279994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc58affd640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c100a90 0x7fc58c104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.929+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c102640 0x7fc58c1035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 -- 172.21.15.28:0/321279994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c100a90 msgr2=0x7fc58c104a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c100a90 0x7fc58c104a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 -- 172.21.15.28:0/321279994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c102640 msgr2=0x7fc58c1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c102640 0x7fc58c1035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 -- 172.21.15.28:0/321279994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc58c103e30 con 0x7fc58c101c80 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc583fff640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc57400e9f0 tx=0x7fc57400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc588ff9640 1 -- 172.21.15.28:0/321279994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57400cd80 con 0x7fc58c101c80 2024-03-22T02:17:42.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc588ff9640 1 -- 172.21.15.28:0/321279994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc574010910 con 0x7fc58c101c80 2024-03-22T02:17:42.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc58c1a1fe0 con 0x7fc58c101c80 2024-03-22T02:17:42.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc588ff9640 1 -- 172.21.15.28:0/321279994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574018950 con 0x7fc58c101c80 2024-03-22T02:17:42.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.930+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc58c1a2440 con 0x7fc58c101c80 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.932+0000 7fc588ff9640 1 -- 172.21.15.28:0/321279994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc574010ac0 con 0x7fc58c101c80 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 Processor -- start 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 -- start start 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6762920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b6765420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5613b66c28c0 con 0x5613b675de20 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5613b666ac20 con 0x5613b675d430 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc59402a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5613b66be870 con 0x5613b6762f90 2024-03-22T02:17:42.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc588ff9640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 0x7fc5780a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc583fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b6765420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37538/0 (socket says 172.21.15.28:37538) 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc58b7fe640 1 -- 172.21.15.28:0/4279763354 learned_addr learned my addr 172.21.15.28:0/4279763354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc58affd640 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6762920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.933+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 0x7fc5780a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc588ff9640 1 -- 172.21.15.28:0/321279994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc5740cfb20 con 0x7fc58c101c80 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 -- 172.21.15.28:0/4279763354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 msgr2=0x5613b6765420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b6765420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 -- 172.21.15.28:0/4279763354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 msgr2=0x5613b6762920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6762920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 -- 172.21.15.28:0/4279763354 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b668f150 con 0x5613b675d430 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc584011e00 tx=0x7fc58400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb670e71e86edc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 0x7fc5780a53e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc58400fc90 tx=0x7fc584018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc5817fa640 1 -- 172.21.15.28:0/4279763354 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584022070 con 0x5613b675d430 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc5817fa640 1 -- 172.21.15.28:0/4279763354 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58400ed10 con 0x5613b675d430 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.934+0000 7fc5817fa640 1 -- 172.21.15.28:0/4279763354 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5840058b0 con 0x5613b675d430 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- 172.21.15.28:0/4279763354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 msgr2=0x5613b675d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc584011e00 tx=0x7fc58400f790 comp rx=0 tx=0).stop 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- 172.21.15.28:0/4279763354 shutdown_connections 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b6765420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6762920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 --2- 172.21.15.28:0/4279763354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b675d850 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- 172.21.15.28:0/4279763354 >> 172.21.15.28:0/4279763354 conn(0x5613b67589a0 msgr2=0x5613b6758da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- 172.21.15.28:0/4279763354 shutdown_connections 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- 172.21.15.28:0/4279763354 wait complete. 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.935+0000 7fc59402a740 1 -- start start 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 Processor -- start 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b6767940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b676eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5613b6765ee0 con 0x5613b675de20 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5613b667e130 con 0x5613b675d430 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5613b6766060 con 0x5613b6762f90 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc59402a740 10 client.0 _ll_register_callbacks cb 0x5613b667db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53026/0 (socket says 172.21.15.28:53026) 2024-03-22T02:17:42.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc58affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b6767940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 -- 172.21.15.28:0/1564288923 learned_addr learned my addr 172.21.15.28:0/1564288923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b676eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 -- 172.21.15.28:0/1564288923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 msgr2=0x5613b676eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b676eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 -- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 msgr2=0x5613b6767940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b6767940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 -- 172.21.15.28:0/1564288923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b64a07d0 con 0x5613b675de20 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc583fff640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc5740cd210 tx=0x7fc574014a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127039e542118168 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc5727fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.936+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57409e070 con 0x5613b675de20 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc5740adb40 con 0x5613b675de20 2024-03-22T02:17:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc58400ed10 con 0x5613b675de20 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc5740b6070 con 0x5613b675de20 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 10 client.16614 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5613b63d63f0 con 0x5613b675de20 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58400ed10 con 0x5613b675de20 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc5740babb0 con 0x5613b675de20 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 10 client.16614 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 10 client.16614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 20 client.16614 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b6773750 0x5613b6775bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 4 client.16614 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613b66cfe00 con 0x5613b6773750 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.937+0000 7fc58affd640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b6773750 0x5613b6775bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.938+0000 7fc5727fc640 10 client.16614 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.938+0000 7fc58affd640 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b6773750 0x5613b6775bd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a430e82afcf3d30a server_cookie=e6ff1d28d1ade649 in_seq=0 out_seq=0 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc5727fc640 1 -- 172.21.15.28:0/1564288923 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5613b66cfe00 con 0x5613b6773750 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc5727fc640 10 client.16614 handle_command_reply: tid=0 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 client.16614 shutdown 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 msgr2=0x5613b6767e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc5740cd210 tx=0x7fc574014a50 comp rx=0 tx=0).stop 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 shutdown_connections 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b6773750 0x5613b6775bd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b6762f90 0x5613b676eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b6767e80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 --2- 172.21.15.28:0/1564288923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675d430 0x5613b6767940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc5727fc640 0 client.16614 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 >> 172.21.15.28:0/1564288923 conn(0x5613b67589a0 msgr2=0x5613b6757c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 shutdown_connections 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 1 -- 172.21.15.28:0/1564288923 wait complete. 2024-03-22T02:17:42.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.940+0000 7fc59402a740 20 client.16614 trim_cache size 0 max 16384 2024-03-22T02:17:42.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 Processor -- start 2024-03-22T02:17:42.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 -- start start 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b675c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b6762f90 0x5613b677a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1af550 con 0x5613b675de20 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ace20 con 0x5613b6762f90 2024-03-22T02:17:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc59402a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1aac70 con 0x5613b675d430 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc583fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53036/0 (socket says 172.21.15.28:53036) 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc583fff640 1 -- 172.21.15.28:0/1608119826 learned_addr learned my addr 172.21.15.28:0/1608119826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b6762f90 0x5613b677a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.959+0000 7fc58affd640 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b675c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 -- 172.21.15.28:0/1608119826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 msgr2=0x5613b675c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b675c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 -- 172.21.15.28:0/1608119826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b6762f90 msgr2=0x5613b677a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b6762f90 0x5613b677a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 -- 172.21.15.28:0/1608119826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b63960f0 con 0x5613b675de20 2024-03-22T02:17:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc583fff640 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc5740988c0 tx=0x7fc57409a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4068a11ea8b276fb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc580ff9640 1 -- 172.21.15.28:0/1608119826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc57409e040 con 0x5613b675de20 2024-03-22T02:17:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc580ff9640 1 -- 172.21.15.28:0/1608119826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57409bef0 con 0x5613b675de20 2024-03-22T02:17:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.960+0000 7fc580ff9640 1 -- 172.21.15.28:0/1608119826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc574026040 con 0x5613b675de20 2024-03-22T02:17:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- 172.21.15.28:0/1608119826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 msgr2=0x5613b676ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fc5740988c0 tx=0x7fc57409a400 comp rx=0 tx=0).stop 2024-03-22T02:17:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- 172.21.15.28:0/1608119826 shutdown_connections 2024-03-22T02:17:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b6762f90 0x5613b677a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b675de20 0x5613b676ee80 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- 172.21.15.28:0/1608119826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b675c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- 172.21.15.28:0/1608119826 >> 172.21.15.28:0/1608119826 conn(0x5613b67589a0 msgr2=0x5613b675bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- 172.21.15.28:0/1608119826 shutdown_connections 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- 172.21.15.28:0/1608119826 wait complete. 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- start start 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 Processor -- start 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b676c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675de20 0x5613b676cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ab2e0 con 0x5613b6762f90 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ac750 con 0x5613b675de20 2024-03-22T02:17:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc58c1ab9b0 con 0x5613b675d430 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.961+0000 7fc59402a740 10 client.0 _ll_register_callbacks cb 0x7fc5740bc0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b676c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53050/0 (socket says 172.21.15.28:53050) 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc583fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675de20 0x5613b676cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 -- 172.21.15.28:0/1973260320 learned_addr learned my addr 172.21.15.28:0/1973260320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 -- 172.21.15.28:0/1973260320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 msgr2=0x5613b676c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b676c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 -- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675de20 msgr2=0x5613b676cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675de20 0x5613b676cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 -- 172.21.15.28:0/1973260320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b676e3d0 con 0x5613b6762f90 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc58b7fe640 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x5613b675e680 tx=0x7fc5840116c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfa4abb6a984d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc5727fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc584022040 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc584003b70 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc57409bef0 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc584037050 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc59402a740 10 client.16620 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.962+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5613b67d9e10 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc57409bef0 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc5840314b0 con 0x5613b6762f90 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 10 client.16620 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 10 client.16620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 20 client.16620 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b67da6a0 0x5613b67dcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 4 client.16620 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5613b66ca150 con 0x5613b67da6a0 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc58affd640 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b67da6a0 0x5613b67dcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc5727fc640 10 client.16620 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.963+0000 7fc58affd640 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b67da6a0 0x5613b67dcac0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=119332c567881145 server_cookie=80477287a444c0cb in_seq=0 out_seq=0 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc5727fc640 1 -- 172.21.15.28:0/1973260320 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5613b66ca150 con 0x5613b67da6a0 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc5727fc640 10 client.16620 handle_command_reply: tid=0 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 client.16620 shutdown 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 msgr2=0x5613b676dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x5613b675e680 tx=0x7fc5840116c0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 shutdown_connections 2024-03-22T02:17:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5613b67da6a0 0x5613b67dcac0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5613b6762f90 0x5613b676dde0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5613b675de20 0x5613b676cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 --2- 172.21.15.28:0/1973260320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5613b675d430 0x5613b676c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc5727fc640 0 client.16620 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 >> 172.21.15.28:0/1973260320 conn(0x5613b67589a0 msgr2=0x5613b675c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 shutdown_connections 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 1 -- 172.21.15.28:0/1973260320 wait complete. 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59402a740 20 client.16620 trim_cache size 0 max 16384 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:17:42.972 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:17:42.973 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 msgr2=0x7fc5780a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 0x7fc5780a53e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc58400fc90 tx=0x7fc584018000 comp rx=0 tx=0).stop 2024-03-22T02:17:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 msgr2=0x7fc58c103060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:42.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc57400e9f0 tx=0x7fc57400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 shutdown_connections 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5780a2f20 0x7fc5780a53e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc58c102640 0x7fc58c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58c101c80 0x7fc58c103060 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 --2- 172.21.15.28:0/321279994 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc58c100a90 0x7fc58c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.964+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 >> 172.21.15.28:0/321279994 conn(0x7fc58c0fc020 msgr2=0x7fc58c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.965+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 shutdown_connections 2024-03-22T02:17:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:42.965+0000 7fc59136b640 1 -- 172.21.15.28:0/321279994 wait complete. 2024-03-22T02:17:42.976 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:17:42.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- 172.21.15.28:0/3094144214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90100ad0 msgr2=0x7fbc90100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 --2- 172.21.15.28:0/3094144214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90100ad0 0x7fbc90100ef0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbc80009900 tx=0x7fbc8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- 172.21.15.28:0/3094144214 shutdown_connections 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 --2- 172.21.15.28:0/3094144214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90102680 0x7fbc90108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 --2- 172.21.15.28:0/3094144214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90101cc0 0x7fbc90102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 --2- 172.21.15.28:0/3094144214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90100ad0 0x7fbc90100ef0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- 172.21.15.28:0/3094144214 >> 172.21.15.28:0/3094144214 conn(0x7fbc900fc040 msgr2=0x7fbc900fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- 172.21.15.28:0/3094144214 shutdown_connections 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- 172.21.15.28:0/3094144214 wait complete. 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 Processor -- start 2024-03-22T02:17:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.082+0000 7fbc96302640 1 -- start start 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90100ad0 0x7fbc90197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90102680 0x7fbc9019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010d870 con 0x7fbc90101cc0 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010d6f0 con 0x7fbc90102680 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9010d9f0 con 0x7fbc90100ad0 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53074/0 (socket says 172.21.15.28:53074) 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90100ad0 0x7fbc90197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 -- 172.21.15.28:0/1010502466 learned_addr learned my addr 172.21.15.28:0/1010502466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc9489f640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90102680 0x7fbc9019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 -- 172.21.15.28:0/1010502466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90100ad0 msgr2=0x7fbc90197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90100ad0 0x7fbc90197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 -- 172.21.15.28:0/1010502466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90102680 msgr2=0x7fbc9019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90102680 0x7fbc9019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 -- 172.21.15.28:0/1010502466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9019faf0 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8f7fe640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400e9f0 tx=0x7fbc8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8400cd80 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.083+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9019fde0 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.084+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc84010910 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.084+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc840188f0 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.084+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc901a02f0 con 0x7fbc90101cc0 2024-03-22T02:17:43.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.084+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc90107df0 con 0x7fbc90101cc0 2024-03-22T02:17:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.085+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fbc84010ac0 con 0x7fbc90101cc0 2024-03-22T02:17:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.086+0000 7fbc8d7fa640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 0x7fbc780a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.086+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fbc840cfb90 con 0x7fbc90101cc0 2024-03-22T02:17:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.086+0000 7fbc8ffff640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 0x7fbc780a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.087+0000 7fbc8ffff640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 0x7fbc780a53e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbc80000c00 tx=0x7fbc80005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.091+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc8409bf60 con 0x7fbc90101cc0 2024-03-22T02:17:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.331+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc9002d3b0 con 0x7fbc90101cc0 2024-03-22T02:17:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.332+0000 7fbc8d7fa640 1 -- 172.21.15.28:0/1010502466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11866 (secure 0 0 0) 0x7fbc840a0000 con 0x7fbc90101cc0 2024-03-22T02:17:43.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:17:43.333 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:17:24.693718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:17:43.333 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 msgr2=0x7fbc780a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 0x7fbc780a53e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fbc80000c00 tx=0x7fbc80005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 msgr2=0x7fbc90198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400e9f0 tx=0x7fbc8400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 shutdown_connections 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbc780a2f20 0x7fbc780a53e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbc90102680 0x7fbc9019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc90101cc0 0x7fbc90198410 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 --2- 172.21.15.28:0/1010502466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc90100ad0 0x7fbc90197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 >> 172.21.15.28:0/1010502466 conn(0x7fbc900fc040 msgr2=0x7fbc90104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.334+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 shutdown_connections 2024-03-22T02:17:43.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.335+0000 7fbc96302640 1 -- 172.21.15.28:0/1010502466 wait complete. 2024-03-22T02:17:43.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:17:43.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 252138 12.14 MB/sec execute 25 sec latency 49.971 ms 2024-03-22T02:17:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.453+0000 7f943e86f640 1 -- 172.21.15.28:0/3651137360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943810e1c0 msgr2=0x7f94381105b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.453+0000 7f943e86f640 1 --2- 172.21.15.28:0/3651137360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943810e1c0 0x7f94381105b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9430009cd0 tx=0x7f943002c0e0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 -- 172.21.15.28:0/3651137360 shutdown_connections 2024-03-22T02:17:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 --2- 172.21.15.28:0/3651137360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f943810e1c0 0x7f94381105b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 --2- 172.21.15.28:0/3651137360 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94380745f0 0x7f9438074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 --2- 172.21.15.28:0/3651137360 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9438075bd0 0x7f9438074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 -- 172.21.15.28:0/3651137360 >> 172.21.15.28:0/3651137360 conn(0x7f94380fd160 msgr2=0x7f94380ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 -- 172.21.15.28:0/3651137360 shutdown_connections 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.455+0000 7f943e86f640 1 -- 172.21.15.28:0/3651137360 wait complete. 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 Processor -- start 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 -- start start 2024-03-22T02:17:43.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 0x7f943819e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9438075bd0 0x7f943819e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9438113ed0 con 0x7f9438075bd0 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9438113d50 con 0x7f94380745f0 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.456+0000 7f943e86f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9438114050 con 0x7f943810e1c0 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f9437fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 0x7f943819e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48766/0 (socket says 172.21.15.28:48766) 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f9437fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 0x7f943819e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37602/0 (socket says 172.21.15.28:37602) 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 -- 172.21.15.28:0/3704559202 learned_addr learned my addr 172.21.15.28:0/3704559202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 -- 172.21.15.28:0/3704559202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 msgr2=0x7f943819e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 0x7f943819e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 -- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9438075bd0 msgr2=0x7f943819e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9438075bd0 0x7f943819e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 -- 172.21.15.28:0/3704559202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94381a5f50 con 0x7f943810e1c0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.457+0000 7f943ce0c640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9430009bf0 tx=0x7f9430035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.458+0000 7f9435ffb640 1 -- 172.21.15.28:0/3704559202 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430039040 con 0x7f943810e1c0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.458+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94381a61e0 con 0x7f943810e1c0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.458+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94381a6720 con 0x7f943810e1c0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.458+0000 7f9435ffb640 1 -- 172.21.15.28:0/3704559202 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9430044070 con 0x7f943810e1c0 2024-03-22T02:17:43.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.458+0000 7f9435ffb640 1 -- 172.21.15.28:0/3704559202 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9430004520 con 0x7f943810e1c0 2024-03-22T02:17:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.461+0000 7f9441458740 1 Processor -- start 2024-03-22T02:17:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9435ffb640 1 -- 172.21.15.28:0/3704559202 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f94300046c0 con 0x7f943810e1c0 2024-03-22T02:17:43.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 -- start start 2024-03-22T02:17:43.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cef2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cefa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c82ce57040 con 0x55c82cef24e0 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c82cdff9e0 con 0x55c82cef8040 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.462+0000 7f9441458740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c82ce52ff0 con 0x55c82cef2ed0 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48774/0 (socket says 172.21.15.28:48774) 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 -- 172.21.15.28:0/2528159274 learned_addr learned my addr 172.21.15.28:0/2528159274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f942ffff640 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cef2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f942ffff640 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cefa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9435ffb640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 0x7f94240a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 -- 172.21.15.28:0/2528159274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 msgr2=0x55c82cefa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cefa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 -- 172.21.15.28:0/2528159274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 msgr2=0x55c82cef2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cef2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9437fff640 1 -- 172.21.15.28:0/2528159274 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c82cb6b210 con 0x55c82cef2ed0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f943ce0c640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 0x7f94240a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.463+0000 7f9435ffb640 1 -- 172.21.15.28:0/3704559202 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9430003d10 con 0x7f943810e1c0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.464+0000 7f943ce0c640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 0x7f94240a56d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f94300bd850 tx=0x7f94300b2fa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.464+0000 7f9437fff640 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f942800bd10 tx=0x7f94280107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89448dd1426549c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.465+0000 7f942e7fc640 1 -- 172.21.15.28:0/2528159274 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428004030 con 0x55c82cef2ed0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.465+0000 7f942e7fc640 1 -- 172.21.15.28:0/2528159274 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94280041d0 con 0x55c82cef2ed0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.465+0000 7f942e7fc640 1 -- 172.21.15.28:0/2528159274 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9428011800 con 0x55c82cef2ed0 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 -- 172.21.15.28:0/2528159274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 msgr2=0x55c82cef79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f942800bd10 tx=0x7f94280107a0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.469 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 -- 172.21.15.28:0/2528159274 shutdown_connections 2024-03-22T02:17:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cefa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cef79d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 --2- 172.21.15.28:0/2528159274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cef2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.466+0000 7f9441458740 1 -- 172.21.15.28:0/2528159274 >> 172.21.15.28:0/2528159274 conn(0x55c82ceeda50 msgr2=0x55c82ceede50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.469+0000 7f9441458740 1 -- 172.21.15.28:0/2528159274 shutdown_connections 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.469+0000 7f9441458740 1 -- 172.21.15.28:0/2528159274 wait complete. 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.469+0000 7f9441458740 1 -- start start 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f9441458740 1 Processor -- start 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f9441458740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f9441458740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cefc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f9441458740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef8040 0x55c82cf038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f942ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.470+0000 7f942ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37616/0 (socket says 172.21.15.28:37616) 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9437fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cefc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9437fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cefc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48782/0 (socket says 172.21.15.28:48782) 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 -- 172.21.15.28:0/2604160606 learned_addr learned my addr 172.21.15.28:0/2604160606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c82cefae30 con 0x55c82cef8040 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c82ce12db0 con 0x55c82cef24e0 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c82cefafb0 con 0x55c82cef2ed0 2024-03-22T02:17:43.472 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f9441458740 10 client.0 _ll_register_callbacks cb 0x55c82cb6aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.473 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef8040 0x55c82cf038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 -- 172.21.15.28:0/2604160606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 msgr2=0x55c82cefc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cefc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 -- 172.21.15.28:0/2604160606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef8040 msgr2=0x55c82cf038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef8040 0x55c82cf038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.471+0000 7f942ffff640 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c82cc37620 con 0x55c82cef24e0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.472+0000 7f942ffff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9420011e30 tx=0x7f9420015040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f053f21470d4df server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.472+0000 7f94177fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.472+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420004030 con 0x55c82cef24e0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.472+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94200041d0 con 0x55c82cef24e0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.472+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94280041d0 con 0x55c82cef24e0 2024-03-22T02:17:43.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.473+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94200043d0 con 0x55c82cef24e0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.473+0000 7f9441458740 10 client.26471 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.473+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c82cf06ab0 con 0x55c82cef24e0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.473+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9420003ad0 con 0x55c82cef24e0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.473+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f942000e790 con 0x55c82cef24e0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 10 client.26471 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 10 client.26471 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 20 client.26471 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf0a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 4 client.26471 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f94280041d0 con 0x55c82cf08150 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9437fff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf0a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f9437fff640 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf0a590 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d5f1bde08c1fe22 server_cookie=941c4fadccf775c4 in_seq=0 out_seq=0 2024-03-22T02:17:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.474+0000 7f94177fe640 10 client.26471 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.475+0000 7f94177fe640 1 -- 172.21.15.28:0/2604160606 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f94280041d0 con 0x55c82cf08150 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.475+0000 7f94177fe640 10 client.26471 handle_command_reply: tid=0 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.475+0000 7f9441458740 1 client.26471 shutdown 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 msgr2=0x55c82cefc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9420011e30 tx=0x7f9420015040 comp rx=0 tx=0).stop 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 shutdown_connections 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf0a590 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f94177fe640 0 client.26471 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef8040 0x55c82cf038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cefc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 --2- 172.21.15.28:0/2604160606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef24e0 0x55c82cefc360 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 >> 172.21.15.28:0/2604160606 conn(0x55c82ceeda50 msgr2=0x55c82ceecc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 shutdown_connections 2024-03-22T02:17:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.476+0000 7f9441458740 1 -- 172.21.15.28:0/2604160606 wait complete. 2024-03-22T02:17:43.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.477+0000 7f9441458740 20 client.26471 trim_cache size 0 max 16384 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 Processor -- start 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 -- start start 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf0b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9418012750 con 0x55c82cef24e0 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f942ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f942ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53114/0 (socket says 172.21.15.28:53114) 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f942ffff640 1 -- 172.21.15.28:0/1025266703 learned_addr learned my addr 172.21.15.28:0/1025266703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9437fff640 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9418010020 con 0x55c82cef8040 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.504+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f941800de70 con 0x55c82cef2ed0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf0b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 -- 172.21.15.28:0/1025266703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 msgr2=0x55c82cf038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 -- 172.21.15.28:0/1025266703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 msgr2=0x55c82cf0b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf0b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 -- 172.21.15.28:0/1025266703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c82cef7160 con 0x55c82cef24e0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942ffff640 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9420011e00 tx=0x7f9420009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec015deceb9294f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942dffb640 1 -- 172.21.15.28:0/1025266703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420052070 con 0x55c82cef24e0 2024-03-22T02:17:43.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942dffb640 1 -- 172.21.15.28:0/1025266703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9420004570 con 0x55c82cef24e0 2024-03-22T02:17:43.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.505+0000 7f942dffb640 1 -- 172.21.15.28:0/1025266703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420057460 con 0x55c82cef24e0 2024-03-22T02:17:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 msgr2=0x55c82cefc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9420011e00 tx=0x7f9420009ac0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 shutdown_connections 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf0b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 --2- 172.21.15.28:0/1025266703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cefc800 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 >> 172.21.15.28:0/1025266703 conn(0x55c82ceeda50 msgr2=0x55c82cf0ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 shutdown_connections 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- 172.21.15.28:0/1025266703 wait complete. 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.506+0000 7f9441458740 1 -- start start 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 Processor -- start 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cf00550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf00a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf6d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f941800e4e0 con 0x55c82cef24e0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f941800f950 con 0x55c82cef8040 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f941800ebb0 con 0x55c82cef2ed0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9441458740 10 client.0 _ll_register_callbacks cb 0x7f9420011cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f942ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cf00550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f942ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cf00550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53128/0 (socket says 172.21.15.28:53128) 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf00a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f942ffff640 1 -- 172.21.15.28:0/1504417776 learned_addr learned my addr 172.21.15.28:0/1504417776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f942ffff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf6d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 -- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 msgr2=0x55c82cf6d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.513 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf6d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 -- 172.21.15.28:0/1504417776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 msgr2=0x55c82cf00550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cf00550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f9437fff640 1 -- 172.21.15.28:0/1504417776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c82cf01080 con 0x55c82cef2ed0 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.507+0000 7f942ffff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf6d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f9437fff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf00a90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55c82ceee9c0 tx=0x7f94280058c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0a403e15d67f401 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94280037f0 con 0x55c82cef2ed0 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9428003990 con 0x55c82cef2ed0 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c82ce64580 con 0x55c82cef2ed0 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9428039050 con 0x55c82cef2ed0 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f9441458740 10 client.26476 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c82cf6ea40 con 0x55c82cef2ed0 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c82ce64580 con 0x55c82cef2ed0 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.508+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f94280292a0 con 0x55c82cef2ed0 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 10 client.26476 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 10 client.26476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 20 client.26476 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf71170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 4 client.26476 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:17:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c82ce5e8d0 con 0x55c82cf08150 2024-03-22T02:17:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f942ffff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf71170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f942ffff640 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf71170 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e9898fdce678aa4 server_cookie=895713b242421fdf in_seq=0 out_seq=0 2024-03-22T02:17:43.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.509+0000 7f94177fe640 10 client.26476 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- 172.21.15.28:0/2463139689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c103770 msgr2=0x7f639c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2463139689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c103770 0x7f639c10a000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6394009be0 tx=0x7f639402bef0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- 172.21.15.28:0/2463139689 shutdown_connections 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2463139689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c103770 0x7f639c10a000 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2463139689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c102db0 0x7f639c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2463139689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- 172.21.15.28:0/2463139689 >> 172.21.15.28:0/2463139689 conn(0x7f639c0fd130 msgr2=0x7f639c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- 172.21.15.28:0/2463139689 shutdown_connections 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- 172.21.15.28:0/2463139689 wait complete. 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 Processor -- start 2024-03-22T02:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.510+0000 7f63a1c78640 1 -- start start 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c102db0 0x7f639c1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c103770 0x7f639c19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f639c10d870 con 0x7f639c101bc0 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f639c10d6f0 con 0x7f639c103770 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f639c10d9f0 con 0x7f639c102db0 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53150/0 (socket says 172.21.15.28:53150) 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 -- 172.21.15.28:0/2209721348 learned_addr learned my addr 172.21.15.28:0/2209721348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639bfff640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c103770 0x7f639c19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639affd640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c102db0 0x7f639c1981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 -- 172.21.15.28:0/2209721348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c102db0 msgr2=0x7f639c1981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c102db0 0x7f639c1981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 -- 172.21.15.28:0/2209721348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c103770 msgr2=0x7f639c19f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c103770 0x7f639c19f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 -- 172.21.15.28:0/2209721348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c19f8d0 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f639b7fe640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f638c00e9f0 tx=0x7f638c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f6398ff9640 1 -- 172.21.15.28:0/2209721348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c00ccc0 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639c19fbc0 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f6398ff9640 1 -- 172.21.15.28:0/2209721348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c0035f0 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.511+0000 7f6398ff9640 1 -- 172.21.15.28:0/2209721348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c00ce60 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.512+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639c1a0160 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.514+0000 7f6398ff9640 1 -- 172.21.15.28:0/2209721348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f638c0037a0 con 0x7f639c101bc0 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.514+0000 7f63a485a740 1 Processor -- start 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.514+0000 7f63a485a740 1 -- start start 2024-03-22T02:17:43.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea680c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea685d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556eea5e58d0 con 0x556eea681210 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556eea58de00 con 0x556eea680820 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f63a485a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556eea5e1880 con 0x556eea686380 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f6398ff9640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 0x7f63840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea680c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea685d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48824/0 (socket says 172.21.15.28:48824) 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 -- 172.21.15.28:0/3717921635 learned_addr learned my addr 172.21.15.28:0/3717921635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639bfff640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 0x7f63840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f6398ff9640 1 -- 172.21.15.28:0/2209721348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f638c0cfa30 con 0x7f639c101bc0 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 -- 172.21.15.28:0/3717921635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 msgr2=0x556eea680c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea680c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 -- 172.21.15.28:0/3717921635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 msgr2=0x556eea685d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea685d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.515+0000 7f639affd640 1 -- 172.21.15.28:0/3717921635 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eea2f93f0 con 0x556eea686380 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.516+0000 7f639affd640 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6390009c90 tx=0x7f6390011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=112edbc3f20125ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.516+0000 7f639bfff640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 0x7f63840a53e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6394004af0 tx=0x7f639402bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.516+0000 7f6381ffb640 1 -- 172.21.15.28:0/3717921635 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639001e070 con 0x556eea686380 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.516+0000 7f6381ffb640 1 -- 172.21.15.28:0/3717921635 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6390019b10 con 0x556eea686380 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.516+0000 7f6381ffb640 1 -- 172.21.15.28:0/3717921635 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000ea60 con 0x556eea686380 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- 172.21.15.28:0/3717921635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 msgr2=0x556eea688810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6390009c90 tx=0x7f6390011810 comp rx=0 tx=0).stop 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- 172.21.15.28:0/3717921635 shutdown_connections 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea688810 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea685d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- 172.21.15.28:0/3717921635 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea680c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- 172.21.15.28:0/3717921635 >> 172.21.15.28:0/3717921635 conn(0x556eea67bd90 msgr2=0x556eea67c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- 172.21.15.28:0/3717921635 shutdown_connections 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- 172.21.15.28:0/3717921635 wait complete. 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- start start 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 Processor -- start 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 0x556eea68ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea68b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 0x556eea692260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556eea689330 con 0x556eea686380 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556eea5a1300 con 0x556eea681210 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556eea6894b0 con 0x556eea680820 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f63a485a740 10 client.0 _ll_register_callbacks cb 0x556eea5dce30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 0x556eea68ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.517+0000 7f639bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 0x556eea692260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 0x556eea68ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48828/0 (socket says 172.21.15.28:48828) 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea68b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639b7fe640 1 -- 172.21.15.28:0/1236907401 learned_addr learned my addr 172.21.15.28:0/1236907401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 -- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 msgr2=0x556eea68ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 0x556eea68ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 -- 172.21.15.28:0/1236907401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 msgr2=0x556eea68b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea68b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 -- 172.21.15.28:0/1236907401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eea3c37d0 con 0x556eea686380 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639affd640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea68b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f639bfff640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 0x556eea692260 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6394009b30 tx=0x7f6394005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa8daf5f4db50a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f637affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6394004040 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f63940041e0 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6390019b10 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f639405c050 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f63a485a740 10 client.16653 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.518+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eea695640 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6390019b10 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f6394043c70 con 0x556eea686380 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 10 client.16653 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 10 client.16653 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 20 client.16653 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea696dd0 0x556eea699230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 4 client.16653 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556eea5f2e10 con 0x556eea696dd0 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f639b7fe640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea696dd0 0x556eea699230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f639b7fe640 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea696dd0 0x556eea699230 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52b68febd46cb9a9 server_cookie=8a3d75f369220c3b in_seq=0 out_seq=0 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.519+0000 7f637affd640 10 client.16653 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f637affd640 1 -- 172.21.15.28:0/1236907401 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556eea5f2e10 con 0x556eea696dd0 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f637affd640 10 client.16653 handle_command_reply: tid=0 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 client.16653 shutdown 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 msgr2=0x556eea692260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 0x556eea692260 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f6394009b30 tx=0x7f6394005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 shutdown_connections 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea686380 0x556eea692260 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea696dd0 0x556eea699230 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea68b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 --2- 172.21.15.28:0/1236907401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea680820 0x556eea68ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f637affd640 0 client.16653 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 >> 172.21.15.28:0/1236907401 conn(0x556eea67bd90 msgr2=0x556eea67afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 shutdown_connections 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 1 -- 172.21.15.28:0/1236907401 wait complete. 2024-03-22T02:17:43.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.520+0000 7f63a485a740 20 client.16653 trim_cache size 0 max 16384 2024-03-22T02:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 Processor -- start 2024-03-22T02:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 -- start start 2024-03-22T02:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea692220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea69db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1ad0b0 con 0x556eea680820 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1aa980 con 0x556eea681210 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f63a485a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1a87d0 con 0x556eea686380 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53182/0 (socket says 172.21.15.28:53182) 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 -- 172.21.15.28:0/3751257774 learned_addr learned my addr 172.21.15.28:0/3751257774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639bfff640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea69db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639affd640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea692220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 -- 172.21.15.28:0/3751257774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 msgr2=0x556eea69db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea69db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 -- 172.21.15.28:0/3751257774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 msgr2=0x556eea692220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea692220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639b7fe640 1 -- 172.21.15.28:0/3751257774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556eea6815f0 con 0x556eea680820 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639bfff640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea69db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.534+0000 7f639affd640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea692220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f638c09bce0 tx=0x7f638c0bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e97fc115ded9ba8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63817fa640 1 -- 172.21.15.28:0/3751257774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c09c450 con 0x556eea680820 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63817fa640 1 -- 172.21.15.28:0/3751257774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c09bd50 con 0x556eea680820 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63817fa640 1 -- 172.21.15.28:0/3751257774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c0983c0 con 0x556eea680820 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 -- 172.21.15.28:0/3751257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 msgr2=0x556eea67ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f638c09bce0 tx=0x7f638c0bc040 comp rx=0 tx=0).stop 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 -- 172.21.15.28:0/3751257774 shutdown_connections 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea69db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea681210 0x556eea692220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 --2- 172.21.15.28:0/3751257774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea680820 0x556eea67ae50 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.535+0000 7f63a485a740 1 -- 172.21.15.28:0/3751257774 >> 172.21.15.28:0/3751257774 conn(0x556eea67bd90 msgr2=0x556eea67a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- 172.21.15.28:0/3751257774 shutdown_connections 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- 172.21.15.28:0/3751257774 wait complete. 2024-03-22T02:17:43.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- start start 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 Processor -- start 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea68e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea691560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1a8e40 con 0x556eea681210 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1aa2b0 con 0x556eea680820 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f639c1a9510 con 0x556eea686380 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f639affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f63a485a740 10 client.0 _ll_register_callbacks cb 0x7f6394009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f639affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53198/0 (socket says 172.21.15.28:53198) 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f639affd640 1 -- 172.21.15.28:0/3539556848 learned_addr learned my addr 172.21.15.28:0/3539556848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.536+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea68e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639bfff640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea691560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 -- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 msgr2=0x556eea691560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea691560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 -- 172.21.15.28:0/3539556848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 msgr2=0x556eea68e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea68e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 -- 172.21.15.28:0/3539556848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556eea691b50 con 0x556eea681210 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f639affd640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x556eea67cd00 tx=0x7f63900054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152bdd0d90cc25c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f637affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639001e040 con 0x556eea681210 2024-03-22T02:17:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f6390019740 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f638c09bd50 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6390019940 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f63a485a740 10 client.16659 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.537+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556eea701c70 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f638c09bd50 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f639002c420 con 0x556eea681210 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 10 client.16659 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 10 client.16659 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 20 client.16659 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea702400 0x556eea704820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 4 client.16659 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:43.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556eea5ed160 con 0x556eea702400 2024-03-22T02:17:43.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.538+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea702400 0x556eea704820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f639b7fe640 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea702400 0x556eea704820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=795fe447278d6bc server_cookie=d560695e03ccfa88 in_seq=0 out_seq=0 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f637affd640 10 client.16659 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f637affd640 1 -- 172.21.15.28:0/3539556848 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556eea5ed160 con 0x556eea702400 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f637affd640 10 client.16659 handle_command_reply: tid=0 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 client.16659 shutdown 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 msgr2=0x556eea68e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x556eea67cd00 tx=0x7f63900054d0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 shutdown_connections 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556eea686380 0x556eea691560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.544+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556eea681210 0x556eea68e590 secure :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0x556eea67cd00 tx=0x7f63900054d0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.545+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556eea702400 0x556eea704820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.545+0000 7f637affd640 0 client.16659 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.545+0000 7f63a485a740 1 --2- 172.21.15.28:0/3539556848 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556eea680820 0x556eea68e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.545+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 >> 172.21.15.28:0/3539556848 conn(0x556eea67bd90 msgr2=0x556eea67afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.546+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 shutdown_connections 2024-03-22T02:17:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.546+0000 7f63a485a740 1 -- 172.21.15.28:0/3539556848 wait complete. 2024-03-22T02:17:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.546+0000 7f63a485a740 20 client.16659 trim_cache size 0 max 16384 2024-03-22T02:17:43.553 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 msgr2=0x7f63840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 0x7f63840a53e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6394004af0 tx=0x7f639402bef0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 msgr2=0x7f639c197c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f638c00e9f0 tx=0x7f638c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 shutdown_connections 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63840a2f20 0x7f63840a53e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f639c103770 0x7f639c19f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f639c102db0 0x7f639c1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 --2- 172.21.15.28:0/2209721348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639c101bc0 0x7f639c197c90 secure :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f638c00e9f0 tx=0x7f638c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.547+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 >> 172.21.15.28:0/2209721348 conn(0x7f639c0fd130 msgr2=0x7f639c105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.548+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 shutdown_connections 2024-03-22T02:17:43.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.548+0000 7f63a1c78640 1 -- 172.21.15.28:0/2209721348 wait complete. 2024-03-22T02:17:43.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:17:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.718+0000 7f94177fe640 1 -- 172.21.15.28:0/1504417776 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c82ce5e8d0 con 0x55c82cf08150 2024-03-22T02:17:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.718+0000 7f94177fe640 10 client.26476 handle_command_reply: tid=0 2024-03-22T02:17:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.718+0000 7f9441458740 1 client.26476 shutdown 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.718+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 msgr2=0x55c82cf00a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.718+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf00a90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x55c82ceee9c0 tx=0x7f94280058c0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 shutdown_connections 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c82cf08150 0x55c82cf71170 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c82cef8040 0x55c82cf6d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c82cef2ed0 0x55c82cf00a90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 --2- 172.21.15.28:0/1504417776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c82cef24e0 0x55c82cf00550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 >> 172.21.15.28:0/1504417776 conn(0x55c82ceeda50 msgr2=0x55c82cf0b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f94177fe640 0 client.26476 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 shutdown_connections 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 1 -- 172.21.15.28:0/1504417776 wait complete. 2024-03-22T02:17:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f9441458740 20 client.26476 trim_cache size 0 max 16384 2024-03-22T02:17:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:17:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:17:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:17:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:17:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.719+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 msgr2=0x7f94240a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 0x7f94240a56d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f94300bd850 tx=0x7f94300b2fa0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 msgr2=0x7f94381a58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9430009bf0 tx=0x7f9430035d10 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 shutdown_connections 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94240a3210 0x7f94240a56d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f943810e1c0 0x7f94381a58e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9438075bd0 0x7f943819e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 --2- 172.21.15.28:0/3704559202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94380745f0 0x7f943819e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 >> 172.21.15.28:0/3704559202 conn(0x7f94380fd160 msgr2=0x7f9438101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 shutdown_connections 2024-03-22T02:17:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:43.720+0000 7f943e86f640 1 -- 172.21.15.28:0/3704559202 wait complete. 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.745+0000 7fc7b4e29640 1 -- 172.21.15.28:0/4109841912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0102d70 msgr2=0x7fc7b01051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.745+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/4109841912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0102d70 0x7fc7b01051e0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc798009900 tx=0x7fc79802bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 -- 172.21.15.28:0/4109841912 shutdown_connections 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/4109841912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0105720 0x7fc7b0105ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/4109841912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0102d70 0x7fc7b01051e0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/4109841912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0100400 0x7fc7b0102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 -- 172.21.15.28:0/4109841912 >> 172.21.15.28:0/4109841912 conn(0x7fc7b00fc020 msgr2=0x7fc7b00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 -- 172.21.15.28:0/4109841912 shutdown_connections 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 -- 172.21.15.28:0/4109841912 wait complete. 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 Processor -- start 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 -- start start 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.746+0000 7fc7b4e29640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0100400 0x7fc7b019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0102d70 0x7fc7b019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0113e80 con 0x7fc7b0105720 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0113d00 con 0x7fc7b0102d70 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b0114000 con 0x7fc7b0100400 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53216/0 (socket says 172.21.15.28:53216) 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 -- 172.21.15.28:0/1665916564 learned_addr learned my addr 172.21.15.28:0/1665916564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0102d70 0x7fc7b019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0100400 0x7fc7b019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 -- 172.21.15.28:0/1665916564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0100400 msgr2=0x7fc7b019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0100400 0x7fc7b019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 -- 172.21.15.28:0/1665916564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0102d70 msgr2=0x7fc7b019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0102d70 0x7fc7b019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 -- 172.21.15.28:0/1665916564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b01a6110 con 0x7fc7b0105720 2024-03-22T02:17:43.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7aed9d640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009fd0 tx=0x7fc7a800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7a77fe640 1 -- 172.21.15.28:0/1665916564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8017070 con 0x7fc7b0105720 2024-03-22T02:17:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b01a6400 con 0x7fc7b0105720 2024-03-22T02:17:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7a77fe640 1 -- 172.21.15.28:0/1665916564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7a8003e50 con 0x7fc7b0105720 2024-03-22T02:17:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7a77fe640 1 -- 172.21.15.28:0/1665916564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a80106c0 con 0x7fc7b0105720 2024-03-22T02:17:43.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.747+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b01a6920 con 0x7fc7b0105720 2024-03-22T02:17:43.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.749+0000 7fc7a77fe640 1 -- 172.21.15.28:0/1665916564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc7a8003970 con 0x7fc7b0105720 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 Processor -- start 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 -- start start 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10670d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe10673810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe105d08d0 con 0x55fe1066c210 2024-03-22T02:17:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe10578e00 con 0x55fe10671380 2024-03-22T02:17:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe105cc880 con 0x55fe1066b820 2024-03-22T02:17:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a77fe640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 0x7fc79c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7ae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10670d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48862/0 (socket says 172.21.15.28:48862) 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7ae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10670d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53218/0 (socket says 172.21.15.28:53218) 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a7fff640 1 -- 172.21.15.28:0/2590020210 learned_addr learned my addr 172.21.15.28:0/2590020210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe10673810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7aed9d640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 0x7fc79c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.750+0000 7fc7a77fe640 1 -- 172.21.15.28:0/1665916564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc7a80cf510 con 0x7fc7b0105720 2024-03-22T02:17:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 -- 172.21.15.28:0/2590020210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 msgr2=0x55fe10673810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe10673810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 -- 172.21.15.28:0/2590020210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 msgr2=0x55fe10670d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10670d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 -- 172.21.15.28:0/2590020210 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe102e43f0 con 0x55fe1066b820 2024-03-22T02:17:43.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7aed9d640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 0x7fc79c0a53e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80982b0 tx=0x7fc7a8086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc79802c8e0 tx=0x7fc798005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4f152f00b36344a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a4ff9640 1 -- 172.21.15.28:0/2590020210 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798004030 con 0x55fe1066b820 2024-03-22T02:17:43.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a4ff9640 1 -- 172.21.15.28:0/2590020210 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798004ba0 con 0x55fe1066b820 2024-03-22T02:17:43.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.751+0000 7fc7a4ff9640 1 -- 172.21.15.28:0/2590020210 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798003950 con 0x55fe1066b820 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/2590020210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 msgr2=0x55fe1066bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc79802c8e0 tx=0x7fc798005fd0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/2590020210 shutdown_connections 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe10673810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10670d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/2590020210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe1066bc40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/2590020210 >> 172.21.15.28:0/2590020210 conn(0x55fe10666d90 msgr2=0x55fe10667190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/2590020210 shutdown_connections 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/2590020210 wait complete. 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 -- start start 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.752+0000 7fc7b7a0c740 1 Processor -- start 2024-03-22T02:17:43.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10675d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe106742d0 con 0x55fe1066c210 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe1058c300 con 0x55fe10671380 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe10674450 con 0x55fe1066b820 2024-03-22T02:17:43.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 10 client.0 _ll_register_callbacks cb 0x55fe105c7e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53230/0 (socket says 172.21.15.28:53230) 2024-03-22T02:17:43.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 -- 172.21.15.28:0/3629726499 learned_addr learned my addr 172.21.15.28:0/3629726499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10675d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 -- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 msgr2=0x55fe10675d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10675d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 -- 172.21.15.28:0/3629726499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 msgr2=0x55fe1067d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 -- 172.21.15.28:0/3629726499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe103ae7d0 con 0x55fe1066c210 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00126e0 tx=0x7fc7a0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5780b0c426d75f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0012190 con 0x55fe1066c210 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc7a000f8c0 con 0x55fe1066c210 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc798004ba0 con 0x55fe1066c210 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7a000fac0 con 0x55fe1066c210 2024-03-22T02:17:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 10 client.16671 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.753+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe10680740 con 0x55fe1066c210 2024-03-22T02:17:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798004ba0 con 0x55fe1066c210 2024-03-22T02:17:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc7a001c420 con 0x55fe1066c210 2024-03-22T02:17:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 10 client.16671 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 10 client.16671 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 20 client.16671 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe10684340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 4 client.16671 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe105dde10 con 0x55fe10681f00 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe10684340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe10684340 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6699659d275867de server_cookie=ce23f4837c875c73 in_seq=0 out_seq=0 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.754+0000 7fc795ffb640 10 client.16671 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc795ffb640 1 -- 172.21.15.28:0/3629726499 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe105dde10 con 0x55fe10681f00 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc795ffb640 10 client.16671 handle_command_reply: tid=0 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 client.16671 shutdown 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 msgr2=0x55fe10676270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00126e0 tx=0x7fc7a0012ba0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 shutdown_connections 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe10684340 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc795ffb640 0 client.16671 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066c210 0x55fe10676270 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/3629726499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10675d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 >> 172.21.15.28:0/3629726499 conn(0x55fe10666d90 msgr2=0x55fe10665fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 shutdown_connections 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/3629726499 wait complete. 2024-03-22T02:17:43.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.755+0000 7fc7b7a0c740 20 client.16671 trim_cache size 0 max 16384 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.770+0000 7fc7b7a0c740 1 Processor -- start 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.770+0000 7fc7b7a0c740 1 -- start start 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.770+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10665740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.770+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe1066c210 0x55fe1067d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.770+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01b3850 con 0x55fe10671380 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01b1120 con 0x55fe1066c210 2024-03-22T02:17:43.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01aef70 con 0x55fe1066b820 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7ae59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe1066c210 0x55fe1067d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53244/0 (socket says 172.21.15.28:53244) 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 -- 172.21.15.28:0/87548528 learned_addr learned my addr 172.21.15.28:0/87548528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10665740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 -- 172.21.15.28:0/87548528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 msgr2=0x55fe10665740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10665740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 -- 172.21.15.28:0/87548528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe1066c210 msgr2=0x55fe1067d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe1066c210 0x55fe1067d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 -- 172.21.15.28:0/87548528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe10667b00 con 0x55fe10671380 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc79802c8e0 tx=0x7fc798045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4e65e986d02cc74 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc797fff640 1 -- 172.21.15.28:0/87548528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc798004940 con 0x55fe10671380 2024-03-22T02:17:43.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc797fff640 1 -- 172.21.15.28:0/87548528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798004ae0 con 0x55fe10671380 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.771+0000 7fc797fff640 1 -- 172.21.15.28:0/87548528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980436e0 con 0x55fe10671380 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/87548528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 msgr2=0x55fe10665c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc79802c8e0 tx=0x7fc798045040 comp rx=0 tx=0).stop 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/87548528 shutdown_connections 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe10671380 0x55fe10665c80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe1066c210 0x55fe1067d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/87548528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066b820 0x55fe10665740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/87548528 >> 172.21.15.28:0/87548528 conn(0x55fe10666d90 msgr2=0x55fe1066d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/87548528 shutdown_connections 2024-03-22T02:17:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/87548528 wait complete. 2024-03-22T02:17:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- start start 2024-03-22T02:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 Processor -- start 2024-03-22T02:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066c210 0x55fe1067b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01af5e0 con 0x55fe1066b820 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01b0a50 con 0x55fe10671380 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.772+0000 7fc7b7a0c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b01afcb0 con 0x55fe1066c210 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7b7a0c740 10 client.0 _ll_register_callbacks cb 0x7fc7a0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53254/0 (socket says 172.21.15.28:53254) 2024-03-22T02:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 -- 172.21.15.28:0/1947108375 learned_addr learned my addr 172.21.15.28:0/1947108375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066c210 0x55fe1067b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 -- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066c210 msgr2=0x55fe1067b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066c210 0x55fe1067b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 -- 172.21.15.28:0/1947108375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 msgr2=0x55fe1067c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 -- 172.21.15.28:0/1947108375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1067c820 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7a7fff640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc798005df0 tx=0x7fc79802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dad5697a7a2421 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc795ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7980050b0 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc798005250 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc798004ae0 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc798005450 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7b7a0c740 10 client.16677 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.773+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fe106e8590 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc798004ae0 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc7980514e0 con 0x55fe1066b820 2024-03-22T02:17:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 10 client.16677 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 10 client.16677 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 20 client.16677 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe106eaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 4 client.16677 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fe105d8160 con 0x55fe10681f00 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe106eaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7ae59c640 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe106eaab0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9876fc3c757a46ec server_cookie=3e1c42dfea64b7c2 in_seq=0 out_seq=0 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc795ffb640 10 client.16677 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc795ffb640 1 -- 172.21.15.28:0/1947108375 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fe105d8160 con 0x55fe10681f00 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc795ffb640 10 client.16677 handle_command_reply: tid=0 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.774+0000 7fc7b7a0c740 1 client.16677 shutdown 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 msgr2=0x55fe1067ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc798005df0 tx=0x7fc79802bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 shutdown_connections 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55fe10681f00 0x55fe106eaab0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe10671380 0x55fe1067c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe1066c210 0x55fe1067b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 --2- 172.21.15.28:0/1947108375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe1066b820 0x55fe1067ad30 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 >> 172.21.15.28:0/1947108375 conn(0x55fe10666d90 msgr2=0x55fe10669ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc795ffb640 0 client.16677 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 shutdown_connections 2024-03-22T02:17:43.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 1 -- 172.21.15.28:0/1947108375 wait complete. 2024-03-22T02:17:43.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b7a0c740 20 client.16677 trim_cache size 0 max 16384 2024-03-22T02:17:43.790 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 msgr2=0x7fc79c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 0x7fc79c0a53e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80982b0 tx=0x7fc7a8086040 comp rx=0 tx=0).stop 2024-03-22T02:17:43.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 msgr2=0x7fc7b01a5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009fd0 tx=0x7fc7a800b260 comp rx=0 tx=0).stop 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 shutdown_connections 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc79c0a2f20 0x7fc79c0a53e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b0105720 0x7fc7b01a5aa0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc7b0102d70 0x7fc7b019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 --2- 172.21.15.28:0/1665916564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc7b0100400 0x7fc7b019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 >> 172.21.15.28:0/1665916564 conn(0x7fc7b00fc020 msgr2=0x7fc7b0100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 shutdown_connections 2024-03-22T02:17:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.775+0000 7fc7b4e29640 1 -- 172.21.15.28:0/1665916564 wait complete. 2024-03-22T02:17:43.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 -- 172.21.15.28:0/989486223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 msgr2=0x7f577c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 --2- 172.21.15.28:0/989486223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c101fc0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f5764009900 tx=0x7f576402bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 -- 172.21.15.28:0/989486223 shutdown_connections 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 --2- 172.21.15.28:0/989486223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c103750 0x7f577c109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 --2- 172.21.15.28:0/989486223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c102d90 0x7f577c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 --2- 172.21.15.28:0/989486223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c101fc0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 -- 172.21.15.28:0/989486223 >> 172.21.15.28:0/989486223 conn(0x7f577c0fd130 msgr2=0x7f577c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 -- 172.21.15.28:0/989486223 shutdown_connections 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.898+0000 7f578322a640 1 -- 172.21.15.28:0/989486223 wait complete. 2024-03-22T02:17:43.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 Processor -- start 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 -- start start 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c102d90 0x7f577c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c103750 0x7f577c19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f577c10d810 con 0x7f577c101ba0 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f577c10d690 con 0x7f577c102d90 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f578322a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f577c10d990 con 0x7f577c103750 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53280/0 (socket says 172.21.15.28:53280) 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f577bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c102d90 0x7f577c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 -- 172.21.15.28:0/3777395797 learned_addr learned my addr 172.21.15.28:0/3777395797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f57817c7640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c103750 0x7f577c19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 -- 172.21.15.28:0/3777395797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c103750 msgr2=0x7f577c19f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c103750 0x7f577c19f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 -- 172.21.15.28:0/3777395797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c102d90 msgr2=0x7f577c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c102d90 0x7f577c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 -- 172.21.15.28:0/3777395797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577c19fb30 con 0x7f577c101ba0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.899+0000 7f5780fc6640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5764000c00 tx=0x7f576402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.900+0000 7f5779ffb640 1 -- 172.21.15.28:0/3777395797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764041070 con 0x7f577c101ba0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.900+0000 7f5779ffb640 1 -- 172.21.15.28:0/3777395797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5764004450 con 0x7f577c101ba0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.900+0000 7f5779ffb640 1 -- 172.21.15.28:0/3777395797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5764034440 con 0x7f577c101ba0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.900+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f577c19fdc0 con 0x7f577c101ba0 2024-03-22T02:17:43.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.900+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f577c1a0220 con 0x7f577c101ba0 2024-03-22T02:17:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.902+0000 7f5779ffb640 1 -- 172.21.15.28:0/3777395797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5764044070 con 0x7f577c101ba0 2024-03-22T02:17:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.902+0000 7f5785e11740 1 Processor -- start 2024-03-22T02:17:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.902+0000 7f5785e11740 1 -- start start 2024-03-22T02:17:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad31fdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad3202d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ad31628d0 con 0x564ad31fd820 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ad310ae00 con 0x564ad3203380 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5785e11740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ad315e880 con 0x564ad31fe210 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5779ffb640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 0x7f57680a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f57817c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad31fdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad3202d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37780/0 (socket says 172.21.15.28:37780) 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 -- 172.21.15.28:0/2002559655 learned_addr learned my addr 172.21.15.28:0/2002559655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f57817c7640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 0x7f57680a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f5779ffb640 1 -- 172.21.15.28:0/3777395797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f57640f1230 con 0x7f577c101ba0 2024-03-22T02:17:43.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 -- 172.21.15.28:0/2002559655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 msgr2=0x564ad3202d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad3202d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 -- 172.21.15.28:0/2002559655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 msgr2=0x564ad31fdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad31fdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 -- 172.21.15.28:0/2002559655 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad2e763f0 con 0x564ad3203380 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f577bfff640 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f576c009ca0 tx=0x7f576c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91179f0f873b495a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.903+0000 7f57817c7640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 0x7f57680a5460 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x564ad31fea70 tx=0x7f5774004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5772ffd640 1 -- 172.21.15.28:0/2002559655 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c01e070 con 0x564ad3203380 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5772ffd640 1 -- 172.21.15.28:0/2002559655 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576c019b10 con 0x564ad3203380 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5772ffd640 1 -- 172.21.15.28:0/2002559655 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c00e9c0 con 0x564ad3203380 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 -- 172.21.15.28:0/2002559655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 msgr2=0x564ad3205810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f576c009ca0 tx=0x7f576c011810 comp rx=0 tx=0).stop 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 -- 172.21.15.28:0/2002559655 shutdown_connections 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad3205810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad3202d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 --2- 172.21.15.28:0/2002559655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad31fdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 -- 172.21.15.28:0/2002559655 >> 172.21.15.28:0/2002559655 conn(0x564ad31f8d90 msgr2=0x564ad31f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 -- 172.21.15.28:0/2002559655 shutdown_connections 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.904+0000 7f5785e11740 1 -- 172.21.15.28:0/2002559655 wait complete. 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 -- start start 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 Processor -- start 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad320f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ad3206330 con 0x564ad31fe210 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ad311e300 con 0x564ad3203380 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ad32064b0 con 0x564ad31fd820 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5785e11740 10 client.0 _ll_register_callbacks cb 0x564ad3159e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f577bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f577bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53298/0 (socket says 172.21.15.28:53298) 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f577bfff640 1 -- 172.21.15.28:0/2237274950 learned_addr learned my addr 172.21.15.28:0/2237274950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.905+0000 7f57817c7640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad320f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 -- 172.21.15.28:0/2237274950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 msgr2=0x564ad3207d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 -- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 msgr2=0x564ad320f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad320f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 -- 172.21.15.28:0/2237274950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad2f407d0 con 0x564ad31fe210 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f5780fc6640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f577bfff640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f576c011d10 tx=0x7f576c00e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62164b99c34c1b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c028020 con 0x564ad31fe210 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f576c005050 con 0x564ad31fe210 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f576c019b10 con 0x564ad31fe210 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f576c01e040 con 0x564ad31fe210 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f5785e11740 10 client.16689 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ad3212650 con 0x564ad31fe210 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f576c019b10 con 0x564ad31fe210 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.906+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f576c0046c0 con 0x564ad31fe210 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 10 client.16689 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 20 client.16689 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad3213a40 0x564ad3215eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 4 client.16689 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ad316fe10 con 0x564ad3213a40 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5780fc6640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad3213a40 0x564ad3215eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f5780fc6640 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad3213a40 0x564ad3215eb0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=980693cacf8a012a server_cookie=fa92898eeac3c37f in_seq=0 out_seq=0 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.907+0000 7f575bfff640 10 client.16689 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f575bfff640 1 -- 172.21.15.28:0/2237274950 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ad316fe10 con 0x564ad3213a40 2024-03-22T02:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f575bfff640 10 client.16689 handle_command_reply: tid=0 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 client.16689 shutdown 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 msgr2=0x564ad32082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f576c011d10 tx=0x7f576c00e3b0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 shutdown_connections 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad3213a40 0x564ad3215eb0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad3203380 0x564ad320f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f575bfff640 0 client.16689 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 --2- 172.21.15.28:0/2237274950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 >> 172.21.15.28:0/2237274950 conn(0x564ad31f8d90 msgr2=0x564ad31f7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 shutdown_connections 2024-03-22T02:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.908+0000 7f5785e11740 1 -- 172.21.15.28:0/2237274950 wait complete. 2024-03-22T02:17:43.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.909+0000 7f5785e11740 20 client.16689 trim_cache size 0 max 16384 2024-03-22T02:17:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 Processor -- start 2024-03-22T02:17:43.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 -- start start 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad3203380 0x564ad320f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1ad2b0 con 0x564ad31fd820 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1aab80 con 0x564ad31fe210 2024-03-22T02:17:43.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5785e11740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1a89d0 con 0x564ad3203380 2024-03-22T02:17:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53304/0 (socket says 172.21.15.28:53304) 2024-03-22T02:17:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 -- 172.21.15.28:0/213775613 learned_addr learned my addr 172.21.15.28:0/213775613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f577bfff640 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f57817c7640 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad3203380 0x564ad320f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 -- 172.21.15.28:0/213775613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad3203380 msgr2=0x564ad320f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad3203380 0x564ad320f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 -- 172.21.15.28:0/213775613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fe210 msgr2=0x564ad32082a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 -- 172.21.15.28:0/213775613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad31fe5f0 con 0x564ad31fd820 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.922+0000 7f5780fc6640 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f57640ee970 tx=0x7f57640a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d21e6cca4e9b8ec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f57727fc640 1 -- 172.21.15.28:0/213775613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576404b040 con 0x564ad31fd820 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f57727fc640 1 -- 172.21.15.28:0/213775613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5764004c80 con 0x564ad31fd820 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f57727fc640 1 -- 172.21.15.28:0/213775613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57640bb7c0 con 0x564ad31fd820 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- 172.21.15.28:0/213775613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 msgr2=0x564ad3207d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f57640ee970 tx=0x7f57640a8040 comp rx=0 tx=0).stop 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- 172.21.15.28:0/213775613 shutdown_connections 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad3203380 0x564ad320f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fe210 0x564ad32082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 --2- 172.21.15.28:0/213775613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad31fd820 0x564ad3207d60 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- 172.21.15.28:0/213775613 >> 172.21.15.28:0/213775613 conn(0x564ad31f8d90 msgr2=0x564ad31f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- 172.21.15.28:0/213775613 shutdown_connections 2024-03-22T02:17:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- 172.21.15.28:0/213775613 wait complete. 2024-03-22T02:17:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.923+0000 7f5785e11740 1 -- start start 2024-03-22T02:17:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 Processor -- start 2024-03-22T02:17:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad320ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 0x564ad320dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1a9040 con 0x564ad3203380 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1aa4b0 con 0x564ad31fd820 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f577c1a9710 con 0x564ad31fe210 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5785e11740 10 client.0 _ll_register_callbacks cb 0x7f576c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 0x564ad320dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5780fc6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37816/0 (socket says 172.21.15.28:37816) 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5780fc6640 1 -- 172.21.15.28:0/772731682 learned_addr learned my addr 172.21.15.28:0/772731682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f577bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad320ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 -- 172.21.15.28:0/772731682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 msgr2=0x564ad320ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad320ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 -- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 msgr2=0x564ad320c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f57817c7640 1 -- 172.21.15.28:0/772731682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad320e250 con 0x564ad3203380 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.924+0000 7f5780fc6640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f57817c7640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 0x564ad320dc60 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x564ad3208850 tx=0x7f5774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4b821b5b697737 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5774004b20 con 0x564ad3203380 2024-03-22T02:17:43.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f577401c070 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5764004c80 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5774004cc0 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 10 client.16695 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ad320f5c0 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5764004c80 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f577402f070 con 0x564ad3203380 2024-03-22T02:17:43.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 10 client.16695 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 20 client.16695 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad327a590 0x564ad327c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 4 client.16695 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ad316a160 con 0x564ad327a590 2024-03-22T02:17:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.925+0000 7f5780fc6640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad327a590 0x564ad327c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f575bfff640 10 client.16695 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5780fc6640 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad327a590 0x564ad327c9b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8d06bc97369ff19 server_cookie=21235beeac4443bb in_seq=0 out_seq=0 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f575bfff640 1 -- 172.21.15.28:0/772731682 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ad316a160 con 0x564ad327a590 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f575bfff640 10 client.16695 handle_command_reply: tid=0 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 client.16695 shutdown 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 msgr2=0x564ad320dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 0x564ad320dc60 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x564ad3208850 tx=0x7f5774004a40 comp rx=0 tx=0).stop 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 shutdown_connections 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ad327a590 0x564ad327c9b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ad3203380 0x564ad320dc60 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ad31fe210 0x564ad320ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f575bfff640 0 client.16695 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 --2- 172.21.15.28:0/772731682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ad31fd820 0x564ad320c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 >> 172.21.15.28:0/772731682 conn(0x564ad31f8d90 msgr2=0x564ad3207d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 shutdown_connections 2024-03-22T02:17:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 1 -- 172.21.15.28:0/772731682 wait complete. 2024-03-22T02:17:43.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.926+0000 7f5785e11740 20 client.16695 trim_cache size 0 max 16384 2024-03-22T02:17:43.948 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:17:43.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 msgr2=0x7f57680a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 0x7f57680a5460 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x564ad31fea70 tx=0x7f5774004ab0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 msgr2=0x7f577c197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f5764000c00 tx=0x7f576402c640 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 shutdown_connections 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57680a2fa0 0x7f57680a5460 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f577c103750 0x7f577c19f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f577c102d90 0x7f577c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 --2- 172.21.15.28:0/3777395797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f577c101ba0 0x7f577c197f10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 >> 172.21.15.28:0/3777395797 conn(0x7f577c0fd130 msgr2=0x7f577c106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 shutdown_connections 2024-03-22T02:17:43.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:43.927+0000 7f578322a640 1 -- 172.21.15.28:0/3777395797 wait complete. 2024-03-22T02:17:44.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:44 smithi028 ceph-mon[31916]: pgmap v482: 129 pgs: 129 active+clean; 48 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-22T02:17:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:44 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1010502466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:44.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 253539 11.89 MB/sec execute 26 sec latency 106.196 ms 2024-03-22T02:17:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:44 smithi120 ceph-mon[36803]: pgmap v482: 129 pgs: 129 active+clean; 48 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-22T02:17:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:44 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1010502466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:44 smithi083 ceph-mon[36340]: pgmap v482: 129 pgs: 129 active+clean; 48 MiB data, 838 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.0 MiB/s wr, 403 op/s 2024-03-22T02:17:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:44 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1010502466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:17:44.951 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:17:44.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:17:44.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 -- 172.21.15.28:0/2907529758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc101c90 msgr2=0x7f94bc102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 --2- 172.21.15.28:0/2907529758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc101c90 0x7f94bc102110 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009900 tx=0x7f94b002bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 -- 172.21.15.28:0/2907529758 shutdown_connections 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 --2- 172.21.15.28:0/2907529758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc102650 0x7f94bc108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 --2- 172.21.15.28:0/2907529758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc101c90 0x7f94bc102110 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 --2- 172.21.15.28:0/2907529758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 -- 172.21.15.28:0/2907529758 >> 172.21.15.28:0/2907529758 conn(0x7f94bc0fc030 msgr2=0x7f94bc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 -- 172.21.15.28:0/2907529758 shutdown_connections 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.060+0000 7f94c0847640 1 -- 172.21.15.28:0/2907529758 wait complete. 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 Processor -- start 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 -- start start 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 0x7f94bc198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10d840 con 0x7f94bc102650 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10d6c0 con 0x7f94bc101c90 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94c0847640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10d9c0 con 0x7f94bc100aa0 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94b9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 0x7f94bc198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48962/0 (socket says 172.21.15.28:48962) 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53334/0 (socket says 172.21.15.28:53334) 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.061+0000 7f94ba59c640 1 -- 172.21.15.28:0/1112338811 learned_addr learned my addr 172.21.15.28:0/1112338811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 -- 172.21.15.28:0/1112338811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 msgr2=0x7f94bc197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 -- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 msgr2=0x7f94bc19f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 -- 172.21.15.28:0/1112338811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc19fb10 con 0x7f94bc101c90 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94bad9d640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 0x7f94bc198430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009230 tx=0x7f94b002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94ab7fe640 1 -- 172.21.15.28:0/1112338811 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0041070 con 0x7f94bc101c90 2024-03-22T02:17:45.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc19fda0 con 0x7f94bc101c90 2024-03-22T02:17:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94ab7fe640 1 -- 172.21.15.28:0/1112338811 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004450 con 0x7f94bc101c90 2024-03-22T02:17:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94ab7fe640 1 -- 172.21.15.28:0/1112338811 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b0003d80 con 0x7f94bc101c90 2024-03-22T02:17:45.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.062+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1a0280 con 0x7f94bc101c90 2024-03-22T02:17:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.064+0000 7f94ab7fe640 1 -- 172.21.15.28:0/1112338811 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f94b0044070 con 0x7f94bc101c90 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.064+0000 7f94c342d740 1 Processor -- start 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 -- start start 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fe4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca00fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddfc959570 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ddfc901e10 con 0x55ddfc9f99c0 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94c342d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddfc955520 con 0x55ddfc9feb30 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94ab7fe640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 0x7f94a40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53336/0 (socket says 172.21.15.28:53336) 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 -- 172.21.15.28:0/990298567 learned_addr learned my addr 172.21.15.28:0/990298567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94ba59c640 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fe4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca00fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94ab7fe640 1 -- 172.21.15.28:0/1112338811 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94b00f1210 con 0x7f94bc101c90 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 0x7f94a40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 -- 172.21.15.28:0/990298567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 msgr2=0x55ddfca00fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca00fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 -- 172.21.15.28:0/990298567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 msgr2=0x55ddfc9fe4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fe4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.065+0000 7f94bad9d640 1 -- 172.21.15.28:0/990298567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddfc7357e0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94bad9d640 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f94b400fa00 tx=0x7f94b400fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd9f63bfe020e532 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94a8ff9640 1 -- 172.21.15.28:0/990298567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b401ccd0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94a8ff9640 1 -- 172.21.15.28:0/990298567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b4010910 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94a8ff9640 1 -- 172.21.15.28:0/990298567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b400e610 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94bad9d640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 0x7f94a40a53e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f94b4023000 tx=0x7f94b4020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 -- 172.21.15.28:0/990298567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 msgr2=0x55ddfc9f93f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f94b400fa00 tx=0x7f94b400fe00 comp rx=0 tx=0).stop 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 -- 172.21.15.28:0/990298567 shutdown_connections 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca00fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fe4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 --2- 172.21.15.28:0/990298567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9f93f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.066+0000 7f94c342d740 1 -- 172.21.15.28:0/990298567 >> 172.21.15.28:0/990298567 conn(0x55ddfc9f0130 msgr2=0x55ddfc9f0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- 172.21.15.28:0/990298567 shutdown_connections 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- 172.21.15.28:0/990298567 wait complete. 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- start start 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 Processor -- start 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfca03710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca0a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddfca01ae0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ddfc915110 con 0x55ddfc9f99c0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddfca01c60 con 0x55ddfc9feb30 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94c342d740 10 client.0 _ll_register_callbacks cb 0x55ddfc951be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53344/0 (socket says 172.21.15.28:53344) 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 -- 172.21.15.28:0/496630252 learned_addr learned my addr 172.21.15.28:0/496630252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfca03710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94bad9d640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca0a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 -- 172.21.15.28:0/496630252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 msgr2=0x55ddfca0a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca0a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 -- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 msgr2=0x55ddfca03710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfca03710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94ba59c640 1 -- 172.21.15.28:0/496630252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddfc795f70 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.067+0000 7f94bad9d640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca0a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94ba59c640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f94ac012760 tx=0x7f94ac012c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fb7ee79aa585ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac004050 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94ac0041f0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b4010910 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94ac0043f0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 10 client.16707 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ddfc62d170 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b4010910 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f94ac003be0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 10 client.16707 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 20 client.16707 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca0f110 0x55ddfca115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 4 client.16707 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddfc966ab0 con 0x55ddfca0f110 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.068+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca0f110 0x55ddfca115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.069+0000 7f94a1ffb640 10 client.16707 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.069+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca0f110 0x55ddfca115d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d0473826a66f3b4 server_cookie=39fc6a42ed57bf25 in_seq=0 out_seq=0 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.070+0000 7f94a1ffb640 1 -- 172.21.15.28:0/496630252 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddfc966ab0 con 0x55ddfca0f110 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.070+0000 7f94a1ffb640 10 client.16707 handle_command_reply: tid=0 2024-03-22T02:17:45.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.070+0000 7f94c342d740 1 client.16707 shutdown 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 msgr2=0x55ddfca031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f94ac012760 tx=0x7f94ac012c20 comp rx=0 tx=0).stop 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 shutdown_connections 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca0f110 0x55ddfca115d0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9feb30 0x55ddfca0a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9f99c0 0x55ddfca03710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 --2- 172.21.15.28:0/496630252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca031d0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94a1ffb640 0 client.16707 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 >> 172.21.15.28:0/496630252 conn(0x55ddfc9f0130 msgr2=0x55ddfc9ef3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 shutdown_connections 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 1 -- 172.21.15.28:0/496630252 wait complete. 2024-03-22T02:17:45.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.071+0000 7f94c342d740 20 client.16707 trim_cache size 0 max 16384 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 Processor -- start 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 -- start start 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9fa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfc9facd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1ab020 con 0x55ddfc9f99c0 2024-03-22T02:17:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1a88f0 con 0x55ddfc9feb30 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94c342d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1a6740 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.088+0000 7f94b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53360/0 (socket says 172.21.15.28:53360) 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 -- 172.21.15.28:0/2337219748 learned_addr learned my addr 172.21.15.28:0/2337219748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94ba59c640 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9fa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94bad9d640 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfc9facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 -- 172.21.15.28:0/2337219748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f8fd0 msgr2=0x55ddfc9fa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9fa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 -- 172.21.15.28:0/2337219748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 msgr2=0x55ddfc9facd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfc9facd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 -- 172.21.15.28:0/2337219748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddfca022c0 con 0x55ddfc9f99c0 2024-03-22T02:17:45.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94b00bd230 tx=0x7f94b008f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92572d0ce9430e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94a3fff640 1 -- 172.21.15.28:0/2337219748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b004b040 con 0x55ddfc9f99c0 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94a3fff640 1 -- 172.21.15.28:0/2337219748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004c80 con 0x55ddfc9f99c0 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.089+0000 7f94a3fff640 1 -- 172.21.15.28:0/2337219748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00c5f40 con 0x55ddfc9f99c0 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- 172.21.15.28:0/2337219748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 msgr2=0x55ddfc9fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94b00bd230 tx=0x7f94b008f1b0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- 172.21.15.28:0/2337219748 shutdown_connections 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfc9facd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f99c0 0x55ddfc9fa790 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- 172.21.15.28:0/2337219748 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f8fd0 0x55ddfc9fa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- 172.21.15.28:0/2337219748 >> 172.21.15.28:0/2337219748 conn(0x55ddfc9f0130 msgr2=0x55ddfc9f3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- 172.21.15.28:0/2337219748 shutdown_connections 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- 172.21.15.28:0/2337219748 wait complete. 2024-03-22T02:17:45.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- start start 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 Processor -- start 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f99c0 0x55ddfca07cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfca08d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1a6db0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1a8220 con 0x55ddfc9feb30 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc1a7480 con 0x55ddfc9f99c0 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94c342d740 10 client.0 _ll_register_callbacks cb 0x7f94ac009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53366/0 (socket says 172.21.15.28:53366) 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94ba59c640 1 -- 172.21.15.28:0/2517025101 learned_addr learned my addr 172.21.15.28:0/2517025101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94bad9d640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfca08d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.090+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f99c0 0x55ddfca07cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 -- 172.21.15.28:0/2517025101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f99c0 msgr2=0x55ddfca07cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f99c0 0x55ddfca07cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 -- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 msgr2=0x55ddfca08d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfca08d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 -- 172.21.15.28:0/2517025101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddfca092f0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94ba59c640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x55ddfc9ebcf0 tx=0x7f94ac005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bacc8bf8aadbc8c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ac005650 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f94ac00f9a0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b0004c80 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94ac00fba0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94c342d740 10 client.16713 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:45.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ddfca75b40 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.091+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b0004c80 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f94ac0047f0 con 0x55ddfc9f8fd0 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 10 client.16713 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 10 client.16713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 20 client.16713 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca76510 0x55ddfca78930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 4 client.16713 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ddfca01440 con 0x55ddfca76510 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca76510 0x55ddfca78930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94a1ffb640 10 client.16713 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.092+0000 7f94b9d9b640 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca76510 0x55ddfca78930 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce929f08bdf93b2 server_cookie=a0c8487ff700e577 in_seq=0 out_seq=0 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94a1ffb640 1 -- 172.21.15.28:0/2517025101 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ddfca01440 con 0x55ddfca76510 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94a1ffb640 10 client.16713 handle_command_reply: tid=0 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 client.16713 shutdown 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 msgr2=0x55ddfca077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x55ddfc9ebcf0 tx=0x7f94ac005ab0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 shutdown_connections 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddfca76510 0x55ddfca78930 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddfc9feb30 0x55ddfca08d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddfc9f99c0 0x55ddfca07cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 --2- 172.21.15.28:0/2517025101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddfc9f8fd0 0x55ddfca077b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 >> 172.21.15.28:0/2517025101 conn(0x55ddfc9f0130 msgr2=0x55ddfc9eeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94a1ffb640 0 client.16713 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 shutdown_connections 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 1 -- 172.21.15.28:0/2517025101 wait complete. 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.095+0000 7f94c342d740 20 client.16713 trim_cache size 0 max 16384 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:17:45.101 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "87b264fd-191c-4d9f-965c-9f9218c814c4", 2024-03-22T02:17:45.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:17:45.102 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:45.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 msgr2=0x7f94a40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 0x7f94a40a53e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f94b4023000 tx=0x7f94b4020000 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 msgr2=0x7f94bc198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 0x7f94bc198430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009230 tx=0x7f94b002c650 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 shutdown_connections 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f94a40a2f20 0x7f94a40a53e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94bc102650 0x7f94bc19f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f94bc101c90 0x7f94bc198430 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 --2- 172.21.15.28:0/1112338811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f94bc100aa0 0x7f94bc197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 >> 172.21.15.28:0/1112338811 conn(0x7f94bc0fc030 msgr2=0x7f94bc1050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 shutdown_connections 2024-03-22T02:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.096+0000 7f94c0847640 1 -- 172.21.15.28:0/1112338811 wait complete. 2024-03-22T02:17:45.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:17:45.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:17:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 -- 172.21.15.28:0/3216090269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24101c80 msgr2=0x7fee24102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/3216090269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24101c80 0x7fee24102100 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:17:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 -- 172.21.15.28:0/3216090269 shutdown_connections 2024-03-22T02:17:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/3216090269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 0x7fee24108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/3216090269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24101c80 0x7fee24102100 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/3216090269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24100a90 0x7fee24100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 -- 172.21.15.28:0/3216090269 >> 172.21.15.28:0/3216090269 conn(0x7fee240fc020 msgr2=0x7fee240fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.213+0000 7fee2b1e9640 1 -- 172.21.15.28:0/3216090269 shutdown_connections 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 -- 172.21.15.28:0/3216090269 wait complete. 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 Processor -- start 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 -- start start 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24101c80 0x7fee24197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 0x7fee2419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410c710 con 0x7fee24100a90 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410c590 con 0x7fee24102640 2024-03-22T02:17:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee2b1e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee2410c890 con 0x7fee24101c80 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee28f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee28f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53378/0 (socket says 172.21.15.28:53378) 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee28f85640 1 -- 172.21.15.28:0/2196554864 learned_addr learned my addr 172.21.15.28:0/2196554864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee29786640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 0x7fee2419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.214+0000 7fee23fff640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24101c80 0x7fee24197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 -- 172.21.15.28:0/2196554864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24101c80 msgr2=0x7fee24197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24101c80 0x7fee24197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 -- 172.21.15.28:0/2196554864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 msgr2=0x7fee2419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 0x7fee2419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 -- 172.21.15.28:0/2196554864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2419e9f0 con 0x7fee24100a90 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee28f85640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00eb70 tx=0x7fee1c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee21ffb640 1 -- 172.21.15.28:0/2196554864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c003b70 con 0x7fee24100a90 2024-03-22T02:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2419ece0 con 0x7fee24100a90 2024-03-22T02:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee21ffb640 1 -- 172.21.15.28:0/2196554864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1c00ee70 con 0x7fee24100a90 2024-03-22T02:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee21ffb640 1 -- 172.21.15.28:0/2196554864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c010830 con 0x7fee24100a90 2024-03-22T02:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.215+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2419f1b0 con 0x7fee24100a90 2024-03-22T02:17:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.217+0000 7fee21ffb640 1 -- 172.21.15.28:0/2196554864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fee1c004020 con 0x7fee24100a90 2024-03-22T02:17:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 Processor -- start 2024-03-22T02:17:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 -- start start 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee21ffb640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687153a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871542420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55687149f8c0 con 0x55687153ae20 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556871447c20 con 0x55687153ff90 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55687149b870 con 0x55687153a430 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee29786640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53384/0 (socket says 172.21.15.28:53384) 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee23fff640 1 -- 172.21.15.28:0/139358662 learned_addr learned my addr 172.21.15.28:0/139358662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee28f85640 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687153a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee29786640 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871542420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.218+0000 7fee21ffb640 1 -- 172.21.15.28:0/2196554864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fee1c0cfed0 con 0x7fee24100a90 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 -- 172.21.15.28:0/139358662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 msgr2=0x55687153a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687153a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 -- 172.21.15.28:0/139358662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 msgr2=0x556871542420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871542420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 -- 172.21.15.28:0/139358662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55687146c150 con 0x55687153ae20 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee29786640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 0x7fee100a53e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fee14009710 tx=0x7fee140098b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee23fff640 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe519626c36f3347 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee1affd640 1 -- 172.21.15.28:0/139358662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c005d50 con 0x55687153ae20 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee1affd640 1 -- 172.21.15.28:0/139358662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0c02cb80 con 0x55687153ae20 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.219+0000 7fee1affd640 1 -- 172.21.15.28:0/139358662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c004920 con 0x55687153ae20 2024-03-22T02:17:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- 172.21.15.28:0/139358662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 msgr2=0x55687153f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c009560 comp rx=0 tx=0).stop 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- 172.21.15.28:0/139358662 shutdown_connections 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871542420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687153f920 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/139358662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687153a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- 172.21.15.28:0/139358662 >> 172.21.15.28:0/139358662 conn(0x5568715359a0 msgr2=0x556871535da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- 172.21.15.28:0/139358662 shutdown_connections 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- 172.21.15.28:0/139358662 wait complete. 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 -- start start 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.220+0000 7fee2ddd1740 1 Processor -- start 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556871542ee0 con 0x55687153ff90 2024-03-22T02:17:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55687145b130 con 0x55687153a430 2024-03-22T02:17:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556871543060 con 0x55687153ae20 2024-03-22T02:17:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 10 client.0 _ll_register_callbacks cb 0x55687145ab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee23fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53390/0 (socket says 172.21.15.28:53390) 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee28f85640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 -- 172.21.15.28:0/3367879513 learned_addr learned my addr 172.21.15.28:0/3367879513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 -- 172.21.15.28:0/3367879513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 msgr2=0x556871544e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 -- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153a430 msgr2=0x5568715448d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 -- 172.21.15.28:0/3367879513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55687127d7d0 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee29786640 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee14004440 tx=0x7fee14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a211573cb8a9e06b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee03fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1400f8f0 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fee1400fa90 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0c02cb80 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.221+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee1400fc90 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 10 client.16725 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568711b33f0 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0c02cb80 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fee1401f070 con 0x55687153ff90 2024-03-22T02:17:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 10 client.16725 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 20 client.16725 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556871550800 0x556871552cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 4 client.16725 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568714ace00 con 0x556871550800 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.222+0000 7fee28f85640 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556871550800 0x556871552cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.223+0000 7fee03fff640 10 client.16725 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.223+0000 7fee28f85640 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556871550800 0x556871552cc0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b089c6db88989039 server_cookie=f9fb3f48d40678a8 in_seq=0 out_seq=0 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.223+0000 7fee03fff640 1 -- 172.21.15.28:0/3367879513 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568714ace00 con 0x556871550800 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.223+0000 7fee03fff640 10 client.16725 handle_command_reply: tid=0 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 client.16725 shutdown 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 msgr2=0x55687154be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fee14004440 tx=0x7fee14004730 comp rx=0 tx=0).stop 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 shutdown_connections 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556871550800 0x556871552cc0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ff90 0x55687154be50 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/3367879513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee03fff640 0 client.16725 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 >> 172.21.15.28:0/3367879513 conn(0x5568715359a0 msgr2=0x556871534c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 shutdown_connections 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 1 -- 172.21.15.28:0/3367879513 wait complete. 2024-03-22T02:17:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.224+0000 7fee2ddd1740 20 client.16725 trim_cache size 0 max 16384 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 Processor -- start 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 -- start start 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871534370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee241ac0b0 con 0x55687153a430 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.241+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee241a9980 con 0x55687153ff90 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53396/0 (socket says 172.21.15.28:53396) 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 -- 172.21.15.28:0/2056295004 learned_addr learned my addr 172.21.15.28:0/2056295004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee241a77d0 con 0x55687153ae20 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee29786640 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871534370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee23fff640 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 -- 172.21.15.28:0/2056295004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 msgr2=0x556871544e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 -- 172.21.15.28:0/2056295004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 msgr2=0x556871534370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871534370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 -- 172.21.15.28:0/2056295004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568711730f0 con 0x55687153a430 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee28f85640 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee1c004730 tx=0x7fee1c0cd430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713c9c903fa4efab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee1a7fc640 1 -- 172.21.15.28:0/2056295004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c09c890 con 0x55687153a430 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee1a7fc640 1 -- 172.21.15.28:0/2056295004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1c09c190 con 0x55687153a430 2024-03-22T02:17:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.242+0000 7fee1a7fc640 1 -- 172.21.15.28:0/2056295004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0cd670 con 0x55687153a430 2024-03-22T02:17:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 msgr2=0x5568715448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fee1c004730 tx=0x7fee1c0cd430 comp rx=0 tx=0).stop 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 shutdown_connections 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x556871534370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153ae20 0x556871544e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/2056295004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153a430 0x5568715448d0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 >> 172.21.15.28:0/2056295004 conn(0x5568715359a0 msgr2=0x556871535d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 shutdown_connections 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- 172.21.15.28:0/2056295004 wait complete. 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 -- start start 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.243+0000 7fee2ddd1740 1 Processor -- start 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687154b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x5568715b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee241a7e40 con 0x55687153ae20 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee241a92b0 con 0x55687153ff90 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee241a8510 con 0x55687153a430 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee2ddd1740 10 client.0 _ll_register_callbacks cb 0x7fee14004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53410/0 (socket says 172.21.15.28:53410) 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 -- 172.21.15.28:0/959887895 learned_addr learned my addr 172.21.15.28:0/959887895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee28f85640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687154b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee29786640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x5568715b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 -- 172.21.15.28:0/959887895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 msgr2=0x55687154b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687154b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 -- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 msgr2=0x5568715b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x5568715b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 -- 172.21.15.28:0/959887895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568715b78b0 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee29786640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x5568715b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee23fff640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50f7b2ca28416dd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee03fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.244+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0c04d060 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee1c09c190 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fee0c005230 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fee0c005d50 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 10 client.16731 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568715b8bf0 con 0x55687153ae20 2024-03-22T02:17:45.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1c09c190 con 0x55687153ae20 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fee0c003680 con 0x55687153ae20 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 10 client.16731 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 10 client.16731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 20 client.16731 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568715b91b0 0x5568715bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 4 client.16731 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5568714a7150 con 0x5568715b91b0 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.245+0000 7fee28f85640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568715b91b0 0x5568715bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee03fff640 10 client.16731 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee28f85640 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568715b91b0 0x5568715bb5a0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa02dd4cbc1631d8 server_cookie=90c55906d59f834 in_seq=0 out_seq=0 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee03fff640 1 -- 172.21.15.28:0/959887895 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5568714a7150 con 0x5568715b91b0 2024-03-22T02:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee03fff640 10 client.16731 handle_command_reply: tid=0 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 client.16731 shutdown 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 msgr2=0x55687154bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fee0c009900 tx=0x7fee0c004270 comp rx=0 tx=0).stop 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 shutdown_connections 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568715b91b0 0x5568715bb5a0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55687153ff90 0x5568715b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55687153ae20 0x55687154bb40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee03fff640 0 client.16731 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 --2- 172.21.15.28:0/959887895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55687153a430 0x55687154b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 >> 172.21.15.28:0/959887895 conn(0x5568715359a0 msgr2=0x556871544870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 shutdown_connections 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 1 -- 172.21.15.28:0/959887895 wait complete. 2024-03-22T02:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.246+0000 7fee2ddd1740 20 client.16731 trim_cache size 0 max 16384 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "87b264fd-191c-4d9f-965c-9f9218c814c4": { 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "87b264fd-191c-4d9f-965c-9f9218c814c4", 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:17:45.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:17:45.256 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:17:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 msgr2=0x7fee100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 0x7fee100a53e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fee14009710 tx=0x7fee140098b0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 msgr2=0x7fee24196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00eb70 tx=0x7fee1c00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 shutdown_connections 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fee100a2f20 0x7fee100a53e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee24102640 0x7fee2419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee24101c80 0x7fee24197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 --2- 172.21.15.28:0/2196554864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24100a90 0x7fee24196dd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 >> 172.21.15.28:0/2196554864 conn(0x7fee240fc020 msgr2=0x7fee24105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 shutdown_connections 2024-03-22T02:17:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:17:45.247+0000 7fee2b1e9640 1 -- 172.21.15.28:0/2196554864 wait complete. 2024-03-22T02:17:45.258 INFO:tasks.cephfs.filesystem:scrub status for tag:87b264fd-191c-4d9f-965c-9f9218c814c4 - {'path': '/', 'tag': '87b264fd-191c-4d9f-965c-9f9218c814c4', 'options': 'recursive,force'} 2024-03-22T02:17:45.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 255972 11.96 MB/sec execute 27 sec latency 9.043 ms 2024-03-22T02:17:46.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 258771 12.05 MB/sec execute 28 sec latency 14.267 ms 2024-03-22T02:17:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: pgmap v483: 129 pgs: 129 active+clean; 51 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 468 op/s 2024-03-22T02:17:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:17:46.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:17:46.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:17:46.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: pgmap v483: 129 pgs: 129 active+clean; 51 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 468 op/s 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: pgmap v483: 129 pgs: 129 active+clean; 51 MiB data, 896 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.8 MiB/s wr, 468 op/s 2024-03-22T02:17:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:17:46.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:17:46.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:17:46.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:46.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:17:47.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 260492 11.97 MB/sec execute 29 sec latency 52.684 ms 2024-03-22T02:17:48.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 262799 12.03 MB/sec execute 30 sec latency 85.755 ms 2024-03-22T02:17:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:48 smithi120 ceph-mon[36803]: pgmap v484: 129 pgs: 129 active+clean; 52 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:17:48.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:48 smithi083 ceph-mon[36340]: pgmap v484: 129 pgs: 129 active+clean; 52 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:17:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:48 smithi028 ceph-mon[31916]: pgmap v484: 129 pgs: 129 active+clean; 52 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:17:49.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 264915 11.88 MB/sec execute 31 sec latency 28.414 ms 2024-03-22T02:17:50.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 266661 11.89 MB/sec execute 32 sec latency 63.275 ms 2024-03-22T02:17:50.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:50 smithi120 ceph-mon[36803]: pgmap v485: 129 pgs: 129 active+clean; 43 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:17:50.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:50 smithi083 ceph-mon[36340]: pgmap v485: 129 pgs: 129 active+clean; 43 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:17:50.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:50 smithi028 ceph-mon[31916]: pgmap v485: 129 pgs: 129 active+clean; 43 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:17:51.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 269435 11.96 MB/sec execute 33 sec latency 26.807 ms 2024-03-22T02:17:52.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 271443 11.91 MB/sec execute 34 sec latency 27.974 ms 2024-03-22T02:17:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:52 smithi120 ceph-mon[36803]: pgmap v486: 129 pgs: 129 active+clean; 47 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:17:52.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:52 smithi083 ceph-mon[36340]: pgmap v486: 129 pgs: 129 active+clean; 47 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:17:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:52 smithi028 ceph-mon[31916]: pgmap v486: 129 pgs: 129 active+clean; 47 MiB data, 872 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:17:53.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 273605 11.96 MB/sec execute 35 sec latency 67.402 ms 2024-03-22T02:17:54.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 276171 11.98 MB/sec execute 36 sec latency 86.343 ms 2024-03-22T02:17:54.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:54 smithi120 ceph-mon[36803]: pgmap v487: 129 pgs: 129 active+clean; 47 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:17:54.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:54 smithi083 ceph-mon[36340]: pgmap v487: 129 pgs: 129 active+clean; 47 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:17:54.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:54 smithi028 ceph-mon[31916]: pgmap v487: 129 pgs: 129 active+clean; 47 MiB data, 847 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:17:55.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 277944 11.89 MB/sec execute 37 sec latency 27.483 ms 2024-03-22T02:17:55.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:55 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:17:55.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:55 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:17:55.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:55 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:17:56.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 280677 12.02 MB/sec execute 38 sec latency 14.959 ms 2024-03-22T02:17:56.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:56 smithi120 ceph-mon[36803]: pgmap v488: 129 pgs: 129 active+clean; 51 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 448 op/s 2024-03-22T02:17:56.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:56 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:17:56.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:56 smithi083 ceph-mon[36340]: pgmap v488: 129 pgs: 129 active+clean; 51 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 448 op/s 2024-03-22T02:17:56.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:56 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:17:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:56 smithi028 ceph-mon[31916]: pgmap v488: 129 pgs: 129 active+clean; 51 MiB data, 870 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.6 MiB/s wr, 448 op/s 2024-03-22T02:17:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:56 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- 172.21.15.28:0/3223413944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c103770 msgr2=0x7f364c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 --2- 172.21.15.28:0/3223413944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c103770 0x7f364c10a000 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f3644009ca0 tx=0x7f364402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- 172.21.15.28:0/3223413944 shutdown_connections 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 --2- 172.21.15.28:0/3223413944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c103770 0x7f364c10a000 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 --2- 172.21.15.28:0/3223413944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c102db0 0x7f364c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 --2- 172.21.15.28:0/3223413944 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- 172.21.15.28:0/3223413944 >> 172.21.15.28:0/3223413944 conn(0x7f364c0fd150 msgr2=0x7f364c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- 172.21.15.28:0/3223413944 shutdown_connections 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- 172.21.15.28:0/3223413944 wait complete. 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 Processor -- start 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.814+0000 7f365393a640 1 -- start start 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c103770 0x7f364c19f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10d840 con 0x7f364c102db0 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10d6c0 con 0x7f364c101bc0 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f364c10d9c0 con 0x7f364c103770 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34432/0 (socket says 172.21.15.28:34432) 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f36516d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 -- 172.21.15.28:0/1362358553 learned_addr learned my addr 172.21.15.28:0/1362358553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3651ed7640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c103770 0x7f364c19f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 -- 172.21.15.28:0/1362358553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c103770 msgr2=0x7f364c19f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c103770 0x7f364c19f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 -- 172.21.15.28:0/1362358553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 msgr2=0x7f364c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 -- 172.21.15.28:0/1362358553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f364c19f9e0 con 0x7f364c102db0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f36516d6640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f363c00e9f0 tx=0x7f363c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f36427fc640 1 -- 172.21.15.28:0/1362358553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c00cd80 con 0x7f364c102db0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f36427fc640 1 -- 172.21.15.28:0/1362358553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f363c005500 con 0x7f364c102db0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f364c19fcd0 con 0x7f364c102db0 2024-03-22T02:17:56.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f36427fc640 1 -- 172.21.15.28:0/1362358553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c010690 con 0x7f364c102db0 2024-03-22T02:17:56.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.815+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f364c1a01d0 con 0x7f364c102db0 2024-03-22T02:17:56.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36427fc640 1 -- 172.21.15.28:0/1362358553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f363c0046c0 con 0x7f364c102db0 2024-03-22T02:17:56.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 Processor -- start 2024-03-22T02:17:56.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 -- start start 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36427fc640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 0x7f36380a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b90de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36427fc640 1 -- 172.21.15.28:0/1362358553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f363c0cfb60 con 0x7f364c102db0 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f3651ed7640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 0x7f36380a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570c4aec040 con 0x5570c4b8c2e0 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5570c4a949e0 con 0x5570c4b91450 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.818+0000 7f36565f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5570c4ae7ff0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36516d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b90de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36516d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60162/0 (socket says 172.21.15.28:60162) 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36516d6640 1 -- 172.21.15.28:0/1306997399 learned_addr learned my addr 172.21.15.28:0/1306997399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3651ed7640 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 -- 172.21.15.28:0/1306997399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 msgr2=0x5570c4b8bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 -- 172.21.15.28:0/1306997399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 msgr2=0x5570c4b938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 -- 172.21.15.28:0/1306997399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c4800210 con 0x5570c4b8c2e0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3650ed5640 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b90de0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f363c0cd250 tx=0x7f363c098250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e246d89ba955e9b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f3651ed7640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 0x7f36380a5500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3644005e30 tx=0x7f364402ce40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36377fe640 1 -- 172.21.15.28:0/1306997399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c09be80 con 0x5570c4b8c2e0 2024-03-22T02:17:56.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36377fe640 1 -- 172.21.15.28:0/1306997399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f363c09ffd0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.819+0000 7f36377fe640 1 -- 172.21.15.28:0/1306997399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c09b2d0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- 172.21.15.28:0/1306997399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 msgr2=0x5570c4b90de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b90de0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f363c0cd250 tx=0x7f363c098250 comp rx=0 tx=0).stop 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- 172.21.15.28:0/1306997399 shutdown_connections 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b90de0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 --2- 172.21.15.28:0/1306997399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- 172.21.15.28:0/1306997399 >> 172.21.15.28:0/1306997399 conn(0x5570c4b82a50 msgr2=0x5570c4b82e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- 172.21.15.28:0/1306997399 shutdown_connections 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- 172.21.15.28:0/1306997399 wait complete. 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 -- start start 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.820+0000 7f36565f8740 1 Processor -- start 2024-03-22T02:17:56.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 0x5570c4b95d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570c4b943a0 con 0x5570c4b91450 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5570c4aa7db0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5570c4b94520 con 0x5570c4b8b8f0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36565f8740 10 client.0 _ll_register_callbacks cb 0x5570c47ffa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3651ed7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36516d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 0x5570c4b95d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3651ed7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34456/0 (socket says 172.21.15.28:34456) 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36516d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60176/0 (socket says 172.21.15.28:60176) 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3651ed7640 1 -- 172.21.15.28:0/2723556351 learned_addr learned my addr 172.21.15.28:0/2723556351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:56.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 -- 172.21.15.28:0/2723556351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 msgr2=0x5570c4b95810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 -- 172.21.15.28:0/2723556351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 msgr2=0x5570c4b9cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 -- 172.21.15.28:0/2723556351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c48cc620 con 0x5570c4b8c2e0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f36516d6640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3651ed7640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3650ed5640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 0x5570c4b95d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x5570c4b93fd0 tx=0x7f363c0984f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d23f53e12e4a195 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3634ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.821+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f363c09b4d0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f363c09ffd0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f363c097bd0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f363c09be80 con 0x5570c4b8c2e0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 10 client.26579 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5570c4ba0100 con 0x5570c4b8c2e0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f363c09ffd0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f363c0b8030 con 0x5570c4b8c2e0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 10 client.26579 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 10 client.26579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 20 client.26579 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4ba1450 0x5570c4ba3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 4 client.26579 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570c4af9580 con 0x5570c4ba1450 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.822+0000 7f36516d6640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4ba1450 0x5570c4ba3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.823+0000 7f36516d6640 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4ba1450 0x5570c4ba3840 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4fe580078ba453 server_cookie=c66f2fe9c1abb207 in_seq=0 out_seq=0 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.823+0000 7f3634ff9640 10 client.26579 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:56.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f3634ff9640 1 -- 172.21.15.28:0/2723556351 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570c4af9580 con 0x5570c4ba1450 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f3634ff9640 10 client.26579 handle_command_reply: tid=0 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 client.26579 shutdown 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 msgr2=0x5570c4b95d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 0x5570c4b95d50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x5570c4b93fd0 tx=0x7f363c0984f0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 shutdown_connections 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4ba1450 0x5570c4ba3840 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b91450 0x5570c4b9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8c2e0 0x5570c4b95d50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f3634ff9640 0 client.26579 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 --2- 172.21.15.28:0/2723556351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8b8f0 0x5570c4b95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 >> 172.21.15.28:0/2723556351 conn(0x5570c4b82a50 msgr2=0x5570c4b81c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 shutdown_connections 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 1 -- 172.21.15.28:0/2723556351 wait complete. 2024-03-22T02:17:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.824+0000 7f36565f8740 20 client.26579 trim_cache size 0 max 16384 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.839+0000 7f36565f8740 1 Processor -- start 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.839+0000 7f36565f8740 1 -- start start 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.839+0000 7f36565f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.839+0000 7f36565f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36565f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b91450 0x5570c4b8d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36565f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1aaf80 con 0x5570c4b8c2e0 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36565f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1a8850 con 0x5570c4b8b8f0 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36565f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1a66a0 con 0x5570c4b91450 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44042/0 (socket says 172.21.15.28:44042) 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3650ed5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 -- 172.21.15.28:0/3393135993 learned_addr learned my addr 172.21.15.28:0/3393135993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3651ed7640 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b91450 0x5570c4b8d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 -- 172.21.15.28:0/3393135993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b91450 msgr2=0x5570c4b8d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b91450 0x5570c4b8d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 -- 172.21.15.28:0/3393135993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 msgr2=0x5570c4b9cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 -- 172.21.15.28:0/3393135993 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c4b948f0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3650ed5640 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f36516d6640 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3648012a50 tx=0x7f3648005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99f939380a549bab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3636ffd640 1 -- 172.21.15.28:0/3393135993 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648028070 con 0x5570c4b8b8f0 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3636ffd640 1 -- 172.21.15.28:0/3393135993 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36480120d0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.840+0000 7f3636ffd640 1 -- 172.21.15.28:0/3393135993 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36480035f0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- 172.21.15.28:0/3393135993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 msgr2=0x5570c4b8cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3648012a50 tx=0x7f3648005f90 comp rx=0 tx=0).stop 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- 172.21.15.28:0/3393135993 shutdown_connections 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b91450 0x5570c4b8d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 --2- 172.21.15.28:0/3393135993 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b8b8f0 0x5570c4b8cd80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- 172.21.15.28:0/3393135993 >> 172.21.15.28:0/3393135993 conn(0x5570c4b82a50 msgr2=0x5570c4b85cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:56.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- 172.21.15.28:0/3393135993 shutdown_connections 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- 172.21.15.28:0/3393135993 wait complete. 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 -- start start 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.841+0000 7f36565f8740 1 Processor -- start 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b9ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1a6d10 con 0x5570c4b8b8f0 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1a8180 con 0x5570c4b91450 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f364c1a73e0 con 0x5570c4b8c2e0 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 10 client.0 _ll_register_callbacks cb 0x7f3648009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34486/0 (socket says 172.21.15.28:34486) 2024-03-22T02:17:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 -- 172.21.15.28:0/472248246 learned_addr learned my addr 172.21.15.28:0/472248246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3650ed5640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3651ed7640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b9ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 -- 172.21.15.28:0/472248246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8c2e0 msgr2=0x5570c4b9aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 -- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 msgr2=0x5570c4b9ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b9ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 -- 172.21.15.28:0/472248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c4b9c000 con 0x5570c4b8b8f0 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3651ed7640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b9ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36516d6640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3648026040 tx=0x7f3648009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08d24d6e2a9bf80 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3634ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:17:56.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3648032060 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36480120d0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f36480226e0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3648028040 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 10 client.16749 fetch_fsmap learned FSMap version 29 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.842+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5570c4c07730 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36480120d0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f36480053b0 con 0x5570c4b8b8f0 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 10 client.16749 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 10 client.16749 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 20 client.16749 populate_metadata read hostname 'smithi028' 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4c08090 0x5570c4c0a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 4 client.16749 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:17:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5570c4af38d0 con 0x5570c4c08090 2024-03-22T02:17:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f3650ed5640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4c08090 0x5570c4c0a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:17:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f3634ff9640 10 client.16749 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:56.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:56.843+0000 7f3650ed5640 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4c08090 0x5570c4c0a480 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=795ee0a42262e1cb server_cookie=71e1820750bb29ba in_seq=0 out_seq=0 2024-03-22T02:17:57.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f3634ff9640 1 -- 172.21.15.28:0/472248246 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5570c4af38d0 con 0x5570c4c08090 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f3634ff9640 10 client.16749 handle_command_reply: tid=0 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 client.16749 shutdown 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 msgr2=0x5570c4b9a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f3648026040 tx=0x7f3648009e20 comp rx=0 tx=0).stop 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 shutdown_connections 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5570c4c08090 0x5570c4c0a480 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5570c4b91450 0x5570c4b9ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5570c4b8c2e0 0x5570c4b9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 --2- 172.21.15.28:0/472248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c4b8b8f0 0x5570c4b9a530 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f3634ff9640 0 client.16749 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.049+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 >> 172.21.15.28:0/472248246 conn(0x5570c4b82a50 msgr2=0x5570c4b90cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 shutdown_connections 2024-03-22T02:17:57.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f36565f8740 1 -- 172.21.15.28:0/472248246 wait complete. 2024-03-22T02:17:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f36565f8740 20 client.16749 trim_cache size 0 max 16384 2024-03-22T02:17:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:17:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:17:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:17:57.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 msgr2=0x7f36380a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 0x7f36380a5500 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3644005e30 tx=0x7f364402ce40 comp rx=0 tx=0).stop 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 msgr2=0x7f364c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f363c00e9f0 tx=0x7f363c00edf0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 shutdown_connections 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f36380a3040 0x7f36380a5500 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f364c103770 0x7f364c19f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f364c102db0 0x7f364c198300 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 --2- 172.21.15.28:0/1362358553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f364c101bc0 0x7f364c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:17:57.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 >> 172.21.15.28:0/1362358553 conn(0x7f364c0fd150 msgr2=0x7f364c105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:17:57.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 shutdown_connections 2024-03-22T02:17:57.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:17:57.050+0000 7f365393a640 1 -- 172.21.15.28:0/1362358553 wait complete. 2024-03-22T02:17:57.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 282739 11.91 MB/sec execute 39 sec latency 136.059 ms 2024-03-22T02:17:58.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 284506 11.90 MB/sec execute 40 sec latency 43.673 ms 2024-03-22T02:17:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:17:58 smithi120 ceph-mon[36803]: pgmap v489: 129 pgs: 129 active+clean; 51 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-03-22T02:17:58.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:17:58 smithi083 ceph-mon[36340]: pgmap v489: 129 pgs: 129 active+clean; 51 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-03-22T02:17:58.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:17:58 smithi028 ceph-mon[31916]: pgmap v489: 129 pgs: 129 active+clean; 51 MiB data, 874 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-03-22T02:17:59.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 287417 12.04 MB/sec execute 41 sec latency 10.038 ms 2024-03-22T02:18:00.413 INFO:tasks.workunit.client.0.smithi028.stdout: 1 289681 11.95 MB/sec execute 42 sec latency 28.287 ms 2024-03-22T02:18:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:00 smithi120 ceph-mon[36803]: pgmap v490: 129 pgs: 129 active+clean; 50 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 432 op/s 2024-03-22T02:18:00.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:00 smithi083 ceph-mon[36340]: pgmap v490: 129 pgs: 129 active+clean; 50 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 432 op/s 2024-03-22T02:18:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:00 smithi028 ceph-mon[31916]: pgmap v490: 129 pgs: 129 active+clean; 50 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 432 op/s 2024-03-22T02:18:01.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 291667 11.97 MB/sec execute 43 sec latency 30.133 ms 2024-03-22T02:18:02.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 294509 12.09 MB/sec execute 44 sec latency 20.004 ms 2024-03-22T02:18:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:02 smithi120 ceph-mon[36803]: pgmap v491: 129 pgs: 129 active+clean; 44 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 412 op/s 2024-03-22T02:18:02.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:02 smithi083 ceph-mon[36340]: pgmap v491: 129 pgs: 129 active+clean; 44 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 412 op/s 2024-03-22T02:18:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:02 smithi028 ceph-mon[31916]: pgmap v491: 129 pgs: 129 active+clean; 44 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 412 op/s 2024-03-22T02:18:03.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 295675 11.90 MB/sec execute 45 sec latency 92.510 ms 2024-03-22T02:18:04.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 298254 12.00 MB/sec execute 46 sec latency 16.864 ms 2024-03-22T02:18:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:04 smithi120 ceph-mon[36803]: pgmap v492: 129 pgs: 129 active+clean; 44 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 364 op/s 2024-03-22T02:18:04.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:04 smithi083 ceph-mon[36340]: pgmap v492: 129 pgs: 129 active+clean; 44 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 364 op/s 2024-03-22T02:18:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:04 smithi028 ceph-mon[31916]: pgmap v492: 129 pgs: 129 active+clean; 44 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 364 op/s 2024-03-22T02:18:05.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 300565 11.92 MB/sec execute 47 sec latency 36.053 ms 2024-03-22T02:18:06.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 302502 11.92 MB/sec execute 48 sec latency 28.601 ms 2024-03-22T02:18:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:06 smithi120 ceph-mon[36803]: pgmap v493: 129 pgs: 129 active+clean; 46 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2024-03-22T02:18:06.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:06 smithi083 ceph-mon[36340]: pgmap v493: 129 pgs: 129 active+clean; 46 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2024-03-22T02:18:06.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:06 smithi028 ceph-mon[31916]: pgmap v493: 129 pgs: 129 active+clean; 46 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.4 MiB/s wr, 448 op/s 2024-03-22T02:18:07.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 305309 12.05 MB/sec execute 49 sec latency 16.814 ms 2024-03-22T02:18:08.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 306980 11.92 MB/sec execute 50 sec latency 64.796 ms 2024-03-22T02:18:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:08 smithi120 ceph-mon[36803]: pgmap v494: 129 pgs: 129 active+clean; 48 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 399 op/s 2024-03-22T02:18:08.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:08 smithi083 ceph-mon[36340]: pgmap v494: 129 pgs: 129 active+clean; 48 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 399 op/s 2024-03-22T02:18:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:08 smithi028 ceph-mon[31916]: pgmap v494: 129 pgs: 129 active+clean; 48 MiB data, 981 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 399 op/s 2024-03-22T02:18:09.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 309063 11.96 MB/sec execute 51 sec latency 21.904 ms 2024-03-22T02:18:09.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:18:09.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:18:09.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:18:09.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:09 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:18:09.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:18:09.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:18:09.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:18:09.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:09 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:18:09.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:18:09.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:18:09.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:18:09.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:09 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:18:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- 172.21.15.28:0/936409052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac101cb0 msgr2=0x7f03ac102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 --2- 172.21.15.28:0/936409052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac101cb0 0x7f03ac102130 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f039c009900 tx=0x7f039c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- 172.21.15.28:0/936409052 shutdown_connections 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 --2- 172.21.15.28:0/936409052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac102670 0x7f03ac108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 --2- 172.21.15.28:0/936409052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac101cb0 0x7f03ac102130 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 --2- 172.21.15.28:0/936409052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac100ac0 0x7f03ac100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- 172.21.15.28:0/936409052 >> 172.21.15.28:0/936409052 conn(0x7f03ac0fc030 msgr2=0x7f03ac0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- 172.21.15.28:0/936409052 shutdown_connections 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- 172.21.15.28:0/936409052 wait complete. 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 Processor -- start 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.140+0000 7f03b3439640 1 -- start start 2024-03-22T02:18:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 0x7f03ac197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac101cb0 0x7f03ac198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac10d860 con 0x7f03ac102670 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac10d6e0 con 0x7f03ac100ac0 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b3439640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac10d9e0 con 0x7f03ac101cb0 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b19d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 0x7f03ac197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b19d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60754/0 (socket says 172.21.15.28:60754) 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac101cb0 0x7f03ac198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b19d6640 1 -- 172.21.15.28:0/4003620589 learned_addr learned my addr 172.21.15.28:0/4003620589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 -- 172.21.15.28:0/4003620589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac101cb0 msgr2=0x7f03ac198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac101cb0 0x7f03ac198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 -- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 msgr2=0x7f03ac19f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 -- 172.21.15.28:0/4003620589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ac19fa40 con 0x7f03ac100ac0 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b19d6640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03b11d5640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 0x7f03ac197e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e9c0 tx=0x7f03a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.141+0000 7f03a27fc640 1 -- 172.21.15.28:0/4003620589 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a800ccc0 con 0x7f03ac100ac0 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.142+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ac19fd30 con 0x7f03ac100ac0 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.142+0000 7f03a27fc640 1 -- 172.21.15.28:0/4003620589 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a80035f0 con 0x7f03ac100ac0 2024-03-22T02:18:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.142+0000 7f03a27fc640 1 -- 172.21.15.28:0/4003620589 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a800ce60 con 0x7f03ac100ac0 2024-03-22T02:18:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.142+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ac1a01a0 con 0x7f03ac100ac0 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03a27fc640 1 -- 172.21.15.28:0/4003620589 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f03a80037a0 con 0x7f03ac100ac0 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 Processor -- start 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 -- start start 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b96859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b96884d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3b95e5040 con 0x55c3b9686040 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c3b958d9e0 con 0x55c3b9680ed0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b6023740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c3b95e0ff0 con 0x55c3b96804e0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b19d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.144+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b96884d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03a27fc640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 0x7f03980a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:41450/0 (socket says 172.21.15.28:41450) 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b11d5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b96859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 -- 172.21.15.28:0/120786867 learned_addr learned my addr 172.21.15.28:0/120786867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 0x7f03980a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 -- 172.21.15.28:0/120786867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 msgr2=0x55c3b96859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b96859d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 -- 172.21.15.28:0/120786867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 msgr2=0x55c3b96884d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b96884d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03a27fc640 1 -- 172.21.15.28:0/4003620589 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f03a80cfd70 con 0x7f03ac100ac0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 -- 172.21.15.28:0/120786867 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b92f9210 con 0x55c3b96804e0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f03a400ff20 tx=0x7f03a40138f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1014fe986459f3bb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03b19d6640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 0x7f03980a5480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f03a401e000 tx=0x7f03a4020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03977fe640 1 -- 172.21.15.28:0/120786867 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a400f470 con 0x55c3b96804e0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03977fe640 1 -- 172.21.15.28:0/120786867 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a4023030 con 0x55c3b96804e0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.145+0000 7f03977fe640 1 -- 172.21.15.28:0/120786867 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4010690 con 0x55c3b96804e0 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- 172.21.15.28:0/120786867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 msgr2=0x55c3b9680900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f03a400ff20 tx=0x7f03a40138f0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- 172.21.15.28:0/120786867 shutdown_connections 2024-03-22T02:18:10.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b96884d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b96859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 --2- 172.21.15.28:0/120786867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b9680900 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- 172.21.15.28:0/120786867 >> 172.21.15.28:0/120786867 conn(0x55c3b967ba50 msgr2=0x55c3b967be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- 172.21.15.28:0/120786867 shutdown_connections 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- 172.21.15.28:0/120786867 wait complete. 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 -- start start 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.146+0000 7f03b6023740 1 Processor -- start 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b9691a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9688f90 con 0x55c3b9686040 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c3b95a0db0 con 0x55c3b9680ed0 2024-03-22T02:18:10.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c3b9689110 con 0x55c3b96804e0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b6023740 10 client.0 _ll_register_callbacks cb 0x55c3b92f8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b19d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b9691a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b11d5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:32984/0 (socket says 172.21.15.28:32984) 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 -- 172.21.15.28:0/3829875868 learned_addr learned my addr 172.21.15.28:0/3829875868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 -- 172.21.15.28:0/3829875868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 msgr2=0x55c3b968a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 -- 172.21.15.28:0/3829875868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 msgr2=0x55c3b9691a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b9691a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b09d4640 1 -- 172.21.15.28:0/3829875868 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b93c5620 con 0x55c3b9680ed0 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.147+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f039c009900 tx=0x7f039c004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe4323b55ab209e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f0394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c005d50 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03a4023030 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f039c005230 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f039c005430 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f03b6023740 10 client.26597 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c3b9694e30 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.148+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a4023030 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f039c003830 con 0x55c3b9680ed0 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 10 client.26597 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 10 client.26597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 20 client.26597 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9696450 0x55c3b9698880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 4 client.26597 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3b95f2580 con 0x55c3b9696450 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9696450 0x55c3b9698880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f0394ff9640 10 client.26597 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.149+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9696450 0x55c3b9698880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d173b472c18927 server_cookie=d6b87e2b79e827d0 in_seq=0 out_seq=0 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f0394ff9640 1 -- 172.21.15.28:0/3829875868 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3b95f2580 con 0x55c3b9696450 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f0394ff9640 10 client.26597 handle_command_reply: tid=0 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 client.26597 shutdown 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 msgr2=0x55c3b968a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f039c009900 tx=0x7f039c004250 comp rx=0 tx=0).stop 2024-03-22T02:18:10.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 shutdown_connections 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9696450 0x55c3b9698880 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9686040 0x55c3b9691a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 --2- 172.21.15.28:0/3829875868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f0394ff9640 0 client.26597 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 >> 172.21.15.28:0/3829875868 conn(0x55c3b967ba50 msgr2=0x55c3b967ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 shutdown_connections 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 1 -- 172.21.15.28:0/3829875868 wait complete. 2024-03-22T02:18:10.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.150+0000 7f03b6023740 20 client.26597 trim_cache size 0 max 16384 2024-03-22T02:18:10.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.165+0000 7f03b6023740 1 Processor -- start 2024-03-22T02:18:10.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.165+0000 7f03b6023740 1 -- start start 2024-03-22T02:18:10.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b9686040 0x55c3b967ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1ad120 con 0x55c3b96804e0 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1aa9f0 con 0x55c3b9680ed0 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b6023740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a8840 con 0x55c3b9686040 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60782/0 (socket says 172.21.15.28:60782) 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 -- 172.21.15.28:0/1874844956 learned_addr learned my addr 172.21.15.28:0/1874844956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b09d4640 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b19d6640 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b9686040 0x55c3b967ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 -- 172.21.15.28:0/1874844956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b9686040 msgr2=0x55c3b967ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b9686040 0x55c3b967ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 -- 172.21.15.28:0/1874844956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 msgr2=0x55c3b968a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 -- 172.21.15.28:0/1874844956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b9680120 con 0x55c3b96804e0 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f03b11d5640 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f03a809bd30 tx=0x7f03a80bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfc9a3299ed6c93 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f0396ffd640 1 -- 172.21.15.28:0/1874844956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a809c730 con 0x55c3b96804e0 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f0396ffd640 1 -- 172.21.15.28:0/1874844956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a80b5120 con 0x55c3b96804e0 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.166+0000 7f0396ffd640 1 -- 172.21.15.28:0/1874844956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a80bc0b0 con 0x55c3b96804e0 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- 172.21.15.28:0/1874844956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 msgr2=0x55c3b968a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f03a809bd30 tx=0x7f03a80bc040 comp rx=0 tx=0).stop 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- 172.21.15.28:0/1874844956 shutdown_connections 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b9686040 0x55c3b967ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9680ed0 0x55c3b968a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 --2- 172.21.15.28:0/1874844956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b96804e0 0x55c3b968a480 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- 172.21.15.28:0/1874844956 >> 172.21.15.28:0/1874844956 conn(0x55c3b967ba50 msgr2=0x55c3b967be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- 172.21.15.28:0/1874844956 shutdown_connections 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- 172.21.15.28:0/1874844956 wait complete. 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 -- start start 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.167+0000 7f03b6023740 1 Processor -- start 2024-03-22T02:18:10.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b96f73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9686040 0x55c3b96fe960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a8eb0 con 0x55c3b9680ed0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1aa320 con 0x55c3b9686040 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03ac1a9580 con 0x55c3b96804e0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b6023740 10 client.0 _ll_register_callbacks cb 0x7f039c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60790/0 (socket says 172.21.15.28:60790) 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 -- 172.21.15.28:0/3243733165 learned_addr learned my addr 172.21.15.28:0/3243733165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b19d6640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9686040 0x55c3b96fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b96f73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 -- 172.21.15.28:0/3243733165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 msgr2=0x55c3b96f73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b96f73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 -- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9686040 msgr2=0x55c3b96fe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9686040 0x55c3b96fe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 -- 172.21.15.28:0/3243733165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b96fef50 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f03b09d4640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f039c02c7b0 tx=0x7f039c02c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485adc48e8fe58eb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.168+0000 7f0394ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c005d50 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03a80b5120 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f039c0037f0 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f039c05b050 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 10 client.16767 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c3b9700300 con 0x55c3b9680ed0 2024-03-22T02:18:10.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a80b5120 con 0x55c3b9680ed0 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f039c0347a0 con 0x55c3b9680ed0 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 10 client.16767 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 10 client.16767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 20 client.16767 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9700d00 0x55c3b9703120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 4 client.16767 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c3b95ec8d0 con 0x55c3b9700d00 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.169+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9700d00 0x55c3b9703120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.170+0000 7f03b11d5640 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9700d00 0x55c3b9703120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ac4e6a71ef3d709 server_cookie=7e158976f7936af9 in_seq=0 out_seq=0 2024-03-22T02:18:10.176 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.170+0000 7f0394ff9640 10 client.16767 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f0394ff9640 1 -- 172.21.15.28:0/3243733165 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c3b95ec8d0 con 0x55c3b9700d00 2024-03-22T02:18:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f0394ff9640 10 client.16767 handle_command_reply: tid=0 2024-03-22T02:18:10.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 client.16767 shutdown 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 msgr2=0x55c3b96f7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f039c02c7b0 tx=0x7f039c02c660 comp rx=0 tx=0).stop 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 shutdown_connections 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c3b9700d00 0x55c3b9703120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3b9686040 0x55c3b96fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3b9680ed0 0x55c3b96f7920 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f0394ff9640 0 client.16767 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 --2- 172.21.15.28:0/3243733165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3b96804e0 0x55c3b96f73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 >> 172.21.15.28:0/3243733165 conn(0x55c3b967ba50 msgr2=0x55c3b9685380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 shutdown_connections 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.393+0000 7f03b6023740 1 -- 172.21.15.28:0/3243733165 wait complete. 2024-03-22T02:18:10.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b6023740 20 client.16767 trim_cache size 0 max 16384 2024-03-22T02:18:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:18:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:18:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:18:10.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 msgr2=0x7f03980a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 0x7f03980a5480 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f03a401e000 tx=0x7f03a4020000 comp rx=0 tx=0).stop 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 msgr2=0x7f03ac197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 0x7f03ac197e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e9c0 tx=0x7f03a800ee80 comp rx=0 tx=0).stop 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 shutdown_connections 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03980a2fc0 0x7f03980a5480 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03ac102670 0x7f03ac19f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f03ac101cb0 0x7f03ac198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 --2- 172.21.15.28:0/4003620589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f03ac100ac0 0x7f03ac197e20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 >> 172.21.15.28:0/4003620589 conn(0x7f03ac0fc030 msgr2=0x7f03ac104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 shutdown_connections 2024-03-22T02:18:10.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:10.394+0000 7f03b3439640 1 -- 172.21.15.28:0/4003620589 wait complete. 2024-03-22T02:18:10.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 311068 11.86 MB/sec execute 52 sec latency 104.384 ms 2024-03-22T02:18:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:10 smithi120 ceph-mon[36803]: pgmap v495: 129 pgs: 129 active+clean; 49 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:10 smithi083 ceph-mon[36340]: pgmap v495: 129 pgs: 129 active+clean; 49 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:10 smithi028 ceph-mon[31916]: pgmap v495: 129 pgs: 129 active+clean; 49 MiB data, 994 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:11.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 312900 11.87 MB/sec execute 53 sec latency 43.669 ms 2024-03-22T02:18:12.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 315818 11.98 MB/sec execute 54 sec latency 11.630 ms 2024-03-22T02:18:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:12 smithi120 ceph-mon[36803]: pgmap v496: 129 pgs: 129 active+clean; 49 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-03-22T02:18:12.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:12 smithi083 ceph-mon[36340]: pgmap v496: 129 pgs: 129 active+clean; 49 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-03-22T02:18:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:12 smithi028 ceph-mon[31916]: pgmap v496: 129 pgs: 129 active+clean; 49 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-03-22T02:18:13.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 318039 11.91 MB/sec execute 55 sec latency 28.341 ms 2024-03-22T02:18:14.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 319751 11.91 MB/sec execute 56 sec latency 43.460 ms 2024-03-22T02:18:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:14 smithi120 ceph-mon[36803]: pgmap v497: 129 pgs: 129 active+clean; 47 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2024-03-22T02:18:14.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:14 smithi083 ceph-mon[36340]: pgmap v497: 129 pgs: 129 active+clean; 47 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2024-03-22T02:18:14.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:14 smithi028 ceph-mon[31916]: pgmap v497: 129 pgs: 129 active+clean; 47 MiB data, 1009 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 302 op/s 2024-03-22T02:18:15.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:18:15.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:18:15.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 -- 172.21.15.28:0/2513309654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60102680 msgr2=0x7f4f60108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 --2- 172.21.15.28:0/2513309654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60102680 0x7f4f60108f10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009ca0 tx=0x7f4f5c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 -- 172.21.15.28:0/2513309654 shutdown_connections 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 --2- 172.21.15.28:0/2513309654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60102680 0x7f4f60108f10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 --2- 172.21.15.28:0/2513309654 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60101cc0 0x7f4f60102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 --2- 172.21.15.28:0/2513309654 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 0x7f4f60100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 -- 172.21.15.28:0/2513309654 >> 172.21.15.28:0/2513309654 conn(0x7f4f600fc040 msgr2=0x7f4f600fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 -- 172.21.15.28:0/2513309654 shutdown_connections 2024-03-22T02:18:15.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.367+0000 7f4f6842a640 1 -- 172.21.15.28:0/2513309654 wait complete. 2024-03-22T02:18:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 Processor -- start 2024-03-22T02:18:15.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 -- start start 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 0x7f4f6019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60102680 0x7f4f601a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6010d870 con 0x7f4f60101cc0 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6010d6f0 con 0x7f4f60102680 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f6842a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f6010d9f0 con 0x7f4f60100ad0 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59772/0 (socket says 172.21.15.28:59772) 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f659c5640 1 -- 172.21.15.28:0/1989238707 learned_addr learned my addr 172.21.15.28:0/1989238707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 0x7f4f6019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f669c7640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60102680 0x7f4f601a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f659c5640 1 -- 172.21.15.28:0/1989238707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 msgr2=0x7f4f6019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.368+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 0x7f4f6019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f659c5640 1 -- 172.21.15.28:0/1989238707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60102680 msgr2=0x7f4f601a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60102680 0x7f4f601a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f659c5640 1 -- 172.21.15.28:0/1989238707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f601a1cf0 con 0x7f4f60101cc0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000e9f0 tx=0x7f4f5000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f577fe640 1 -- 172.21.15.28:0/1989238707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5000cd80 con 0x7f4f60101cc0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f577fe640 1 -- 172.21.15.28:0/1989238707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f50010910 con 0x7f4f60101cc0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f601a1fe0 con 0x7f4f60101cc0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f577fe640 1 -- 172.21.15.28:0/1989238707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f50018990 con 0x7f4f60101cc0 2024-03-22T02:18:15.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.369+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f601a24a0 con 0x7f4f60101cc0 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.371+0000 7f4f577fe640 1 -- 172.21.15.28:0/1989238707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4f50010ac0 con 0x7f4f60101cc0 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f577fe640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 0x7f4f4c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 Processor -- start 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f669c7640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 0x7f4f4c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f577fe640 1 -- 172.21.15.28:0/1989238707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4f500cfae0 con 0x7f4f60101cc0 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 -- start start 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x561761302920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x561761305420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617612628c0 con 0x5617612fde20 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56176120ac20 con 0x5617612fd430 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.372+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56176125e870 con 0x561761302f90 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46438/0 (socket says 172.21.15.28:46438) 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x561761302920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f669c7640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 0x7f4f4c0a53e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c006010 tx=0x7f4f5c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 -- 172.21.15.28:0/1714475763 learned_addr learned my addr 172.21.15.28:0/1714475763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f669c7640 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x561761305420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 -- 172.21.15.28:0/1714475763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 msgr2=0x561761305420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x561761305420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 -- 172.21.15.28:0/1714475763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 msgr2=0x561761302920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x561761302920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 -- 172.21.15.28:0/1714475763 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56176122f150 con 0x5617612fd430 2024-03-22T02:18:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009ca0 tx=0x7f4f58011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c69c289551ffda3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f54bfe640 1 -- 172.21.15.28:0/1714475763 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58004220 con 0x5617612fd430 2024-03-22T02:18:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f54bfe640 1 -- 172.21.15.28:0/1714475763 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f580043c0 con 0x5617612fd430 2024-03-22T02:18:15.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.373+0000 7f4f54bfe640 1 -- 172.21.15.28:0/1714475763 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f58019900 con 0x5617612fd430 2024-03-22T02:18:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1714475763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 msgr2=0x5617612fd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4f58009ca0 tx=0x7f4f58011810 comp rx=0 tx=0).stop 2024-03-22T02:18:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1714475763 shutdown_connections 2024-03-22T02:18:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x561761305420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x561761302920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1714475763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x5617612fd850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.374+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1714475763 >> 172.21.15.28:0/1714475763 conn(0x5617612f89a0 msgr2=0x5617612f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.375+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1714475763 shutdown_connections 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.375+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1714475763 wait complete. 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.375+0000 7f4f6b00e740 1 -- start start 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 Processor -- start 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 0x5617613078c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 0x561761307e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x56176130ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561761305f40 con 0x5617612fd430 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56176121e130 con 0x5617612fde20 2024-03-22T02:18:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617613060c0 con 0x561761302f90 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f6b00e740 10 client.0 _ll_register_callbacks cb 0x56176121db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 0x5617613078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 0x5617613078c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59778/0 (socket says 172.21.15.28:59778) 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 0x561761307e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f661c6640 1 -- 172.21.15.28:0/3739063691 learned_addr learned my addr 172.21.15.28:0/3739063691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f669c7640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x56176130ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 -- 172.21.15.28:0/3739063691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 msgr2=0x56176130ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x56176130ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 -- 172.21.15.28:0/3739063691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 msgr2=0x5617613078c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 0x5617613078c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.376+0000 7f4f659c5640 1 -- 172.21.15.28:0/3739063691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617610407d0 con 0x5617612fde20 2024-03-22T02:18:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f669c7640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x56176130ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 0x561761307e00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cd220 tx=0x7f4f5009bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38508af07e61f815 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500bad70 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4f50018b30 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f580043c0 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4f50018d30 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f6b00e740 10 client.26615 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561760f763f0 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.377+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f580043c0 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f4f50097d20 con 0x5617612fde20 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 10 client.26615 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 10 client.26615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 20 client.26615 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761313760 0x561761315bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 4 client.26615 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56176126fe00 con 0x561761313760 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f661c6640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761313760 0x561761315bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f661c6640 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761313760 0x561761315bc0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cbdd735340885c server_cookie=cdbebe201470b8c8 in_seq=0 out_seq=0 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.378+0000 7f4f49ffb640 10 client.26615 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.379+0000 7f4f49ffb640 1 -- 172.21.15.28:0/3739063691 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56176126fe00 con 0x561761313760 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.379+0000 7f4f49ffb640 10 client.26615 handle_command_reply: tid=0 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.379+0000 7f4f6b00e740 1 client.26615 shutdown 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.379+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 msgr2=0x561761307e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.379+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 0x561761307e00 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cd220 tx=0x7f4f5009bf70 comp rx=0 tx=0).stop 2024-03-22T02:18:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 shutdown_connections 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761313760 0x561761315bc0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561761302f90 0x56176130ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fde20 0x561761307e00 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3739063691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fd430 0x5617613078c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 >> 172.21.15.28:0/3739063691 conn(0x5617612f89a0 msgr2=0x5617612f7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f49ffb640 0 client.26615 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 shutdown_connections 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3739063691 wait complete. 2024-03-22T02:18:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.380+0000 7f4f6b00e740 20 client.26615 trim_cache size 0 max 16384 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 Processor -- start 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 -- start start 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fd430 0x5617612ff320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561761302f90 0x5617612ff860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601af4f0 con 0x5617612fde20 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601acdc0 con 0x561761302f90 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601aac10 con 0x5617612fd430 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.399+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59780/0 (socket says 172.21.15.28:59780) 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fd430 0x5617612ff320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 -- 172.21.15.28:0/3985987369 learned_addr learned my addr 172.21.15.28:0/3985987369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f669c7640 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561761302f90 0x5617612ff860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 -- 172.21.15.28:0/3985987369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fd430 msgr2=0x5617612ff320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fd430 0x5617612ff320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 -- 172.21.15.28:0/3985987369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561761302f90 msgr2=0x5617612ff860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561761302f90 0x5617612ff860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 -- 172.21.15.28:0/3985987369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561760f360f0 con 0x5617612fde20 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f659c5640 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cd540 tx=0x7f4f5009a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bd617f1da7a2a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f4bfff640 1 -- 172.21.15.28:0/3985987369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500bad80 con 0x5617612fde20 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f4bfff640 1 -- 172.21.15.28:0/3985987369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f50097c00 con 0x5617612fde20 2024-03-22T02:18:15.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.400+0000 7f4f4bfff640 1 -- 172.21.15.28:0/3985987369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f500a1390 con 0x5617612fde20 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3985987369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 msgr2=0x56176130ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f4f500cd540 tx=0x7f4f5009a3c0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3985987369 shutdown_connections 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561761302f90 0x5617612ff860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617612fde20 0x56176130ee00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/3985987369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fd430 0x5617612ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3985987369 >> 172.21.15.28:0/3985987369 conn(0x5617612f89a0 msgr2=0x5617612fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3985987369 shutdown_connections 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- 172.21.15.28:0/3985987369 wait complete. 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 -- start start 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 Processor -- start 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 0x56176130c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.401+0000 7f4f6b00e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601ab280 con 0x561761302f90 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601ac6f0 con 0x5617612fd430 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f6b00e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f601ab950 con 0x5617612fde20 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f6b00e740 10 client.0 _ll_register_callbacks cb 0x7f4f5009b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f669c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 0x56176130c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f669c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59788/0 (socket says 172.21.15.28:59788) 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f661c6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46470/0 (socket says 172.21.15.28:46470) 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f669c7640 1 -- 172.21.15.28:0/1947594281 learned_addr learned my addr 172.21.15.28:0/1947594281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 -- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 msgr2=0x56176130bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 -- 172.21.15.28:0/1947594281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 msgr2=0x561761377ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 -- 172.21.15.28:0/1947594281 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617613780a0 con 0x5617612fde20 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f669c7640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f659c5640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 0x56176130c2b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4f5009ade0 tx=0x7f4f50098280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ec65b88c880a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5009b630 con 0x5617612fde20 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f50097c00 con 0x5617612fde20 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.402+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f4f500b0170 con 0x5617612fde20 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4f5008e050 con 0x5617612fde20 2024-03-22T02:18:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 10 client.26620 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561761379350 con 0x5617612fde20 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f50097c00 con 0x5617612fde20 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f4f50088c70 con 0x5617612fde20 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 10 client.26620 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 10 client.26620 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 20 client.26620 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761379a60 0x56176137be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 4 client.26620 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56176126a150 con 0x561761379a60 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.403+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761379a60 0x56176137be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f49ffb640 10 client.26620 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f661c6640 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761379a60 0x56176137be80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7c910bc43b715e server_cookie=365954816c116cfb in_seq=0 out_seq=0 2024-03-22T02:18:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f49ffb640 1 -- 172.21.15.28:0/1947594281 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56176126a150 con 0x561761379a60 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f49ffb640 10 client.26620 handle_command_reply: tid=0 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 client.26620 shutdown 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 msgr2=0x56176130c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 0x56176130c2b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4f5009ade0 tx=0x7f4f50098280 comp rx=0 tx=0).stop 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 shutdown_connections 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561761379a60 0x56176137be80 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561761302f90 0x561761377ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617612fde20 0x56176130c2b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 --2- 172.21.15.28:0/1947594281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617612fd430 0x56176130bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f49ffb640 0 client.26620 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 >> 172.21.15.28:0/1947594281 conn(0x5617612f89a0 msgr2=0x5617612fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 shutdown_connections 2024-03-22T02:18:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 1 -- 172.21.15.28:0/1947594281 wait complete. 2024-03-22T02:18:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6b00e740 20 client.26620 trim_cache size 0 max 16384 2024-03-22T02:18:15.412 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:18:15.412 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:18:15.412 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 msgr2=0x7f4f4c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 0x7f4f4c0a53e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c006010 tx=0x7f4f5c032590 comp rx=0 tx=0).stop 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 msgr2=0x7f4f6019a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.404+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f4f5000e9f0 tx=0x7f4f5000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 shutdown_connections 2024-03-22T02:18:15.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f4c0a2f20 0x7f4f4c0a53e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f60102680 0x7f4f601a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f60101cc0 0x7f4f6019a610 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 --2- 172.21.15.28:0/1989238707 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f60100ad0 0x7f4f6019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 >> 172.21.15.28:0/1989238707 conn(0x7f4f600fc040 msgr2=0x7f4f601098d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 shutdown_connections 2024-03-22T02:18:15.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.405+0000 7f4f6842a640 1 -- 172.21.15.28:0/1989238707 wait complete. 2024-03-22T02:18:15.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 322609 11.96 MB/sec execute 57 sec latency 27.258 ms 2024-03-22T02:18:15.416 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:18:15.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- 172.21.15.28:0/1771577662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 msgr2=0x7f463c10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1771577662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c10a010 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4634009ca0 tx=0x7f463402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- 172.21.15.28:0/1771577662 shutdown_connections 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1771577662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c10a010 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1771577662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c102dc0 0x7f463c103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1771577662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c101bd0 0x7f463c101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- 172.21.15.28:0/1771577662 >> 172.21.15.28:0/1771577662 conn(0x7f463c0fd140 msgr2=0x7f463c0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- 172.21.15.28:0/1771577662 shutdown_connections 2024-03-22T02:18:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- 172.21.15.28:0/1771577662 wait complete. 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 Processor -- start 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.523+0000 7f4643ec3640 1 -- start start 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c101bd0 0x7f463c075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c102dc0 0x7f463c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f463c10d850 con 0x7f463c103780 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f463c10d6d0 con 0x7f463c101bd0 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f463c10d9d0 con 0x7f463c102dc0 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59794/0 (socket says 172.21.15.28:59794) 2024-03-22T02:18:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 -- 172.21.15.28:0/1192716155 learned_addr learned my addr 172.21.15.28:0/1192716155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f464145e640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c102dc0 0x7f463c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4641c5f640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c101bd0 0x7f463c075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 -- 172.21.15.28:0/1192716155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c102dc0 msgr2=0x7f463c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c102dc0 0x7f463c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 -- 172.21.15.28:0/1192716155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c101bd0 msgr2=0x7f463c075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c101bd0 0x7f463c075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 -- 172.21.15.28:0/1192716155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c074d50 con 0x7f463c103780 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4642460640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4634037040 tx=0x7f4634032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634041020 con 0x7f463c103780 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4634034cd0 con 0x7f463c103780 2024-03-22T02:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c1a1f20 con 0x7f463c103780 2024-03-22T02:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634039040 con 0x7f463c103780 2024-03-22T02:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.524+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c1a23e0 con 0x7f463c103780 2024-03-22T02:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.526+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c108ef0 con 0x7f463c103780 2024-03-22T02:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.527+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4634044020 con 0x7f463c103780 2024-03-22T02:18:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.527+0000 7f4632ffd640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 0x7f46280a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.527+0000 7f4641c5f640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 0x7f46280a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.527+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f46340f1580 con 0x7f463c103780 2024-03-22T02:18:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.528+0000 7f4641c5f640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 0x7f46280a5460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f463c102c20 tx=0x7f4638005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.532+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f463404b020 con 0x7f463c103780 2024-03-22T02:18:15.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.780+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f463c062830 con 0x7f463c103780 2024-03-22T02:18:15.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.780+0000 7f4632ffd640 1 -- 172.21.15.28:0/1192716155 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11866 (secure 0 0 0) 0x7f46340be030 con 0x7f463c103780 2024-03-22T02:18:15.781 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:18:15.781 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:17:24.693718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:18:15.781 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 msgr2=0x7f46280a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 0x7f46280a5460 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f463c102c20 tx=0x7f4638005f90 comp rx=0 tx=0).stop 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 msgr2=0x7f463c0744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4634037040 tx=0x7f4634032f40 comp rx=0 tx=0).stop 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 shutdown_connections 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46280a2fa0 0x7f46280a5460 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f463c103780 0x7f463c0744c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f463c102dc0 0x7f463c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 --2- 172.21.15.28:0/1192716155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f463c101bd0 0x7f463c075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.782+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 >> 172.21.15.28:0/1192716155 conn(0x7f463c0fd140 msgr2=0x7f463c1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.783+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 shutdown_connections 2024-03-22T02:18:15.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.783+0000 7f4643ec3640 1 -- 172.21.15.28:0/1192716155 wait complete. 2024-03-22T02:18:15.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:18:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.900+0000 7f08ac343640 1 -- 172.21.15.28:0/21737867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4102640 msgr2=0x7f08a4108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.900+0000 7f08ac343640 1 --2- 172.21.15.28:0/21737867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4102640 0x7f08a4108ed0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f089c009ca0 tx=0x7f089c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- 172.21.15.28:0/21737867 shutdown_connections 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- 172.21.15.28:0/21737867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4102640 0x7f08a4108ed0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- 172.21.15.28:0/21737867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4101c80 0x7f08a4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- 172.21.15.28:0/21737867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4100a90 0x7f08a4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- 172.21.15.28:0/21737867 >> 172.21.15.28:0/21737867 conn(0x7f08a40fc020 msgr2=0x7f08a40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- 172.21.15.28:0/21737867 shutdown_connections 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- 172.21.15.28:0/21737867 wait complete. 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 Processor -- start 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- start start 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 0x7f08a41a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08a410c6f0 con 0x7f08a4101c80 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08a410c570 con 0x7f08a4100a90 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08ac343640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08a410c870 con 0x7f08a4102640 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08a98de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08aa0df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08aa0df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46492/0 (socket says 172.21.15.28:46492) 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08aa0df640 1 -- 172.21.15.28:0/1911436192 learned_addr learned my addr 172.21.15.28:0/1911436192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08a98de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59804/0 (socket says 172.21.15.28:59804) 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.901+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 0x7f08a41a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 -- 172.21.15.28:0/1911436192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 msgr2=0x7f08a41a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 0x7f08a41a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 -- 172.21.15.28:0/1911436192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 msgr2=0x7f08a40698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 -- 172.21.15.28:0/1911436192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a41a1ca0 con 0x7f08a4101c80 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08aa0df640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 0x7f08a41a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08a98de640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f089400e9f0 tx=0x7f089400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f089b7fe640 1 -- 172.21.15.28:0/1911436192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089400ccc0 con 0x7f08a4101c80 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f089b7fe640 1 -- 172.21.15.28:0/1911436192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0894010910 con 0x7f08a4101c80 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a41a1f90 con 0x7f08a4101c80 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f089b7fe640 1 -- 172.21.15.28:0/1911436192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894018960 con 0x7f08a4101c80 2024-03-22T02:18:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.902+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a41a24a0 con 0x7f08a4101c80 2024-03-22T02:18:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.904+0000 7f089b7fe640 1 -- 172.21.15.28:0/1911436192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0894010ac0 con 0x7f08a4101c80 2024-03-22T02:18:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 Processor -- start 2024-03-22T02:18:15.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 -- start start 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 0x55d6aa258d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa259380 0x55d6aa25b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa1b88d0 con 0x55d6aa259380 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa160e00 con 0x55d6aa253820 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aeffd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa1b4880 con 0x55d6aa254210 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f089b7fe640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 0x7f08900a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aa8e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aa8e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46494/0 (socket says 172.21.15.28:46494) 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aa8e0640 1 -- 172.21.15.28:0/3405402788 learned_addr learned my addr 172.21.15.28:0/3405402788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08a98de640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa259380 0x55d6aa25b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 0x55d6aa258d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.905+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 0x7f08900a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f089b7fe640 1 -- 172.21.15.28:0/1911436192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f08940cfb10 con 0x7f08a4101c80 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 -- 172.21.15.28:0/3405402788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 msgr2=0x55d6aa253c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 -- 172.21.15.28:0/3405402788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa259380 msgr2=0x55d6aa25b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa259380 0x55d6aa25b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 -- 172.21.15.28:0/3405402788 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6a9ecc3f0 con 0x55d6aa254210 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 0x7f08900a5460 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55d6aa254a70 tx=0x7f089c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 0x55d6aa258d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08a00114c0 tx=0x7f08a0011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb56a2bffc596e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f0898bfe640 1 -- 172.21.15.28:0/3405402788 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0005020 con 0x55d6aa254210 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f0898bfe640 1 -- 172.21.15.28:0/3405402788 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a00047a0 con 0x55d6aa254210 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.906+0000 7f0898bfe640 1 -- 172.21.15.28:0/3405402788 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a000f8b0 con 0x55d6aa254210 2024-03-22T02:18:15.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- 172.21.15.28:0/3405402788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 msgr2=0x55d6aa258d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 0x55d6aa258d10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f08a00114c0 tx=0x7f08a0011980 comp rx=0 tx=0).stop 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- 172.21.15.28:0/3405402788 shutdown_connections 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa259380 0x55d6aa25b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa254210 0x55d6aa258d10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3405402788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa253c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- 172.21.15.28:0/3405402788 >> 172.21.15.28:0/3405402788 conn(0x55d6aa24ed90 msgr2=0x55d6aa24f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- 172.21.15.28:0/3405402788 shutdown_connections 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- 172.21.15.28:0/3405402788 wait complete. 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 -- start start 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.907+0000 7f08aeffd740 1 Processor -- start 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa25e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa259380 0x55d6aa265260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa25c330 con 0x55d6aa253820 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa174300 con 0x55d6aa254210 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d6aa25c4b0 con 0x55d6aa259380 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 10 client.0 _ll_register_callbacks cb 0x55d6aa1afe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59830/0 (socket says 172.21.15.28:59830) 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08a98de640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa25e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 -- 172.21.15.28:0/3786745035 learned_addr learned my addr 172.21.15.28:0/3786745035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa259380 0x55d6aa265260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 -- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa259380 msgr2=0x55d6aa265260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa259380 0x55d6aa265260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 -- 172.21.15.28:0/3786745035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 msgr2=0x55d6aa25e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa25e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 -- 172.21.15.28:0/3786745035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6a9f967d0 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08a98de640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa25e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aa0df640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f08a001b040 tx=0x7f08a0005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19100905f3410918 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f088dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a0028020 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f08a0005410 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.908+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08a00047a0 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f08a0033050 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 10 client.16803 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6aa2685d0 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08a00047a0 con 0x55d6aa253820 2024-03-22T02:18:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f08a0023630 con 0x55d6aa253820 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 10 client.16803 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 10 client.16803 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 20 client.16803 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa269b20 0x55d6aa26bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 4 client.16803 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6aa1c5e10 con 0x55d6aa269b20 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.909+0000 7f08a98de640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa269b20 0x55d6aa26bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.910+0000 7f088dffb640 10 client.16803 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.910+0000 7f08a98de640 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa269b20 0x55d6aa26bf90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a970ea01c26c93c server_cookie=bc8757eac11f7981 in_seq=0 out_seq=0 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.910+0000 7f088dffb640 1 -- 172.21.15.28:0/3786745035 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d6aa1c5e10 con 0x55d6aa269b20 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.910+0000 7f088dffb640 10 client.16803 handle_command_reply: tid=0 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 client.16803 shutdown 2024-03-22T02:18:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 msgr2=0x55d6aa25dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f08a001b040 tx=0x7f08a0005220 comp rx=0 tx=0).stop 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 shutdown_connections 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa259380 0x55d6aa265260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa269b20 0x55d6aa26bf90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa25e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 --2- 172.21.15.28:0/3786745035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa25dce0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f088dffb640 0 client.16803 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 >> 172.21.15.28:0/3786745035 conn(0x55d6aa24ed90 msgr2=0x55d6aa24dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 shutdown_connections 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 1 -- 172.21.15.28:0/3786745035 wait complete. 2024-03-22T02:18:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.911+0000 7f08aeffd740 20 client.16803 trim_cache size 0 max 16384 2024-03-22T02:18:15.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 Processor -- start 2024-03-22T02:18:15.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 -- start start 2024-03-22T02:18:15.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa252700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa270850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41af4e0 con 0x55d6aa253820 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41acdb0 con 0x55d6aa254210 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.925+0000 7f08aeffd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41aac00 con 0x55d6aa24ed90 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08aa8e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa270850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59840/0 (socket says 172.21.15.28:59840) 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08aa0df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa252700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 -- 172.21.15.28:0/2045220742 learned_addr learned my addr 172.21.15.28:0/2045220742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 -- 172.21.15.28:0/2045220742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 msgr2=0x55d6aa252700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa252700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 -- 172.21.15.28:0/2045220742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 msgr2=0x55d6aa270850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa270850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 -- 172.21.15.28:0/2045220742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6aa255fc0 con 0x55d6aa253820 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08aa0df640 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa252700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f08a98de640 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0894098850 tx=0x7f08940669d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2591dada7f27bc7f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f088ffff640 1 -- 172.21.15.28:0/2045220742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089409a180 con 0x55d6aa253820 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f088ffff640 1 -- 172.21.15.28:0/2045220742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08940cd2b0 con 0x55d6aa253820 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.926+0000 7f088ffff640 1 -- 172.21.15.28:0/2045220742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08940bad80 con 0x55d6aa253820 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- 172.21.15.28:0/2045220742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 msgr2=0x55d6aa265220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0894098850 tx=0x7f08940669d0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- 172.21.15.28:0/2045220742 shutdown_connections 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa254210 0x55d6aa270850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa253820 0x55d6aa265220 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 --2- 172.21.15.28:0/2045220742 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa252700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- 172.21.15.28:0/2045220742 >> 172.21.15.28:0/2045220742 conn(0x55d6aa254ff0 msgr2=0x55d6aa251ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- 172.21.15.28:0/2045220742 shutdown_connections 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- 172.21.15.28:0/2045220742 wait complete. 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 -- start start 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.927+0000 7f08aeffd740 1 Processor -- start 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa262850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa262d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41ab270 con 0x55d6aa254210 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41ac6e0 con 0x55d6aa253820 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08a41ab940 con 0x55d6aa24ed90 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aeffd740 10 client.0 _ll_register_callbacks cb 0x7f0894010e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:15.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59852/0 (socket says 172.21.15.28:59852) 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 -- 172.21.15.28:0/349656960 learned_addr learned my addr 172.21.15.28:0/349656960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08a98de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa262d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa0df640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa262850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 -- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 msgr2=0x55d6aa262850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa262850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 -- 172.21.15.28:0/349656960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 msgr2=0x55d6aa262d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa262d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 -- 172.21.15.28:0/349656960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6aa264390 con 0x55d6aa254210 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa0df640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa262850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f08aa8e0640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f089c02c630 tx=0x7f089c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16375ed041867236 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f088dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:15.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.928+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089c054070 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f089c034c00 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08940cd2b0 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f089c039040 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 10 client.16809 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6aa2cfe90 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08940cd2b0 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f089c005020 con 0x55d6aa254210 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 10 client.16809 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 10 client.16809 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 20 client.16809 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa259380 0x55d6aa2d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 4 client.16809 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:15.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d6aa1c0160 con 0x55d6aa259380 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.929+0000 7f08aa0df640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa259380 0x55d6aa2d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aa0df640 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa259380 0x55d6aa2d28a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=120c71f595e9e5a1 server_cookie=c81b279a3d209e43 in_seq=0 out_seq=0 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f088dffb640 10 client.16809 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f088dffb640 1 -- 172.21.15.28:0/349656960 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d6aa1c0160 con 0x55d6aa259380 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f088dffb640 10 client.16809 handle_command_reply: tid=0 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 client.16809 shutdown 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 msgr2=0x55d6aa263da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f089c02c630 tx=0x7f089c0095c0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 shutdown_connections 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d6aa254210 0x55d6aa263da0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55d6aa259380 0x55d6aa2d28a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d6aa253820 0x55d6aa262d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 --2- 172.21.15.28:0/349656960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d6aa24ed90 0x55d6aa262850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f088dffb640 0 client.16809 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:15.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 >> 172.21.15.28:0/349656960 conn(0x55d6aa254ff0 msgr2=0x55d6aa252880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 shutdown_connections 2024-03-22T02:18:15.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 1 -- 172.21.15.28:0/349656960 wait complete. 2024-03-22T02:18:15.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.930+0000 7f08aeffd740 20 client.16809 trim_cache size 0 max 16384 2024-03-22T02:18:15.938 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 msgr2=0x7f08900a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 0x7f08900a5460 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55d6aa254a70 tx=0x7f089c032590 comp rx=0 tx=0).stop 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 msgr2=0x7f08a419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f089400e9f0 tx=0x7f089400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 shutdown_connections 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08900a2fa0 0x7f08900a5460 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08a4102640 0x7f08a41a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a4101c80 0x7f08a419ef50 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 --2- 172.21.15.28:0/1911436192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08a4100a90 0x7f08a40698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:15.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 >> 172.21.15.28:0/1911436192 conn(0x7f08a40fc020 msgr2=0x7f08a4104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:15.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 shutdown_connections 2024-03-22T02:18:15.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:15.931+0000 7f08ac343640 1 -- 172.21.15.28:0/1911436192 wait complete. 2024-03-22T02:18:15.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 -- 172.21.15.28:0/2895135184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 msgr2=0x7fefc41011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 --2- 172.21.15.28:0/2895135184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc41011f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009900 tx=0x7fefb402bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 -- 172.21.15.28:0/2895135184 shutdown_connections 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 --2- 172.21.15.28:0/2895135184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 --2- 172.21.15.28:0/2895135184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc41011f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 --2- 172.21.15.28:0/2895135184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 0x7fefc4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 -- 172.21.15.28:0/2895135184 >> 172.21.15.28:0/2895135184 conn(0x7fefc40fb7a0 msgr2=0x7fefc40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 -- 172.21.15.28:0/2895135184 shutdown_connections 2024-03-22T02:18:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.049+0000 7fefcc476640 1 -- 172.21.15.28:0/2895135184 wait complete. 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 Processor -- start 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 -- start start 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 0x7fefc419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c740 con 0x7fefc4100d70 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c5c0 con 0x7fefc4101730 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcc476640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc410c8c0 con 0x7fefc4102040 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59874/0 (socket says 172.21.15.28:59874) 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 -- 172.21.15.28:0/2985381143 learned_addr learned my addr 172.21.15.28:0/2985381143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 0x7fefc419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 -- 172.21.15.28:0/2985381143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 msgr2=0x7fefc419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 0x7fefc419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 -- 172.21.15.28:0/2985381143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 msgr2=0x7fefc4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 -- 172.21.15.28:0/2985381143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc419fb00 con 0x7fefc4100d70 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.050+0000 7fefca212640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fefc000e9c0 tx=0x7fefc000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.051+0000 7fefbb7fe640 1 -- 172.21.15.28:0/2985381143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc000cd80 con 0x7fefc4100d70 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.051+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefc419fdf0 con 0x7fefc4100d70 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.051+0000 7fefbb7fe640 1 -- 172.21.15.28:0/2985381143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefc00035f0 con 0x7fefc4100d70 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.051+0000 7fefbb7fe640 1 -- 172.21.15.28:0/2985381143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc0010690 con 0x7fefc4100d70 2024-03-22T02:18:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.051+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefc41a0200 con 0x7fefc4100d70 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.053+0000 7fefcf05d740 1 Processor -- start 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 -- start start 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce17ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce0de8d0 con 0x55a7ce17a210 2024-03-22T02:18:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce086e00 con 0x55a7ce179820 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcf05d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce0da880 con 0x55a7ce17f380 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefbb7fe640 1 -- 172.21.15.28:0/2985381143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fefc00037a0 con 0x7fefc4100d70 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46024/0 (socket says 172.21.15.28:46024) 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefc9a11640 1 -- 172.21.15.28:0/2352543680 learned_addr learned my addr 172.21.15.28:0/2352543680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46560/0 (socket says 172.21.15.28:46560) 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.054+0000 7fefca212640 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce17ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 -- 172.21.15.28:0/2352543680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 msgr2=0x55a7ce181810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 -- 172.21.15.28:0/2352543680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 msgr2=0x55a7ce17ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce17ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 -- 172.21.15.28:0/2352543680 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7cddf23f0 con 0x55a7ce179820 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009f00 tx=0x7fefbc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91b0b1ee082299c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefbb7fe640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 0x7fefb00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefb8bfe640 1 -- 172.21.15.28:0/2352543680 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc004d20 con 0x55a7ce179820 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.055+0000 7fefb8bfe640 1 -- 172.21.15.28:0/2352543680 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefbc004030 con 0x55a7ce179820 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefb8bfe640 1 -- 172.21.15.28:0/2352543680 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc005260 con 0x55a7ce179820 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefca212640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 0x7fefb00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefbb7fe640 1 -- 172.21.15.28:0/2985381143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fefc00cfb60 con 0x7fefc4100d70 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 -- 172.21.15.28:0/2352543680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 msgr2=0x55a7ce179c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefca212640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 0x7fefb00a53e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fefc00cd5c0 tx=0x7fefc00d3640 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fefbc009f00 tx=0x7fefbc011810 comp rx=0 tx=0).stop 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 -- 172.21.15.28:0/2352543680 shutdown_connections 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce181810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce17ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2352543680 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce179c40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.056+0000 7fefcf05d740 1 -- 172.21.15.28:0/2352543680 >> 172.21.15.28:0/2352543680 conn(0x55a7ce174d90 msgr2=0x55a7ce175190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- 172.21.15.28:0/2352543680 shutdown_connections 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- 172.21.15.28:0/2352543680 wait complete. 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- start start 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 Processor -- start 2024-03-22T02:18:16.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce183d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce182330 con 0x55a7ce17a210 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce09a300 con 0x55a7ce179820 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a7ce1824b0 con 0x55a7ce17f380 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcf05d740 10 client.0 _ll_register_callbacks cb 0x55a7ce0d5e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59900/0 (socket says 172.21.15.28:59900) 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 -- 172.21.15.28:0/1558675062 learned_addr learned my addr 172.21.15.28:0/1558675062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce183d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefcaa13640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 -- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 msgr2=0x55a7ce18b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 -- 172.21.15.28:0/1558675062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 msgr2=0x55a7ce183d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.057+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce183d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefc9a11640 1 -- 172.21.15.28:0/1558675062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7cdebc7d0 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009230 tx=0x7fefb4005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a382a0e91c252b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefadffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4005d50 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fefb40046c0 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefbc004030 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fefb404f050 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefcf05d740 10 client.16821 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a7ce18e6c0 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.058+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefbc004030 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fefb4003930 con 0x55a7ce17a210 2024-03-22T02:18:16.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 10 client.16821 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 10 client.16821 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 20 client.16821 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce18fd70 0x55a7ce1921d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 4 client.16821 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7ce0ebe10 con 0x55a7ce18fd70 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcaa13640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce18fd70 0x55a7ce1921d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefadffb640 10 client.16821 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.059+0000 7fefcaa13640 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce18fd70 0x55a7ce1921d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a6b1678af746c75b server_cookie=80a92f24c4099d91 in_seq=0 out_seq=0 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.060+0000 7fefadffb640 1 -- 172.21.15.28:0/1558675062 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7ce0ebe10 con 0x55a7ce18fd70 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.060+0000 7fefadffb640 10 client.16821 handle_command_reply: tid=0 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.060+0000 7fefcf05d740 1 client.16821 shutdown 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.060+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 msgr2=0x55a7ce1842a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009230 tx=0x7fefb4005270 comp rx=0 tx=0).stop 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 shutdown_connections 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce18fd70 0x55a7ce1921d0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce17a210 0x55a7ce1842a0 secure :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fefb4009230 tx=0x7fefb4005270 comp rx=0 tx=0).stop 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1558675062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce179820 0x55a7ce183d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 >> 172.21.15.28:0/1558675062 conn(0x55a7ce174d90 msgr2=0x55a7ce173fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefadffb640 0 client.16821 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 shutdown_connections 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 1 -- 172.21.15.28:0/1558675062 wait complete. 2024-03-22T02:18:16.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.061+0000 7fefcf05d740 20 client.16821 trim_cache size 0 max 16384 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 Processor -- start 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 -- start start 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce173740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce17be80 0x55a7ce173c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41ad260 con 0x55a7ce179820 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41aab30 con 0x55a7ce17be80 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.078+0000 7fefcf05d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41a8980 con 0x55a7ce17f380 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce173740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46038/0 (socket says 172.21.15.28:46038) 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 -- 172.21.15.28:0/1951338396 learned_addr learned my addr 172.21.15.28:0/1951338396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce173740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59904/0 (socket says 172.21.15.28:59904) 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefcaa13640 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce17be80 0x55a7ce173c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 -- 172.21.15.28:0/1951338396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce17be80 msgr2=0x55a7ce173c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce17be80 0x55a7ce173c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 -- 172.21.15.28:0/1951338396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 msgr2=0x55a7ce173740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce173740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 -- 172.21.15.28:0/1951338396 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7ce17aaa0 con 0x55a7ce17f380 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefc9a11640 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4003f70 tx=0x7fefb402c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ace7673c64ba49f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefaffff640 1 -- 172.21.15.28:0/1951338396 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4000e60 con 0x55a7ce17f380 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefaffff640 1 -- 172.21.15.28:0/1951338396 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4041070 con 0x55a7ce17f380 2024-03-22T02:18:16.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.079+0000 7fefaffff640 1 -- 172.21.15.28:0/1951338396 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4003ad0 con 0x55a7ce17f380 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- 172.21.15.28:0/1951338396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 msgr2=0x55a7ce18b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fefb4003f70 tx=0x7fefb402c330 comp rx=0 tx=0).stop 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- 172.21.15.28:0/1951338396 shutdown_connections 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce17be80 0x55a7ce173c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b2a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 --2- 172.21.15.28:0/1951338396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce173740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- 172.21.15.28:0/1951338396 >> 172.21.15.28:0/1951338396 conn(0x55a7ce174d90 msgr2=0x55a7ce17b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- 172.21.15.28:0/1951338396 shutdown_connections 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- 172.21.15.28:0/1951338396 wait complete. 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 -- start start 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.080+0000 7fefcf05d740 1 Processor -- start 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce18abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce1891e0 0x55a7ce189660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41a8ff0 con 0x55a7ce179820 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41aa460 con 0x55a7ce1891e0 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fefc41a96c0 con 0x55a7ce17f380 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcf05d740 10 client.0 _ll_register_callbacks cb 0x7fefb4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce18abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcaa13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce18abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59918/0 (socket says 172.21.15.28:59918) 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcaa13640 1 -- 172.21.15.28:0/2784448901 learned_addr learned my addr 172.21.15.28:0/2784448901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce1891e0 0x55a7ce189660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 -- 172.21.15.28:0/2784448901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce1891e0 msgr2=0x55a7ce189660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce1891e0 0x55a7ce189660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 -- 172.21.15.28:0/2784448901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 msgr2=0x55a7ce18abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce18abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 -- 172.21.15.28:0/2784448901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7ce189ca0 con 0x55a7ce17f380 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefc9a11640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fefb402c170 tx=0x7fefb4005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01da8de96007440 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefadffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb4060060 con 0x55a7ce17f380 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.081+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fefb4003800 con 0x55a7ce17f380 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fefb4041070 con 0x55a7ce17f380 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fefb4067050 con 0x55a7ce17f380 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 10 client.26662 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:16.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a7ce1f6e80 con 0x55a7ce17f380 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefb4041070 con 0x55a7ce17f380 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fefb4046470 con 0x55a7ce17f380 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 10 client.26662 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 10 client.26662 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 20 client.26662 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce1f7770 0x55a7ce1f9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 4 client.26662 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7ce0e6160 con 0x55a7ce1f7770 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.082+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce1f7770 0x55a7ce1f9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefadffb640 10 client.26662 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcaa13640 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce1f7770 0x55a7ce1f9b90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93d038cdc74438b6 server_cookie=d1e75bdf10e71342 in_seq=0 out_seq=0 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefadffb640 1 -- 172.21.15.28:0/2784448901 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7ce0e6160 con 0x55a7ce1f7770 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefadffb640 10 client.26662 handle_command_reply: tid=0 2024-03-22T02:18:16.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 client.26662 shutdown 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 msgr2=0x55a7ce18b110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fefb402c170 tx=0x7fefb4005b50 comp rx=0 tx=0).stop 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 shutdown_connections 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55a7ce1f7770 0x55a7ce1f9b90 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a7ce1891e0 0x55a7ce189660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a7ce17f380 0x55a7ce18b110 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 --2- 172.21.15.28:0/2784448901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a7ce179820 0x55a7ce18abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefadffb640 0 client.26662 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 >> 172.21.15.28:0/2784448901 conn(0x55a7ce174d90 msgr2=0x55a7ce177ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 shutdown_connections 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 1 -- 172.21.15.28:0/2784448901 wait complete. 2024-03-22T02:18:16.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcf05d740 20 client.26662 trim_cache size 0 max 16384 2024-03-22T02:18:16.091 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:16.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 msgr2=0x7fefb00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 0x7fefb00a53e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fefc00cd5c0 tx=0x7fefc00d3640 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 msgr2=0x7fefc4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fefc000e9c0 tx=0x7fefc000ee80 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.083+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 shutdown_connections 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fefb00a2f20 0x7fefb00a53e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fefc4102040 0x7fefc419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fefc4101730 0x7fefc4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 --2- 172.21.15.28:0/2985381143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefc4100d70 0x7fefc4197ee0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 >> 172.21.15.28:0/2985381143 conn(0x7fefc40fb7a0 msgr2=0x7fefc40fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 shutdown_connections 2024-03-22T02:18:16.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.084+0000 7fefcc476640 1 -- 172.21.15.28:0/2985381143 wait complete. 2024-03-22T02:18:16.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 -- 172.21.15.28:0/3892046865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 msgr2=0x7f8ca4108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/3892046865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca4108ed0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c009ca0 tx=0x7f8c9c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 -- 172.21.15.28:0/3892046865 shutdown_connections 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/3892046865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca4108ed0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/3892046865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4101c80 0x7f8ca4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/3892046865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4100a90 0x7f8ca4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 -- 172.21.15.28:0/3892046865 >> 172.21.15.28:0/3892046865 conn(0x7f8ca40fc020 msgr2=0x7f8ca40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 -- 172.21.15.28:0/3892046865 shutdown_connections 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.209+0000 7f8cabd7a640 1 -- 172.21.15.28:0/3892046865 wait complete. 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 Processor -- start 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 -- start start 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4100a90 0x7f8ca4197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4101c80 0x7f8ca41983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410d830 con 0x7f8ca4102640 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410d6b0 con 0x7f8ca4101c80 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8cabd7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410d9b0 con 0x7f8ca4100a90 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59932/0 (socket says 172.21.15.28:59932) 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4100a90 0x7f8ca4197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 -- 172.21.15.28:0/2490389598 learned_addr learned my addr 172.21.15.28:0/2490389598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8ca9315640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4101c80 0x7f8ca41983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 -- 172.21.15.28:0/2490389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4100a90 msgr2=0x7f8ca4197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4100a90 0x7f8ca4197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 -- 172.21.15.28:0/2490389598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4101c80 msgr2=0x7f8ca41983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4101c80 0x7f8ca41983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.210+0000 7f8caa317640 1 -- 172.21.15.28:0/2490389598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca419fa80 con 0x7f8ca4102640 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8caa317640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c02c630 tx=0x7f8c9c032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8c9affd640 1 -- 172.21.15.28:0/2490389598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c041020 con 0x7f8ca4102640 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca419fd10 con 0x7f8ca4102640 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8c9affd640 1 -- 172.21.15.28:0/2490389598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9c034cd0 con 0x7f8ca4102640 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8c9affd640 1 -- 172.21.15.28:0/2490389598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c039040 con 0x7f8ca4102640 2024-03-22T02:18:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.211+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca41a01f0 con 0x7f8ca4102640 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.213+0000 7f8c9affd640 1 -- 172.21.15.28:0/2490389598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8c9c044020 con 0x7f8ca4102640 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.213+0000 7f8cae965740 1 Processor -- start 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.213+0000 7f8cae965740 1 -- start start 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce18c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce20810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c4cd7d8d0 con 0x563c4ce1e380 2024-03-22T02:18:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c4cd25e00 con 0x563c4ce19210 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8cae965740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563c4cd79880 con 0x563c4ce18820 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8c9affd640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 0x7f8c900a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce18c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46616/0 (socket says 172.21.15.28:46616) 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce18c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46058/0 (socket says 172.21.15.28:46058) 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 -- 172.21.15.28:0/3551329440 learned_addr learned my addr 172.21.15.28:0/3551329440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9315640 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce20810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8caa317640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 0x7f8c900a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8c9affd640 1 -- 172.21.15.28:0/2490389598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8c9c0f1600 con 0x7f8ca4102640 2024-03-22T02:18:16.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 -- 172.21.15.28:0/3551329440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 msgr2=0x563c4ce18c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce18c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 -- 172.21.15.28:0/3551329440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 msgr2=0x563c4ce20810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce20810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 -- 172.21.15.28:0/3551329440 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c4ca913f0 con 0x563c4ce19210 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.214+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009ca0 tx=0x7f8ca0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5147a6fea92bdb09 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8caa317640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 0x7f8c900a53e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0b9e10 tx=0x7f8c9c08d4d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8c8ffff640 1 -- 172.21.15.28:0/3551329440 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0004220 con 0x563c4ce19210 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8c8ffff640 1 -- 172.21.15.28:0/3551329440 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca00043c0 con 0x563c4ce19210 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8c8ffff640 1 -- 172.21.15.28:0/3551329440 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca00198d0 con 0x563c4ce19210 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 -- 172.21.15.28:0/3551329440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 msgr2=0x563c4ce1dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009ca0 tx=0x7f8ca0011810 comp rx=0 tx=0).stop 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 -- 172.21.15.28:0/3551329440 shutdown_connections 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce20810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce1dd10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 --2- 172.21.15.28:0/3551329440 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce18c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.215+0000 7f8cae965740 1 -- 172.21.15.28:0/3551329440 >> 172.21.15.28:0/3551329440 conn(0x563c4ce13d90 msgr2=0x563c4ce14190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- 172.21.15.28:0/3551329440 shutdown_connections 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- 172.21.15.28:0/3551329440 wait complete. 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- start start 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 Processor -- start 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce22df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce23330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c4ce21330 con 0x563c4ce1e380 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c4cd39300 con 0x563c4ce19210 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563c4ce214b0 con 0x563c4ce18820 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8cae965740 10 client.0 _ll_register_callbacks cb 0x563c4cd74e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce23330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59954/0 (socket says 172.21.15.28:59954) 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9315640 1 -- 172.21.15.28:0/1379160751 learned_addr learned my addr 172.21.15.28:0/1379160751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce23330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46628/0 (socket says 172.21.15.28:46628) 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.216+0000 7f8ca9315640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce22df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 -- 172.21.15.28:0/1379160751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 msgr2=0x563c4ce22df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce22df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 -- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 msgr2=0x563c4ce23330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce23330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 -- 172.21.15.28:0/1379160751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c4cb5b7d0 con 0x563c4ce1e380 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8ca9315640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8c94011c90 tx=0x7f8c94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51304b562e8422bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8c8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:16.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94027060 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8c94004030 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ca00043c0 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c9402e050 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8cae965740 10 client.16839 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c4ce2d6e0 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.217+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ca00043c0 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8c94003ad0 con 0x563c4ce1e380 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 10 client.16839 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 10 client.16839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 20 client.16839 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce2ed90 0x563c4ce31230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 4 client.16839 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c4cd8ae10 con 0x563c4ce2ed90 2024-03-22T02:18:16.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce2ed90 0x563c4ce31230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8c8d7fa640 10 client.16839 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.218+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce2ed90 0x563c4ce31230 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dffec3481d974b0 server_cookie=28b544c6869821ef in_seq=0 out_seq=0 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/1379160751 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c4cd8ae10 con 0x563c4ce2ed90 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8c8d7fa640 10 client.16839 handle_command_reply: tid=0 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 client.16839 shutdown 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 msgr2=0x563c4ce2a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8c94011c90 tx=0x7f8c94004730 comp rx=0 tx=0).stop 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 shutdown_connections 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce2ed90 0x563c4ce31230 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce2a300 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce23330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8c8d7fa640 0 client.16839 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 --2- 172.21.15.28:0/1379160751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce22df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 >> 172.21.15.28:0/1379160751 conn(0x563c4ce13d90 msgr2=0x563c4ce12fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 shutdown_connections 2024-03-22T02:18:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 1 -- 172.21.15.28:0/1379160751 wait complete. 2024-03-22T02:18:16.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.219+0000 7f8cae965740 20 client.16839 trim_cache size 0 max 16384 2024-03-22T02:18:16.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 Processor -- start 2024-03-22T02:18:16.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 -- start start 2024-03-22T02:18:16.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce12740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce2a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41ad2c0 con 0x563c4ce1e380 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41aab90 con 0x563c4ce19210 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8cae965740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41a89e0 con 0x563c4ce18820 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59960/0 (socket says 172.21.15.28:59960) 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce2a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 -- 172.21.15.28:0/133242283 learned_addr learned my addr 172.21.15.28:0/133242283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce12740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 -- 172.21.15.28:0/133242283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 msgr2=0x563c4ce12740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce12740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 -- 172.21.15.28:0/133242283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 msgr2=0x563c4ce2a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce2a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 -- 172.21.15.28:0/133242283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c4ce0e3e0 con 0x563c4ce1e380 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.233+0000 7f8ca9315640 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8c94020510 tx=0x7f8c940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921bce29f68b778e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8c8f7fe640 1 -- 172.21.15.28:0/133242283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94027040 con 0x563c4ce1e380 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8c8f7fe640 1 -- 172.21.15.28:0/133242283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9400fe60 con 0x563c4ce1e380 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8c8f7fe640 1 -- 172.21.15.28:0/133242283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94020bf0 con 0x563c4ce1e380 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 -- 172.21.15.28:0/133242283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 msgr2=0x563c4ce12c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8c94020510 tx=0x7f8c940046c0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 -- 172.21.15.28:0/133242283 shutdown_connections 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce1e380 0x563c4ce12c80 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce2a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 --2- 172.21.15.28:0/133242283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce18820 0x563c4ce12740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 -- 172.21.15.28:0/133242283 >> 172.21.15.28:0/133242283 conn(0x563c4ce13d90 msgr2=0x563c4ce1a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 -- 172.21.15.28:0/133242283 shutdown_connections 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.234+0000 7f8cae965740 1 -- 172.21.15.28:0/133242283 wait complete. 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 -- start start 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 Processor -- start 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce28110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce1e380 0x563c4ce290d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41a9050 con 0x563c4ce18820 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41aa4c0 con 0x563c4ce19210 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41a9720 con 0x563c4ce1e380 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8cae965740 10 client.0 _ll_register_callbacks cb 0x7f8c94011ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59962/0 (socket says 172.21.15.28:59962) 2024-03-22T02:18:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 -- 172.21.15.28:0/4198042698 learned_addr learned my addr 172.21.15.28:0/4198042698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce28110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce1e380 0x563c4ce290d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 -- 172.21.15.28:0/4198042698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce1e380 msgr2=0x563c4ce290d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce1e380 0x563c4ce290d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 -- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 msgr2=0x563c4ce28110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce28110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.235+0000 7f8ca9315640 1 -- 172.21.15.28:0/4198042698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c4ce296c0 con 0x563c4ce18820 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8ca9315640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25abaa91bc1c7b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c94027070 con 0x563c4ce18820 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8c94020da0 con 0x563c4ce18820 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c9400fe60 con 0x563c4ce18820 2024-03-22T02:18:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c94045050 con 0x563c4ce18820 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 10 client.16845 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c4ce952b0 con 0x563c4ce18820 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c9400fe60 con 0x563c4ce18820 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8c94003920 con 0x563c4ce18820 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 10 client.16845 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 20 client.16845 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce95ba0 0x563c4ce97fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 4 client.16845 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.236+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c4cd85160 con 0x563c4ce95ba0 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.237+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce95ba0 0x563c4ce97fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.237+0000 7f8ca9b16640 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce95ba0 0x563c4ce97fc0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=342a18971ec11974 server_cookie=a53c34cd126ee0f0 in_seq=0 out_seq=0 2024-03-22T02:18:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.237+0000 7f8c8d7fa640 10 client.16845 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8c8d7fa640 1 -- 172.21.15.28:0/4198042698 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c4cd85160 con 0x563c4ce95ba0 2024-03-22T02:18:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8c8d7fa640 10 client.16845 handle_command_reply: tid=0 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 client.16845 shutdown 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 msgr2=0x563c4ce27bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94005850 comp rx=0 tx=0).stop 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 shutdown_connections 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c4ce95ba0 0x563c4ce97fc0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c4ce1e380 0x563c4ce290d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c4ce19210 0x563c4ce28110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 --2- 172.21.15.28:0/4198042698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c4ce18820 0x563c4ce27bd0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8c8d7fa640 0 client.16845 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 >> 172.21.15.28:0/4198042698 conn(0x563c4ce13d90 msgr2=0x563c4ce16ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 shutdown_connections 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 1 -- 172.21.15.28:0/4198042698 wait complete. 2024-03-22T02:18:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.238+0000 7f8cae965740 20 client.16845 trim_cache size 0 max 16384 2024-03-22T02:18:16.245 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 msgr2=0x7f8c900a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 0x7f8c900a53e0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c0b9e10 tx=0x7f8c9c08d4d0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 msgr2=0x7f8ca419f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c02c630 tx=0x7f8c9c032ea0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 shutdown_connections 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c900a2f20 0x7f8c900a53e0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ca4102640 0x7f8ca419f410 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ca4101c80 0x7f8ca41983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 --2- 172.21.15.28:0/2490389598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ca4100a90 0x7f8ca4197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 >> 172.21.15.28:0/2490389598 conn(0x7f8ca40fc020 msgr2=0x7f8ca4104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 shutdown_connections 2024-03-22T02:18:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:16.239+0000 7f8cabd7a640 1 -- 172.21.15.28:0/2490389598 wait complete. 2024-03-22T02:18:16.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 324463 11.92 MB/sec execute 58 sec latency 27.945 ms 2024-03-22T02:18:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:16 smithi120 ceph-mon[36803]: pgmap v498: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-03-22T02:18:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:16 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1192716155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:16.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:16 smithi083 ceph-mon[36340]: pgmap v498: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-03-22T02:18:16.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:16 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1192716155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:16 smithi028 ceph-mon[31916]: pgmap v498: 129 pgs: 129 active+clean; 42 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-03-22T02:18:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:16 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1192716155' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:17.251 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:18:17.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:18:17.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:18:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 -- 172.21.15.28:0/3358287615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc530100d20 msgr2=0x7fc5301011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 --2- 172.21.15.28:0/3358287615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc530100d20 0x7fc5301011a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc518009900 tx=0x7fc51802bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 -- 172.21.15.28:0/3358287615 shutdown_connections 2024-03-22T02:18:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 --2- 172.21.15.28:0/3358287615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301016e0 0x7fc530108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 --2- 172.21.15.28:0/3358287615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc530100d20 0x7fc5301011a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 --2- 172.21.15.28:0/3358287615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5301020d0 0x7fc5301024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.361+0000 7fc534f1e640 1 -- 172.21.15.28:0/3358287615 >> 172.21.15.28:0/3358287615 conn(0x7fc5300fb790 msgr2=0x7fc5300fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- 172.21.15.28:0/3358287615 shutdown_connections 2024-03-22T02:18:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- 172.21.15.28:0/3358287615 wait complete. 2024-03-22T02:18:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 Processor -- start 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- start start 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc530100d20 0x7fc530197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301020d0 0x7fc53019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc53010c760 con 0x7fc5301016e0 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc53010c5e0 con 0x7fc530100d20 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.362+0000 7fc534f1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc53010c8e0 con 0x7fc5301020d0 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59978/0 (socket says 172.21.15.28:59978) 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 -- 172.21.15.28:0/2193836806 learned_addr learned my addr 172.21.15.28:0/2193836806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52e59c640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc530100d20 0x7fc530197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301020d0 0x7fc53019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 -- 172.21.15.28:0/2193836806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301020d0 msgr2=0x7fc53019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301020d0 0x7fc53019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 -- 172.21.15.28:0/2193836806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc530100d20 msgr2=0x7fc530197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc530100d20 0x7fc530197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 -- 172.21.15.28:0/2193836806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc53019fb30 con 0x7fc5301016e0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc518000c00 tx=0x7fc51802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc5277fe640 1 -- 172.21.15.28:0/2193836806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518041070 con 0x7fc5301016e0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc5277fe640 1 -- 172.21.15.28:0/2193836806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc518004450 con 0x7fc5301016e0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc53019fdc0 con 0x7fc5301016e0 2024-03-22T02:18:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc5277fe640 1 -- 172.21.15.28:0/2193836806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc518034440 con 0x7fc5301016e0 2024-03-22T02:18:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.363+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5301a01f0 con 0x7fc5301016e0 2024-03-22T02:18:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.365+0000 7fc5277fe640 1 -- 172.21.15.28:0/2193836806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc518044070 con 0x7fc5301016e0 2024-03-22T02:18:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 Processor -- start 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 -- start start 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc5277fe640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 0x7fc51c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x56294707a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x56294707cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562946fd9570 con 0x562947074bc0 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562946f81e10 con 0x56294707a720 2024-03-22T02:18:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc537be8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562946fd5520 con 0x5629470755b0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 0x7fc51c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x56294707a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59982/0 (socket says 172.21.15.28:59982) 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52e59c640 1 -- 172.21.15.28:0/1926589503 learned_addr learned my addr 172.21.15.28:0/1926589503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc5277fe640 1 -- 172.21.15.28:0/2193836806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc5180f1280 con 0x7fc5301016e0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.366+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x56294707cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 -- 172.21.15.28:0/1926589503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 msgr2=0x56294707cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x56294707cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 -- 172.21.15.28:0/1926589503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 msgr2=0x562947074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 -- 172.21.15.28:0/1926589503 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562946db57e0 con 0x5629470755b0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52e59c640 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 0x7fc51c0a53e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc520005fd0 tx=0x7fc520009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x56294707a0b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc518006010 tx=0x7fc518049000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=310aaaf922938c94 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc524ff9640 1 -- 172.21.15.28:0/1926589503 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51804b070 con 0x5629470755b0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc524ff9640 1 -- 172.21.15.28:0/1926589503 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180bd650 con 0x5629470755b0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.367+0000 7fc524ff9640 1 -- 172.21.15.28:0/1926589503 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180b9cc0 con 0x5629470755b0 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- 172.21.15.28:0/1926589503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 msgr2=0x56294707a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x56294707a0b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc518006010 tx=0x7fc518049000 comp rx=0 tx=0).stop 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- 172.21.15.28:0/1926589503 shutdown_connections 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x56294707cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x56294707a0b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- 172.21.15.28:0/1926589503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562947074bc0 0x562947074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- 172.21.15.28:0/1926589503 >> 172.21.15.28:0/1926589503 conn(0x562947070130 msgr2=0x562947070530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- 172.21.15.28:0/1926589503 shutdown_connections 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- 172.21.15.28:0/1926589503 wait complete. 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- start start 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 Processor -- start 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 0x56294707efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947086020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56294707d490 con 0x56294707a720 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562946f95110 con 0x5629470755b0 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.368+0000 7fc537be8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56294707d610 con 0x562947074bc0 2024-03-22T02:18:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc537be8740 10 client.0 _ll_register_callbacks cb 0x562946fd1be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947086020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46128/0 (socket says 172.21.15.28:46128) 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52e59c640 1 -- 172.21.15.28:0/3523319064 learned_addr learned my addr 172.21.15.28:0/3523319064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 0x56294707efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3523319064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 msgr2=0x56294707eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 msgr2=0x56294707efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 0x56294707efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3523319064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562946e15f70 con 0x56294707a720 2024-03-22T02:18:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 0x56294707efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947086020 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc520009dd0 tx=0x7fc52001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0979ffa82bb4e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc515ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520004b20 con 0x56294707a720 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc52001c040 con 0x56294707a720 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5180bd650 con 0x56294707a720 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc520031050 con 0x56294707a720 2024-03-22T02:18:17.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc537be8740 10 client.16857 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.369+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562946cad170 con 0x56294707a720 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180bd650 con 0x56294707a720 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc520028070 con 0x56294707a720 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 10 client.16857 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 10 client.16857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 20 client.16857 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56294708a9c0 0x56294708ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 4 client.16857 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.370+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562946fe6ab0 con 0x56294708a9c0 2024-03-22T02:18:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.396+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56294708a9c0 0x56294708ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.396+0000 7fc515ffb640 10 client.16857 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.396+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56294708a9c0 0x56294708ce20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e46825de517f4898 server_cookie=ea5d631a88b8f7b2 in_seq=0 out_seq=0 2024-03-22T02:18:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.397+0000 7fc515ffb640 1 -- 172.21.15.28:0/3523319064 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562946fe6ab0 con 0x56294708a9c0 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.397+0000 7fc515ffb640 10 client.16857 handle_command_reply: tid=0 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 client.16857 shutdown 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 msgr2=0x562947086020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947086020 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fc520009dd0 tx=0x7fc52001d000 comp rx=0 tx=0).stop 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 shutdown_connections 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56294708a9c0 0x56294708ce20 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947086020 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5629470755b0 0x56294707efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 --2- 172.21.15.28:0/3523319064 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x56294707eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc515ffb640 0 client.16857 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 >> 172.21.15.28:0/3523319064 conn(0x562947070130 msgr2=0x56294706f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 shutdown_connections 2024-03-22T02:18:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 1 -- 172.21.15.28:0/3523319064 wait complete. 2024-03-22T02:18:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.398+0000 7fc537be8740 20 client.16857 trim_cache size 0 max 16384 2024-03-22T02:18:17.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 326814 11.96 MB/sec execute 59 sec latency 33.518 ms 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.415+0000 7fc537be8740 1 Processor -- start 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.415+0000 7fc537be8740 1 -- start start 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.415+0000 7fc537be8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x562947073aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.415+0000 7fc537be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc537be8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x562947076050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc537be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301ad2c0 con 0x5629470755b0 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc537be8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301aab90 con 0x56294707a720 2024-03-22T02:18:17.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc537be8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301a89e0 con 0x562947074bc0 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59994/0 (socket says 172.21.15.28:59994) 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 -- 172.21.15.28:0/3279640478 learned_addr learned my addr 172.21.15.28:0/3279640478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x562947073aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x562947076050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 -- 172.21.15.28:0/3279640478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 msgr2=0x562947073aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x562947073aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 -- 172.21.15.28:0/3279640478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 msgr2=0x562947076050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x562947076050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 -- 172.21.15.28:0/3279640478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56294707f330 con 0x5629470755b0 2024-03-22T02:18:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x562947076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5180bd350 tx=0x7fc51802c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ccfca6de8c3c4d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc517fff640 1 -- 172.21.15.28:0/3279640478 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180d6070 con 0x5629470755b0 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc517fff640 1 -- 172.21.15.28:0/3279640478 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180bd650 con 0x5629470755b0 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.416+0000 7fc517fff640 1 -- 172.21.15.28:0/3279640478 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51804b040 con 0x5629470755b0 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- 172.21.15.28:0/3279640478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 msgr2=0x562947085fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5180bd350 tx=0x7fc51802c1f0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- 172.21.15.28:0/3279640478 shutdown_connections 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56294707a720 0x562947076050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5629470755b0 0x562947085fe0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- 172.21.15.28:0/3279640478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562947074bc0 0x562947073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- 172.21.15.28:0/3279640478 >> 172.21.15.28:0/3279640478 conn(0x562947070130 msgr2=0x562947073390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- 172.21.15.28:0/3279640478 shutdown_connections 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- 172.21.15.28:0/3279640478 wait complete. 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- start start 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 Processor -- start 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x562947083d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947084da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301a9050 con 0x56294707a720 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301aa4c0 con 0x562947074bc0 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.417+0000 7fc537be8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5301a9720 con 0x5629470755b0 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc537be8740 10 client.0 _ll_register_callbacks cb 0x7fc5200136c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947084da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46706/0 (socket says 172.21.15.28:46706) 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x562947083d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52e59c640 1 -- 172.21.15.28:0/3332577091 learned_addr learned my addr 172.21.15.28:0/3332577091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3332577091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 msgr2=0x562947083d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x562947083d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 msgr2=0x562947083850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 -- 172.21.15.28:0/3332577091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562947085390 con 0x56294707a720 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52dd9b640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x562947083d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc52ed9d640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947084da0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc520000f80 tx=0x7fc52000fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f644f8287af7722 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc515ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52002e070 con 0x56294707a720 2024-03-22T02:18:17.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fc520009040 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5180bd650 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc520010bb0 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc537be8740 10 client.16863 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.418+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5629470f0ed0 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5180bd650 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fc520034070 con 0x56294707a720 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 10 client.16863 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 10 client.16863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 20 client.16863 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5629470f18d0 0x5629470f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 4 client.16863 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:18:17.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562946fe0e00 con 0x5629470f18d0 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5629470f18d0 0x5629470f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc52e59c640 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5629470f18d0 0x5629470f3cf0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8fac3c08aab8633 server_cookie=34b8de765747ab22 in_seq=0 out_seq=0 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.419+0000 7fc515ffb640 10 client.16863 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc515ffb640 1 -- 172.21.15.28:0/3332577091 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562946fe0e00 con 0x5629470f18d0 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc515ffb640 10 client.16863 handle_command_reply: tid=0 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 client.16863 shutdown 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 msgr2=0x562947084da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947084da0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc520000f80 tx=0x7fc52000fca0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 shutdown_connections 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5629470f18d0 0x5629470f3cf0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.420+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56294707a720 0x562947084da0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc515ffb640 0 client.16863 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5629470755b0 0x562947083d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 1 --2- 172.21.15.28:0/3332577091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562947074bc0 0x562947083850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 >> 172.21.15.28:0/3332577091 conn(0x562947070130 msgr2=0x562947073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 shutdown_connections 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 1 -- 172.21.15.28:0/3332577091 wait complete. 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc537be8740 20 client.16863 trim_cache size 0 max 16384 2024-03-22T02:18:17.430 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:17.431 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:18:17.431 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e03d6d67-661e-4993-9bdc-2233e4a8f6cf", 2024-03-22T02:18:17.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:18:17.431 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 msgr2=0x7fc51c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 0x7fc51c0a53e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc520005fd0 tx=0x7fc520009e00 comp rx=0 tx=0).stop 2024-03-22T02:18:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 msgr2=0x7fc5301984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc518000c00 tx=0x7fc51802c650 comp rx=0 tx=0).stop 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 shutdown_connections 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc51c0a2f20 0x7fc51c0a53e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5301020d0 0x7fc53019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5301016e0 0x7fc5301984a0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 --2- 172.21.15.28:0/2193836806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc530100d20 0x7fc530197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 >> 172.21.15.28:0/2193836806 conn(0x7fc5300fb790 msgr2=0x7fc5300ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 shutdown_connections 2024-03-22T02:18:17.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.421+0000 7fc534f1e640 1 -- 172.21.15.28:0/2193836806 wait complete. 2024-03-22T02:18:17.435 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:18:17.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:18:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 -- 172.21.15.28:0/3834461937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40100a90 msgr2=0x7f7c40100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 --2- 172.21.15.28:0/3834461937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40100a90 0x7f7c40100eb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009860 tx=0x7f7c3402bb70 comp rx=0 tx=0).stop 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 -- 172.21.15.28:0/3834461937 shutdown_connections 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 --2- 172.21.15.28:0/3834461937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40102640 0x7f7c40108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 --2- 172.21.15.28:0/3834461937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 0x7f7c40102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 --2- 172.21.15.28:0/3834461937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40100a90 0x7f7c40100eb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 -- 172.21.15.28:0/3834461937 >> 172.21.15.28:0/3834461937 conn(0x7f7c400fc020 msgr2=0x7f7c400fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 -- 172.21.15.28:0/3834461937 shutdown_connections 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.543+0000 7f7c46acf640 1 -- 172.21.15.28:0/3834461937 wait complete. 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 Processor -- start 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 -- start start 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 0x7f7c40197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40102640 0x7f7c4019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4010c700 con 0x7f7c40100a90 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4010c580 con 0x7f7c40102640 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c46acf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4010c880 con 0x7f7c40101c80 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60028/0 (socket says 172.21.15.28:60028) 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c4486b640 1 -- 172.21.15.28:0/2232991427 learned_addr learned my addr 172.21.15.28:0/2232991427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c4506c640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40102640 0x7f7c4019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.544+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 0x7f7c40197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 -- 172.21.15.28:0/2232991427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 msgr2=0x7f7c40197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 0x7f7c40197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 -- 172.21.15.28:0/2232991427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40102640 msgr2=0x7f7c4019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40102640 0x7f7c4019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 -- 172.21.15.28:0/2232991427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4019ea60 con 0x7f7c40100a90 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7c3402c0b0 tx=0x7f7c34032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c3dffb640 1 -- 172.21.15.28:0/2232991427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34041020 con 0x7f7c40100a90 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4019ecf0 con 0x7f7c40100a90 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c3dffb640 1 -- 172.21.15.28:0/2232991427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3402c9f0 con 0x7f7c40100a90 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c3dffb640 1 -- 172.21.15.28:0/2232991427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c34039040 con 0x7f7c40100a90 2024-03-22T02:18:17.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.545+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4019f180 con 0x7f7c40100a90 2024-03-22T02:18:17.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.547+0000 7f7c3dffb640 1 -- 172.21.15.28:0/2232991427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7c34044020 con 0x7f7c40100a90 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 Processor -- start 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 -- start start 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3dffb640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 0x7f7c300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ef51268c0 con 0x556ef51c4030 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ef50cec20 con 0x556ef51c91a0 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4978b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ef5122870 con 0x556ef51c3640 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4506c640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 0x7f7c300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3dffb640 1 -- 172.21.15.28:0/2232991427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7c340f1140 con 0x7f7c40100a90 2024-03-22T02:18:17.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60044/0 (socket says 172.21.15.28:60044) 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46170/0 (socket says 172.21.15.28:46170) 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3ffff640 1 -- 172.21.15.28:0/736923825 learned_addr learned my addr 172.21.15.28:0/736923825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4506c640 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 -- 172.21.15.28:0/736923825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 msgr2=0x556ef51cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 -- 172.21.15.28:0/736923825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 msgr2=0x556ef51c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c4486b640 1 -- 172.21.15.28:0/736923825 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ef50f3150 con 0x556ef51c3640 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.548+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4506c640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 0x7f7c300a53e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7c38005fd0 tx=0x7f7c38009e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4486b640 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c34004650 tx=0x7f7c340045c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=599a74997051a361 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c2affd640 1 -- 172.21.15.28:0/736923825 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c340bcd60 con 0x556ef51c3640 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c2affd640 1 -- 172.21.15.28:0/736923825 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340bdb00 con 0x556ef51c3640 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c2affd640 1 -- 172.21.15.28:0/736923825 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c340d6730 con 0x556ef51c3640 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 -- 172.21.15.28:0/736923825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 msgr2=0x556ef51c3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7c34004650 tx=0x7f7c340045c0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 -- 172.21.15.28:0/736923825 shutdown_connections 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 --2- 172.21.15.28:0/736923825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51c3a60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.549+0000 7f7c4978b740 1 -- 172.21.15.28:0/736923825 >> 172.21.15.28:0/736923825 conn(0x556ef51bebb0 msgr2=0x556ef51befb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- 172.21.15.28:0/736923825 shutdown_connections 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- 172.21.15.28:0/736923825 wait complete. 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- start start 2024-03-22T02:18:17.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 Processor -- start 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 0x556ef51cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 0x556ef51d4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ef51cbf10 con 0x556ef51c4030 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ef50e2130 con 0x556ef51c3640 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ef51cc090 con 0x556ef51c91a0 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4978b740 10 client.0 _ll_register_callbacks cb 0x556ef50e1b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 0x556ef51cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60058/0 (socket says 172.21.15.28:60058) 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c3ffff640 1 -- 172.21.15.28:0/3674582119 learned_addr learned my addr 172.21.15.28:0/3674582119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.550+0000 7f7c4506c640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 0x556ef51d4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 -- 172.21.15.28:0/3674582119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 msgr2=0x556ef51d4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 0x556ef51d4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 -- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 msgr2=0x556ef51cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 0x556ef51cd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 -- 172.21.15.28:0/3674582119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ef4f047d0 con 0x556ef51c4030 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 0x556ef51cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c4506c640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 0x556ef51d4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c012700 tx=0x7f7c2c012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa373ee6192bf4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2c004050 con 0x556ef51c4030 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7c2c0041f0 con 0x556ef51c4030 2024-03-22T02:18:17.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c340bdb00 con 0x556ef51c4030 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c2c0043f0 con 0x556ef51c4030 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c4978b740 10 client.16875 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ef4e3a3f0 con 0x556ef51c4030 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.551+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340bdb00 con 0x556ef51c4030 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f7c2c003c70 con 0x556ef51c4030 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 10 client.16875 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 20 client.16875 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef51d9810 0x556ef51dbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 4 client.16875 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ef5133e00 con 0x556ef51d9810 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef51d9810 0x556ef51dbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c1ffff640 10 client.16875 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.552+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef51d9810 0x556ef51dbc50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7479c59b7b53712b server_cookie=1bde95b8f4f550b7 in_seq=0 out_seq=0 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c1ffff640 1 -- 172.21.15.28:0/3674582119 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ef5133e00 con 0x556ef51d9810 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c1ffff640 10 client.16875 handle_command_reply: tid=0 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 client.16875 shutdown 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 msgr2=0x556ef51cde10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c012700 tx=0x7f7c2c012bc0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 shutdown_connections 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef51d9810 0x556ef51dbc50 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c91a0 0x556ef51d4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cde10 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3674582119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c3640 0x556ef51cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c1ffff640 0 client.16875 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 >> 172.21.15.28:0/3674582119 conn(0x556ef51bebb0 msgr2=0x556ef51bbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 shutdown_connections 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 1 -- 172.21.15.28:0/3674582119 wait complete. 2024-03-22T02:18:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.555+0000 7f7c4978b740 20 client.16875 trim_cache size 0 max 16384 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 Processor -- start 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 -- start start 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c4030 0x556ef51cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51bd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401ab0c0 con 0x556ef51c3640 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401a8990 con 0x556ef51c91a0 2024-03-22T02:18:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.583+0000 7f7c4978b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401a67e0 con 0x556ef51c4030 2024-03-22T02:18:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.584+0000 7f7c3ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c4030 0x556ef51cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.585+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.585+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60074/0 (socket says 172.21.15.28:60074) 2024-03-22T02:18:17.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.585+0000 7f7c4486b640 1 -- 172.21.15.28:0/3586009226 learned_addr learned my addr 172.21.15.28:0/3586009226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.585+0000 7f7c4506c640 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51bd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 -- 172.21.15.28:0/3586009226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c4030 msgr2=0x556ef51cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c4030 0x556ef51cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 -- 172.21.15.28:0/3586009226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 msgr2=0x556ef51bd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51bd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 -- 172.21.15.28:0/3586009226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ef4dfa0f0 con 0x556ef51c3640 2024-03-22T02:18:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.586+0000 7f7c4486b640 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009830 tx=0x7f7c34032970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22cdc7795cc38b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.587+0000 7f7c2a7fc640 1 -- 172.21.15.28:0/3586009226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c340d6910 con 0x556ef51c3640 2024-03-22T02:18:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.587+0000 7f7c2a7fc640 1 -- 172.21.15.28:0/3586009226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340bdb00 con 0x556ef51c3640 2024-03-22T02:18:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.587+0000 7f7c2a7fc640 1 -- 172.21.15.28:0/3586009226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c340042c0 con 0x556ef51c3640 2024-03-22T02:18:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 -- 172.21.15.28:0/3586009226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 msgr2=0x556ef51cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009830 tx=0x7f7c34032970 comp rx=0 tx=0).stop 2024-03-22T02:18:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 -- 172.21.15.28:0/3586009226 shutdown_connections 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef51bd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c4030 0x556ef51cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 --2- 172.21.15.28:0/3586009226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c3640 0x556ef51cc380 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 -- 172.21.15.28:0/3586009226 >> 172.21.15.28:0/3586009226 conn(0x556ef51bebb0 msgr2=0x556ef51c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.588+0000 7f7c4978b740 1 -- 172.21.15.28:0/3586009226 shutdown_connections 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.589+0000 7f7c4978b740 1 -- 172.21.15.28:0/3586009226 wait complete. 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.589+0000 7f7c4978b740 1 -- start start 2024-03-22T02:18:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.589+0000 7f7c4978b740 1 Processor -- start 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef5244380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401a6e50 con 0x556ef51c4030 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401a82c0 con 0x556ef51c91a0 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7c401a7520 con 0x556ef51c3640 2024-03-22T02:18:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4978b740 10 client.0 _ll_register_callbacks cb 0x7f7c2c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4486b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46200/0 (socket says 172.21.15.28:46200) 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4486b640 1 -- 172.21.15.28:0/2876072802 learned_addr learned my addr 172.21.15.28:0/2876072802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.590+0000 7f7c4506c640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef5244380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 -- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 msgr2=0x556ef5244380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef5244380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 -- 172.21.15.28:0/2876072802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 msgr2=0x556ef51cfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 -- 172.21.15.28:0/2876072802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ef51d4e40 con 0x556ef51c3640 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c4486b640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c340bd100 tx=0x7f7c340b7f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a3d7c790c7a4fbb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.591+0000 7f7c1ffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:18:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c340ee850 con 0x556ef51c3640 2024-03-22T02:18:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f7c340b9a70 con 0x556ef51c3640 2024-03-22T02:18:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c340bdb00 con 0x556ef51c3640 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7c340c11b0 con 0x556ef51c3640 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c4978b740 10 client.26716 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.592+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ef52457c0 con 0x556ef51c3640 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c340db1c0 con 0x556ef51c3640 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f7c3403c030 con 0x556ef51c3640 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 10 client.26716 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 10 client.26716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 20 client.26716 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef5245d40 0x556ef5248130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 4 client.26716 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.593+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f7c340bdb00 con 0x556ef5245d40 2024-03-22T02:18:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.594+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef5245d40 0x556ef5248130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.597+0000 7f7c3ffff640 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef5245d40 0x556ef5248130 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccff4b49e5560428 server_cookie=d8eefa5e7f3629d3 in_seq=0 out_seq=0 2024-03-22T02:18:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c1ffff640 10 client.26716 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c1ffff640 1 -- 172.21.15.28:0/2876072802 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f7c340bdb00 con 0x556ef5245d40 2024-03-22T02:18:17.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c1ffff640 10 client.26716 handle_command_reply: tid=0 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c4978b740 1 client.26716 shutdown 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 msgr2=0x556ef51cf820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c340bd100 tx=0x7f7c340b7f90 comp rx=0 tx=0).stop 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.598+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 shutdown_connections 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ef5245d40 0x556ef5248130 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ef51c91a0 0x556ef5244380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ef51c4030 0x556ef51cfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 --2- 172.21.15.28:0/2876072802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ef51c3640 0x556ef51cf820 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 >> 172.21.15.28:0/2876072802 conn(0x556ef51bebb0 msgr2=0x556ef51cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c1ffff640 0 client.26716 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 shutdown_connections 2024-03-22T02:18:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 1 -- 172.21.15.28:0/2876072802 wait complete. 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.599+0000 7f7c4978b740 20 client.26716 trim_cache size 0 max 16384 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stdout: "e03d6d67-661e-4993-9bdc-2233e4a8f6cf": { 2024-03-22T02:18:17.601 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:18:17.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e03d6d67-661e-4993-9bdc-2233e4a8f6cf", 2024-03-22T02:18:17.602 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:18:17.602 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:18:17.602 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:18:17.602 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.600+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 msgr2=0x7f7c300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.600+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 0x7f7c300a53e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7c38005fd0 tx=0x7f7c38009e30 comp rx=0 tx=0).stop 2024-03-22T02:18:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.600+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 msgr2=0x7f7c40196e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.600+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f7c3402c0b0 tx=0x7f7c34032430 comp rx=0 tx=0).stop 2024-03-22T02:18:17.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 shutdown_connections 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7c300a2f20 0x7f7c300a53e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7c40102640 0x7f7c4019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c40101c80 0x7f7c40197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 --2- 172.21.15.28:0/2232991427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c40100a90 0x7f7c40196e40 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.602+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 >> 172.21.15.28:0/2232991427 conn(0x7f7c400fc020 msgr2=0x7f7c401050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.603+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 shutdown_connections 2024-03-22T02:18:17.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:17.603+0000 7f7c46acf640 1 -- 172.21.15.28:0/2232991427 wait complete. 2024-03-22T02:18:17.617 INFO:tasks.cephfs.filesystem:scrub status for tag:e03d6d67-661e-4993-9bdc-2233e4a8f6cf - {'path': '/', 'tag': 'e03d6d67-661e-4993-9bdc-2233e4a8f6cf', 'options': 'recursive,force'} 2024-03-22T02:18:18.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 329511 11.98 MB/sec execute 60 sec latency 25.528 ms 2024-03-22T02:18:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:18 smithi120 ceph-mon[36803]: pgmap v499: 129 pgs: 129 active+clean; 42 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-03-22T02:18:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:18 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:18:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:18 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:18.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:18 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:18:18.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:18 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:18.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:18 smithi083 ceph-mon[36340]: pgmap v499: 129 pgs: 129 active+clean; 42 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-03-22T02:18:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:18 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:18:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:18 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:18 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:18:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:18 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:18 smithi028 ceph-mon[31916]: pgmap v499: 129 pgs: 129 active+clean; 42 MiB data, 1011 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 397 op/s 2024-03-22T02:18:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:18 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:18:18.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:18 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:18.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:18 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:18:18.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:18 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:19.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 330753 11.89 MB/sec execute 61 sec latency 74.752 ms 2024-03-22T02:18:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:19 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:18:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:19 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:19.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:19 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:18:19.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:19 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:19 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:18:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:19 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:20.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 333547 11.97 MB/sec execute 62 sec latency 19.915 ms 2024-03-22T02:18:20.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:20 smithi028 ceph-mon[31916]: pgmap v500: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:18:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:20 smithi120 ceph-mon[36803]: pgmap v500: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:18:20.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:20 smithi083 ceph-mon[36340]: pgmap v500: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.3 MiB/s wr, 398 op/s 2024-03-22T02:18:21.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 335805 11.91 MB/sec execute 63 sec latency 24.862 ms 2024-03-22T02:18:22.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 337718 11.92 MB/sec execute 64 sec latency 29.491 ms 2024-03-22T02:18:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:22 smithi120 ceph-mon[36803]: pgmap v501: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-03-22T02:18:22.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:22 smithi083 ceph-mon[36340]: pgmap v501: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-03-22T02:18:22.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:22 smithi028 ceph-mon[31916]: pgmap v501: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 375 op/s 2024-03-22T02:18:23.414 INFO:tasks.workunit.client.0.smithi028.stdout: 1 340499 12.00 MB/sec execute 65 sec latency 101.260 ms 2024-03-22T02:18:23.486 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 -- 172.21.15.28:0/1778302375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0102dd0 msgr2=0x7fecc0103250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 --2- 172.21.15.28:0/1778302375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0102dd0 0x7fecc0103250 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009860 tx=0x7fecb402bb70 comp rx=0 tx=0).stop 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 -- 172.21.15.28:0/1778302375 shutdown_connections 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 --2- 172.21.15.28:0/1778302375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0103790 0x7fecc010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 --2- 172.21.15.28:0/1778302375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0102dd0 0x7fecc0103250 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 --2- 172.21.15.28:0/1778302375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0101be0 0x7fecc0102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 -- 172.21.15.28:0/1778302375 >> 172.21.15.28:0/1778302375 conn(0x7fecc00fd150 msgr2=0x7fecc00ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 -- 172.21.15.28:0/1778302375 shutdown_connections 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.483+0000 7fecc5884640 1 -- 172.21.15.28:0/1778302375 wait complete. 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 Processor -- start 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 -- start start 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0102dd0 0x7fecc019a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0103790 0x7fecc01a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d860 con 0x7fecc0101be0 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d6e0 con 0x7fecc0102dd0 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecc5884640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecc010d9e0 con 0x7fecc0103790 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32850/0 (socket says 172.21.15.28:32850) 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 -- 172.21.15.28:0/3769374615 learned_addr learned my addr 172.21.15.28:0/3769374615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0103790 0x7fecc01a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0102dd0 0x7fecc019a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 -- 172.21.15.28:0/3769374615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0103790 msgr2=0x7fecc01a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0103790 0x7fecc01a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.484+0000 7fecbeffd640 1 -- 172.21.15.28:0/3769374615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0102dd0 msgr2=0x7fecc019a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecbeffd640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0102dd0 0x7fecc019a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecbeffd640 1 -- 172.21.15.28:0/3769374615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc01a1c50 con 0x7fecc0101be0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecbeffd640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fecb800e9f0 tx=0x7fecb800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecc4882640 1 -- 172.21.15.28:0/3769374615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb800ccc0 con 0x7fecc0101be0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecc4882640 1 -- 172.21.15.28:0/3769374615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb80035f0 con 0x7fecc0101be0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc01a1f40 con 0x7fecc0101be0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecc4882640 1 -- 172.21.15.28:0/3769374615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb800ce60 con 0x7fecc0101be0 2024-03-22T02:18:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.485+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc01a2350 con 0x7fecc0101be0 2024-03-22T02:18:23.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.487+0000 7fecc4882640 1 -- 172.21.15.28:0/3769374615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fecb8003b10 con 0x7fecc0101be0 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 Processor -- start 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 -- start start 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef91d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef91e040 0x5616ef9204d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5616ef87d040 con 0x5616ef91e040 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5616ef8259e0 con 0x5616ef9184e0 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc846d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5616ef878ff0 con 0x5616ef918ed0 2024-03-22T02:18:23.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc4882640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 0x7feca80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef91d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51800/0 (socket says 172.21.15.28:51800) 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbf7fe640 1 -- 172.21.15.28:0/2617396291 learned_addr learned my addr 172.21.15.28:0/2617396291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef91e040 0x5616ef9204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 0x7feca80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.488+0000 7fecc4882640 1 -- 172.21.15.28:0/3769374615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fecb80cfab0 con 0x7fecc0101be0 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 -- 172.21.15.28:0/2617396291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 msgr2=0x5616ef918900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 -- 172.21.15.28:0/2617396291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef91e040 msgr2=0x5616ef9204d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef91e040 0x5616ef9204d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 -- 172.21.15.28:0/2617396291 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ef591210 con 0x5616ef918ed0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef91d9d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd170 tx=0x7fecb801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a0c4ad63211159 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 0x7feca80a5480 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5616ef919730 tx=0x7fecb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7feca5ffb640 1 -- 172.21.15.28:0/2617396291 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb809bdd0 con 0x5616ef918ed0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7feca5ffb640 1 -- 172.21.15.28:0/2617396291 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb809ff20 con 0x5616ef918ed0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.489+0000 7feca5ffb640 1 -- 172.21.15.28:0/2617396291 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8098440 con 0x5616ef918ed0 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- 172.21.15.28:0/2617396291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 msgr2=0x5616ef91d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef91d9d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd170 tx=0x7fecb801d000 comp rx=0 tx=0).stop 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- 172.21.15.28:0/2617396291 shutdown_connections 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef91e040 0x5616ef9204d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef91d9d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 --2- 172.21.15.28:0/2617396291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef9184e0 0x5616ef918900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- 172.21.15.28:0/2617396291 >> 172.21.15.28:0/2617396291 conn(0x5616ef913a50 msgr2=0x5616ef913e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- 172.21.15.28:0/2617396291 shutdown_connections 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- 172.21.15.28:0/2617396291 wait complete. 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 -- start start 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.490+0000 7fecc846d740 1 Processor -- start 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9229c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef929a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5616ef920f90 con 0x5616ef9184e0 2024-03-22T02:18:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5616ef838db0 con 0x5616ef918ed0 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5616ef921110 con 0x5616ef91e040 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 10 client.0 _ll_register_callbacks cb 0x5616ef590a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32860/0 (socket says 172.21.15.28:32860) 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 -- 172.21.15.28:0/2565524495 learned_addr learned my addr 172.21.15.28:0/2565524495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbe7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9229c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef929a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 -- 172.21.15.28:0/2565524495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 msgr2=0x5616ef929a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef929a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 -- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 msgr2=0x5616ef9229c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9229c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 -- 172.21.15.28:0/2565524495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ef65d620 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecbeffd640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd1f0 tx=0x7fecb80039b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b771412fff10a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fec9effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8099e70 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fecb80bab40 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb809ff20 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fecb809bdd0 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 10 client.16893 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.491+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5616ef92cd70 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb809ff20 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fecb809ad20 con 0x5616ef9184e0 2024-03-22T02:18:23.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 10 client.16893 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 10 client.16893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 20 client.16893 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef92e0c0 0x5616ef9304b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 4 client.16893 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616ef88a580 con 0x5616ef92e0c0 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.492+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef92e0c0 0x5616ef9304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef92e0c0 0x5616ef9304b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f150a035034d37fc server_cookie=23d81ebcf43441b3 in_seq=0 out_seq=0 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fec9effd640 10 client.16893 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fec9effd640 1 -- 172.21.15.28:0/2565524495 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5616ef88a580 con 0x5616ef92e0c0 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fec9effd640 10 client.16893 handle_command_reply: tid=0 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 client.16893 shutdown 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 msgr2=0x5616ef922480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd1f0 tx=0x7fecb80039b0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 shutdown_connections 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef92e0c0 0x5616ef9304b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef929a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9229c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fec9effd640 0 client.16893 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 --2- 172.21.15.28:0/2565524495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef922480 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 >> 172.21.15.28:0/2565524495 conn(0x5616ef913a50 msgr2=0x5616ef912c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 shutdown_connections 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.493+0000 7fecc846d740 1 -- 172.21.15.28:0/2565524495 wait complete. 2024-03-22T02:18:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.494+0000 7fecc846d740 20 client.16893 trim_cache size 0 max 16384 2024-03-22T02:18:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 Processor -- start 2024-03-22T02:18:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 -- start start 2024-03-22T02:18:23.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9299c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef9123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01af360 con 0x5616ef9184e0 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01acc30 con 0x5616ef918ed0 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecc846d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01aaa80 con 0x5616ef91e040 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32868/0 (socket says 172.21.15.28:32868) 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 -- 172.21.15.28:0/236422884 learned_addr learned my addr 172.21.15.28:0/236422884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbf7fe640 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef9123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 -- 172.21.15.28:0/236422884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 msgr2=0x5616ef9123c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef9123c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 -- 172.21.15.28:0/236422884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 msgr2=0x5616ef9299c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9299c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 -- 172.21.15.28:0/236422884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616ef91d130 con 0x5616ef9184e0 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.509+0000 7fecbeffd640 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd510 tx=0x7fecb809b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52fd9aff43a0e815 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.510+0000 7feca57fa640 1 -- 172.21.15.28:0/236422884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb807f070 con 0x5616ef9184e0 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.510+0000 7feca57fa640 1 -- 172.21.15.28:0/236422884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8099e70 con 0x5616ef9184e0 2024-03-22T02:18:23.512 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.510+0000 7feca57fa640 1 -- 172.21.15.28:0/236422884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb809ac90 con 0x5616ef9184e0 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.510+0000 7fecc846d740 1 -- 172.21.15.28:0/236422884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 msgr2=0x5616ef9227a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.510+0000 7fecc846d740 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fecb80cd510 tx=0x7fecb809b7f0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- 172.21.15.28:0/236422884 shutdown_connections 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef91e040 0x5616ef9123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef918ed0 0x5616ef9299c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- 172.21.15.28:0/236422884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9227a0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- 172.21.15.28:0/236422884 >> 172.21.15.28:0/236422884 conn(0x5616ef913a50 msgr2=0x5616ef922090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- 172.21.15.28:0/236422884 shutdown_connections 2024-03-22T02:18:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- 172.21.15.28:0/236422884 wait complete. 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- start start 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 Processor -- start 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 0x5616ef928a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ab0f0 con 0x5616ef9184e0 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ac560 con 0x5616ef91e040 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecc01ab7c0 con 0x5616ef918ed0 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecc846d740 10 client.0 _ll_register_callbacks cb 0x7fecb00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 0x5616ef928a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbe7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55150/0 (socket says 172.21.15.28:55150) 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32874/0 (socket says 172.21.15.28:32874) 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 0x5616ef928a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51832/0 (socket says 172.21.15.28:51832) 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.511+0000 7fecbe7fc640 1 -- 172.21.15.28:0/1323343372 learned_addr learned my addr 172.21.15.28:0/1323343372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 -- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 msgr2=0x5616ef928a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 0x5616ef928a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 -- 172.21.15.28:0/1323343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 msgr2=0x5616ef9274c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 -- 172.21.15.28:0/1323343372 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616ef929000 con 0x5616ef918ed0 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbeffd640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecbe7fc640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fecb40057c0 tx=0x7fecb402c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97f8d09d40af4d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fec9effd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4039040 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5616ef8848d0 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fecb4004040 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fecb4059050 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecc846d740 10 client.26734 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.512+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5616ef99cda0 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5616ef8848d0 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fecb40326b0 con 0x5616ef918ed0 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 10 client.26734 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 10 client.26734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 20 client.26734 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef99d780 0x5616ef99fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 4 client.26734 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:18:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5616ef802770 con 0x5616ef99d780 2024-03-22T02:18:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecbeffd640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef99d780 0x5616ef99fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fec9effd640 10 client.26734 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.513+0000 7fecbeffd640 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef99d780 0x5616ef99fba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea554bc9e27408e5 server_cookie=6bc13d7c88543243 in_seq=0 out_seq=0 2024-03-22T02:18:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fec9effd640 1 -- 172.21.15.28:0/1323343372 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5616ef802770 con 0x5616ef99d780 2024-03-22T02:18:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fec9effd640 10 client.26734 handle_command_reply: tid=0 2024-03-22T02:18:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 client.26734 shutdown 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 msgr2=0x5616ef927a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fecb40057c0 tx=0x7fecb402c320 comp rx=0 tx=0).stop 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 shutdown_connections 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5616ef99d780 0x5616ef99fba0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5616ef91e040 0x5616ef928a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5616ef918ed0 0x5616ef927a00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fec9effd640 0 client.26734 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 --2- 172.21.15.28:0/1323343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5616ef9184e0 0x5616ef9274c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 >> 172.21.15.28:0/1323343372 conn(0x5616ef913a50 msgr2=0x5616ef922890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 shutdown_connections 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 1 -- 172.21.15.28:0/1323343372 wait complete. 2024-03-22T02:18:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.795+0000 7fecc846d740 20 client.26734 trim_cache size 0 max 16384 2024-03-22T02:18:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:18:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:18:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:18:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:18:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 msgr2=0x7feca80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 0x7feca80a5480 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5616ef919730 tx=0x7fecb0009500 comp rx=0 tx=0).stop 2024-03-22T02:18:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 msgr2=0x7fecc019a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fecb800e9f0 tx=0x7fecb800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 shutdown_connections 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feca80a2fc0 0x7feca80a5480 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecc0103790 0x7fecc01a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecc0102dd0 0x7fecc019a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 --2- 172.21.15.28:0/3769374615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecc0101be0 0x7fecc019a030 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 >> 172.21.15.28:0/3769374615 conn(0x7fecc00fd150 msgr2=0x7fecc0073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 shutdown_connections 2024-03-22T02:18:23.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:23.796+0000 7fecc5884640 1 -- 172.21.15.28:0/3769374615 wait complete. 2024-03-22T02:18:24.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 341328 11.85 MB/sec execute 66 sec latency 136.849 ms 2024-03-22T02:18:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:24 smithi120 ceph-mon[36803]: pgmap v502: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 434 op/s 2024-03-22T02:18:24.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:24 smithi083 ceph-mon[36340]: pgmap v502: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 434 op/s 2024-03-22T02:18:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:24 smithi028 ceph-mon[31916]: pgmap v502: 129 pgs: 129 active+clean; 50 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.4 MiB/s wr, 434 op/s 2024-03-22T02:18:25.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 344072 11.93 MB/sec execute 67 sec latency 20.962 ms 2024-03-22T02:18:25.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:25 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:18:25.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:25 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:18:25.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:25 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:18:26.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 346464 11.88 MB/sec execute 68 sec latency 26.632 ms 2024-03-22T02:18:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:26 smithi120 ceph-mon[36803]: pgmap v503: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-03-22T02:18:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:26 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:18:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:26 smithi083 ceph-mon[36340]: pgmap v503: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-03-22T02:18:26.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:26 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:18:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:26 smithi028 ceph-mon[31916]: pgmap v503: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-03-22T02:18:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:26 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:18:27.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 348401 11.89 MB/sec execute 69 sec latency 28.567 ms 2024-03-22T02:18:28.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 351295 11.97 MB/sec execute 70 sec latency 30.704 ms 2024-03-22T02:18:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:28 smithi120 ceph-mon[36803]: pgmap v504: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2024-03-22T02:18:28.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:28 smithi083 ceph-mon[36340]: pgmap v504: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2024-03-22T02:18:28.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:28 smithi028 ceph-mon[31916]: pgmap v504: 129 pgs: 129 active+clean; 42 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 316 op/s 2024-03-22T02:18:29.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 352853 11.89 MB/sec execute 71 sec latency 52.204 ms 2024-03-22T02:18:30.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 355166 11.92 MB/sec execute 72 sec latency 17.014 ms 2024-03-22T02:18:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:30 smithi120 ceph-mon[36803]: pgmap v505: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 412 op/s 2024-03-22T02:18:30.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:30 smithi083 ceph-mon[36340]: pgmap v505: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 412 op/s 2024-03-22T02:18:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:30 smithi028 ceph-mon[31916]: pgmap v505: 129 pgs: 129 active+clean; 44 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.2 MiB/s wr, 412 op/s 2024-03-22T02:18:31.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 357691 11.93 MB/sec execute 73 sec latency 26.143 ms 2024-03-22T02:18:32.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 359507 11.89 MB/sec execute 74 sec latency 42.028 ms 2024-03-22T02:18:32.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:32 smithi120 ceph-mon[36803]: pgmap v506: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 428 op/s 2024-03-22T02:18:32.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:32 smithi083 ceph-mon[36340]: pgmap v506: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 428 op/s 2024-03-22T02:18:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:32 smithi028 ceph-mon[31916]: pgmap v506: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 428 op/s 2024-03-22T02:18:33.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 362241 11.95 MB/sec execute 75 sec latency 11.694 ms 2024-03-22T02:18:34.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 364569 11.94 MB/sec execute 76 sec latency 53.717 ms 2024-03-22T02:18:34.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:34 smithi120 ceph-mon[36803]: pgmap v507: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 422 op/s 2024-03-22T02:18:34.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:34 smithi083 ceph-mon[36340]: pgmap v507: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 422 op/s 2024-03-22T02:18:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:34 smithi028 ceph-mon[31916]: pgmap v507: 129 pgs: 129 active+clean; 46 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 422 op/s 2024-03-22T02:18:35.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 366252 11.90 MB/sec execute 77 sec latency 28.305 ms 2024-03-22T02:18:36.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 369138 11.98 MB/sec execute 78 sec latency 17.644 ms 2024-03-22T02:18:36.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:36 smithi120 ceph-mon[36803]: pgmap v508: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:36.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:36 smithi083 ceph-mon[36340]: pgmap v508: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:36.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:36 smithi028 ceph-mon[31916]: pgmap v508: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 426 op/s 2024-03-22T02:18:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.890+0000 7f75096e0640 1 -- 172.21.15.28:0/2870401376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75040ffe70 msgr2=0x7f7504100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.890+0000 7f75096e0640 1 --2- 172.21.15.28:0/2870401376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75040ffe70 0x7f7504100290 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009930 tx=0x7f74f402bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 -- 172.21.15.28:0/2870401376 shutdown_connections 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 --2- 172.21.15.28:0/2870401376 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7504103800 0x7f750410a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 --2- 172.21.15.28:0/2870401376 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504102e40 0x7f75041032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 --2- 172.21.15.28:0/2870401376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75040ffe70 0x7f7504100290 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 -- 172.21.15.28:0/2870401376 >> 172.21.15.28:0/2870401376 conn(0x7f75040fb7e0 msgr2=0x7f75040fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 -- 172.21.15.28:0/2870401376 shutdown_connections 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.891+0000 7f75096e0640 1 -- 172.21.15.28:0/2870401376 wait complete. 2024-03-22T02:18:36.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 Processor -- start 2024-03-22T02:18:36.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 -- start start 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040ffe70 0x7f7504197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504103800 0x7f750419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f750410d820 con 0x7f7504102e40 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f750410d6a0 con 0x7f7504103800 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75096e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f750410d9a0 con 0x7f75040ffe70 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38710/0 (socket says 172.21.15.28:38710) 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 -- 172.21.15.28:0/4215665466 learned_addr learned my addr 172.21.15.28:0/4215665466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f7502ffd640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040ffe70 0x7f7504197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75037fe640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504103800 0x7f750419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 -- 172.21.15.28:0/4215665466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040ffe70 msgr2=0x7f7504197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040ffe70 0x7f7504197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 -- 172.21.15.28:0/4215665466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504103800 msgr2=0x7f750419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504103800 0x7f750419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.892+0000 7f75027fc640 1 -- 172.21.15.28:0/4215665466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f750419faa0 con 0x7f7504102e40 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f75027fc640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f74ebfff640 1 -- 172.21.15.28:0/4215665466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f800cd80 con 0x7f7504102e40 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f750419fd90 con 0x7f7504102e40 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f74ebfff640 1 -- 172.21.15.28:0/4215665466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f8010910 con 0x7f7504102e40 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f74ebfff640 1 -- 172.21.15.28:0/4215665466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f8018990 con 0x7f7504102e40 2024-03-22T02:18:36.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.893+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75041a0280 con 0x7f7504102e40 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f74ebfff640 1 -- 172.21.15.28:0/4215665466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f74f8018b30 con 0x7f7504102e40 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 Processor -- start 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 -- start start 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cf38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d90c4c040 con 0x555d90cec2e0 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555d90bf49e0 con 0x555d90cf1450 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.895+0000 7f750c39d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555d90c47ff0 con 0x555d90ceb8f0 2024-03-22T02:18:36.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f75037fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38716/0 (socket says 172.21.15.28:38716) 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 -- 172.21.15.28:0/1700267208 learned_addr learned my addr 172.21.15.28:0/1700267208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f75027fc640 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cf38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f74ebfff640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 0x7f74ec0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 -- 172.21.15.28:0/1700267208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 msgr2=0x555d90cebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 -- 172.21.15.28:0/1700267208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 msgr2=0x555d90cf38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cf38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 -- 172.21.15.28:0/1700267208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d90960210 con 0x555d90cec2e0 2024-03-22T02:18:36.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f75037fe640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 0x7f74ec0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f74ebfff640 1 -- 172.21.15.28:0/4215665466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f74f80cfb10 con 0x7f7504102e40 2024-03-22T02:18:36.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f7502ffd640 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009900 tx=0x7f74f402c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57c42790bf1e953 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f74e97fa640 1 -- 172.21.15.28:0/1700267208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f402cb80 con 0x555d90cec2e0 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f74e97fa640 1 -- 172.21.15.28:0/1700267208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4005040 con 0x555d90cec2e0 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.896+0000 7f74e97fa640 1 -- 172.21.15.28:0/1700267208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4004920 con 0x555d90cec2e0 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 -- 172.21.15.28:0/1700267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 msgr2=0x555d90cf0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f75037fe640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 0x7f74ec0a5480 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009fd0 tx=0x7f74fc004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009900 tx=0x7f74f402c660 comp rx=0 tx=0).stop 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 -- 172.21.15.28:0/1700267208 shutdown_connections 2024-03-22T02:18:36.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cf38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf0de0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 --2- 172.21.15.28:0/1700267208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 -- 172.21.15.28:0/1700267208 >> 172.21.15.28:0/1700267208 conn(0x555d90ce2a50 msgr2=0x555d90ce2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 -- 172.21.15.28:0/1700267208 shutdown_connections 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.897+0000 7f750c39d740 1 -- 172.21.15.28:0/1700267208 wait complete. 2024-03-22T02:18:36.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 -- start start 2024-03-22T02:18:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 Processor -- start 2024-03-22T02:18:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 0x555d90cf58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90cfce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555d90cf43a0 con 0x555d90cec2e0 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555d90c07db0 con 0x555d90ceb8f0 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555d90cf4520 con 0x555d90cf1450 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f750c39d740 10 client.0 _ll_register_callbacks cb 0x555d9095fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 0x555d90cf58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38726/0 (socket says 172.21.15.28:38726) 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75037fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90cfce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 -- 172.21.15.28:0/444166261 learned_addr learned my addr 172.21.15.28:0/444166261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 -- 172.21.15.28:0/444166261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 msgr2=0x555d90cfce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90cfce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 -- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 msgr2=0x555d90cf58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 0x555d90cf58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 -- 172.21.15.28:0/444166261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d90a2c620 con 0x555d90cec2e0 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f7502ffd640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 0x555d90cf58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f75027fc640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f74f80988b0 tx=0x7f74f8098240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb4f39d6efad3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f74e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f809bee0 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.898+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f74f80baba0 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f4005040 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74f80d5070 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 10 client.16911 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d90d00250 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4005040 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f74f809b330 con 0x555d90cec2e0 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 10 client.16911 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 10 client.16911 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 20 client.16911 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d018d0 0x555d90d03d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 4 client.16911 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d90c59580 con 0x555d90d018d0 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.899+0000 7f7502ffd640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d018d0 0x555d90d03d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.900+0000 7f74e6ff5640 10 client.16911 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.900+0000 7f7502ffd640 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d018d0 0x555d90d03d20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a59829c0bce5ed server_cookie=9353028a904dfe32 in_seq=0 out_seq=0 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.901+0000 7f74e6ff5640 1 -- 172.21.15.28:0/444166261 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d90c59580 con 0x555d90d018d0 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.901+0000 7f74e6ff5640 10 client.16911 handle_command_reply: tid=0 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.901+0000 7f750c39d740 1 client.16911 shutdown 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.901+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 msgr2=0x555d90cf5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.901+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f74f80988b0 tx=0x7f74f8098240 comp rx=0 tx=0).stop 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 shutdown_connections 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d018d0 0x555d90d03d20 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90cfce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cf5de0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 --2- 172.21.15.28:0/444166261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90ceb8f0 0x555d90cf58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f74e6ff5640 0 client.16911 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 >> 172.21.15.28:0/444166261 conn(0x555d90ce2a50 msgr2=0x555d90ce1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 shutdown_connections 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 1 -- 172.21.15.28:0/444166261 wait complete. 2024-03-22T02:18:36.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.902+0000 7f750c39d740 20 client.16911 trim_cache size 0 max 16384 2024-03-22T02:18:36.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.918+0000 7f750c39d740 1 Processor -- start 2024-03-22T02:18:36.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.918+0000 7f750c39d740 1 -- start start 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cec2e0 0x555d90cfcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90d08620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75041aaeb0 con 0x555d90ceb8f0 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a8780 con 0x555d90cec2e0 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f750c39d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a65d0 con 0x555d90cf1450 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f75027fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cec2e0 0x555d90cfcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38728/0 (socket says 172.21.15.28:38728) 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 -- 172.21.15.28:0/4285223947 learned_addr learned my addr 172.21.15.28:0/4285223947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f75037fe640 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90d08620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 -- 172.21.15.28:0/4285223947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 msgr2=0x555d90d08620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90d08620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 -- 172.21.15.28:0/4285223947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cec2e0 msgr2=0x555d90cfcde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cec2e0 0x555d90cfcde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 -- 172.21.15.28:0/4285223947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d90cecb70 con 0x555d90ceb8f0 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f7502ffd640 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f74f402c170 tx=0x7f74f402c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfa4dd6f631f1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f74e8ff9640 1 -- 172.21.15.28:0/4285223947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4005630 con 0x555d90ceb8f0 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f74e8ff9640 1 -- 172.21.15.28:0/4285223947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4009520 con 0x555d90ceb8f0 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.919+0000 7f74e8ff9640 1 -- 172.21.15.28:0/4285223947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4003850 con 0x555d90ceb8f0 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 -- 172.21.15.28:0/4285223947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 msgr2=0x555d90ced490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f74f402c170 tx=0x7f74f402c330 comp rx=0 tx=0).stop 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 -- 172.21.15.28:0/4285223947 shutdown_connections 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90cf1450 0x555d90d08620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cec2e0 0x555d90cfcde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 --2- 172.21.15.28:0/4285223947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90ceb8f0 0x555d90ced490 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 -- 172.21.15.28:0/4285223947 >> 172.21.15.28:0/4285223947 conn(0x555d90ce2a50 msgr2=0x555d90cecd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 -- 172.21.15.28:0/4285223947 shutdown_connections 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.920+0000 7f750c39d740 1 -- 172.21.15.28:0/4285223947 wait complete. 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 -- start start 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 Processor -- start 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cfa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cfbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a6c40 con 0x555d90cec2e0 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a80b0 con 0x555d90cf1450 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75041a7310 con 0x555d90ceb8f0 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f750c39d740 10 client.0 _ll_register_callbacks cb 0x7f74f80d32e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:36.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38738/0 (socket says 172.21.15.28:38738) 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f7502ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cfa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 -- 172.21.15.28:0/1898356276 learned_addr learned my addr 172.21.15.28:0/1898356276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75037fe640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cfbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 -- 172.21.15.28:0/1898356276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 msgr2=0x555d90cfa840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cfa840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 -- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 msgr2=0x555d90cfbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cfbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 -- 172.21.15.28:0/1898356276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d90cfc380 con 0x555d90cec2e0 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75037fe640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cfbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:18:36.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f75027fc640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f74f80a4310 tx=0x7f74f809a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2340e9076802987b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.921+0000 7f74e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f809ba20 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f74f8010430 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f4009520 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74f8097fa0 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 10 client.16917 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d90d68070 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f4009520 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f74f80a4a90 con 0x555d90cec2e0 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 10 client.16917 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 10 client.16917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:36.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 20 client.16917 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d689c0 0x555d90d6ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 4 client.16917 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d90c538d0 con 0x555d90d689c0 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.922+0000 7f7502ffd640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d689c0 0x555d90d6ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.923+0000 7f7502ffd640 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d689c0 0x555d90d6ade0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5570cc056b3ace1 server_cookie=69e83be4f3c93413 in_seq=0 out_seq=0 2024-03-22T02:18:36.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:36.923+0000 7f74e6ff5640 10 client.16917 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f74e6ff5640 1 -- 172.21.15.28:0/1898356276 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d90c538d0 con 0x555d90d689c0 2024-03-22T02:18:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f74e6ff5640 10 client.16917 handle_command_reply: tid=0 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 client.16917 shutdown 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 msgr2=0x555d90cfad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f74f80a4310 tx=0x7f74f809a3f0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 shutdown_connections 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555d90d689c0 0x555d90d6ade0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555d90cf1450 0x555d90cfbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555d90cec2e0 0x555d90cfad80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 --2- 172.21.15.28:0/1898356276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555d90ceb8f0 0x555d90cfa840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 >> 172.21.15.28:0/1898356276 conn(0x555d90ce2a50 msgr2=0x555d90ced610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f74e6ff5640 0 client.16917 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 shutdown_connections 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.169+0000 7f750c39d740 1 -- 172.21.15.28:0/1898356276 wait complete. 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f750c39d740 20 client.16917 trim_cache size 0 max 16384 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:18:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:18:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 msgr2=0x7f74ec0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 0x7f74ec0a5480 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009fd0 tx=0x7f74fc004a40 comp rx=0 tx=0).stop 2024-03-22T02:18:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 msgr2=0x7f7504198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f74f800e9f0 tx=0x7f74f800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 shutdown_connections 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74ec0a2fc0 0x7f74ec0a5480 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7504103800 0x7f750419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7504102e40 0x7f7504198410 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 --2- 172.21.15.28:0/4215665466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75040ffe70 0x7f7504197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.170+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 >> 172.21.15.28:0/4215665466 conn(0x7f75040fb7e0 msgr2=0x7f75041061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.171+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 shutdown_connections 2024-03-22T02:18:37.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:37.171+0000 7f75096e0640 1 -- 172.21.15.28:0/4215665466 wait complete. 2024-03-22T02:18:37.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 370544 11.90 MB/sec execute 79 sec latency 112.912 ms 2024-03-22T02:18:38.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 372900 11.92 MB/sec execute 80 sec latency 15.518 ms 2024-03-22T02:18:38.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:38 smithi120 ceph-mon[36803]: pgmap v509: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 372 op/s 2024-03-22T02:18:38.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:38 smithi083 ceph-mon[36340]: pgmap v509: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 372 op/s 2024-03-22T02:18:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:38 smithi028 ceph-mon[31916]: pgmap v509: 129 pgs: 129 active+clean; 49 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 372 op/s 2024-03-22T02:18:39.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 375305 11.92 MB/sec execute 81 sec latency 51.952 ms 2024-03-22T02:18:40.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 376887 11.88 MB/sec execute 82 sec latency 42.960 ms 2024-03-22T02:18:40.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:40 smithi120 ceph-mon[36803]: pgmap v510: 129 pgs: 129 active+clean; 53 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 476 op/s 2024-03-22T02:18:40.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:40 smithi083 ceph-mon[36340]: pgmap v510: 129 pgs: 129 active+clean; 53 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 476 op/s 2024-03-22T02:18:40.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:40 smithi028 ceph-mon[31916]: pgmap v510: 129 pgs: 129 active+clean; 53 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.7 MiB/s wr, 476 op/s 2024-03-22T02:18:41.415 INFO:tasks.workunit.client.0.smithi028.stdout: 1 379845 11.95 MB/sec execute 83 sec latency 14.115 ms 2024-03-22T02:18:42.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 382011 11.90 MB/sec execute 84 sec latency 76.417 ms 2024-03-22T02:18:42.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:42 smithi120 ceph-mon[36803]: pgmap v511: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:18:42.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:42 smithi083 ceph-mon[36340]: pgmap v511: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:18:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:42 smithi028 ceph-mon[31916]: pgmap v511: 129 pgs: 129 active+clean; 48 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:18:43.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 383972 11.91 MB/sec execute 85 sec latency 29.997 ms 2024-03-22T02:18:44.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 386641 11.96 MB/sec execute 86 sec latency 84.055 ms 2024-03-22T02:18:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:44 smithi120 ceph-mon[36803]: pgmap v512: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-22T02:18:44.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:44 smithi083 ceph-mon[36340]: pgmap v512: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-22T02:18:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:44 smithi028 ceph-mon[31916]: pgmap v512: 129 pgs: 129 active+clean; 46 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.0 MiB/s wr, 412 op/s 2024-03-22T02:18:45.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 388169 11.90 MB/sec execute 87 sec latency 31.495 ms 2024-03-22T02:18:46.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 390653 11.93 MB/sec execute 88 sec latency 13.965 ms 2024-03-22T02:18:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:46 smithi120 ceph-mon[36803]: pgmap v513: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 439 op/s 2024-03-22T02:18:46.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:46 smithi083 ceph-mon[36340]: pgmap v513: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 439 op/s 2024-03-22T02:18:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:46 smithi028 ceph-mon[31916]: pgmap v513: 129 pgs: 129 active+clean; 46 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 439 op/s 2024-03-22T02:18:47.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 393147 11.93 MB/sec execute 89 sec latency 32.413 ms 2024-03-22T02:18:47.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:18:47.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:18:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 -- 172.21.15.28:0/3121771392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 msgr2=0x7f2db4102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 --2- 172.21.15.28:0/3121771392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4102110 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009860 tx=0x7f2da802bb70 comp rx=0 tx=0).stop 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 -- 172.21.15.28:0/3121771392 shutdown_connections 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 --2- 172.21.15.28:0/3121771392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4102650 0x7f2db4108ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 --2- 172.21.15.28:0/3121771392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4102110 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 --2- 172.21.15.28:0/3121771392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4100aa0 0x7f2db4100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 -- 172.21.15.28:0/3121771392 >> 172.21.15.28:0/3121771392 conn(0x7f2db40fc030 msgr2=0x7f2db40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 -- 172.21.15.28:0/3121771392 shutdown_connections 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 -- 172.21.15.28:0/3121771392 wait complete. 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.742+0000 7f2db9381640 1 Processor -- start 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 -- start start 2024-03-22T02:18:47.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4102650 0x7f2db419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410c700 con 0x7f2db4100aa0 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410c580 con 0x7f2db4101c90 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db9381640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2db410c880 con 0x7f2db4102650 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34442/0 (socket says 172.21.15.28:34442) 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 -- 172.21.15.28:0/439335227 learned_addr learned my addr 172.21.15.28:0/439335227 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db27fc640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db37fe640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4102650 0x7f2db419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 -- 172.21.15.28:0/439335227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4102650 msgr2=0x7f2db419e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4102650 0x7f2db419e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 -- 172.21.15.28:0/439335227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 msgr2=0x7f2db4197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 -- 172.21.15.28:0/439335227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db419ea20 con 0x7f2db4100aa0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db27fc640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.743+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e9a0 tx=0x7f2da400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.744+0000 7f2d9bfff640 1 -- 172.21.15.28:0/439335227 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da400ccc0 con 0x7f2db4100aa0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.744+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db419ed10 con 0x7f2db4100aa0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.744+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db419f1d0 con 0x7f2db4100aa0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.744+0000 7f2d9bfff640 1 -- 172.21.15.28:0/439335227 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da4009520 con 0x7f2db4100aa0 2024-03-22T02:18:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.744+0000 7f2d9bfff640 1 -- 172.21.15.28:0/439335227 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40187e0 con 0x7f2db4100aa0 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2d9bfff640 1 -- 172.21.15.28:0/439335227 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2da401f020 con 0x7f2db4100aa0 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2dbbf6b740 1 Processor -- start 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2d9bfff640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 0x7f2d9c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2dbbf6b740 1 -- start start 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2d9bfff640 1 -- 172.21.15.28:0/439335227 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2da40cf960 con 0x7f2db4100aa0 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.746+0000 7f2db37fe640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 0x7f2d9c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d33e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d343920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c19d2a38c0 con 0x55c19d343f90 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c19d24bc20 con 0x55c19d33ee20 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c19d29f870 con 0x55c19d33e430 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34450/0 (socket says 172.21.15.28:34450) 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db37fe640 1 -- 172.21.15.28:0/4057862451 learned_addr learned my addr 172.21.15.28:0/4057862451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d33e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db27fc640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d343920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 -- 172.21.15.28:0/4057862451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 msgr2=0x55c19d343920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d343920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 -- 172.21.15.28:0/4057862451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 msgr2=0x55c19d346420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 -- 172.21.15.28:0/4057862451 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c19d270150 con 0x55c19d33e430 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db27fc640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d343920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db37fe640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db37fe640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 0x7f2d9c0a5460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2dac013d10 tx=0x7f2dac013800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d33e850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2da409b920 tx=0x7f2da409c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18da496ce2ac5ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2d997fa640 1 -- 172.21.15.28:0/4057862451 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40a4630 con 0x55c19d33e430 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2d997fa640 1 -- 172.21.15.28:0/4057862451 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da40cd100 con 0x55c19d33e430 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.747+0000 7f2d997fa640 1 -- 172.21.15.28:0/4057862451 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40a4400 con 0x55c19d33e430 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/4057862451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 msgr2=0x55c19d33e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d33e850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2da409b920 tx=0x7f2da409c3e0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/4057862451 shutdown_connections 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d346420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d343920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/4057862451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d33e850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/4057862451 >> 172.21.15.28:0/4057862451 conn(0x55c19d3399a0 msgr2=0x55c19d339da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/4057862451 shutdown_connections 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/4057862451 wait complete. 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.748+0000 7f2dbbf6b740 1 -- start start 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 Processor -- start 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d348900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d348e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c19d346f40 con 0x55c19d343f90 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c19d25f130 con 0x55c19d33ee20 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c19d3470c0 con 0x55c19d33e430 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 10 client.0 _ll_register_callbacks cb 0x55c19d25eb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34452/0 (socket says 172.21.15.28:34452) 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 -- 172.21.15.28:0/3626157026 learned_addr learned my addr 172.21.15.28:0/3626157026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d348900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db27fc640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d348e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 -- 172.21.15.28:0/3626157026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 msgr2=0x55c19d348900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d348900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 -- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 msgr2=0x55c19d348e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d348e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 -- 172.21.15.28:0/3626157026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c19d0817d0 con 0x55c19d343f90 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2db37fe640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2dac003ba0 tx=0x7f2dac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3950bee6e6d9ae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2d96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac025070 con 0x55c19d343f90 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2dac010430 con 0x55c19d343f90 2024-03-22T02:18:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.749+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2da40cd100 con 0x55c19d343f90 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2dac031050 con 0x55c19d343f90 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 10 client.16929 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c19cfb73f0 con 0x55c19d343f90 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da40cd100 con 0x55c19d343f90 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f2dac01f040 con 0x55c19d343f90 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 10 client.16929 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 10 client.16929 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 20 client.16929 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3547e0 0x55c19d356c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 4 client.16929 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c19d2b0e00 con 0x55c19d3547e0 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.750+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3547e0 0x55c19d356c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.751+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3547e0 0x55c19d356c40 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8512b14d88243cf2 server_cookie=c09cf72c7fde8e6d in_seq=0 out_seq=0 2024-03-22T02:18:47.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.751+0000 7f2d96ff5640 10 client.16929 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.751+0000 7f2d96ff5640 1 -- 172.21.15.28:0/3626157026 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c19d2b0e00 con 0x55c19d3547e0 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.751+0000 7f2d96ff5640 10 client.16929 handle_command_reply: tid=0 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.751+0000 7f2dbbf6b740 1 client.16929 shutdown 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 msgr2=0x55c19d34fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2dac003ba0 tx=0x7f2dac004ab0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 shutdown_connections 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3547e0 0x55c19d356c40 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d34fe80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33ee20 0x55c19d348e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/3626157026 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33e430 0x55c19d348900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 >> 172.21.15.28:0/3626157026 conn(0x55c19d3399a0 msgr2=0x55c19d338c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2d96ff5640 0 client.16929 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 shutdown_connections 2024-03-22T02:18:47.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/3626157026 wait complete. 2024-03-22T02:18:47.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.752+0000 7f2dbbf6b740 20 client.16929 trim_cache size 0 max 16384 2024-03-22T02:18:47.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 Processor -- start 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 -- start start 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d348320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 0x55c19d338370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0012690 con 0x55c19d33e430 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2da000ff60 con 0x55c19d343f90 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2da000ddb0 con 0x55c19d33ee20 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34458/0 (socket says 172.21.15.28:34458) 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 -- 172.21.15.28:0/2519511392 learned_addr learned my addr 172.21.15.28:0/2519511392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db27fc640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d348320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db37fe640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 0x55c19d338370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 -- 172.21.15.28:0/2519511392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 msgr2=0x55c19d348320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d348320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 -- 172.21.15.28:0/2519511392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 msgr2=0x55c19d338370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 0x55c19d338370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db2ffd640 1 -- 172.21.15.28:0/2519511392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c19cf770f0 con 0x55c19d33e430 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.770+0000 7f2db37fe640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 0x55c19d338370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2da40058c0 tx=0x7f2da4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a2b85f8765823e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2d98ff9640 1 -- 172.21.15.28:0/2519511392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4010aa0 con 0x55c19d33e430 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2d98ff9640 1 -- 172.21.15.28:0/2519511392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da4064070 con 0x55c19d33e430 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2d98ff9640 1 -- 172.21.15.28:0/2519511392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da409b070 con 0x55c19d33e430 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/2519511392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 msgr2=0x55c19d33fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2da40058c0 tx=0x7f2da4000bc0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/2519511392 shutdown_connections 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d343f90 0x55c19d338370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d348320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/2519511392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d33e430 0x55c19d33fb10 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.771+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/2519511392 >> 172.21.15.28:0/2519511392 conn(0x55c19d3399a0 msgr2=0x55c19d347c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/2519511392 shutdown_connections 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/2519511392 wait complete. 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- start start 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 Processor -- start 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33e430 0x55c19d34f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d34f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2da000e420 con 0x55c19d343f90 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2da000f890 con 0x55c19d33e430 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2da000eaf0 con 0x55c19d33ee20 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2dbbf6b740 10 client.0 _ll_register_callbacks cb 0x7f2dac0139a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34460/0 (socket says 172.21.15.28:34460) 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 -- 172.21.15.28:0/1272566025 learned_addr learned my addr 172.21.15.28:0/1272566025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33e430 0x55c19d34f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db27fc640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d34f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 -- 172.21.15.28:0/1272566025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 msgr2=0x55c19d34f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d34f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 -- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33e430 msgr2=0x55c19d34f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33e430 0x55c19d34f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.772+0000 7f2db37fe640 1 -- 172.21.15.28:0/1272566025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c19d3bb230 con 0x55c19d343f90 2024-03-22T02:18:47.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2db37fe640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2dac006010 tx=0x7f2dac013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68688c53ae3b412 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dac03a060 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2da4064070 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2dac02a640 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2dac041050 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 10 client.16935 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c19d3bc540 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da4064070 con 0x55c19d343f90 2024-03-22T02:18:47.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f2dac020050 con 0x55c19d343f90 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 10 client.16935 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 10 client.16935 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 20 client.16935 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3bce30 0x55c19d3bf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 4 client.16935 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.773+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c19d2ab150 con 0x55c19d3bce30 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.774+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3bce30 0x55c19d3bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.774+0000 7f2db2ffd640 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3bce30 0x55c19d3bf250 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1ea82586e92f67 server_cookie=21819d2f82e763e6 in_seq=0 out_seq=0 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.774+0000 7f2d96ff5640 10 client.16935 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2d96ff5640 1 -- 172.21.15.28:0/1272566025 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c19d2ab150 con 0x55c19d3bce30 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2d96ff5640 10 client.16935 handle_command_reply: tid=0 2024-03-22T02:18:47.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 client.16935 shutdown 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 msgr2=0x55c19d3bac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f2dac006010 tx=0x7f2dac013b70 comp rx=0 tx=0).stop 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 shutdown_connections 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c19d3bce30 0x55c19d3bf250 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c19d343f90 0x55c19d3bac70 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c19d33ee20 0x55c19d34f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2d96ff5640 0 client.16935 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 --2- 172.21.15.28:0/1272566025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c19d33e430 0x55c19d34f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 >> 172.21.15.28:0/1272566025 conn(0x55c19d3399a0 msgr2=0x55c19d33cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 shutdown_connections 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 1 -- 172.21.15.28:0/1272566025 wait complete. 2024-03-22T02:18:47.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2dbbf6b740 20 client.16935 trim_cache size 0 max 16384 2024-03-22T02:18:47.784 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:18:47.784 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:18:47.784 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:47.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 msgr2=0x7f2d9c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 0x7f2d9c0a5460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2dac013d10 tx=0x7f2dac013800 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 msgr2=0x7f2db4196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f2da400e9a0 tx=0x7f2da400eda0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 shutdown_connections 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d9c0a2fa0 0x7f2d9c0a5460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2db4102650 0x7f2db419e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.775+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2db4101c90 0x7f2db4197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.776+0000 7f2db9381640 1 --2- 172.21.15.28:0/439335227 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2db4100aa0 0x7f2db4196e00 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.776+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 >> 172.21.15.28:0/439335227 conn(0x7f2db40fc030 msgr2=0x7f2db4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.776+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 shutdown_connections 2024-03-22T02:18:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.776+0000 7f2db9381640 1 -- 172.21.15.28:0/439335227 wait complete. 2024-03-22T02:18:47.787 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:18:47.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- 172.21.15.28:0/3368732174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc100ab0 msgr2=0x7f2ccc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- 172.21.15.28:0/3368732174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc100ed0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc009900 tx=0x7f2cbc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- 172.21.15.28:0/3368732174 shutdown_connections 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- 172.21.15.28:0/3368732174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc102660 0x7f2ccc108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- 172.21.15.28:0/3368732174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- 172.21.15.28:0/3368732174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc100ed0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- 172.21.15.28:0/3368732174 >> 172.21.15.28:0/3368732174 conn(0x7f2ccc0fc020 msgr2=0x7f2ccc0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- 172.21.15.28:0/3368732174 shutdown_connections 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- 172.21.15.28:0/3368732174 wait complete. 2024-03-22T02:18:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 Processor -- start 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 -- start start 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.891+0000 7f2cd1681640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc102660 0x7f2ccc19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cd1681640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10c720 con 0x7f2ccc101ca0 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cd1681640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10c5a0 con 0x7f2ccc100ab0 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cd1681640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc10c8a0 con 0x7f2ccc102660 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34480/0 (socket says 172.21.15.28:34480) 2024-03-22T02:18:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 -- 172.21.15.28:0/457057741 learned_addr learned my addr 172.21.15.28:0/457057741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2ccaffd640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2ccb7fe640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc102660 0x7f2ccc19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 -- 172.21.15.28:0/457057741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc102660 msgr2=0x7f2ccc19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc102660 0x7f2ccc19f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 -- 172.21.15.28:0/457057741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc100ab0 msgr2=0x7f2ccc197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 -- 172.21.15.28:0/457057741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc19fb60 con 0x7f2ccc101ca0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc3fff640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400e9f0 tx=0x7f2cb400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb400cd80 con 0x7f2ccc101ca0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc19fe50 con 0x7f2ccc101ca0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb4005500 con 0x7f2ccc101ca0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc1a0310 con 0x7f2ccc101ca0 2024-03-22T02:18:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.892+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4018630 con 0x7f2ccc101ca0 2024-03-22T02:18:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.893+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc1a05a0 con 0x7f2ccc101ca0 2024-03-22T02:18:47.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.894+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2cb40046c0 con 0x7f2ccc101ca0 2024-03-22T02:18:47.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.895+0000 7f2cc8ff9640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 0x7f2cb80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:47.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.895+0000 7f2ccaffd640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 0x7f2cb80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:47.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.895+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f2cb40d0310 con 0x7f2ccc101ca0 2024-03-22T02:18:47.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.896+0000 7f2ccaffd640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 0x7f2cb80a53e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc000c00 tx=0x7f2cbc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:47.899+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2cb409ccd0 con 0x7f2ccc101ca0 2024-03-22T02:18:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.124+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ccc1a0770 con 0x7f2ccc101ca0 2024-03-22T02:18:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.125+0000 7f2cc8ff9640 1 -- 172.21.15.28:0/457057741 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11866 (secure 0 0 0) 0x7f2cb409c5d0 con 0x7f2ccc101ca0 2024-03-22T02:18:48.125 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:18:48.125 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:17:24.693718+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3509701876","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3509701876},{"type":"v1","addr":"172.21.15.83:6837","nonce":3509701876}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-03-22T02:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 msgr2=0x7f2cb80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 0x7f2cb80a53e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc000c00 tx=0x7f2cbc005fd0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 msgr2=0x7f2ccc198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400e9f0 tx=0x7f2cb400edf0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 shutdown_connections 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2cb80a2f20 0x7f2cb80a53e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ccc102660 0x7f2ccc19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ccc101ca0 0x7f2ccc198480 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 --2- 172.21.15.28:0/457057741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2ccc100ab0 0x7f2ccc197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 >> 172.21.15.28:0/457057741 conn(0x7f2ccc0fc020 msgr2=0x7f2ccc0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 shutdown_connections 2024-03-22T02:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.127+0000 7f2cd1681640 1 -- 172.21.15.28:0/457057741 wait complete. 2024-03-22T02:18:48.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 -- 172.21.15.28:0/3425307506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 msgr2=0x7f617c108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 --2- 172.21.15.28:0/3425307506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c108ef0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6174009ca0 tx=0x7f617402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 -- 172.21.15.28:0/3425307506 shutdown_connections 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 --2- 172.21.15.28:0/3425307506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c108ef0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 --2- 172.21.15.28:0/3425307506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 0x7f617c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 --2- 172.21.15.28:0/3425307506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 -- 172.21.15.28:0/3425307506 >> 172.21.15.28:0/3425307506 conn(0x7f617c0fc040 msgr2=0x7f617c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 -- 172.21.15.28:0/3425307506 shutdown_connections 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.247+0000 7f6182d7a640 1 -- 172.21.15.28:0/3425307506 wait complete. 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 Processor -- start 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 -- start start 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 0x7f617c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10c710 con 0x7f617c102660 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10c590 con 0x7f617c101ca0 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6182d7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f617c10c890 con 0x7f617c100ab0 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6181317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6180b16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f617bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 0x7f617c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6181317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34504/0 (socket says 172.21.15.28:34504) 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6180b16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38804/0 (socket says 172.21.15.28:38804) 2024-03-22T02:18:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.248+0000 7f6181317640 1 -- 172.21.15.28:0/2956091055 learned_addr learned my addr 172.21.15.28:0/2956091055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 -- 172.21.15.28:0/2956091055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 msgr2=0x7f617c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 -- 172.21.15.28:0/2956091055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 msgr2=0x7f617c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 0x7f617c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 -- 172.21.15.28:0/2956091055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c19fa70 con 0x7f617c102660 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6181317640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f6174032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6179ffb640 1 -- 172.21.15.28:0/2956091055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174041020 con 0x7f617c102660 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c19fd00 con 0x7f617c102660 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6179ffb640 1 -- 172.21.15.28:0/2956091055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6174034cd0 con 0x7f617c102660 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6179ffb640 1 -- 172.21.15.28:0/2956091055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174039040 con 0x7f617c102660 2024-03-22T02:18:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.249+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c1a01a0 con 0x7f617c102660 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.251+0000 7f6179ffb640 1 -- 172.21.15.28:0/2956091055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6174044020 con 0x7f617c102660 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 Processor -- start 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 -- start start 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e873c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e879380 0x557c4e87b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c4e7d88d0 con 0x557c4e874210 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557c4e780e00 con 0x557c4e879380 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6185967740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c4e7d4880 con 0x557c4e873820 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34510/0 (socket says 172.21.15.28:34510) 2024-03-22T02:18:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 -- 172.21.15.28:0/1376839917 learned_addr learned my addr 172.21.15.28:0/1376839917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f617bfff640 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e873c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6179ffb640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 0x7f616c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f617bfff640 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e879380 0x557c4e87b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 -- 172.21.15.28:0/1376839917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 msgr2=0x557c4e873c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e873c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6181317640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 0x7f616c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 -- 172.21.15.28:0/1376839917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e879380 msgr2=0x557c4e87b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e879380 0x557c4e87b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6180b16640 1 -- 172.21.15.28:0/1376839917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c4e4ec3f0 con 0x557c4e874210 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.252+0000 7f6179ffb640 1 -- 172.21.15.28:0/2956091055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f61740f1610 con 0x7f617c102660 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6180b16640 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6170009c80 tx=0x7f6170011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a862e1dbc62b55dc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6166ffd640 1 -- 172.21.15.28:0/1376839917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617001e070 con 0x557c4e874210 2024-03-22T02:18:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6166ffd640 1 -- 172.21.15.28:0/1376839917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6170009d70 con 0x557c4e874210 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6166ffd640 1 -- 172.21.15.28:0/1376839917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170018870 con 0x557c4e874210 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6181317640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 0x7f616c0a53e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61740b9e20 tx=0x7f61740e85f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6185967740 1 -- 172.21.15.28:0/1376839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 msgr2=0x557c4e878d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.253+0000 7f6185967740 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6170009c80 tx=0x7f6170011810 comp rx=0 tx=0).stop 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- 172.21.15.28:0/1376839917 shutdown_connections 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e879380 0x557c4e87b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e874210 0x557c4e878d10 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- 172.21.15.28:0/1376839917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e873c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- 172.21.15.28:0/1376839917 >> 172.21.15.28:0/1376839917 conn(0x557c4e86ed90 msgr2=0x557c4e86f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- 172.21.15.28:0/1376839917 shutdown_connections 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- 172.21.15.28:0/1376839917 wait complete. 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- start start 2024-03-22T02:18:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 Processor -- start 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e8852e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557c4e87c330 con 0x557c4e879380 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557c4e794300 con 0x557c4e874210 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557c4e87c4b0 con 0x557c4e873820 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6185967740 10 client.0 _ll_register_callbacks cb 0x557c4e7cfe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f6180b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.254+0000 7f617bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:39766/0 (socket says 172.21.15.28:39766) 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 -- 172.21.15.28:0/3402810608 learned_addr learned my addr 172.21.15.28:0/3402810608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f617bfff640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e8852e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 -- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 msgr2=0x557c4e87ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 -- 172.21.15.28:0/3402810608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 msgr2=0x557c4e8852e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e8852e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 -- 172.21.15.28:0/3402810608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c4e5b67d0 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f617bfff640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6180b16640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61700097e0 tx=0x7f6170005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3b46f7227324210 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f615bfff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170028030 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6170009d70 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f6170016040 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6170032050 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6185967740 10 client.26786 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.255+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c4e8886a0 con 0x557c4e874210 2024-03-22T02:18:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6170009d70 con 0x557c4e874210 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f6170021030 con 0x557c4e874210 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 10 client.26786 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 10 client.26786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 20 client.26786 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e889b00 0x557c4e88bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 4 client.26786 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c4e7e5e10 con 0x557c4e889b00 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f617bfff640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e889b00 0x557c4e88bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f617bfff640 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e889b00 0x557c4e88bf40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2150f3bd0e20d11b server_cookie=e77fec82e104e4d1 in_seq=0 out_seq=0 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.256+0000 7f615bfff640 10 client.26786 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f615bfff640 1 -- 172.21.15.28:0/3402810608 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c4e7e5e10 con 0x557c4e889b00 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f615bfff640 10 client.26786 handle_command_reply: tid=0 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 client.26786 shutdown 2024-03-22T02:18:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 msgr2=0x557c4e87e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f61700097e0 tx=0x7f6170005740 comp rx=0 tx=0).stop 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 shutdown_connections 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e8852e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e889b00 0x557c4e88bf40 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.257+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e87e310 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f6185967740 1 --2- 172.21.15.28:0/3402810608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f615bfff640 0 client.26786 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 >> 172.21.15.28:0/3402810608 conn(0x557c4e86ed90 msgr2=0x557c4e86dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 shutdown_connections 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f6185967740 1 -- 172.21.15.28:0/3402810608 wait complete. 2024-03-22T02:18:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.258+0000 7f6185967740 20 client.26786 trim_cache size 0 max 16384 2024-03-22T02:18:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 Processor -- start 2024-03-22T02:18:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 -- start start 2024-03-22T02:18:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e8852a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1ad260 con 0x557c4e879380 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1aab30 con 0x557c4e874210 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f6185967740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1a8980 con 0x557c4e873820 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f617bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f617bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34530/0 (socket says 172.21.15.28:34530) 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f617bfff640 1 -- 172.21.15.28:0/1057500580 learned_addr learned my addr 172.21.15.28:0/1057500580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.271+0000 7f617bfff640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f6180b16640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e8852a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 -- 172.21.15.28:0/1057500580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 msgr2=0x557c4e87d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 -- 172.21.15.28:0/1057500580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 msgr2=0x557c4e8852a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e8852a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 -- 172.21.15.28:0/1057500580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c4e88c5c0 con 0x557c4e879380 2024-03-22T02:18:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f6180b16640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e8852a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:18:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f617bfff640 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f617c199360 tx=0x7f6168005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1deb60fa1b7c37 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f61667fc640 1 -- 172.21.15.28:0/1057500580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680284d0 con 0x557c4e879380 2024-03-22T02:18:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f61667fc640 1 -- 172.21.15.28:0/1057500580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6168028670 con 0x557c4e879380 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.272+0000 7f61667fc640 1 -- 172.21.15.28:0/1057500580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61680047a0 con 0x557c4e879380 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- 172.21.15.28:0/1057500580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 msgr2=0x557c4e86d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f617c199360 tx=0x7f6168005f90 comp rx=0 tx=0).stop 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- 172.21.15.28:0/1057500580 shutdown_connections 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e879380 0x557c4e86d740 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e8852a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 --2- 172.21.15.28:0/1057500580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e873820 0x557c4e87d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- 172.21.15.28:0/1057500580 >> 172.21.15.28:0/1057500580 conn(0x557c4e86ed90 msgr2=0x557c4e87c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- 172.21.15.28:0/1057500580 shutdown_connections 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- 172.21.15.28:0/1057500580 wait complete. 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 -- start start 2024-03-22T02:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.273+0000 7f6185967740 1 Processor -- start 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e883160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e879380 0x557c4e884170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1a8ff0 con 0x557c4e873820 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1aa460 con 0x557c4e874210 2024-03-22T02:18:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f617c1a96c0 con 0x557c4e879380 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 10 client.0 _ll_register_callbacks cb 0x7f617001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34532/0 (socket says 172.21.15.28:34532) 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 -- 172.21.15.28:0/1442865520 learned_addr learned my addr 172.21.15.28:0/1442865520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6180b16640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e883160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e879380 0x557c4e884170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 -- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e879380 msgr2=0x557c4e884170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e879380 0x557c4e884170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 -- 172.21.15.28:0/1442865520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 msgr2=0x557c4e883160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e883160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 -- 172.21.15.28:0/1442865520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c4e884760 con 0x557c4e873820 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f617bfff640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6168009ef0 tx=0x7f6168005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b19ba1f5feabdb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f615bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6168028820 con 0x557c4e873820 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f6168004040 con 0x557c4e873820 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6168028670 con 0x557c4e873820 2024-03-22T02:18:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.274+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f616803d050 con 0x557c4e873820 2024-03-22T02:18:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 10 client.16959 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557c4e8f00b0 con 0x557c4e873820 2024-03-22T02:18:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6168028670 con 0x557c4e873820 2024-03-22T02:18:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f6168020040 con 0x557c4e873820 2024-03-22T02:18:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 10 client.16959 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 10 client.16959 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 20 client.16959 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e8f09c0 0x557c4e8f2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 4 client.16959 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c4e87dbc0 con 0x557c4e8f09c0 2024-03-22T02:18:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6180b16640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e8f09c0 0x557c4e8f2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f6180b16640 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e8f09c0 0x557c4e8f2de0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b47dccd10c2eedbe server_cookie=327f022e96d52409 in_seq=0 out_seq=0 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.275+0000 7f615bfff640 10 client.16959 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f615bfff640 1 -- 172.21.15.28:0/1442865520 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c4e87dbc0 con 0x557c4e8f09c0 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f615bfff640 10 client.16959 handle_command_reply: tid=0 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 client.16959 shutdown 2024-03-22T02:18:48.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 msgr2=0x557c4e882c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f6168009ef0 tx=0x7f6168005850 comp rx=0 tx=0).stop 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 shutdown_connections 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557c4e879380 0x557c4e884170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x557c4e8f09c0 0x557c4e8f2de0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557c4e874210 0x557c4e883160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 --2- 172.21.15.28:0/1442865520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c4e873820 0x557c4e882c20 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f615bfff640 0 client.16959 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:18:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 >> 172.21.15.28:0/1442865520 conn(0x557c4e86ed90 msgr2=0x557c4e871ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 shutdown_connections 2024-03-22T02:18:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 1 -- 172.21.15.28:0/1442865520 wait complete. 2024-03-22T02:18:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6185967740 20 client.16959 trim_cache size 0 max 16384 2024-03-22T02:18:48.292 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 msgr2=0x7f616c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 0x7f616c0a53e0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f61740b9e20 tx=0x7f61740e85f0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 msgr2=0x7f617c19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.276+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f6174032f40 comp rx=0 tx=0).stop 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 shutdown_connections 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f616c0a2f20 0x7f616c0a53e0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f617c102660 0x7f617c19f400 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f617c101ca0 0x7f617c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 --2- 172.21.15.28:0/2956091055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f617c100ab0 0x7f617c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 >> 172.21.15.28:0/2956091055 conn(0x7f617c0fc040 msgr2=0x7f617c06a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 shutdown_connections 2024-03-22T02:18:48.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.277+0000 7f6182d7a640 1 -- 172.21.15.28:0/2956091055 wait complete. 2024-03-22T02:18:48.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 -- 172.21.15.28:0/2811637248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c102640 msgr2=0x7feb2c108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 --2- 172.21.15.28:0/2811637248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c102640 0x7feb2c108ed0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb2402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 -- 172.21.15.28:0/2811637248 shutdown_connections 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 --2- 172.21.15.28:0/2811637248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c102640 0x7feb2c108ed0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 --2- 172.21.15.28:0/2811637248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c101c80 0x7feb2c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 --2- 172.21.15.28:0/2811637248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c100a90 0x7feb2c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 -- 172.21.15.28:0/2811637248 >> 172.21.15.28:0/2811637248 conn(0x7feb2c0fc020 msgr2=0x7feb2c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 -- 172.21.15.28:0/2811637248 shutdown_connections 2024-03-22T02:18:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.407+0000 7feb33cf4640 1 -- 172.21.15.28:0/2811637248 wait complete. 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.408+0000 7feb33cf4640 1 Processor -- start 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.408+0000 7feb33cf4640 1 -- start start 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.408+0000 7feb33cf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb33cf4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c101c80 0x7feb2c103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb33cf4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 0x7feb2c1035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb33cf4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c10c710 con 0x7feb2c100a90 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb33cf4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c10c590 con 0x7feb2c102640 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb33cf4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c10c890 con 0x7feb2c101c80 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34548/0 (socket says 172.21.15.28:34548) 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 -- 172.21.15.28:0/770813487 learned_addr learned my addr 172.21.15.28:0/770813487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb3128f640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c101c80 0x7feb2c103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb32291640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 0x7feb2c1035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 -- 172.21.15.28:0/770813487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c101c80 msgr2=0x7feb2c103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c101c80 0x7feb2c103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 -- 172.21.15.28:0/770813487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 msgr2=0x7feb2c1035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 0x7feb2c1035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 -- 172.21.15.28:0/770813487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c103e30 con 0x7feb2c100a90 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb32291640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 0x7feb2c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb31a90640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e9c0 tx=0x7feb2800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.409+0000 7feb22ffd640 1 -- 172.21.15.28:0/770813487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb2800cd80 con 0x7feb2c100a90 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.410+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb2c197b30 con 0x7feb2c100a90 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.410+0000 7feb22ffd640 1 -- 172.21.15.28:0/770813487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb280035f0 con 0x7feb2c100a90 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.410+0000 7feb22ffd640 1 -- 172.21.15.28:0/770813487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb28010690 con 0x7feb2c100a90 2024-03-22T02:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.410+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb2c198080 con 0x7feb2c100a90 2024-03-22T02:18:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.413+0000 7feb22ffd640 1 -- 172.21.15.28:0/770813487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7feb28010830 con 0x7feb2c100a90 2024-03-22T02:18:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.413+0000 7feb369b8740 1 Processor -- start 2024-03-22T02:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.413+0000 7feb369b8740 1 -- start start 2024-03-22T02:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d5ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d5b380 0x55f161d5d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f161cba8d0 con 0x55f161d5b380 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f161c62e00 con 0x55f161d56210 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb369b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f161cb6880 con 0x55f161d55820 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb31a90640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d5ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38890/0 (socket says 172.21.15.28:38890) 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 -- 172.21.15.28:0/3666538766 learned_addr learned my addr 172.21.15.28:0/3666538766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb3128f640 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d5b380 0x55f161d5d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 -- 172.21.15.28:0/3666538766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 msgr2=0x55f161d5ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d5ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 -- 172.21.15.28:0/3666538766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d5b380 msgr2=0x55f161d5d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d5b380 0x55f161d5d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 -- 172.21.15.28:0/3666538766 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1619ce3f0 con 0x55f161d55820 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb22ffd640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 0x7feb180adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb31a90640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 0x7feb180adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.414+0000 7feb32291640 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb24005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9b6d3913d206ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.415+0000 7feb22ffd640 1 -- 172.21.15.28:0/770813487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7feb280cfba0 con 0x7feb2c100a90 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.415+0000 7feb17fff640 1 -- 172.21.15.28:0/3666538766 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24043070 con 0x55f161d55820 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.415+0000 7feb17fff640 1 -- 172.21.15.28:0/3666538766 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb24032c90 con 0x55f161d55820 2024-03-22T02:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.415+0000 7feb17fff640 1 -- 172.21.15.28:0/3666538766 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24039040 con 0x55f161d55820 2024-03-22T02:18:48.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 394574 11.88 MB/sec execute 90 sec latency 104.528 ms 2024-03-22T02:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb31a90640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 0x7feb180adc70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c6a0 tx=0x7feb280d3620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- 172.21.15.28:0/3666538766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 msgr2=0x55f161d55c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb24005ef0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- 172.21.15.28:0/3666538766 shutdown_connections 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d5b380 0x55f161d5d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d5ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 --2- 172.21.15.28:0/3666538766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d55820 0x55f161d55c40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- 172.21.15.28:0/3666538766 >> 172.21.15.28:0/3666538766 conn(0x55f161d50d90 msgr2=0x55f161d51190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- 172.21.15.28:0/3666538766 shutdown_connections 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- 172.21.15.28:0/3666538766 wait complete. 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.416+0000 7feb369b8740 1 -- start start 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 Processor -- start 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d5fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d67280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f161d5e2d0 con 0x55f161d55820 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f161c76300 con 0x55f161d5b380 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f161d5e450 con 0x55f161d56210 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb32291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d5fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb3128f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb3128f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38902/0 (socket says 172.21.15.28:38902) 2024-03-22T02:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb3128f640 1 -- 172.21.15.28:0/3294936529 learned_addr learned my addr 172.21.15.28:0/3294936529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb369b8740 10 client.0 _ll_register_callbacks cb 0x55f161cb1e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.417+0000 7feb32291640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d67280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 -- 172.21.15.28:0/3294936529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 msgr2=0x55f161d67280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d67280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 -- 172.21.15.28:0/3294936529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 msgr2=0x55f161d5fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d5fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 -- 172.21.15.28:0/3294936529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f161a987d0 con 0x55f161d56210 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb3128f640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb1c012720 tx=0x7feb1c012be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e6ec9181d22ef8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb157fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c00f8c0 con 0x55f161d56210 2024-03-22T02:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7feb1c00fa60 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb24032c90 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb1c00fca0 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb369b8740 10 client.26806 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f161d6a6a0 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.418+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb24032c90 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7feb1c010550 con 0x55f161d56210 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 10 client.26806 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 10 client.26806 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 20 client.26806 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161d6bdd0 0x55f161d6e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 4 client.26806 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f161cc7e10 con 0x55f161d6bdd0 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb32291640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161d6bdd0 0x55f161d6e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb157fa640 10 client.26806 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.419+0000 7feb32291640 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161d6bdd0 0x55f161d6e270 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b73b419f49b85fbf server_cookie=ed70cd60dbb7c6bb in_seq=0 out_seq=0 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb157fa640 1 -- 172.21.15.28:0/3294936529 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f161cc7e10 con 0x55f161d6bdd0 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb157fa640 10 client.26806 handle_command_reply: tid=0 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 client.26806 shutdown 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 msgr2=0x55f161d60240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feb1c012720 tx=0x7feb1c012be0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 shutdown_connections 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161d6bdd0 0x55f161d6e270 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d67280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d60240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb157fa640 0 client.26806 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 --2- 172.21.15.28:0/3294936529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d5fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.420+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 >> 172.21.15.28:0/3294936529 conn(0x55f161d50d90 msgr2=0x55f161d4ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.421+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 shutdown_connections 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.421+0000 7feb369b8740 1 -- 172.21.15.28:0/3294936529 wait complete. 2024-03-22T02:18:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.421+0000 7feb369b8740 20 client.26806 trim_cache size 0 max 16384 2024-03-22T02:18:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 Processor -- start 2024-03-22T02:18:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 -- start start 2024-03-22T02:18:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d4f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1ab090 con 0x55f161d55820 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1a8960 con 0x55f161d5b380 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb369b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1a67b0 con 0x55f161d56210 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb3128f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34576/0 (socket says 172.21.15.28:34576) 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb3128f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38918/0 (socket says 172.21.15.28:38918) 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 -- 172.21.15.28:0/731032378 learned_addr learned my addr 172.21.15.28:0/731032378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d4f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 -- 172.21.15.28:0/731032378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 msgr2=0x55f161d67240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 -- 172.21.15.28:0/731032378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 msgr2=0x55f161d4f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d4f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb32291640 1 -- 172.21.15.28:0/731032378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f161d51b00 con 0x55f161d55820 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.436+0000 7feb3128f640 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb32291640 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3024490cfbdf7970 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb177fe640 1 -- 172.21.15.28:0/731032378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24039040 con 0x55f161d55820 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb177fe640 1 -- 172.21.15.28:0/731032378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb24032c90 con 0x55f161d55820 2024-03-22T02:18:48.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb177fe640 1 -- 172.21.15.28:0/731032378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb24043040 con 0x55f161d55820 2024-03-22T02:18:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb369b8740 1 -- 172.21.15.28:0/731032378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 msgr2=0x55f161d54700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.437+0000 7feb369b8740 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7feb24009ca0 tx=0x7feb240095c0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- 172.21.15.28:0/731032378 shutdown_connections 2024-03-22T02:18:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d5b380 0x55f161d4f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d56210 0x55f161d67240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- 172.21.15.28:0/731032378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d54700 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- 172.21.15.28:0/731032378 >> 172.21.15.28:0/731032378 conn(0x55f161d50d90 msgr2=0x55f161d53ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- 172.21.15.28:0/731032378 shutdown_connections 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- 172.21.15.28:0/731032378 wait complete. 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- start start 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 Processor -- start 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d65220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d5b380 0x55f161d66230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1a6e20 con 0x55f161d55820 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1a8290 con 0x55f161d56210 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c1a74f0 con 0x55f161d5b380 2024-03-22T02:18:48.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb369b8740 10 client.0 _ll_register_callbacks cb 0x7feb1c009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb32291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb3128f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d65220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb32291640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34582/0 (socket says 172.21.15.28:34582) 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb32291640 1 -- 172.21.15.28:0/1765387133 learned_addr learned my addr 172.21.15.28:0/1765387133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.438+0000 7feb32291640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d5b380 0x55f161d66230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 -- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d5b380 msgr2=0x55f161d66230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d5b380 0x55f161d66230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 -- 172.21.15.28:0/1765387133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 msgr2=0x55f161d64ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 -- 172.21.15.28:0/1765387133 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f161d66820 con 0x55f161d56210 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb32291640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:48.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb3128f640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d65220 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb1c000c00 tx=0x7feb1c00f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b88d6b3bb6c37dd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb157fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb1c02e060 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb24032c90 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7feb1c00fe40 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feb1c010640 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb369b8740 10 client.26810 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f161dd2560 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.439+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb24032c90 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7feb1c003730 con 0x55f161d56210 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 10 client.26810 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 10 client.26810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 20 client.26810 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161dd2eb0 0x55f161dd52d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 4 client.26810 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f161cc2160 con 0x55f161dd2eb0 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb32291640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161dd2eb0 0x55f161dd52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb32291640 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161dd2eb0 0x55f161dd52d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=289613e70791a2b4 server_cookie=464dcbd621ccbc92 in_seq=0 out_seq=0 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb157fa640 10 client.26810 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb157fa640 1 -- 172.21.15.28:0/1765387133 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f161cc2160 con 0x55f161dd2eb0 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb157fa640 10 client.26810 handle_command_reply: tid=0 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 1 client.26810 shutdown 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 msgr2=0x55f161d65220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.440+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d65220 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb1c000c00 tx=0x7feb1c00f9c0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 shutdown_connections 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f161dd2eb0 0x55f161dd52d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f161d5b380 0x55f161d66230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f161d56210 0x55f161d65220 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb157fa640 0 client.26810 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 --2- 172.21.15.28:0/1765387133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f161d55820 0x55f161d64ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 >> 172.21.15.28:0/1765387133 conn(0x55f161d50d90 msgr2=0x55f161d54880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 shutdown_connections 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 1 -- 172.21.15.28:0/1765387133 wait complete. 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb369b8740 20 client.26810 trim_cache size 0 max 16384 2024-03-22T02:18:48.449 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 msgr2=0x7feb180adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 0x7feb180adc70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7feb2800c6a0 tx=0x7feb280d3620 comp rx=0 tx=0).stop 2024-03-22T02:18:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 msgr2=0x7feb2c104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb2800e9c0 tx=0x7feb2800ee80 comp rx=0 tx=0).stop 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 shutdown_connections 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7feb180ab7b0 0x7feb180adc70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7feb2c102640 0x7feb2c1035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feb2c101c80 0x7feb2c103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 --2- 172.21.15.28:0/770813487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb2c100a90 0x7feb2c104a20 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 >> 172.21.15.28:0/770813487 conn(0x7feb2c0fc020 msgr2=0x7feb2c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 shutdown_connections 2024-03-22T02:18:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.441+0000 7feb33cf4640 1 -- 172.21.15.28:0/770813487 wait complete. 2024-03-22T02:18:48.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:48 smithi028 ceph-mon[31916]: pgmap v514: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 420 op/s 2024-03-22T02:18:48.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:48 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/457057741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:48.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:18:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:48 smithi120 ceph-mon[36803]: pgmap v514: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 420 op/s 2024-03-22T02:18:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:48 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/457057741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:48.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:48 smithi083 ceph-mon[36340]: pgmap v514: 129 pgs: 129 active+clean; 49 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 420 op/s 2024-03-22T02:18:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:48 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/457057741' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:18:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 -- 172.21.15.28:0/3787500845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 msgr2=0x7f8bd4103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 --2- 172.21.15.28:0/3787500845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4103270 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009930 tx=0x7f8bc402bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 -- 172.21.15.28:0/3787500845 shutdown_connections 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 --2- 172.21.15.28:0/3787500845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 0x7f8bd410a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 --2- 172.21.15.28:0/3787500845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4103270 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 --2- 172.21.15.28:0/3787500845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 0x7f8bd41001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 -- 172.21.15.28:0/3787500845 >> 172.21.15.28:0/3787500845 conn(0x7f8bd40fb690 msgr2=0x7f8bd40fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 -- 172.21.15.28:0/3787500845 shutdown_connections 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.559+0000 7f8bdb554640 1 -- 172.21.15.28:0/3787500845 wait complete. 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 Processor -- start 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 -- start start 2024-03-22T02:18:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 0x7f8bd4197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 0x7f8bd419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410d730 con 0x7f8bd4102df0 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410d5b0 con 0x7f8bd41037b0 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdb554640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd410d8b0 con 0x7f8bd40ffd80 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34594/0 (socket says 172.21.15.28:34594) 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 -- 172.21.15.28:0/4270329348 learned_addr learned my addr 172.21.15.28:0/4270329348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bda552640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 0x7f8bd4197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bdad53640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 0x7f8bd419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 -- 172.21.15.28:0/4270329348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 msgr2=0x7f8bd4197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 0x7f8bd4197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 -- 172.21.15.28:0/4270329348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 msgr2=0x7f8bd419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 0x7f8bd419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.560+0000 7f8bd9d51640 1 -- 172.21.15.28:0/4270329348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd419f9f0 con 0x7f8bd4102df0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000c00 tx=0x7f8bc402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bcb7fe640 1 -- 172.21.15.28:0/4270329348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4041070 con 0x7f8bd4102df0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bcb7fe640 1 -- 172.21.15.28:0/4270329348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4004450 con 0x7f8bd4102df0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd419fc80 con 0x7f8bd4102df0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bcb7fe640 1 -- 172.21.15.28:0/4270329348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4034440 con 0x7f8bd4102df0 2024-03-22T02:18:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.561+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd41a0170 con 0x7f8bd4102df0 2024-03-22T02:18:48.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bcb7fe640 1 -- 172.21.15.28:0/4270329348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8bc4044070 con 0x7f8bd4102df0 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 Processor -- start 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 -- start start 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe8400 0x55cd31fecf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fed570 0x55cd31fefa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd31f4c8d0 con 0x55cd31fed570 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd31ef4e00 con 0x55cd31fe7a10 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdf396740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cd31f48880 con 0x55cd31fe8400 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdad53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fed570 0x55cd31fefa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bda552640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe8400 0x55cd31fecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdad53640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:39896/0 (socket says 172.21.15.28:39896) 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdad53640 1 -- 172.21.15.28:0/2650446911 learned_addr learned my addr 172.21.15.28:0/2650446911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bcb7fe640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 0x7f8bc00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.564+0000 7f8bdad53640 1 -- 172.21.15.28:0/2650446911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe8400 msgr2=0x55cd31fecf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe8400 0x55cd31fecf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 -- 172.21.15.28:0/2650446911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fed570 msgr2=0x55cd31fefa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fed570 0x55cd31fefa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 -- 172.21.15.28:0/2650446911 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd31c603f0 con 0x55cd31fe7a10 2024-03-22T02:18:48.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bcb7fe640 1 -- 172.21.15.28:0/4270329348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8bc40f12c0 con 0x7f8bd4102df0 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 0x7f8bc00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bdad53640 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fd0 tx=0x7f8bcc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=720cfbed9cad187 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bc8ff9640 1 -- 172.21.15.28:0/2650446911 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc004050 con 0x55cd31fe7a10 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bc8ff9640 1 -- 172.21.15.28:0/2650446911 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bcc004800 con 0x55cd31fe7a10 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.565+0000 7f8bc8ff9640 1 -- 172.21.15.28:0/2650446911 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc0196f0 con 0x55cd31fe7a10 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- 172.21.15.28:0/2650446911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 msgr2=0x55cd31fe7e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009fd0 tx=0x7f8bcc0118d0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdad53640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 0x7f8bc00a53e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01d000 tx=0x7f8bcc01e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- 172.21.15.28:0/2650446911 shutdown_connections 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fed570 0x55cd31fefa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe8400 0x55cd31fecf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 --2- 172.21.15.28:0/2650446911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe7a10 0x55cd31fe7e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- 172.21.15.28:0/2650446911 >> 172.21.15.28:0/2650446911 conn(0x55cd31fe2f80 msgr2=0x55cd31fe3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- 172.21.15.28:0/2650446911 shutdown_connections 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- 172.21.15.28:0/2650446911 wait complete. 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 -- start start 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.566+0000 7f8bdf396740 1 Processor -- start 2024-03-22T02:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd31ff04c0 con 0x55cd31fe7a10 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cd31f08300 con 0x55cd31fe8400 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cd31ff0640 con 0x55cd31fed570 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 10 client.0 _ll_register_callbacks cb 0x55cd31f43e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34612/0 (socket says 172.21.15.28:34612) 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 -- 172.21.15.28:0/1793851110 learned_addr learned my addr 172.21.15.28:0/1793851110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdad53640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 -- 172.21.15.28:0/1793851110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 msgr2=0x55cd31ff9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 -- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 msgr2=0x55cd31ff2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 -- 172.21.15.28:0/1793851110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd31d2a7d0 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bda552640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0012700 tx=0x7f8bd0012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dabf2c1f0efcef1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd000f480 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8bd000f620 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bcc004800 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.567+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8bd000f820 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 10 client.16989 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd31ffc8a0 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bcc004800 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8bd0010c00 con 0x55cd31fe7a10 2024-03-22T02:18:48.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 10 client.16989 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 10 client.16989 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 20 client.16989 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd31ffdfc0 0x55cd32000460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 4 client.16989 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd31f59e10 con 0x55cd31ffdfc0 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd31ffdfc0 0x55cd32000460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bbdffb640 10 client.16989 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.568+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd31ffdfc0 0x55cd32000460 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e71b827761ba5a97 server_cookie=a3779e8976b6e51d in_seq=0 out_seq=0 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bbdffb640 1 -- 172.21.15.28:0/1793851110 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cd31f59e10 con 0x55cd31ffdfc0 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bbdffb640 10 client.16989 handle_command_reply: tid=0 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 client.16989 shutdown 2024-03-22T02:18:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 msgr2=0x55cd31ff1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0012700 tx=0x7f8bd0012b00 comp rx=0 tx=0).stop 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 shutdown_connections 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd31ffdfc0 0x55cd32000460 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bbdffb640 0 client.16989 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 --2- 172.21.15.28:0/1793851110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff1ee0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 >> 172.21.15.28:0/1793851110 conn(0x55cd31fe2f80 msgr2=0x55cd31fe2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 shutdown_connections 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 1 -- 172.21.15.28:0/1793851110 wait complete. 2024-03-22T02:18:48.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.570+0000 7f8bdf396740 20 client.16989 trim_cache size 0 max 16384 2024-03-22T02:18:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.583+0000 7f8bdf396740 1 Processor -- start 2024-03-22T02:18:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.583+0000 7f8bdf396740 1 -- start start 2024-03-22T02:18:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 0x55cd31fe68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fed570 0x55cd31ff11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41ad120 con 0x55cd31fe8400 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41aa9f0 con 0x55cd31fed570 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdf396740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41a8840 con 0x55cd31fe7a10 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34626/0 (socket says 172.21.15.28:34626) 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 -- 172.21.15.28:0/3499989139 learned_addr learned my addr 172.21.15.28:0/3499989139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bda552640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 0x55cd31fe68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bdad53640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fed570 0x55cd31ff11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 -- 172.21.15.28:0/3499989139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 msgr2=0x55cd31fe68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 0x55cd31fe68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 -- 172.21.15.28:0/3499989139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fed570 msgr2=0x55cd31ff11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fed570 0x55cd31ff11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 -- 172.21.15.28:0/3499989139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd31fe87e0 con 0x55cd31fe8400 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bda552640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 0x55cd31fe68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40b9b80 tx=0x7f8bc403f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a215b1a9f4489db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bbffff640 1 -- 172.21.15.28:0/3499989139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc404b040 con 0x55cd31fe8400 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bbffff640 1 -- 172.21.15.28:0/3499989139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4004c80 con 0x55cd31fe8400 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.584+0000 7f8bbffff640 1 -- 172.21.15.28:0/3499989139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc40a80b0 con 0x55cd31fe8400 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- 172.21.15.28:0/3499989139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 msgr2=0x55cd31ff9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8bc40b9b80 tx=0x7f8bc403f040 comp rx=0 tx=0).stop 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- 172.21.15.28:0/3499989139 shutdown_connections 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fed570 0x55cd31ff11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe8400 0x55cd31ff9420 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- 172.21.15.28:0/3499989139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fe7a10 0x55cd31fe68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- 172.21.15.28:0/3499989139 >> 172.21.15.28:0/3499989139 conn(0x55cd31fe2f80 msgr2=0x55cd31fe61e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- 172.21.15.28:0/3499989139 shutdown_connections 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- 172.21.15.28:0/3499989139 wait complete. 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- start start 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 Processor -- start 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41a8eb0 con 0x55cd31fe7a10 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41aa320 con 0x55cd31fe8400 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.585+0000 7f8bdf396740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd41a9580 con 0x55cd31fed570 2024-03-22T02:18:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bdf396740 10 client.0 _ll_register_callbacks cb 0x7f8bd0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34636/0 (socket says 172.21.15.28:34636) 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bd9d51640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 -- 172.21.15.28:0/66122415 learned_addr learned my addr 172.21.15.28:0/66122415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bdad53640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 -- 172.21.15.28:0/66122415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 msgr2=0x55cd31ff7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 -- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 msgr2=0x55cd31ff6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 -- 172.21.15.28:0/66122415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd31ff84b0 con 0x55cd31fe7a10 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bda552640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0012420 tx=0x7f8bd001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970b3ea16a4f25a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bbdffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd002f060 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f8bd001cc10 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8bc4004c80 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8bd0009e60 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bdf396740 10 client.16995 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd31ff9820 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.586+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bc4004c80 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f8bd0005480 con 0x55cd31fe7a10 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 10 client.16995 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 10 client.16995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 20 client.16995 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd32064bc0 0x55cd32066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 4 client.16995 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd31f54160 con 0x55cd32064bc0 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd32064bc0 0x55cd32066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bd9d51640 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd32064bc0 0x55cd32066fe0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f410002cbcbae07e server_cookie=8b4234b8221fbf93 in_seq=0 out_seq=0 2024-03-22T02:18:48.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.587+0000 7f8bbdffb640 10 client.16995 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.590+0000 7f8bbdffb640 1 -- 172.21.15.28:0/66122415 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd31f54160 con 0x55cd32064bc0 2024-03-22T02:18:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.590+0000 7f8bbdffb640 10 client.16995 handle_command_reply: tid=0 2024-03-22T02:18:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.590+0000 7f8bdf396740 1 client.16995 shutdown 2024-03-22T02:18:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 msgr2=0x55cd31ff69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0012420 tx=0x7f8bd001c860 comp rx=0 tx=0).stop 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 shutdown_connections 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cd32064bc0 0x55cd32066fe0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cd31fed570 0x55cd31ff7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cd31fe8400 0x55cd31ff6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bbdffb640 0 client.16995 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 --2- 172.21.15.28:0/66122415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd31fe7a10 0x55cd31ff69c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 >> 172.21.15.28:0/66122415 conn(0x55cd31fe2f80 msgr2=0x55cd31fe6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 shutdown_connections 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 1 -- 172.21.15.28:0/66122415 wait complete. 2024-03-22T02:18:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdf396740 20 client.16995 trim_cache size 0 max 16384 2024-03-22T02:18:48.597 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:18:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 msgr2=0x7f8bc00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 0x7f8bc00a53e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc01d000 tx=0x7f8bcc01e000 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 msgr2=0x7f8bd4198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4000c00 tx=0x7f8bc402c640 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 shutdown_connections 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8bc00a2f20 0x7f8bc00a53e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bd41037b0 0x7f8bd419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bd4102df0 0x7f8bd4198310 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 --2- 172.21.15.28:0/4270329348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bd40ffd80 0x7f8bd4197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 >> 172.21.15.28:0/4270329348 conn(0x7f8bd40fb690 msgr2=0x7f8bd41060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 shutdown_connections 2024-03-22T02:18:48.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:48.591+0000 7f8bdb554640 1 -- 172.21.15.28:0/4270329348 wait complete. 2024-03-22T02:18:49.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 397266 11.94 MB/sec execute 91 sec latency 96.744 ms 2024-03-22T02:18:49.603 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:18:49.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:18:49.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:18:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 -- 172.21.15.28:0/2112571506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102640 msgr2=0x7faed8108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 --2- 172.21.15.28:0/2112571506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102640 0x7faed8108ed0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7faed0009be0 tx=0x7faed002bf10 comp rx=0 tx=0).stop 2024-03-22T02:18:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 -- 172.21.15.28:0/2112571506 shutdown_connections 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 --2- 172.21.15.28:0/2112571506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8102640 0x7faed8108ed0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 --2- 172.21.15.28:0/2112571506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8101c80 0x7faed8102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 --2- 172.21.15.28:0/2112571506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8100a90 0x7faed8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 -- 172.21.15.28:0/2112571506 >> 172.21.15.28:0/2112571506 conn(0x7faed80fc020 msgr2=0x7faed80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 -- 172.21.15.28:0/2112571506 shutdown_connections 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.742+0000 7faede665640 1 -- 172.21.15.28:0/2112571506 wait complete. 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.743+0000 7faede665640 1 Processor -- start 2024-03-22T02:18:49.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.743+0000 7faede665640 1 -- start start 2024-03-22T02:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.743+0000 7faede665640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.743+0000 7faede665640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8101c80 0x7faed81972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faede665640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8102640 0x7faed819e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faede665640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed810c740 con 0x7faed8100a90 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faede665640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faed810c5c0 con 0x7faed8101c80 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faede665640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faed810c8c0 con 0x7faed8102640 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34656/0 (socket says 172.21.15.28:34656) 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 -- 172.21.15.28:0/4069746412 learned_addr learned my addr 172.21.15.28:0/4069746412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed77fe640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8101c80 0x7faed81972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faedcc02640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8102640 0x7faed819e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 -- 172.21.15.28:0/4069746412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8102640 msgr2=0x7faed819e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8102640 0x7faed819e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 -- 172.21.15.28:0/4069746412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8101c80 msgr2=0x7faed81972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8101c80 0x7faed81972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 -- 172.21.15.28:0/4069746412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed819e980 con 0x7faed8100a90 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.744+0000 7faed7fff640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faecc00e9f0 tx=0x7faecc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.745+0000 7faed57fa640 1 -- 172.21.15.28:0/4069746412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc00ccc0 con 0x7faed8100a90 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.745+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed8068d50 con 0x7faed8100a90 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.745+0000 7faed57fa640 1 -- 172.21.15.28:0/4069746412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faecc0035f0 con 0x7faed8100a90 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.745+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faed80692a0 con 0x7faed8100a90 2024-03-22T02:18:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.746+0000 7faed57fa640 1 -- 172.21.15.28:0/4069746412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc00ce60 con 0x7faed8100a90 2024-03-22T02:18:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.748+0000 7faed57fa640 1 -- 172.21.15.28:0/4069746412 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faecc0037a0 con 0x7faed8100a90 2024-03-22T02:18:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 Processor -- start 2024-03-22T02:18:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 -- start start 2024-03-22T02:18:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3019f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db301d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 0x555db302a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faee1245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555db2f7e570 con 0x555db3019b40 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faedcc02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3019f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faedcc02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3019f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34662/0 (socket says 172.21.15.28:34662) 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.749+0000 7faedcc02640 1 -- 172.21.15.28:0/2324390589 learned_addr learned my addr 172.21.15.28:0/2324390589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555db2f26e10 con 0x555db301d690 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555db2f7a520 con 0x555db301a530 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 0x555db302a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed57fa640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 0x7faec00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed7fff640 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db301d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faedcc02640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 0x7faec00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 -- 172.21.15.28:0/2324390589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 msgr2=0x555db301d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db301d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 -- 172.21.15.28:0/2324390589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 msgr2=0x555db3019f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3019f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 -- 172.21.15.28:0/2324390589 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555db2d5a7e0 con 0x555db301d690 2024-03-22T02:18:49.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed57fa640 1 -- 172.21.15.28:0/4069746412 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7faecc0cfa30 con 0x7faed8100a90 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.750+0000 7faed77fe640 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 0x555db302a500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faec8009b70 tx=0x7faec80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262d888c560b50e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.751+0000 7faebe7fc640 1 -- 172.21.15.28:0/2324390589 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec8009d20 con 0x555db301d690 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.751+0000 7faedcc02640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 0x7faec00a53e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7faed0009370 tx=0x7faed002bf10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.751+0000 7faebe7fc640 1 -- 172.21.15.28:0/2324390589 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faec80035f0 con 0x555db301d690 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.751+0000 7faebe7fc640 1 -- 172.21.15.28:0/2324390589 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec800f6e0 con 0x555db301d690 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 msgr2=0x555db302a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 0x555db302a500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faec8009b70 tx=0x7faec80107a0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 shutdown_connections 2024-03-22T02:18:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301d690 0x555db302a500 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db301d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 --2- 172.21.15.28:0/2324390589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3019f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.752+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 >> 172.21.15.28:0/2324390589 conn(0x555db3015130 msgr2=0x555db3015530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.753+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 shutdown_connections 2024-03-22T02:18:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.753+0000 7faee1245740 1 -- 172.21.15.28:0/2324390589 wait complete. 2024-03-22T02:18:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.753+0000 7faee1245740 1 -- start start 2024-03-22T02:18:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.753+0000 7faee1245740 1 Processor -- start 2024-03-22T02:18:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faee1245740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db3019b40 0x555db302c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faee1245740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faee1245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 0x555db30339f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faed77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faed77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39008/0 (socket says 172.21.15.28:39008) 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faed77fe640 1 -- 172.21.15.28:0/804911508 learned_addr learned my addr 172.21.15.28:0/804911508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.754+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555db302af00 con 0x555db301d690 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 0x555db30339f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555db2f3a110 con 0x555db3019b40 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555db302b080 con 0x555db301a530 2024-03-22T02:18:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faee1245740 10 client.0 _ll_register_callbacks cb 0x555db2f76be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faed7fff640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db3019b40 0x555db302c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 -- 172.21.15.28:0/804911508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 msgr2=0x555db302c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 -- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db3019b40 msgr2=0x555db302c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db3019b40 0x555db302c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faedcc02640 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555db2dbaf70 con 0x555db301d690 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.755+0000 7faed77fe640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:49.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.756+0000 7faedcc02640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 0x555db30339f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faed0005c40 tx=0x7faed0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeb06295419c51c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.756+0000 7faeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:49.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.756+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0047070 con 0x555db301d690 2024-03-22T02:18:49.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.756+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faec80035f0 con 0x555db301d690 2024-03-22T02:18:49.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.757+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7faed0003e00 con 0x555db301d690 2024-03-22T02:18:49.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.757+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faed005c050 con 0x555db301d690 2024-03-22T02:18:49.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faee1245740 10 client.17007 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:49.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555db2c52170 con 0x555db301d690 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faec80035f0 con 0x555db301d690 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7faed00347b0 con 0x555db301d690 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faee1245740 10 client.17007 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faee1245740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.758+0000 7faee1245740 20 client.17007 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.759+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db303a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.759+0000 7faee1245740 4 client.17007 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:49.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.759+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555db2f8bab0 con 0x555db3038310 2024-03-22T02:18:49.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.761+0000 7faed7fff640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db303a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.761+0000 7faed7fff640 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db303a760 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91bd668a0c3aa2ed server_cookie=2c5325a9dc4f91e8 in_seq=0 out_seq=0 2024-03-22T02:18:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.761+0000 7faeb77fe640 10 client.17007 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.764+0000 7faeb77fe640 1 -- 172.21.15.28:0/804911508 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555db2f8bab0 con 0x555db3038310 2024-03-22T02:18:49.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.764+0000 7faeb77fe640 10 client.17007 handle_command_reply: tid=0 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.764+0000 7faee1245740 1 client.17007 shutdown 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.764+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 msgr2=0x555db30339f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.764+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 0x555db30339f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7faed0005c40 tx=0x7faed0009500 comp rx=0 tx=0).stop 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 shutdown_connections 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db303a760 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faeb77fe640 0 client.17007 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db301d690 0x555db30339f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301a530 0x555db302c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 --2- 172.21.15.28:0/804911508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db3019b40 0x555db302c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 >> 172.21.15.28:0/804911508 conn(0x555db3015130 msgr2=0x555db30143f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 shutdown_connections 2024-03-22T02:18:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 1 -- 172.21.15.28:0/804911508 wait complete. 2024-03-22T02:18:49.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.765+0000 7faee1245740 20 client.17007 trim_cache size 0 max 16384 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.797+0000 7faee1245740 1 Processor -- start 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.797+0000 7faee1245740 1 -- start start 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.797+0000 7faee1245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3026bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faee1245740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30339b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faee1245740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db303f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3026bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3026bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34674/0 (socket says 172.21.15.28:34674) 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faed7fff640 1 -- 172.21.15.28:0/1637146099 learned_addr learned my addr 172.21.15.28:0/1637146099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faee1245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed81ab140 con 0x555db3019b40 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faed81a8a10 con 0x555db301a530 2024-03-22T02:18:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faed81a6860 con 0x555db301d690 2024-03-22T02:18:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db303f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faed77fe640 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 -- 172.21.15.28:0/1637146099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 msgr2=0x555db30339b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30339b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 -- 172.21.15.28:0/1637146099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 msgr2=0x555db3026bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3026bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.798+0000 7faedcc02640 1 -- 172.21.15.28:0/1637146099 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555db3010c10 con 0x555db301d690 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.799+0000 7faedcc02640 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db303f020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faed0034e10 tx=0x7faed0005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6da20e639e6e301 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.799+0000 7faebdffb640 1 -- 172.21.15.28:0/1637146099 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0064070 con 0x555db301d690 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.799+0000 7faebdffb640 1 -- 172.21.15.28:0/1637146099 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faed0053070 con 0x555db301d690 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.799+0000 7faebdffb640 1 -- 172.21.15.28:0/1637146099 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0047040 con 0x555db301d690 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.800+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 msgr2=0x555db303f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.800+0000 7faee1245740 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db303f020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faed0034e10 tx=0x7faed0005cf0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 shutdown_connections 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db303f020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30339b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 --2- 172.21.15.28:0/1637146099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3026bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 >> 172.21.15.28:0/1637146099 conn(0x555db3015130 msgr2=0x555db30264e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 shutdown_connections 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.801+0000 7faee1245740 1 -- 172.21.15.28:0/1637146099 wait complete. 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.802+0000 7faee1245740 1 -- start start 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.802+0000 7faee1245740 1 Processor -- start 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.802+0000 7faee1245740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.802+0000 7faee1245740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30332f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faee1245740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db3023ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faee1245740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faed81a6ed0 con 0x555db3019b40 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faee1245740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faed81a8340 con 0x555db301a530 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faee1245740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faed81a75a0 con 0x555db301d690 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faee1245740 10 client.0 _ll_register_callbacks cb 0x7faec8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34676/0 (socket says 172.21.15.28:34676) 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 -- 172.21.15.28:0/1701596423 learned_addr learned my addr 172.21.15.28:0/1701596423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faedcc02640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db3023ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 -- 172.21.15.28:0/1701596423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 msgr2=0x555db3023ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db3023ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 -- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 msgr2=0x555db30332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30332f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 -- 172.21.15.28:0/1701596423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555db3024490 con 0x555db3019b40 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faedcc02640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db3023ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.803+0000 7faed7fff640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faecc0cd170 tx=0x7faecc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb675c95f07efbe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faecc003b10 con 0x555db3019b40 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7faecc0983c0 con 0x555db3019b40 2024-03-22T02:18:49.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faed0053070 con 0x555db3019b40 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faecc0a42d0 con 0x555db3019b40 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 10 client.17013 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555db3025840 con 0x555db3019b40 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faed0053070 con 0x555db3019b40 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7faecc09b1a0 con 0x555db3019b40 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 10 client.17013 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 10 client.17013 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 20 client.17013 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db3026240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 4 client.17013 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.804+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555db2f85e00 con 0x555db3038310 2024-03-22T02:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.805+0000 7faed77fe640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db3026240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.805+0000 7faeb77fe640 10 client.17013 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.805+0000 7faed77fe640 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db3026240 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8617a2f0db71f16 server_cookie=1e84c8ed4533fe1c in_seq=0 out_seq=0 2024-03-22T02:18:49.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.809+0000 7faeb77fe640 1 -- 172.21.15.28:0/1701596423 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555db2f85e00 con 0x555db3038310 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.809+0000 7faeb77fe640 10 client.17013 handle_command_reply: tid=0 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.809+0000 7faee1245740 1 client.17013 shutdown 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.809+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 msgr2=0x555db3032db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.809+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7faecc0cd170 tx=0x7faecc003f10 comp rx=0 tx=0).stop 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 shutdown_connections 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555db3038310 0x555db3026240 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555db301d690 0x555db3023ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faeb77fe640 0 client.17013 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555db301a530 0x555db30332f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 --2- 172.21.15.28:0/1701596423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555db3019b40 0x555db3032db0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 >> 172.21.15.28:0/1701596423 conn(0x555db3015130 msgr2=0x555db30143f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.810+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 shutdown_connections 2024-03-22T02:18:49.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.811+0000 7faee1245740 1 -- 172.21.15.28:0/1701596423 wait complete. 2024-03-22T02:18:49.813 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:49.813 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:18:49.813 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "f42e5d2f-1deb-435f-bf63-3c2be6dc7448", 2024-03-22T02:18:49.813 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:18:49.813 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.811+0000 7faee1245740 20 client.17013 trim_cache size 0 max 16384 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 msgr2=0x7faec00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 0x7faec00a53e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7faed0009370 tx=0x7faed002bf10 comp rx=0 tx=0).stop 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 msgr2=0x7faed8196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7faecc00e9f0 tx=0x7faecc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 shutdown_connections 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faec00a2f20 0x7faec00a53e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faed8102640 0x7faed819e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faed8101c80 0x7faed81972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 --2- 172.21.15.28:0/4069746412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faed8100a90 0x7faed8196d60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 >> 172.21.15.28:0/4069746412 conn(0x7faed80fc020 msgr2=0x7faed8104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 shutdown_connections 2024-03-22T02:18:49.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.812+0000 7faede665640 1 -- 172.21.15.28:0/4069746412 wait complete. 2024-03-22T02:18:49.825 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:18:49.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 -- 172.21.15.28:0/3542218075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8101c80 msgr2=0x7fdfd8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 --2- 172.21.15.28:0/3542218075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8101c80 0x7fdfd8102100 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fdfc0009900 tx=0x7fdfc002bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 -- 172.21.15.28:0/3542218075 shutdown_connections 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 --2- 172.21.15.28:0/3542218075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8102640 0x7fdfd8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 --2- 172.21.15.28:0/3542218075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8101c80 0x7fdfd8102100 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 --2- 172.21.15.28:0/3542218075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8100a90 0x7fdfd8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.934+0000 7fdfdd579640 1 -- 172.21.15.28:0/3542218075 >> 172.21.15.28:0/3542218075 conn(0x7fdfd80fc020 msgr2=0x7fdfd80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- 172.21.15.28:0/3542218075 shutdown_connections 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- 172.21.15.28:0/3542218075 wait complete. 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 Processor -- start 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- start start 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8101c80 0x7fdfd819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8102640 0x7fdfd81a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810c710 con 0x7fdfd8100a90 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810c590 con 0x7fdfd8101c80 2024-03-22T02:18:49.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfdd579640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810c890 con 0x7fdfd8102640 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.935+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8101c80 0x7fdfd819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34698/0 (socket says 172.21.15.28:34698) 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/573315573 learned_addr learned my addr 172.21.15.28:0/573315573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8102640 0x7fdfd81a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/573315573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8102640 msgr2=0x7fdfd81a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8102640 0x7fdfd81a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/573315573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8101c80 msgr2=0x7fdfd819f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8101c80 0x7fdfd819f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/573315573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81a1d90 con 0x7fdfd8100a90 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800eb70 tx=0x7fdfc800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfcffff640 1 -- 172.21.15.28:0/573315573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8003b70 con 0x7fdfd8100a90 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfcffff640 1 -- 172.21.15.28:0/573315573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc800ee70 con 0x7fdfd8100a90 2024-03-22T02:18:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd81a1fc0 con 0x7fdfd8100a90 2024-03-22T02:18:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfcffff640 1 -- 172.21.15.28:0/573315573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8010830 con 0x7fdfd8100a90 2024-03-22T02:18:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.936+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81a2430 con 0x7fdfd8100a90 2024-03-22T02:18:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.938+0000 7fdfcffff640 1 -- 172.21.15.28:0/573315573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdfc8004020 con 0x7fdfd8100a90 2024-03-22T02:18:49.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.939+0000 7fdfe0233740 1 Processor -- start 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.939+0000 7fdfcffff640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 0x7fdfc40a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.939+0000 7fdfe0233740 1 -- start start 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.939+0000 7fdfcffff640 1 -- 172.21.15.28:0/573315573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdfc80cfe50 con 0x7fdfd8100a90 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.939+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 0x7fdfc40a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46512920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c46515420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c464728c0 con 0x564c46512f90 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564c4641ac20 con 0x564c4650de20 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfe0233740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c4646e870 con 0x564c4650d430 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46512920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39052/0 (socket says 172.21.15.28:39052) 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/1056291020 learned_addr learned my addr 172.21.15.28:0/1056291020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c46515420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 0x7fdfc40a5410 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009c00 tx=0x7fdfd0011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 -- 172.21.15.28:0/1056291020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 msgr2=0x564c4650d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 -- 172.21.15.28:0/1056291020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 msgr2=0x564c46512920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46512920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 -- 172.21.15.28:0/1056291020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c4643f150 con 0x564c46512f90 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd67fc640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46512920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c46515420 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011f30 tx=0x7fdfd001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3140d4da6b21bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfcd7fa640 1 -- 172.21.15.28:0/1056291020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0017040 con 0x564c46512f90 2024-03-22T02:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.940+0000 7fdfcd7fa640 1 -- 172.21.15.28:0/1056291020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd0019050 con 0x564c46512f90 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfcd7fa640 1 -- 172.21.15.28:0/1056291020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0004d70 con 0x564c46512f90 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 -- 172.21.15.28:0/1056291020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 msgr2=0x564c46515420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c46515420 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011f30 tx=0x7fdfd001c000 comp rx=0 tx=0).stop 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 -- 172.21.15.28:0/1056291020 shutdown_connections 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c46515420 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46512920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 --2- 172.21.15.28:0/1056291020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 -- 172.21.15.28:0/1056291020 >> 172.21.15.28:0/1056291020 conn(0x564c465089a0 msgr2=0x564c46508da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 -- 172.21.15.28:0/1056291020 shutdown_connections 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.941+0000 7fdfe0233740 1 -- 172.21.15.28:0/1056291020 wait complete. 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 -- start start 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 Processor -- start 2024-03-22T02:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c46517970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46517eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c46515f40 con 0x564c46512f90 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564c4642e130 con 0x564c4650de20 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c465160c0 con 0x564c4650d430 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfe0233740 10 client.0 _ll_register_callbacks cb 0x564c4642db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46517eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46517eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40020/0 (socket says 172.21.15.28:40020) 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c46517970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd67fc640 1 -- 172.21.15.28:0/4123911161 learned_addr learned my addr 172.21.15.28:0/4123911161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 -- 172.21.15.28:0/4123911161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 msgr2=0x564c46517970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c46517970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 -- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 msgr2=0x564c46517eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46517eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.942+0000 7fdfd77fe640 1 -- 172.21.15.28:0/4123911161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c462507d0 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011940 tx=0x7fdfd0011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0ef32c2cc59404 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0017050 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdfd0004ce0 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd0019050 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdfd003c050 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfe0233740 10 client.17025 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c461863f0 con 0x564c46512f90 2024-03-22T02:18:49.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd0019050 con 0x564c46512f90 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fdfd0037030 con 0x564c46512f90 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfe0233740 10 client.17025 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.943+0000 7fdfe0233740 10 client.17025 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfe0233740 20 client.17025 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c465235a0 0x564c46525a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfe0233740 4 client.17025 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c4647fe00 con 0x564c465235a0 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c465235a0 0x564c46525a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c465235a0 0x564c46525a00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2124dec772da1c6 server_cookie=cebf9b774ff47b9b in_seq=0 out_seq=0 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.944+0000 7fdfbe7fc640 10 client.17025 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.945+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/4123911161 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564c4647fe00 con 0x564c465235a0 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.945+0000 7fdfbe7fc640 10 client.17025 handle_command_reply: tid=0 2024-03-22T02:18:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 client.17025 shutdown 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 msgr2=0x564c4651eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0011940 tx=0x7fdfd0011d80 comp rx=0 tx=0).stop 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 shutdown_connections 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c465235a0 0x564c46525a00 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eef0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c46517eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfbe7fc640 0 client.17025 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 --2- 172.21.15.28:0/4123911161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c46517970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 >> 172.21.15.28:0/4123911161 conn(0x564c465089a0 msgr2=0x564c46507c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 shutdown_connections 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 1 -- 172.21.15.28:0/4123911161 wait complete. 2024-03-22T02:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.946+0000 7fdfe0233740 20 client.17025 trim_cache size 0 max 16384 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.965+0000 7fdfe0233740 1 Processor -- start 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.965+0000 7fdfe0233740 1 -- start start 2024-03-22T02:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4650f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81af460 con 0x564c46512f90 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81acd30 con 0x564c4650de20 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfe0233740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81aab80 con 0x564c4650d430 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4650f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4650f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40028/0 (socket says 172.21.15.28:40028) 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd67fc640 1 -- 172.21.15.28:0/2908530660 learned_addr learned my addr 172.21.15.28:0/2908530660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2908530660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 msgr2=0x564c4650f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4650f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2908530660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 msgr2=0x564c4651eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.966+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2908530660 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c461460f0 con 0x564c4650d430 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650f320 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdfc809be10 tx=0x7fdfc80cf040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a53b5e5da475d634 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfccff9640 1 -- 172.21.15.28:0/2908530660 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc809c810 con 0x564c4650d430 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfccff9640 1 -- 172.21.15.28:0/2908530660 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc809c110 con 0x564c4650d430 2024-03-22T02:18:49.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfccff9640 1 -- 172.21.15.28:0/2908530660 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8004310 con 0x564c4650d430 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfe0233740 1 -- 172.21.15.28:0/2908530660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 msgr2=0x564c4650f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.967+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650f320 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdfc809be10 tx=0x7fdfc80cf040 comp rx=0 tx=0).stop 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- 172.21.15.28:0/2908530660 shutdown_connections 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c46512f90 0x564c4651eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4650f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2908530660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c4650d430 0x564c4650f320 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- 172.21.15.28:0/2908530660 >> 172.21.15.28:0/2908530660 conn(0x564c465089a0 msgr2=0x564c46516460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- 172.21.15.28:0/2908530660 shutdown_connections 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- 172.21.15.28:0/2908530660 wait complete. 2024-03-22T02:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- start start 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 Processor -- start 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4651c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c46512f90 0x564c4651d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81ab1f0 con 0x564c4650d430 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81ac660 con 0x564c4650de20 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81ab8c0 con 0x564c46512f90 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfe0233740 10 client.0 _ll_register_callbacks cb 0x7fdfc8004680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34728/0 (socket says 172.21.15.28:34728) 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2998249627 learned_addr learned my addr 172.21.15.28:0/2998249627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:49.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfd67fc640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4651c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.968+0000 7fdfd77fe640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c46512f90 0x564c4651d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2998249627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c46512f90 msgr2=0x564c4651d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c46512f90 0x564c4651d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 msgr2=0x564c4651c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4651c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 -- 172.21.15.28:0/2998249627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c4651daa0 con 0x564c4650d430 2024-03-22T02:18:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfd6ffd640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8098ae0 tx=0x7fdfc800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e93798fddd5ff4c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80a4b20 con 0x564c4650d430 2024-03-22T02:18:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc809c110 con 0x564c4650d430 2024-03-22T02:18:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7fdfc8098780 con 0x564c4650d430 2024-03-22T02:18:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdfc80a4690 con 0x564c4650d430 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 10 client.17031 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564c4651ee00 con 0x564c4650d430 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc809c110 con 0x564c4650d430 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7fdfc809b560 con 0x564c4650d430 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 10 client.17031 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 10 client.17031 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 20 client.17031 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c46589ba0 0x564c4658bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 4 client.17031 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.969+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c4647a150 con 0x564c46589ba0 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.970+0000 7fdfd67fc640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c46589ba0 0x564c4658bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.970+0000 7fdfbe7fc640 10 client.17031 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.970+0000 7fdfd67fc640 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c46589ba0 0x564c4658bfc0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e475523412fe1b82 server_cookie=ef1a544016bce1e5 in_seq=0 out_seq=0 2024-03-22T02:18:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfbe7fc640 1 -- 172.21.15.28:0/2998249627 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564c4647a150 con 0x564c46589ba0 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfbe7fc640 10 client.17031 handle_command_reply: tid=0 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 client.17031 shutdown 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 msgr2=0x564c4651bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8098ae0 tx=0x7fdfc800cf90 comp rx=0 tx=0).stop 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 shutdown_connections 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c46589ba0 0x564c4658bfc0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c46512f90 0x564c4651d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c4650de20 0x564c4651c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfbe7fc640 0 client.17031 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 --2- 172.21.15.28:0/2998249627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c4650d430 0x564c4651bf60 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 >> 172.21.15.28:0/2998249627 conn(0x564c465089a0 msgr2=0x564c4650f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 shutdown_connections 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 1 -- 172.21.15.28:0/2998249627 wait complete. 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfe0233740 20 client.17031 trim_cache size 0 max 16384 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "f42e5d2f-1deb-435f-bf63-3c2be6dc7448": { 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "f42e5d2f-1deb-435f-bf63-3c2be6dc7448", 2024-03-22T02:18:49.983 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:18:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:18:49.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:18:49.984 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 msgr2=0x7fdfc40a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 0x7fdfc40a5410 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fdfd0009c00 tx=0x7fdfd0011440 comp rx=0 tx=0).stop 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 msgr2=0x7fdfd8069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800eb70 tx=0x7fdfc800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 shutdown_connections 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdfc40a2f50 0x7fdfc40a5410 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdfd8102640 0x7fdfd81a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdfd8101c80 0x7fdfd819f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 --2- 172.21.15.28:0/573315573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8100a90 0x7fdfd8069860 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 >> 172.21.15.28:0/573315573 conn(0x7fdfd80fc020 msgr2=0x7fdfd8105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.971+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 shutdown_connections 2024-03-22T02:18:49.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:18:49.972+0000 7fdfdd579640 1 -- 172.21.15.28:0/573315573 wait complete. 2024-03-22T02:18:49.986 INFO:tasks.cephfs.filesystem:scrub status for tag:f42e5d2f-1deb-435f-bf63-3c2be6dc7448 - {'path': '/', 'tag': 'f42e5d2f-1deb-435f-bf63-3c2be6dc7448', 'options': 'recursive,force'} 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 -- 172.21.15.28:0/1200134207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4101ca0 msgr2=0x7ffbe4102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/1200134207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4102120 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009900 tx=0x7ffbd802bc30 comp rx=0 tx=0).stop 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 -- 172.21.15.28:0/1200134207 shutdown_connections 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/1200134207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4102660 0x7ffbe4108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/1200134207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4102120 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/1200134207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 -- 172.21.15.28:0/1200134207 >> 172.21.15.28:0/1200134207 conn(0x7ffbe40fc020 msgr2=0x7ffbe40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 -- 172.21.15.28:0/1200134207 shutdown_connections 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.264+0000 7ffbea4ed640 1 -- 172.21.15.28:0/1200134207 wait complete. 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 Processor -- start 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 -- start start 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4102660 0x7ffbe41035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe410c760 con 0x7ffbe4100ab0 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe410c5e0 con 0x7ffbe4102660 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbea4ed640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe410c8e0 con 0x7ffbe4101ca0 2024-03-22T02:18:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34740/0 (socket says 172.21.15.28:34740) 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbe3fff640 1 -- 172.21.15.28:0/814946618 learned_addr learned my addr 172.21.15.28:0/814946618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.265+0000 7ffbe8a8a640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4102660 0x7ffbe41035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 -- 172.21.15.28:0/814946618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 msgr2=0x7ffbe4103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 -- 172.21.15.28:0/814946618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4102660 msgr2=0x7ffbe41035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4102660 0x7ffbe41035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 -- 172.21.15.28:0/814946618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe4103e50 con 0x7ffbe4100ab0 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9c0 tx=0x7ffbd400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe17fa640 1 -- 172.21.15.28:0/814946618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400cd80 con 0x7ffbe4100ab0 2024-03-22T02:18:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe17fa640 1 -- 172.21.15.28:0/814946618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd40035f0 con 0x7ffbe4100ab0 2024-03-22T02:18:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe41a1f10 con 0x7ffbe4100ab0 2024-03-22T02:18:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbe17fa640 1 -- 172.21.15.28:0/814946618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010690 con 0x7ffbe4100ab0 2024-03-22T02:18:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.266+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe41a2310 con 0x7ffbe4100ab0 2024-03-22T02:18:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.268+0000 7ffbe17fa640 1 -- 172.21.15.28:0/814946618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ffbd40037a0 con 0x7ffbe4100ab0 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe17fa640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 0x7ffbcc0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 0x7ffbcc0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 Processor -- start 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe17fa640 1 -- 172.21.15.28:0/814946618 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffbd40cfb20 con 0x7ffbe4100ab0 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 -- start start 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d3459ed0 0x5578d345e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d34614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5578d33be040 con 0x5578d34594e0 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5578d33669e0 con 0x5578d345f040 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5578d33b9ff0 con 0x5578d3459ed0 2024-03-22T02:18:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34750/0 (socket says 172.21.15.28:34750) 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe3fff640 1 -- 172.21.15.28:0/77302479 learned_addr learned my addr 172.21.15.28:0/77302479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe8a8a640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d34614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 0x7ffbcc0a5480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c240 tx=0x7ffbd802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.269+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d3459ed0 0x5578d345e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 -- 172.21.15.28:0/77302479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d3459ed0 msgr2=0x5578d345e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d3459ed0 0x5578d345e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 -- 172.21.15.28:0/77302479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 msgr2=0x5578d34614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d34614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 -- 172.21.15.28:0/77302479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d30d2210 con 0x5578d34594e0 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe8a8a640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d34614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40a4320 tx=0x7ffbd409b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad14f86b43314ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbca7fc640 1 -- 172.21.15.28:0/77302479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40b6070 con 0x5578d34594e0 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbca7fc640 1 -- 172.21.15.28:0/77302479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd409ff90 con 0x5578d34594e0 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.270+0000 7ffbca7fc640 1 -- 172.21.15.28:0/77302479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40b4020 con 0x5578d34594e0 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- 172.21.15.28:0/77302479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 msgr2=0x5578d3459900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40a4320 tx=0x7ffbd409b860 comp rx=0 tx=0).stop 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- 172.21.15.28:0/77302479 shutdown_connections 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d34614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d3459ed0 0x5578d345e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/77302479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d3459900 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- 172.21.15.28:0/77302479 >> 172.21.15.28:0/77302479 conn(0x5578d3454a50 msgr2=0x5578d3454e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- 172.21.15.28:0/77302479 shutdown_connections 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- 172.21.15.28:0/77302479 wait complete. 2024-03-22T02:18:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 -- start start 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.271+0000 7ffbed1b4740 1 Processor -- start 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34634f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d346aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5578d3461ff0 con 0x5578d3459ed0 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5578d3379db0 con 0x5578d345f040 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5578d3462170 con 0x5578d34594e0 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 10 client.0 _ll_register_callbacks cb 0x5578d30d1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34764/0 (socket says 172.21.15.28:34764) 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 -- 172.21.15.28:0/86026370 learned_addr learned my addr 172.21.15.28:0/86026370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe8a8a640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d346aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 -- 172.21.15.28:0/86026370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 msgr2=0x5578d34634f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34634f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 -- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 msgr2=0x5578d346aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d346aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 -- 172.21.15.28:0/86026370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d319e620 con 0x5578d3459ed0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802ceb0 tx=0x7ffbd8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a012ec77f639a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8050030 con 0x5578d3459ed0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffbd803a040 con 0x5578d3459ed0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.272+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbd409ff90 con 0x5578d3459ed0 2024-03-22T02:18:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbd8004ed0 con 0x5578d3459ed0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 10 client.17043 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5578d346de50 con 0x5578d3459ed0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd409ff90 con 0x5578d3459ed0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ffbd804b040 con 0x5578d3459ed0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 10 client.17043 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 20 client.17043 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d346f5e0 0x5578d34719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 4 client.17043 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d33cb580 con 0x5578d346f5e0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.273+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d346f5e0 0x5578d34719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbc37fe640 10 client.17043 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d346f5e0 0x5578d34719d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=307077e10ddb7f06 server_cookie=2a90c571e2e317ea in_seq=0 out_seq=0 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbc37fe640 1 -- 172.21.15.28:0/86026370 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578d33cb580 con 0x5578d346f5e0 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbc37fe640 10 client.17043 handle_command_reply: tid=0 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 client.17043 shutdown 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 msgr2=0x5578d3463a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802ceb0 tx=0x7ffbd8009260 comp rx=0 tx=0).stop 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 shutdown_connections 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d346f5e0 0x5578d34719d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d346aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3463a30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbc37fe640 0 client.17043 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/86026370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 >> 172.21.15.28:0/86026370 conn(0x5578d3454a50 msgr2=0x5578d3453c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.274+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 shutdown_connections 2024-03-22T02:18:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.275+0000 7ffbed1b4740 1 -- 172.21.15.28:0/86026370 wait complete. 2024-03-22T02:18:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.275+0000 7ffbed1b4740 20 client.17043 trim_cache size 0 max 16384 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 Processor -- start 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 -- start start 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 0x5578d346aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d345f040 0x5578d34533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41af440 con 0x5578d34594e0 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41acd10 con 0x5578d3459ed0 2024-03-22T02:18:50.296 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41aab60 con 0x5578d345f040 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34768/0 (socket says 172.21.15.28:34768) 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 0x5578d346aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbe3fff640 1 -- 172.21.15.28:0/347515992 learned_addr learned my addr 172.21.15.28:0/347515992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.291+0000 7ffbe8a8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d345f040 0x5578d34533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 -- 172.21.15.28:0/347515992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d345f040 msgr2=0x5578d34533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d345f040 0x5578d34533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 -- 172.21.15.28:0/347515992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 msgr2=0x5578d345e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 -- 172.21.15.28:0/347515992 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d3459120 con 0x5578d3459ed0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 0x5578d346aa30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009900 tx=0x7ffbd802c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65bd4404c285a767 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbc9ffb640 1 -- 172.21.15.28:0/347515992 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8050030 con 0x5578d3459ed0 2024-03-22T02:18:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbc9ffb640 1 -- 172.21.15.28:0/347515992 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd8046070 con 0x5578d3459ed0 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.292+0000 7ffbc9ffb640 1 -- 172.21.15.28:0/347515992 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80052d0 con 0x5578d3459ed0 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- 172.21.15.28:0/347515992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 msgr2=0x5578d346aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 0x5578d346aa30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009900 tx=0x7ffbd802c8a0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- 172.21.15.28:0/347515992 shutdown_connections 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d345f040 0x5578d34533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d3459ed0 0x5578d346aa30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/347515992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d34594e0 0x5578d345e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- 172.21.15.28:0/347515992 >> 172.21.15.28:0/347515992 conn(0x5578d3454a50 msgr2=0x5578d3458540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- 172.21.15.28:0/347515992 shutdown_connections 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- 172.21.15.28:0/347515992 wait complete. 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 -- start start 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.293+0000 7ffbed1b4740 1 Processor -- start 2024-03-22T02:18:50.298 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34685e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d3469b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ab1d0 con 0x5578d3459ed0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ac640 con 0x5578d345f040 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe41ab8a0 con 0x5578d34594e0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 10 client.0 _ll_register_callbacks cb 0x7ffbd8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34685e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34774/0 (socket says 172.21.15.28:34774) 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 -- 172.21.15.28:0/1126595874 learned_addr learned my addr 172.21.15.28:0/1126595874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe8a8a640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d3469b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 -- 172.21.15.28:0/1126595874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 msgr2=0x5578d34685e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34685e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.299 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 -- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 msgr2=0x5578d3469b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d3469b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 -- 172.21.15.28:0/1126595874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d346a120 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbe37fe640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c170 tx=0x7ffbd8003b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3bf51b43c34f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8050070 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbd8046070 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.294+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ffbd8005480 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffbd8003e70 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 10 client.17049 fetch_fsmap learned FSMap version 29 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5578d34d5cc0 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd8046070 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ffbd804b030 con 0x5578d3459ed0 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 10 client.17049 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:18:50.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 10 client.17049 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 20 client.17049 populate_metadata read hostname 'smithi028' 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d34d6360 0x5578d34d8780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 4 client.17049 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578d33c58d0 con 0x5578d34d6360 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d34d6360 0x5578d34d8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.295+0000 7ffbc37fe640 10 client.17049 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.296+0000 7ffbe3fff640 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d34d6360 0x5578d34d8780 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bdf7615080697f7 server_cookie=bebf4b0086b00224 in_seq=0 out_seq=0 2024-03-22T02:18:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:50 smithi028 ceph-mon[31916]: pgmap v515: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 441 op/s 2024-03-22T02:18:50.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 398417 11.85 MB/sec execute 92 sec latency 62.765 ms 2024-03-22T02:18:50.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:50 smithi120 ceph-mon[36803]: pgmap v515: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 441 op/s 2024-03-22T02:18:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.513+0000 7ffbc37fe640 1 -- 172.21.15.28:0/1126595874 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578d33c58d0 con 0x5578d34d6360 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.513+0000 7ffbc37fe640 10 client.17049 handle_command_reply: tid=0 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.513+0000 7ffbed1b4740 1 client.17049 shutdown 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.513+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 msgr2=0x5578d3468b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.513+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c170 tx=0x7ffbd8003b70 comp rx=0 tx=0).stop 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 shutdown_connections 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5578d34d6360 0x5578d34d8780 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbc37fe640 0 client.17049 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5578d345f040 0x5578d3469b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5578d3459ed0 0x5578d3468b20 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 --2- 172.21.15.28:0/1126595874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5578d34594e0 0x5578d34685e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 >> 172.21.15.28:0/1126595874 conn(0x5578d3454a50 msgr2=0x5578d3458540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 shutdown_connections 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 1 -- 172.21.15.28:0/1126595874 wait complete. 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbed1b4740 20 client.17049 trim_cache size 0 max 16384 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:18:50.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:18:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 msgr2=0x7ffbcc0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 0x7ffbcc0a5480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c240 tx=0x7ffbd802df90 comp rx=0 tx=0).stop 2024-03-22T02:18:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 msgr2=0x7ffbe4104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:18:50.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.514+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9c0 tx=0x7ffbd400ee80 comp rx=0 tx=0).stop 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 shutdown_connections 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffbcc0a2fc0 0x7ffbcc0a5480 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffbe4102660 0x7ffbe41035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbe4101ca0 0x7ffbe4103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 --2- 172.21.15.28:0/814946618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbe4100ab0 0x7ffbe4104a40 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 >> 172.21.15.28:0/814946618 conn(0x7ffbe40fc020 msgr2=0x7ffbe40fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 shutdown_connections 2024-03-22T02:18:50.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:18:50.515+0000 7ffbea4ed640 1 -- 172.21.15.28:0/814946618 wait complete. 2024-03-22T02:18:50.519 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:50 smithi083 ceph-mon[36340]: pgmap v515: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 441 op/s 2024-03-22T02:18:51.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 401162 11.90 MB/sec execute 93 sec latency 152.619 ms 2024-03-22T02:18:51.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:18:51.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:18:51.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:18:51.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:51 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:18:51.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:18:51.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:18:51.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:51 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:18:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:18:51.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:18:51.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:51.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:51 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:18:52.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 403758 11.90 MB/sec execute 94 sec latency 46.075 ms 2024-03-22T02:18:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:52 smithi120 ceph-mon[36803]: pgmap v516: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2024-03-22T02:18:52.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:52 smithi083 ceph-mon[36340]: pgmap v516: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2024-03-22T02:18:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:52 smithi028 ceph-mon[31916]: pgmap v516: 129 pgs: 129 active+clean; 50 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 314 op/s 2024-03-22T02:18:53.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 405542 11.87 MB/sec execute 95 sec latency 36.626 ms 2024-03-22T02:18:54.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 408242 11.93 MB/sec execute 96 sec latency 42.658 ms 2024-03-22T02:18:54.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:54 smithi120 ceph-mon[36803]: pgmap v517: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-03-22T02:18:54.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:54 smithi083 ceph-mon[36340]: pgmap v517: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-03-22T02:18:54.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:54 smithi028 ceph-mon[31916]: pgmap v517: 129 pgs: 129 active+clean; 44 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 308 op/s 2024-03-22T02:18:55.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 410399 11.88 MB/sec execute 97 sec latency 57.292 ms 2024-03-22T02:18:56.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 412404 11.89 MB/sec execute 98 sec latency 28.691 ms 2024-03-22T02:18:56.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:56 smithi120 ceph-mon[36803]: pgmap v518: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:18:56.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:56 smithi083 ceph-mon[36340]: pgmap v518: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:18:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:56 smithi028 ceph-mon[31916]: pgmap v518: 129 pgs: 129 active+clean; 43 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:18:57.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 415325 11.95 MB/sec execute 99 sec latency 12.316 ms 2024-03-22T02:18:58.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 417538 11.91 MB/sec execute 100 sec latency 50.517 ms 2024-03-22T02:18:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:18:58 smithi120 ceph-mon[36803]: pgmap v519: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:18:58.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:18:58 smithi083 ceph-mon[36340]: pgmap v519: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:18:58.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:18:58 smithi028 ceph-mon[31916]: pgmap v519: 129 pgs: 129 active+clean; 43 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:18:59.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 419410 11.91 MB/sec execute 101 sec latency 29.130 ms 2024-03-22T02:19:00.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 421804 11.92 MB/sec execute 102 sec latency 100.528 ms 2024-03-22T02:19:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:00 smithi120 ceph-mon[36803]: pgmap v520: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:19:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:00 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:19:00.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:00 smithi083 ceph-mon[36340]: pgmap v520: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:19:00.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:00 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:19:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:00 smithi028 ceph-mon[31916]: pgmap v520: 129 pgs: 129 active+clean; 46 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:19:00.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:00 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:19:01.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 423557 11.91 MB/sec execute 103 sec latency 44.601 ms 2024-03-22T02:19:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:01 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:19:01.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:01 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:19:01.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:01 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:19:02.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 426092 11.93 MB/sec execute 104 sec latency 10.711 ms 2024-03-22T02:19:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:02 smithi120 ceph-mon[36803]: pgmap v521: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:02.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:02 smithi083 ceph-mon[36340]: pgmap v521: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:02.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:02 smithi028 ceph-mon[31916]: pgmap v521: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:03.416 INFO:tasks.workunit.client.0.smithi028.stdout: 1 428553 11.92 MB/sec execute 105 sec latency 88.991 ms 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- 172.21.15.28:0/1571619689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 msgr2=0x7f9dd810a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/1571619689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd810a030 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009ca0 tx=0x7f9dd002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- 172.21.15.28:0/1571619689 shutdown_connections 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/1571619689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd810a030 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/1571619689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 0x7f9dd8103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/1571619689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- 172.21.15.28:0/1571619689 >> 172.21.15.28:0/1571619689 conn(0x7f9dd80fd160 msgr2=0x7f9dd80ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- 172.21.15.28:0/1571619689 shutdown_connections 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- 172.21.15.28:0/1571619689 wait complete. 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 Processor -- start 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 -- start start 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.609+0000 7f9ddf25c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 0x7f9dd8073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d870 con 0x7f9dd81037a0 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d6f0 con 0x7f9dd8101bf0 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd810d9f0 con 0x7f9dd8102de0 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddd7f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddd7f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50770/0 (socket says 172.21.15.28:50770) 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddd7f9640 1 -- 172.21.15.28:0/4216997056 learned_addr learned my addr 172.21.15.28:0/4216997056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 0x7f9dd8073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 -- 172.21.15.28:0/4216997056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 msgr2=0x7f9dd8073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 0x7f9dd8073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 -- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 msgr2=0x7f9dd80744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 -- 172.21.15.28:0/4216997056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd8074d50 con 0x7f9dd8101bf0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8075940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e9c0 tx=0x7f9dcc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9dd5ffb640 1 -- 172.21.15.28:0/4216997056 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc00cd80 con 0x7f9dd8101bf0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9dd5ffb640 1 -- 172.21.15.28:0/4216997056 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dcc010910 con 0x7f9dd8101bf0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd81a1ee0 con 0x7f9dd8101bf0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.610+0000 7f9dd5ffb640 1 -- 172.21.15.28:0/4216997056 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc018980 con 0x7f9dd8101bf0 2024-03-22T02:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.611+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd81a22f0 con 0x7f9dd8101bf0 2024-03-22T02:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 Processor -- start 2024-03-22T02:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9dd5ffb640 1 -- 172.21.15.28:0/4216997056 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9dcc01f020 con 0x7f9dd8101bf0 2024-03-22T02:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 -- start start 2024-03-22T02:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e494d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d271da6040 con 0x55d271e414e0 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d271d4e9e0 con 0x55d271e41ed0 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9de1e42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d271da1ff0 con 0x55d271e47040 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9ddd7f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9ddd7f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50776/0 (socket says 172.21.15.28:50776) 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9ddd7f9640 1 -- 172.21.15.28:0/3409972486 learned_addr learned my addr 172.21.15.28:0/3409972486 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.613+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 -- 172.21.15.28:0/3409972486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 msgr2=0x55d271e494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 -- 172.21.15.28:0/3409972486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 msgr2=0x55d271e469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dd5ffb640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 0x7f9dc80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 -- 172.21.15.28:0/3409972486 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d271aba210 con 0x55d271e414e0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 0x7f9dc80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dd5ffb640 1 -- 172.21.15.28:0/4216997056 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9dcc0cf720 con 0x7f9dd8101bf0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009430 tx=0x7f9dd0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a12009e7fc209f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dc2ffd640 1 -- 172.21.15.28:0/3409972486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0043070 con 0x55d271e414e0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dc2ffd640 1 -- 172.21.15.28:0/3409972486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dd0032c90 con 0x55d271e414e0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.614+0000 7f9dc2ffd640 1 -- 172.21.15.28:0/3409972486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0039040 con 0x55d271e414e0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- 172.21.15.28:0/3409972486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 msgr2=0x55d271e41900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0009430 tx=0x7f9dd0005ef0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 0x7f9dc80a5480 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80756b0 tx=0x7f9dc4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- 172.21.15.28:0/3409972486 shutdown_connections 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3409972486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e41900 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- 172.21.15.28:0/3409972486 >> 172.21.15.28:0/3409972486 conn(0x55d271e3ca50 msgr2=0x55d271e3ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- 172.21.15.28:0/3409972486 shutdown_connections 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- 172.21.15.28:0/3409972486 wait complete. 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 -- start start 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.615+0000 7f9de1e42740 1 Processor -- start 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 0x55d271e4b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e52970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d271e49f90 con 0x55d271e41ed0 2024-03-22T02:19:03.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d271d61db0 con 0x55d271e414e0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d271e4a110 con 0x55d271e47040 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 10 client.0 _ll_register_callbacks cb 0x55d271ab9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50778/0 (socket says 172.21.15.28:50778) 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9dd7fff640 1 -- 172.21.15.28:0/1392793234 learned_addr learned my addr 172.21.15.28:0/1392793234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 0x55d271e4b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e52970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 -- 172.21.15.28:0/1392793234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 msgr2=0x55d271e52970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e52970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 -- 172.21.15.28:0/1392793234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 msgr2=0x55d271e4b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 -- 172.21.15.28:0/1392793234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d271b86620 con 0x55d271e414e0 2024-03-22T02:19:03.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 0x55d271e4b3f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0cce60 tx=0x7f9dcc009550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97488df25b9e8652 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9db7fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc09b580 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9dcc010ac0 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.616+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dd0032c90 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9dcc09b3a0 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 10 client.26894 fetch_fsmap learned FSMap version 29 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d271e55d50 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dd0032c90 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f9dcc0cf3d0 con 0x55d271e414e0 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 10 client.26894 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 10 client.26894 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:03.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 20 client.26894 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271e573c0 0x55d271e597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 4 client.26894 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d271db3580 con 0x55d271e573c0 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.617+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271e573c0 0x55d271e597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271e573c0 0x55d271e597b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ec620e3e81be7c7 server_cookie=7422292f0cf8097 in_seq=0 out_seq=0 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9db7fff640 10 client.26894 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9db7fff640 1 -- 172.21.15.28:0/1392793234 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d271db3580 con 0x55d271e573c0 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9db7fff640 10 client.26894 handle_command_reply: tid=0 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 client.26894 shutdown 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 msgr2=0x55d271e4b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 0x55d271e4b3f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0cce60 tx=0x7f9dcc009550 comp rx=0 tx=0).stop 2024-03-22T02:19:03.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 shutdown_connections 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271e573c0 0x55d271e597b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e52970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.618+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e4b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9de1e42740 1 --2- 172.21.15.28:0/1392793234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e414e0 0x55d271e4b3f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9db7fff640 0 client.26894 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 >> 172.21.15.28:0/1392793234 conn(0x55d271e3ca50 msgr2=0x55d271e3bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 shutdown_connections 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9de1e42740 1 -- 172.21.15.28:0/1392793234 wait complete. 2024-03-22T02:19:03.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.619+0000 7f9de1e42740 20 client.26894 trim_cache size 0 max 16384 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.633+0000 7f9de1e42740 1 Processor -- start 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.633+0000 7f9de1e42740 1 -- start start 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e3b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81af390 con 0x55d271e414e0 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81acc60 con 0x55d271e41ed0 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9de1e42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81aaab0 con 0x55d271e47040 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9ddcff8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:60706/0 (socket says 172.21.15.28:60706) 2024-03-22T02:19:03.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 -- 172.21.15.28:0/3800263999 learned_addr learned my addr 172.21.15.28:0/3800263999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9ddcff8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50784/0 (socket says 172.21.15.28:50784) 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9ddd7f9640 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e3b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 -- 172.21.15.28:0/3800263999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 msgr2=0x55d271e3b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e3b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 -- 172.21.15.28:0/3800263999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 msgr2=0x55d271e4b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 -- 172.21.15.28:0/3800263999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d271e46d90 con 0x55d271e41ed0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4012cb0 tx=0x7f9dc401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=967642e2026b6135 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dc27fc640 1 -- 172.21.15.28:0/3800263999 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4003dd0 con 0x55d271e41ed0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dc27fc640 1 -- 172.21.15.28:0/3800263999 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc402bb90 con 0x55d271e41ed0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.634+0000 7f9dc27fc640 1 -- 172.21.15.28:0/3800263999 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4005020 con 0x55d271e41ed0 2024-03-22T02:19:03.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- 172.21.15.28:0/3800263999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 msgr2=0x55d271e52900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4012cb0 tx=0x7f9dc401b000 comp rx=0 tx=0).stop 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- 172.21.15.28:0/3800263999 shutdown_connections 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e47040 0x55d271e3b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e41ed0 0x55d271e52900 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 --2- 172.21.15.28:0/3800263999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e414e0 0x55d271e4b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- 172.21.15.28:0/3800263999 >> 172.21.15.28:0/3800263999 conn(0x55d271e3ca50 msgr2=0x55d271e4b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- 172.21.15.28:0/3800263999 shutdown_connections 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- 172.21.15.28:0/3800263999 wait complete. 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.635+0000 7f9de1e42740 1 -- start start 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 Processor -- start 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e47040 0x55d271e51730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81ab120 con 0x55d271e41ed0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81ac590 con 0x55d271e47040 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd81ab7f0 con 0x55d271e414e0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9de1e42740 10 client.0 _ll_register_callbacks cb 0x7f9dcc009f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddd7f9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e47040 0x55d271e51730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35424/0 (socket says 172.21.15.28:35424) 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9dd7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50800/0 (socket says 172.21.15.28:50800) 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 -- 172.21.15.28:0/2889950042 learned_addr learned my addr 172.21.15.28:0/2889950042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 -- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e47040 msgr2=0x55d271e51730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e47040 0x55d271e51730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 -- 172.21.15.28:0/2889950042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 msgr2=0x55d271e50720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9ddcff8640 1 -- 172.21.15.28:0/2889950042 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d271e51d20 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.636+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9ddcff8640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00cf60 tx=0x7f9dcc00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2fb78e6cb919a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc09c140 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7f9dcc0cf3d0 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dc402bb90 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9dcc0a4450 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 10 client.26899 fetch_fsmap learned FSMap version 29 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d271ebd850 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dc402bb90 con 0x55d271e414e0 2024-03-22T02:19:03.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7f9dcc09ae90 con 0x55d271e414e0 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 10 client.26899 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 10 client.26899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 20 client.26899 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271ebe220 0x55d271ec0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 4 client.26899 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.637+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d271dad8d0 con 0x55d271ebe220 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.638+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271ebe220 0x55d271ec0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.638+0000 7f9dd7fff640 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271ebe220 0x55d271ec0640 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dae9fa5502c2607 server_cookie=e336ed49c950eef2 in_seq=0 out_seq=0 2024-03-22T02:19:03.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.638+0000 7f9db7fff640 10 client.26899 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:03.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9db7fff640 1 -- 172.21.15.28:0/2889950042 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d271dad8d0 con 0x55d271ebe220 2024-03-22T02:19:03.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9db7fff640 10 client.26899 handle_command_reply: tid=0 2024-03-22T02:19:03.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 client.26899 shutdown 2024-03-22T02:19:03.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 msgr2=0x55d271e501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00cf60 tx=0x7f9dcc00c840 comp rx=0 tx=0).stop 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 shutdown_connections 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d271ebe220 0x55d271ec0640 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d271e47040 0x55d271e51730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d271e41ed0 0x55d271e50720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 --2- 172.21.15.28:0/2889950042 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d271e414e0 0x55d271e501e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 >> 172.21.15.28:0/2889950042 conn(0x55d271e3ca50 msgr2=0x55d271e3b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9db7fff640 0 client.26899 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 shutdown_connections 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 1 -- 172.21.15.28:0/2889950042 wait complete. 2024-03-22T02:19:03.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.885+0000 7f9de1e42740 20 client.26899 trim_cache size 0 max 16384 2024-03-22T02:19:03.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:19:03.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:19:03.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:19:03.888 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:19:03.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 msgr2=0x7f9dc80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 0x7f9dc80a5480 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80756b0 tx=0x7f9dc4005f90 comp rx=0 tx=0).stop 2024-03-22T02:19:03.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 msgr2=0x7f9dd8075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8075940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc00e9c0 tx=0x7f9dcc00ee80 comp rx=0 tx=0).stop 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 shutdown_connections 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9dc80a2fc0 0x7f9dc80a5480 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dd81037a0 0x7f9dd80744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9dd8102de0 0x7f9dd8073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 --2- 172.21.15.28:0/4216997056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9dd8101bf0 0x7f9dd8075940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 >> 172.21.15.28:0/4216997056 conn(0x7f9dd80fd160 msgr2=0x7f9dd8106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 shutdown_connections 2024-03-22T02:19:03.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:03.886+0000 7f9ddf25c640 1 -- 172.21.15.28:0/4216997056 wait complete. 2024-03-22T02:19:04.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 429828 11.89 MB/sec execute 106 sec latency 98.449 ms 2024-03-22T02:19:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:04 smithi120 ceph-mon[36803]: pgmap v522: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:04.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:04 smithi083 ceph-mon[36340]: pgmap v522: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:04 smithi028 ceph-mon[31916]: pgmap v522: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:05.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 431817 11.85 MB/sec execute 107 sec latency 59.464 ms 2024-03-22T02:19:06.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 433734 11.85 MB/sec execute 108 sec latency 27.214 ms 2024-03-22T02:19:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:06 smithi120 ceph-mon[36803]: pgmap v523: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 462 op/s 2024-03-22T02:19:06.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:06 smithi083 ceph-mon[36340]: pgmap v523: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 462 op/s 2024-03-22T02:19:06.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:06 smithi028 ceph-mon[31916]: pgmap v523: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.6 MiB/s wr, 462 op/s 2024-03-22T02:19:07.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 436685 11.91 MB/sec execute 109 sec latency 13.363 ms 2024-03-22T02:19:08.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 439311 11.92 MB/sec execute 110 sec latency 35.121 ms 2024-03-22T02:19:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:08 smithi120 ceph-mon[36803]: pgmap v524: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 333 op/s 2024-03-22T02:19:08.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:08 smithi083 ceph-mon[36340]: pgmap v524: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 333 op/s 2024-03-22T02:19:08.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:08 smithi028 ceph-mon[31916]: pgmap v524: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 333 op/s 2024-03-22T02:19:09.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 441043 11.89 MB/sec execute 111 sec latency 30.048 ms 2024-03-22T02:19:10.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 443353 11.92 MB/sec execute 112 sec latency 89.320 ms 2024-03-22T02:19:10.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:10 smithi083 ceph-mon[36340]: pgmap v525: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 431 op/s 2024-03-22T02:19:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:19:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:19:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:19:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:10 smithi028 ceph-mon[31916]: pgmap v525: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 431 op/s 2024-03-22T02:19:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:10.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:19:10.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:19:10.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:19:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:10 smithi120 ceph-mon[36803]: pgmap v525: 129 pgs: 129 active+clean; 45 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.2 MiB/s wr, 431 op/s 2024-03-22T02:19:10.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:19:10.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:19:10.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:19:10.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:19:11.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 444989 11.87 MB/sec execute 113 sec latency 37.604 ms 2024-03-22T02:19:12.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 447508 11.89 MB/sec execute 114 sec latency 16.184 ms 2024-03-22T02:19:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:12 smithi083 ceph-mon[36340]: pgmap v526: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-03-22T02:19:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:12 smithi028 ceph-mon[31916]: pgmap v526: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-03-22T02:19:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:12 smithi120 ceph-mon[36803]: pgmap v526: 129 pgs: 129 active+clean; 47 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 410 op/s 2024-03-22T02:19:13.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 450466 11.94 MB/sec execute 115 sec latency 27.977 ms 2024-03-22T02:19:14.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 452135 11.90 MB/sec execute 116 sec latency 37.690 ms 2024-03-22T02:19:14.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:14 smithi083 ceph-mon[36340]: pgmap v527: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2024-03-22T02:19:14.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:14 smithi028 ceph-mon[31916]: pgmap v527: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2024-03-22T02:19:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:14 smithi120 ceph-mon[36803]: pgmap v527: 129 pgs: 129 active+clean; 48 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 395 op/s 2024-03-22T02:19:15.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 454471 11.91 MB/sec execute 117 sec latency 60.647 ms 2024-03-22T02:19:16.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 457004 11.92 MB/sec execute 118 sec latency 30.846 ms 2024-03-22T02:19:16.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:16 smithi083 ceph-mon[36340]: pgmap v528: 129 pgs: 129 active+clean; 51 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-03-22T02:19:16.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:16 smithi028 ceph-mon[31916]: pgmap v528: 129 pgs: 129 active+clean; 51 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-03-22T02:19:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:16 smithi120 ceph-mon[36803]: pgmap v528: 129 pgs: 129 active+clean; 51 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.8 MiB/s wr, 479 op/s 2024-03-22T02:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 -- 172.21.15.28:0/945841368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0100ac0 msgr2=0x7ff2c0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/945841368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0100ee0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009900 tx=0x7ff2bc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 -- 172.21.15.28:0/945841368 shutdown_connections 2024-03-22T02:19:16.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/945841368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0102670 0x7ff2c0108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/945841368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0101cb0 0x7ff2c0102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/945841368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0100ee0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 -- 172.21.15.28:0/945841368 >> 172.21.15.28:0/945841368 conn(0x7ff2c00fc050 msgr2=0x7ff2c00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 -- 172.21.15.28:0/945841368 shutdown_connections 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 -- 172.21.15.28:0/945841368 wait complete. 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.977+0000 7ff2c7dac640 1 Processor -- start 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 -- start start 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0101cb0 0x7ff2c0198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c006a800 con 0x7ff2c0102670 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c006a680 con 0x7ff2c0100ac0 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c006a980 con 0x7ff2c0101cb0 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34014/0 (socket says 172.21.15.28:34014) 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 -- 172.21.15.28:0/1793158256 learned_addr learned my addr 172.21.15.28:0/1793158256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0101cb0 0x7ff2c0198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 -- 172.21.15.28:0/1793158256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0101cb0 msgr2=0x7ff2c0198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0101cb0 0x7ff2c0198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 -- 172.21.15.28:0/1793158256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0100ac0 msgr2=0x7ff2c0197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:16.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 -- 172.21.15.28:0/1793158256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2c019fa30 con 0x7ff2c0102670 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c6349640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800eb10 tx=0x7ff2b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2b6ffd640 1 -- 172.21.15.28:0/1793158256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8004b20 con 0x7ff2c0102670 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c019fd20 con 0x7ff2c0102670 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2b6ffd640 1 -- 172.21.15.28:0/1793158256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b8004cc0 con 0x7ff2c0102670 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.978+0000 7ff2b6ffd640 1 -- 172.21.15.28:0/1793158256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8010760 con 0x7ff2c0102670 2024-03-22T02:19:16.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.979+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2c01a0210 con 0x7ff2c0102670 2024-03-22T02:19:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.981+0000 7ff2b6ffd640 1 -- 172.21.15.28:0/1793158256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff2b8005020 con 0x7ff2c0102670 2024-03-22T02:19:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.981+0000 7ff2ca98d740 1 Processor -- start 2024-03-22T02:19:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.981+0000 7ff2ca98d740 1 -- start start 2024-03-22T02:19:16.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2b6ffd640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 0x7ff2ac0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad5769d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad5794d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad4d6040 con 0x55f6ad5714e0 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad47e9e0 con 0x55f6ad577040 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad4d1ff0 con 0x55f6ad571ed0 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 0x7ff2ac0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad5794d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34018/0 (socket says 172.21.15.28:34018) 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5b48640 1 -- 172.21.15.28:0/434305454 learned_addr learned my addr 172.21.15.28:0/434305454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5347640 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad5769d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2b6ffd640 1 -- 172.21.15.28:0/1793158256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff2b80cfea0 con 0x7ff2c0102670 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 -- 172.21.15.28:0/434305454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 msgr2=0x55f6ad5769d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad5769d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 -- 172.21.15.28:0/434305454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 msgr2=0x55f6ad571900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 -- 172.21.15.28:0/434305454 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6ad1ea210 con 0x55f6ad577040 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 0x7ff2ac0a5480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0199320 tx=0x7ff2b0009ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:16.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.982+0000 7ff2c6349640 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad5794d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80049d0 tx=0x7ff2b801bb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c964a8d0b9908ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2abfff640 1 -- 172.21.15.28:0/434305454 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b809bf90 con 0x55f6ad577040 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2abfff640 1 -- 172.21.15.28:0/434305454 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b809c8c0 con 0x55f6ad577040 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2abfff640 1 -- 172.21.15.28:0/434305454 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8064350 con 0x55f6ad577040 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 -- 172.21.15.28:0/434305454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 msgr2=0x55f6ad5794d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad5794d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff2b80049d0 tx=0x7ff2b801bb30 comp rx=0 tx=0).stop 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 -- 172.21.15.28:0/434305454 shutdown_connections 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad5794d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad5769d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/434305454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad571900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 -- 172.21.15.28:0/434305454 >> 172.21.15.28:0/434305454 conn(0x55f6ad56ca50 msgr2=0x55f6ad56ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 -- 172.21.15.28:0/434305454 shutdown_connections 2024-03-22T02:19:16.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.983+0000 7ff2ca98d740 1 -- 172.21.15.28:0/434305454 wait complete. 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 -- start start 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 Processor -- start 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad57b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad57baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad579ff0 con 0x55f6ad577040 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad491db0 con 0x55f6ad5714e0 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f6ad57a170 con 0x55f6ad571ed0 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2ca98d740 10 client.0 _ll_register_callbacks cb 0x55f6ad1e9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:16.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.984+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34024/0 (socket says 172.21.15.28:34024) 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad57b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 -- 172.21.15.28:0/4224311337 learned_addr learned my addr 172.21.15.28:0/4224311337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c5347640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad57baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 -- 172.21.15.28:0/4224311337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 msgr2=0x55f6ad57baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad57baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 -- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 msgr2=0x55f6ad57b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad57b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 -- 172.21.15.28:0/4224311337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6ad2b6620 con 0x55f6ad577040 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad57b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2c6349640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8005910 tx=0x7ff2b80a0310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a605d2f6fa68ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:16.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b809bb20 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff2b80642e0 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b809c8c0 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2b809bd00 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2ca98d740 10 client.17079 fetch_fsmap learned FSMap version 29 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6ad585e30 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.985+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b809c8c0 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ff2b8005310 con 0x55f6ad577040 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 10 client.17079 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 10 client.17079 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 20 client.17079 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:16.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad587180 0x55f6ad589570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 4 client.17079 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f6ad4e3580 con 0x55f6ad587180 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad587180 0x55f6ad589570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad587180 0x55f6ad589570 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f284f565a405aabd server_cookie=3028794ca1c5ea4f in_seq=0 out_seq=0 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.986+0000 7ff2a97fa640 10 client.17079 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.988+0000 7ff2a97fa640 1 -- 172.21.15.28:0/4224311337 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f6ad4e3580 con 0x55f6ad587180 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.988+0000 7ff2a97fa640 10 client.17079 handle_command_reply: tid=0 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.988+0000 7ff2ca98d740 1 client.17079 shutdown 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.988+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 msgr2=0x55f6ad582ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.988+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8005910 tx=0x7ff2b80a0310 comp rx=0 tx=0).stop 2024-03-22T02:19:16.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 shutdown_connections 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad587180 0x55f6ad589570 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad582ac0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad57baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/4224311337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad57b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2a97fa640 0 client.17079 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 >> 172.21.15.28:0/4224311337 conn(0x55f6ad56ca50 msgr2=0x55f6ad56bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 shutdown_connections 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 1 -- 172.21.15.28:0/4224311337 wait complete. 2024-03-22T02:19:16.998 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:16.989+0000 7ff2ca98d740 20 client.17079 trim_cache size 0 max 16384 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 Processor -- start 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 -- start start 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad573080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad582a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01ad220 con 0x55f6ad577040 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01aaaf0 con 0x55f6ad5714e0 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01a8940 con 0x55f6ad571ed0 2024-03-22T02:19:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34032/0 (socket says 172.21.15.28:34032) 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 -- 172.21.15.28:0/3246859466 learned_addr learned my addr 172.21.15.28:0/3246859466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad573080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c5347640 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad582a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 -- 172.21.15.28:0/3246859466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 msgr2=0x55f6ad582a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad582a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 -- 172.21.15.28:0/3246859466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 msgr2=0x55f6ad573080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad573080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.004+0000 7ff2c6349640 1 -- 172.21.15.28:0/3246859466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6ad57a2b0 con 0x55f6ad577040 2024-03-22T02:19:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2c6349640 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8000f80 tx=0x7ff2b8098520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6444314d60f25659 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ab7fe640 1 -- 172.21.15.28:0/3246859466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b809bb20 con 0x55f6ad577040 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ab7fe640 1 -- 172.21.15.28:0/3246859466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b809c8c0 con 0x55f6ad577040 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ab7fe640 1 -- 172.21.15.28:0/3246859466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b809bf90 con 0x55f6ad577040 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 -- 172.21.15.28:0/3246859466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 msgr2=0x55f6ad58dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8000f80 tx=0x7ff2b8098520 comp rx=0 tx=0).stop 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 -- 172.21.15.28:0/3246859466 shutdown_connections 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad577040 0x55f6ad58dcd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad582a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/3246859466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad5714e0 0x55f6ad573080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 -- 172.21.15.28:0/3246859466 >> 172.21.15.28:0/3246859466 conn(0x55f6ad56ca50 msgr2=0x55f6ad572970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 -- 172.21.15.28:0/3246859466 shutdown_connections 2024-03-22T02:19:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.005+0000 7ff2ca98d740 1 -- 172.21.15.28:0/3246859466 wait complete. 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 -- start start 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 Processor -- start 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad580b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad581ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01a8fb0 con 0x55f6ad5714e0 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01aa420 con 0x55f6ad577040 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2c01a9680 con 0x55f6ad571ed0 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2ca98d740 10 client.0 _ll_register_callbacks cb 0x7ff2b80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2c5b48640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34046/0 (socket says 172.21.15.28:34046) 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2c5b48640 1 -- 172.21.15.28:0/1850400055 learned_addr learned my addr 172.21.15.28:0/1850400055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2c6349640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad581ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.006+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad580b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 -- 172.21.15.28:0/1850400055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 msgr2=0x55f6ad580b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad580b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 -- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 msgr2=0x55f6ad581ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad581ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 -- 172.21.15.28:0/1850400055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6ad5820c0 con 0x55f6ad5714e0 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2c5b48640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc036040 tx=0x7ff2bc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed29f6d48773dc55 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc004040 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9274+0+0 (secure 0 0 0) 0x7ff2bc0041e0 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff2b809c8c0 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff2bc054050 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2ca98d740 10 client.17085 fetch_fsmap learned FSMap version 29 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f6ad5f1b50 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2b809c8c0 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 29) ==== 8346+0+0 (secure 0 0 0) 0x7ff2bc0054b0 con 0x55f6ad5714e0 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2ca98d740 10 client.17085 fetch_fsmap finished waiting for FSMap version 29 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.007+0000 7ff2ca98d740 10 client.17085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2ca98d740 20 client.17085 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad5f2550 0x55f6ad5f4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2ca98d740 4 client.17085 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f6ad576db0 con 0x55f6ad5f2550 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad5f2550 0x55f6ad5f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2c5347640 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad5f2550 0x55f6ad5f4970 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d419a4ee6ed3227e server_cookie=2e98da90300790b8 in_seq=0 out_seq=0 2024-03-22T02:19:17.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.008+0000 7ff2a97fa640 10 client.17085 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.220+0000 7ff2a97fa640 1 -- 172.21.15.28:0/1850400055 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f6ad576db0 con 0x55f6ad5f2550 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.220+0000 7ff2a97fa640 10 client.17085 handle_command_reply: tid=0 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.220+0000 7ff2ca98d740 1 client.17085 shutdown 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.220+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 msgr2=0x55f6ad5805f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.220+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc036040 tx=0x7ff2bc005fd0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 shutdown_connections 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f6ad5f2550 0x55f6ad5f4970 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f6ad577040 0x55f6ad581ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f6ad571ed0 0x55f6ad580b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 --2- 172.21.15.28:0/1850400055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6ad5714e0 0x55f6ad5805f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 >> 172.21.15.28:0/1850400055 conn(0x55f6ad56ca50 msgr2=0x55f6ad573200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2a97fa640 0 client.17085 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 shutdown_connections 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 1 -- 172.21.15.28:0/1850400055 wait complete. 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.221+0000 7ff2ca98d740 20 client.17085 trim_cache size 0 max 16384 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:19:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:19:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:19:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 msgr2=0x7ff2ac0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 0x7ff2ac0a5480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0199320 tx=0x7ff2b0009ab0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 msgr2=0x7ff2c019f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7ff2b800eb10 tx=0x7ff2b800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 shutdown_connections 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff2ac0a2fc0 0x7ff2ac0a5480 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2c0102670 0x7ff2c019f3c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff2c0101cb0 0x7ff2c0198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 --2- 172.21.15.28:0/1793158256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff2c0100ac0 0x7ff2c0197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 >> 172.21.15.28:0/1793158256 conn(0x7ff2c00fc050 msgr2=0x7ff2c00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 shutdown_connections 2024-03-22T02:19:17.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:17.222+0000 7ff2c7dac640 1 -- 172.21.15.28:0/1793158256 wait complete. 2024-03-22T02:19:17.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 458313 11.87 MB/sec execute 119 sec latency 101.975 ms 2024-03-22T02:19:18.015 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:19:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: starting mds.h at 2024-03-22T02:19:18.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 460712 12.01 MB/sec execute 120 sec latency 60.344 ms 2024-03-22T02:19:18.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:18 smithi083 ceph-mon[36340]: pgmap v529: 129 pgs: 129 active+clean; 50 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2024-03-22T02:19:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:18 smithi083 ceph-mon[36340]: daemon mds.h restarted 2024-03-22T02:19:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:18 smithi028 ceph-mon[31916]: pgmap v529: 129 pgs: 129 active+clean; 50 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2024-03-22T02:19:18.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:18 smithi028 ceph-mon[31916]: daemon mds.h restarted 2024-03-22T02:19:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:18 smithi120 ceph-mon[36803]: pgmap v529: 129 pgs: 129 active+clean; 50 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 366 op/s 2024-03-22T02:19:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:18 smithi120 ceph-mon[36803]: daemon mds.h restarted 2024-03-22T02:19:19.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 462730 12.01 MB/sec execute 121 sec latency 28.905 ms 2024-03-22T02:19:19.514 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:19 smithi083 ceph-mon[36340]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T02:19:19.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:19 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/2468455656,v1:172.21.15.83:6837/2468455656] up:boot 2024-03-22T02:19:19.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:19 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:19:19.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:19:19.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:19 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:19 smithi028 ceph-mon[31916]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T02:19:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:19 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/2468455656,v1:172.21.15.83:6837/2468455656] up:boot 2024-03-22T02:19:19.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:19 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:19:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:19:19.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:19 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:19 smithi120 ceph-mon[36803]: osdmap e95: 12 total, 12 up, 12 in 2024-03-22T02:19:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:19 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/2468455656,v1:172.21.15.83:6837/2468455656] up:boot 2024-03-22T02:19:19.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:19 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:19:19.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:19:19.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:19 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:19.987 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:19:19.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 -- 172.21.15.28:0/3811903059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818101c80 msgr2=0x7f9818102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 --2- 172.21.15.28:0/3811903059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818101c80 0x7f9818102100 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f980c009900 tx=0x7f980c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 -- 172.21.15.28:0/3811903059 shutdown_connections 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 --2- 172.21.15.28:0/3811903059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818102640 0x7f9818108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 --2- 172.21.15.28:0/3811903059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818101c80 0x7f9818102100 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 --2- 172.21.15.28:0/3811903059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818100a90 0x7f9818100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 -- 172.21.15.28:0/3811903059 >> 172.21.15.28:0/3811903059 conn(0x7f98180fc020 msgr2=0x7f98180fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 -- 172.21.15.28:0/3811903059 shutdown_connections 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.095+0000 7f981e393640 1 -- 172.21.15.28:0/3811903059 wait complete. 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 Processor -- start 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 -- start start 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818100a90 0x7f9818197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818101c80 0x7f9818198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f981810c710 con 0x7f9818102640 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f981810c590 con 0x7f9818100a90 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981e393640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f981810c890 con 0x7f9818101c80 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981c930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981c930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34074/0 (socket says 172.21.15.28:34074) 2024-03-22T02:19:20.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f981c930640 1 -- 172.21.15.28:0/2253797973 learned_addr learned my addr 172.21.15.28:0/2253797973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f9817fff640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818100a90 0x7f9818197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.096+0000 7f98177fe640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818101c80 0x7f9818198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 -- 172.21.15.28:0/2253797973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818101c80 msgr2=0x7f9818198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818101c80 0x7f9818198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 -- 172.21.15.28:0/2253797973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818100a90 msgr2=0x7f9818197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818100a90 0x7f9818197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 -- 172.21.15.28:0/2253797973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981819fae0 con 0x7f9818102640 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981c930640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f981000eb10 tx=0x7f981000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f98157fa640 1 -- 172.21.15.28:0/2253797973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9810004b20 con 0x7f9818102640 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f98157fa640 1 -- 172.21.15.28:0/2253797973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9810004cc0 con 0x7f9818102640 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981819fdd0 con 0x7f9818102640 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f98157fa640 1 -- 172.21.15.28:0/2253797973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9810010760 con 0x7f9818102640 2024-03-22T02:19:20.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.097+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98181a0240 con 0x7f9818102640 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.099+0000 7f98157fa640 1 -- 172.21.15.28:0/2253797973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9810005020 con 0x7f9818102640 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 Processor -- start 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 -- start start 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f98157fa640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 0x7f98000a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed11920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed14420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f98177fe640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 0x7f98000a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d9ec718c0 con 0x559d9ed0ce20 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559d9ec19c20 con 0x559d9ed11f90 2024-03-22T02:19:20.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9820f7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d9ec6d870 con 0x559d9ed0c430 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f98157fa640 1 -- 172.21.15.28:0/2253797973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f98100cfd50 con 0x7f9818102640 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9817fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f98177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed11920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed14420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9817fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44928/0 (socket says 172.21.15.28:44928) 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f9817fff640 1 -- 172.21.15.28:0/2473204060 learned_addr learned my addr 172.21.15.28:0/2473204060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 -- 172.21.15.28:0/2473204060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 msgr2=0x559d9ed0c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 -- 172.21.15.28:0/2473204060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 msgr2=0x559d9ed11920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed11920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f98177fe640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 0x7f98000a53e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f980c02c7b0 tx=0x7f980c02cb90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.100+0000 7f981c930640 1 -- 172.21.15.28:0/2473204060 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d9ec3e150 con 0x559d9ed11f90 2024-03-22T02:19:20.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.101+0000 7f981c930640 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed14420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f98100cd490 tx=0x7f981009bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac37652620d1a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.101+0000 7f97fe7fc640 1 -- 172.21.15.28:0/2473204060 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981009c070 con 0x559d9ed11f90 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.101+0000 7f97fe7fc640 1 -- 172.21.15.28:0/2473204060 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98100a01c0 con 0x559d9ed11f90 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.101+0000 7f97fe7fc640 1 -- 172.21.15.28:0/2473204060 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981009c2d0 con 0x559d9ed11f90 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- 172.21.15.28:0/2473204060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 msgr2=0x559d9ed14420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed14420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f98100cd490 tx=0x7f981009bf00 comp rx=0 tx=0).stop 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- 172.21.15.28:0/2473204060 shutdown_connections 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed14420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed11920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- 172.21.15.28:0/2473204060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- 172.21.15.28:0/2473204060 >> 172.21.15.28:0/2473204060 conn(0x559d9ed079a0 msgr2=0x559d9ed07da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- 172.21.15.28:0/2473204060 shutdown_connections 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- 172.21.15.28:0/2473204060 wait complete. 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- start start 2024-03-22T02:19:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 Processor -- start 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 0x559d9ed169a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed11f90 0x559d9ed1df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559d9ed14f40 con 0x559d9ed0ce20 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559d9ec2d130 con 0x559d9ed0c430 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.102+0000 7f9820f7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559d9ed150c0 con 0x559d9ed11f90 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9820f7f740 10 client.0 _ll_register_callbacks cb 0x559d9ec2cb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34092/0 (socket says 172.21.15.28:34092) 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9817fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 0x559d9ed169a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 -- 172.21.15.28:0/737764411 learned_addr learned my addr 172.21.15.28:0/737764411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f981c930640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed11f90 0x559d9ed1df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 -- 172.21.15.28:0/737764411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed11f90 msgr2=0x559d9ed1df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed11f90 0x559d9ed1df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 -- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 msgr2=0x559d9ed169a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 0x559d9ed169a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 -- 172.21.15.28:0/737764411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d9ea4f7d0 con 0x559d9ed0ce20 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9817fff640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 0x559d9ed169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f98177fe640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f980c02cfd0 tx=0x7f980c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61b3c0273985e64 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f97f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c0048c0 con 0x559d9ed0ce20 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f980c004a60 con 0x559d9ed0ce20 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98100a01c0 con 0x559d9ed0ce20 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f980c05a050 con 0x559d9ed0ce20 2024-03-22T02:19:20.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9820f7f740 10 client.17100 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.103+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d9e9853f0 con 0x559d9ed0ce20 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98100a01c0 con 0x559d9ed0ce20 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f980c04c460 con 0x559d9ed0ce20 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 10 client.17100 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 20 client.17100 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed22850 0x559d9ed24cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 4 client.17100 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d9ec7ee00 con 0x559d9ed22850 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9817fff640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed22850 0x559d9ed24cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f9817fff640 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed22850 0x559d9ed24cf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e8951dfb897bc23 server_cookie=334732494150dda5 in_seq=0 out_seq=0 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.104+0000 7f97f77fe640 10 client.17100 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f97f77fe640 1 -- 172.21.15.28:0/737764411 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559d9ec7ee00 con 0x559d9ed22850 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f97f77fe640 10 client.17100 handle_command_reply: tid=0 2024-03-22T02:19:20.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 client.17100 shutdown 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 msgr2=0x559d9ed16ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f980c02cfd0 tx=0x7f980c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 shutdown_connections 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed22850 0x559d9ed24cf0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed11f90 0x559d9ed1df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed16ee0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f97f77fe640 0 client.17100 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 --2- 172.21.15.28:0/737764411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0c430 0x559d9ed169a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 >> 172.21.15.28:0/737764411 conn(0x559d9ed079a0 msgr2=0x559d9ed06c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 shutdown_connections 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 1 -- 172.21.15.28:0/737764411 wait complete. 2024-03-22T02:19:20.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.105+0000 7f9820f7f740 20 client.17100 trim_cache size 0 max 16384 2024-03-22T02:19:20.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 Processor -- start 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 -- start start 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0ce20 0x559d9ed0e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98181ad230 con 0x559d9ed11f90 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f98181aab00 con 0x559d9ed0ce20 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.123+0000 7f9820f7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98181a8950 con 0x559d9ed0c430 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f9817fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34094/0 (socket says 172.21.15.28:34094) 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 -- 172.21.15.28:0/4107884471 learned_addr learned my addr 172.21.15.28:0/4107884471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f98177fe640 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0ce20 0x559d9ed0e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 -- 172.21.15.28:0/4107884471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 msgr2=0x559d9ed0e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 -- 172.21.15.28:0/4107884471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0ce20 msgr2=0x559d9ed0e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0ce20 0x559d9ed0e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 -- 172.21.15.28:0/4107884471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d9e9450f0 con 0x559d9ed11f90 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f981c930640 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f98100cd440 tx=0x7f9810009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f305eb524735725 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f97fdffb640 1 -- 172.21.15.28:0/4107884471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98100a01c0 con 0x559d9ed11f90 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f97fdffb640 1 -- 172.21.15.28:0/4107884471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98100b5170 con 0x559d9ed11f90 2024-03-22T02:19:20.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.124+0000 7f97fdffb640 1 -- 172.21.15.28:0/4107884471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98100bc0c0 con 0x559d9ed11f90 2024-03-22T02:19:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- 172.21.15.28:0/4107884471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 msgr2=0x559d9ed1dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f98100cd440 tx=0x7f9810009a70 comp rx=0 tx=0).stop 2024-03-22T02:19:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- 172.21.15.28:0/4107884471 shutdown_connections 2024-03-22T02:19:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed11f90 0x559d9ed1dee0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed0ce20 0x559d9ed0e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 --2- 172.21.15.28:0/4107884471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed0e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- 172.21.15.28:0/4107884471 >> 172.21.15.28:0/4107884471 conn(0x559d9ed079a0 msgr2=0x559d9ed15490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- 172.21.15.28:0/4107884471 shutdown_connections 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- 172.21.15.28:0/4107884471 wait complete. 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 -- start start 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.125+0000 7f9820f7f740 1 Processor -- start 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed1b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed1c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98181a8fc0 con 0x559d9ed0ce20 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f98181aa430 con 0x559d9ed11f90 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f98181a9690 con 0x559d9ed0c430 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 10 client.0 _ll_register_callbacks cb 0x7f980c004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34106/0 (socket says 172.21.15.28:34106) 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 -- 172.21.15.28:0/497167399 learned_addr learned my addr 172.21.15.28:0/497167399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f981c930640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed1c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9817fff640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed1b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 -- 172.21.15.28:0/497167399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 msgr2=0x559d9ed1b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed1b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 -- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 msgr2=0x559d9ed1c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed1c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 -- 172.21.15.28:0/497167399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d9ed1cc20 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f98177fe640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f980c009900 tx=0x7f980c02cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10da718ca741950 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f97f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f980c063060 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f980c04cb40 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98100b5170 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.126+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f980c06a050 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 10 client.17106 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559d9ed1df30 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98100b5170 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f980c003710 con 0x559d9ed0ce20 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 10 client.17106 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 20 client.17106 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed89110 0x559d9ed8b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 4 client.17106 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559d9ec79150 con 0x559d9ed89110 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9817fff640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed89110 0x559d9ed8b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f9817fff640 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed89110 0x559d9ed8b530 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26aed223a41193bd server_cookie=972429e231e5698d in_seq=0 out_seq=0 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.127+0000 7f97f77fe640 10 client.17106 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.128+0000 7f97f77fe640 1 -- 172.21.15.28:0/497167399 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559d9ec79150 con 0x559d9ed89110 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.128+0000 7f97f77fe640 10 client.17106 handle_command_reply: tid=0 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.128+0000 7f9820f7f740 1 client.17106 shutdown 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 msgr2=0x559d9ed1b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f980c009900 tx=0x7f980c02cca0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 shutdown_connections 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559d9ed89110 0x559d9ed8b530 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559d9ed11f90 0x559d9ed1c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d9ed0ce20 0x559d9ed1b620 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 --2- 172.21.15.28:0/497167399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d9ed0c430 0x559d9ed1b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f97f77fe640 0 client.17106 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 >> 172.21.15.28:0/497167399 conn(0x559d9ed079a0 msgr2=0x559d9ed16960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 shutdown_connections 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 1 -- 172.21.15.28:0/497167399 wait complete. 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f9820f7f740 20 client.17106 trim_cache size 0 max 16384 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:19:20.135 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 msgr2=0x7f98000a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 0x7f98000a53e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f980c02c7b0 tx=0x7f980c02cb90 comp rx=0 tx=0).stop 2024-03-22T02:19:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 msgr2=0x7f981819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.129+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f981000eb10 tx=0x7f981000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 shutdown_connections 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98000a2f20 0x7f98000a53e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9818102640 0x7f981819f470 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9818101c80 0x7f9818198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 --2- 172.21.15.28:0/2253797973 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9818100a90 0x7f9818197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 >> 172.21.15.28:0/2253797973 conn(0x7f98180fc020 msgr2=0x7f9818068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 shutdown_connections 2024-03-22T02:19:20.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.130+0000 7f981e393640 1 -- 172.21.15.28:0/2253797973 wait complete. 2024-03-22T02:19:20.140 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:19:20.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:19:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- 172.21.15.28:0/664806912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560100ab0 msgr2=0x7fd560100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 --2- 172.21.15.28:0/664806912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560100ab0 0x7fd560100ed0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd550009900 tx=0x7fd55002bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- 172.21.15.28:0/664806912 shutdown_connections 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 --2- 172.21.15.28:0/664806912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560102660 0x7fd560108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 --2- 172.21.15.28:0/664806912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560101ca0 0x7fd560102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 --2- 172.21.15.28:0/664806912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560100ab0 0x7fd560100ed0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- 172.21.15.28:0/664806912 >> 172.21.15.28:0/664806912 conn(0x7fd5600fc020 msgr2=0x7fd5600fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- 172.21.15.28:0/664806912 shutdown_connections 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- 172.21.15.28:0/664806912 wait complete. 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 Processor -- start 2024-03-22T02:19:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.249+0000 7fd5665e9640 1 -- start start 2024-03-22T02:19:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560100ab0 0x7fd560197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560101ca0 0x7fd560198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010c700 con 0x7fd560102660 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010c580 con 0x7fd560100ab0 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd56010c880 con 0x7fd560101ca0 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34114/0 (socket says 172.21.15.28:34114) 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 -- 172.21.15.28:0/394363433 learned_addr learned my addr 172.21.15.28:0/394363433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd55f7fe640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560101ca0 0x7fd560198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd55ffff640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560100ab0 0x7fd560197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 -- 172.21.15.28:0/394363433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560101ca0 msgr2=0x7fd560198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560101ca0 0x7fd560198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 -- 172.21.15.28:0/394363433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560100ab0 msgr2=0x7fd560197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560100ab0 0x7fd560197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 -- 172.21.15.28:0/394363433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56019fb40 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd564b86640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd55800efd0 tx=0x7fd55800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558003b70 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd558003d10 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd56019fe30 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580109f0 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.250+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5601a02a0 con 0x7fd560102660 2024-03-22T02:19:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.252+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5601a0560 con 0x7fd560102660 2024-03-22T02:19:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.253+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd558010b90 con 0x7fd560102660 2024-03-22T02:19:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.254+0000 7fd55d7fa640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 0x7fd5480a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.254+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fd5580cffa0 con 0x7fd560102660 2024-03-22T02:19:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.254+0000 7fd55ffff640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 0x7fd5480a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.254+0000 7fd55ffff640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 0x7fd5480a53e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd550000c00 tx=0x7fd550005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.258+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd55809c9c0 con 0x7fd560102660 2024-03-22T02:19:20.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 465429 12.05 MB/sec execute 122 sec latency 39.841 ms 2024-03-22T02:19:20.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.495+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd5601a0730 con 0x7fd560102660 2024-03-22T02:19:20.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.496+0000 7fd55d7fa640 1 -- 172.21.15.28:0/394363433 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11866 (secure 0 0 0) 0x7fd55809c2c0 con 0x7fd560102660 2024-03-22T02:19:20.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:19:20.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:19:18.264292+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26924":{"gid":26924,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2468455656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2468455656},{"type":"v1","addr":"172.21.15.83:6837","nonce":2468455656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:19:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 31 2024-03-22T02:19:20.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 msgr2=0x7fd5480a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 0x7fd5480a53e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd550000c00 tx=0x7fd550005fd0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 msgr2=0x7fd56019f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd55800efd0 tx=0x7fd55800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 shutdown_connections 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd5480a2f20 0x7fd5480a53e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd560102660 0x7fd56019f4d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd560101ca0 0x7fd560198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 --2- 172.21.15.28:0/394363433 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd560100ab0 0x7fd560197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 >> 172.21.15.28:0/394363433 conn(0x7fd5600fc020 msgr2=0x7fd5600fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 shutdown_connections 2024-03-22T02:19:20.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.498+0000 7fd5665e9640 1 -- 172.21.15.28:0/394363433 wait complete. 2024-03-22T02:19:20.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:19:20.533 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:20 smithi028 ceph-mon[31916]: pgmap v531: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/448448173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68100a90 msgr2=0x7f4c68100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/448448173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68100a90 0x7f4c68100eb0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c5802bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/448448173 shutdown_connections 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/448448173 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 0x7f4c68108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/448448173 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68101c80 0x7f4c68102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/448448173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68100a90 0x7f4c68100eb0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.615+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/448448173 >> 172.21.15.28:0/448448173 conn(0x7f4c680fc020 msgr2=0x7f4c680fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/448448173 shutdown_connections 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/448448173 wait complete. 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 Processor -- start 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- start start 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68100a90 0x7f4c68197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 0x7f4c6819f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810c700 con 0x7f4c68101c80 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810c580 con 0x7f4c68102640 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.616+0000 7f4c6d9bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6810c880 con 0x7f4c68100a90 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34130/0 (socket says 172.21.15.28:34130) 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 -- 172.21.15.28:0/4108907527 learned_addr learned my addr 172.21.15.28:0/4108907527 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68100a90 0x7f4c68197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c677fe640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 0x7f4c6819f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 -- 172.21.15.28:0/4108907527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68100a90 msgr2=0x7f4c68197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68100a90 0x7f4c68197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 -- 172.21.15.28:0/4108907527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 msgr2=0x7f4c6819f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 0x7f4c6819f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 -- 172.21.15.28:0/4108907527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6819fb40 con 0x7f4c68101c80 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c667fc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e9f0 tx=0x7f4c5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c6c9ba640 1 -- 172.21.15.28:0/4108907527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00cd80 con 0x7f4c68101c80 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c6c9ba640 1 -- 172.21.15.28:0/4108907527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c005500 con 0x7f4c68101c80 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6819fe30 con 0x7f4c68101c80 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c6c9ba640 1 -- 172.21.15.28:0/4108907527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c010690 con 0x7f4c68101c80 2024-03-22T02:19:20.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.617+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c681a02a0 con 0x7f4c68101c80 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.619+0000 7f4c6c9ba640 1 -- 172.21.15.28:0/4108907527 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4c5c0046c0 con 0x7f4c68101c80 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 Processor -- start 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 -- start start 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c6c9ba640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 0x7f4c500a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b248fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2492810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2b23ef8d0 con 0x55f2b248a820 2024-03-22T02:19:20.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f2b2397e00 con 0x55f2b248b210 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c705a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2b23eb880 con 0x55f2b2490380 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c6c9ba640 1 -- 172.21.15.28:0/4108907527 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4c5c0cfc20 con 0x7f4c68101c80 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c677fe640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 0x7f4c500a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34136/0 (socket says 172.21.15.28:34136) 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3241536368 learned_addr learned my addr 172.21.15.28:0/3241536368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c677fe640 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2492810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c667fc640 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b248fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3241536368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 msgr2=0x55f2b2492810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.620+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2492810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3241536368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 msgr2=0x55f2b248fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b248fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3241536368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b21033f0 con 0x55f2b248a820 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c677fe640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 0x7f4c500a53e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005fd0 tx=0x7f4c60009e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c58009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e5870bd476a121 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c4dffb640 1 -- 172.21.15.28:0/3241536368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5802cb80 con 0x55f2b248a820 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c4dffb640 1 -- 172.21.15.28:0/3241536368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c58005040 con 0x55f2b248a820 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c4dffb640 1 -- 172.21.15.28:0/3241536368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c58004920 con 0x55f2b248a820 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c705a4740 1 -- 172.21.15.28:0/3241536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 msgr2=0x55f2b248ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.621+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009900 tx=0x7f4c58009560 comp rx=0 tx=0).stop 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- 172.21.15.28:0/3241536368 shutdown_connections 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2492810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b248fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3241536368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b248ac40 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- 172.21.15.28:0/3241536368 >> 172.21.15.28:0/3241536368 conn(0x55f2b2485d90 msgr2=0x55f2b2486190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- 172.21.15.28:0/3241536368 shutdown_connections 2024-03-22T02:19:20.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- 172.21.15.28:0/3241536368 wait complete. 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- start start 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 Processor -- start 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b248a820 0x55f2b2494ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b2490380 0x55f2b249c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2b24932d0 con 0x55f2b248b210 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f2b23ab300 con 0x55f2b2490380 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2b2493450 con 0x55f2b248a820 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c705a4740 10 client.0 _ll_register_callbacks cb 0x55f2b23e6e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b248a820 0x55f2b2494ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c677fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b2490380 0x55f2b249c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34138/0 (socket says 172.21.15.28:34138) 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.622+0000 7f4c667fc640 1 -- 172.21.15.28:0/2659330257 learned_addr learned my addr 172.21.15.28:0/2659330257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 -- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b248a820 msgr2=0x55f2b2494ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b248a820 0x55f2b2494ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 -- 172.21.15.28:0/2659330257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b2490380 msgr2=0x55f2b249c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b2490380 0x55f2b249c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 -- 172.21.15.28:0/2659330257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b21cd7d0 con 0x55f2b248b210 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e9c0 tx=0x7f4c5c09c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318e286711c518e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0b4070 con 0x55f2b248b210 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4c5c0a4660 con 0x55f2b248b210 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c58005040 con 0x55f2b248b210 2024-03-22T02:19:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4c5c0ab050 con 0x55f2b248b210 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c705a4740 10 client.17124 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2b249f660 con 0x55f2b248b210 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.623+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c58005040 con 0x55f2b248b210 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4c5c09b330 con 0x55f2b248b210 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 10 client.17124 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 10 client.17124 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 20 client.17124 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b24a0d30 0x55f2b24a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 4 client.17124 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2b23fce10 con 0x55f2b24a0d30 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b24a0d30 0x55f2b24a31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c46ffd640 10 client.17124 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.624+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b24a0d30 0x55f2b24a31c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8094b388c2dd0e64 server_cookie=a93cfba7a06a2fb4 in_seq=0 out_seq=0 2024-03-22T02:19:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c46ffd640 1 -- 172.21.15.28:0/2659330257 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2b23fce10 con 0x55f2b24a0d30 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c46ffd640 10 client.17124 handle_command_reply: tid=0 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 client.17124 shutdown 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 msgr2=0x55f2b24951e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e9c0 tx=0x7f4c5c09c6a0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 shutdown_connections 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b24a0d30 0x55f2b24a31c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b2490380 0x55f2b249c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b24951e0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2659330257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b248a820 0x55f2b2494ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 >> 172.21.15.28:0/2659330257 conn(0x55f2b2485d90 msgr2=0x55f2b2484fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c46ffd640 0 client.17124 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 shutdown_connections 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 1 -- 172.21.15.28:0/2659330257 wait complete. 2024-03-22T02:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.625+0000 7f4c705a4740 20 client.17124 trim_cache size 0 max 16384 2024-03-22T02:19:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 Processor -- start 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 -- start start 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248a820 0x55f2b2480e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2484740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681ad270 con 0x55f2b248b210 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681aab40 con 0x55f2b248a820 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c705a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681a8970 con 0x55f2b2490380 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248a820 0x55f2b2480e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34154/0 (socket says 172.21.15.28:34154) 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 -- 172.21.15.28:0/2474491451 learned_addr learned my addr 172.21.15.28:0/2474491451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c677fe640 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2484740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 -- 172.21.15.28:0/2474491451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 msgr2=0x55f2b2484740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2484740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 -- 172.21.15.28:0/2474491451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248a820 msgr2=0x55f2b2480e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248a820 0x55f2b2480e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 -- 172.21.15.28:0/2474491451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2b248b5f0 con 0x55f2b248b210 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.639+0000 7f4c667fc640 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c09a190 tx=0x7f4c5c09a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e123ee81f772ef01 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c4d7fa640 1 -- 172.21.15.28:0/2474491451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0b4070 con 0x55f2b248b210 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c4d7fa640 1 -- 172.21.15.28:0/2474491451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c0a48f0 con 0x55f2b248b210 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c4d7fa640 1 -- 172.21.15.28:0/2474491451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c0a46c0 con 0x55f2b248b210 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 -- 172.21.15.28:0/2474491451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 msgr2=0x55f2b249c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c09a190 tx=0x7f4c5c09a3f0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 -- 172.21.15.28:0/2474491451 shutdown_connections 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b2484740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248b210 0x55f2b249c1e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 --2- 172.21.15.28:0/2474491451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248a820 0x55f2b2480e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 -- 172.21.15.28:0/2474491451 >> 172.21.15.28:0/2474491451 conn(0x55f2b2485d90 msgr2=0x55f2b248c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 -- 172.21.15.28:0/2474491451 shutdown_connections 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.640+0000 7f4c705a4740 1 -- 172.21.15.28:0/2474491451 wait complete. 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 -- start start 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 Processor -- start 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b2499ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b249aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681a9000 con 0x55f2b248a820 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681aa470 con 0x55f2b248b210 2024-03-22T02:19:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4c681a96d0 con 0x55f2b2490380 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c705a4740 10 client.0 _ll_register_callbacks cb 0x7f4c5c09bcb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34156/0 (socket says 172.21.15.28:34156) 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3892256177 learned_addr learned my addr 172.21.15.28:0/3892256177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c667fc640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b2499ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c677fe640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b249aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 msgr2=0x55f2b249aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b249aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3892256177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 msgr2=0x55f2b2499ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b2499ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c66ffd640 1 -- 172.21.15.28:0/3892256177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2b249b490 con 0x55f2b248a820 2024-03-22T02:19:20.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.641+0000 7f4c677fe640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b249aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c66ffd640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4c5802cfd0 tx=0x7f4c580058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11de40d1e56d58e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c58054020 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4c58005040 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c5c0a48f0 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4c5805f050 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 10 client.17130 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f2b25070c0 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c0a48f0 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4c58050050 con 0x55f2b248a820 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 10 client.17130 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 10 client.17130 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 20 client.17130 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b2507690 0x55f2b2509a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 4 client.17130 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f2b23f7160 con 0x55f2b2507690 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.642+0000 7f4c667fc640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b2507690 0x55f2b2509a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c667fc640 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b2507690 0x55f2b2509a80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36a369106a008b9d server_cookie=6541a4d6dc55199c in_seq=0 out_seq=0 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c46ffd640 10 client.17130 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c46ffd640 1 -- 172.21.15.28:0/3892256177 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f2b23f7160 con 0x55f2b2507690 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c46ffd640 10 client.17130 handle_command_reply: tid=0 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 client.17130 shutdown 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 msgr2=0x55f2b24999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f4c5802cfd0 tx=0x7f4c580058e0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 shutdown_connections 2024-03-22T02:19:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2b2490380 0x55f2b249aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f2b2507690 0x55f2b2509a80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2b248b210 0x55f2b2499ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 --2- 172.21.15.28:0/3892256177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2b248a820 0x55f2b24999a0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c46ffd640 0 client.17130 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 >> 172.21.15.28:0/3892256177 conn(0x55f2b2485d90 msgr2=0x55f2b2488ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 shutdown_connections 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 1 -- 172.21.15.28:0/3892256177 wait complete. 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.643+0000 7f4c705a4740 20 client.17130 trim_cache size 0 max 16384 2024-03-22T02:19:20.651 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 msgr2=0x7f4c500a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 0x7f4c500a53e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005fd0 tx=0x7f4c60009e60 comp rx=0 tx=0).stop 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 msgr2=0x7f4c68198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e9f0 tx=0x7f4c5c00edf0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 shutdown_connections 2024-03-22T02:19:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4c500a2f20 0x7f4c500a53e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4c68102640 0x7f4c6819f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c68101c80 0x7f4c68198460 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 --2- 172.21.15.28:0/4108907527 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4c68100a90 0x7f4c68197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 >> 172.21.15.28:0/4108907527 conn(0x7f4c680fc020 msgr2=0x7f4c680fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 shutdown_connections 2024-03-22T02:19:20.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.644+0000 7f4c6d9bc640 1 -- 172.21.15.28:0/4108907527 wait complete. 2024-03-22T02:19:20.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:19:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:20 smithi120 ceph-mon[36803]: pgmap v531: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-22T02:19:20.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:20 smithi083 ceph-mon[36340]: pgmap v531: 129 pgs: 129 active+clean; 45 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 370 op/s 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 -- 172.21.15.28:0/1200671012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0101bc0 msgr2=0x7f45d0101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/1200671012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0101bc0 0x7f45d0101fe0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c402bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 -- 172.21.15.28:0/1200671012 shutdown_connections 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/1200671012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0103770 0x7f45d010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/1200671012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 0x7f45d0103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/1200671012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0101bc0 0x7f45d0101fe0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 -- 172.21.15.28:0/1200671012 >> 172.21.15.28:0/1200671012 conn(0x7f45d00fd130 msgr2=0x7f45d00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 -- 172.21.15.28:0/1200671012 shutdown_connections 2024-03-22T02:19:20.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.803+0000 7f45d6ba1640 1 -- 172.21.15.28:0/1200671012 wait complete. 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 Processor -- start 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 -- start start 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0101bc0 0x7f45d0197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 0x7f45d0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d810 con 0x7f45d0103770 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d690 con 0x7f45d0102db0 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d6ba1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d010d990 con 0x7f45d0101bc0 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d513e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d493d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0101bc0 0x7f45d0197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d513e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34184/0 (socket says 172.21.15.28:34184) 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45d513e640 1 -- 172.21.15.28:0/2302207077 learned_addr learned my addr 172.21.15.28:0/2302207077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.804+0000 7f45cffff640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 0x7f45d0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 -- 172.21.15.28:0/2302207077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0101bc0 msgr2=0x7f45d0197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0101bc0 0x7f45d0197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 -- 172.21.15.28:0/2302207077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 msgr2=0x7f45d0198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 0x7f45d0198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 -- 172.21.15.28:0/2302207077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d019fb30 con 0x7f45d0103770 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d513e640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009fd0 tx=0x7f45c800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45cdffb640 1 -- 172.21.15.28:0/2302207077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c8017070 con 0x7f45d0103770 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45cdffb640 1 -- 172.21.15.28:0/2302207077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c8003e50 con 0x7f45d0103770 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45cdffb640 1 -- 172.21.15.28:0/2302207077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80106c0 con 0x7f45d0103770 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d019fe20 con 0x7f45d0103770 2024-03-22T02:19:20.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.805+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d01a02e0 con 0x7f45d0103770 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45cdffb640 1 -- 172.21.15.28:0/2302207077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f45c8003970 con 0x7f45d0103770 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 Processor -- start 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 -- start start 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66a60c8d0 con 0x55b66a6ac620 2024-03-22T02:19:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b66a5b4e00 con 0x55b66a6b1790 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.808+0000 7f45d9789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b66a608880 con 0x55b66a6abc30 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45cdffb640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 0x7f45c00a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34186/0 (socket says 172.21.15.28:34186) 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 -- 172.21.15.28:0/2988198018 learned_addr learned my addr 172.21.15.28:0/2988198018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45cffff640 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45070/0 (socket says 172.21.15.28:45070) 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45cffff640 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45cdffb640 1 -- 172.21.15.28:0/2302207077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f45c80cf3f0 con 0x7f45d0103770 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 -- 172.21.15.28:0/2988198018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 msgr2=0x55b66a6ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 -- 172.21.15.28:0/2988198018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 msgr2=0x55b66a6b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6b3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 -- 172.21.15.28:0/2988198018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66a3203f0 con 0x55b66a6ac620 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.809+0000 7f45d493d640 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd8b4a299160fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.811+0000 7f45d513e640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 0x7f45c00a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.811+0000 7f45baffd640 1 -- 172.21.15.28:0/2988198018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c402cb80 con 0x55b66a6ac620 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.811+0000 7f45baffd640 1 -- 172.21.15.28:0/2988198018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c4005040 con 0x55b66a6ac620 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.811+0000 7f45baffd640 1 -- 172.21.15.28:0/2988198018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4004900 con 0x55b66a6ac620 2024-03-22T02:19:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.812+0000 7f45d513e640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 0x7f45c00a5460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f45c800dfd0 tx=0x7f45c8004020 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.813+0000 7f45d9789740 1 -- 172.21.15.28:0/2988198018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 msgr2=0x55b66a6b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.813+0000 7f45d9789740 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c40095e0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 -- 172.21.15.28:0/2988198018 shutdown_connections 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6b3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6ac620 0x55b66a6b1120 secure :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f45c4009900 tx=0x7f45c40095e0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 --2- 172.21.15.28:0/2988198018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 -- 172.21.15.28:0/2988198018 >> 172.21.15.28:0/2988198018 conn(0x55b66a6a2d90 msgr2=0x55b66a6a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 -- 172.21.15.28:0/2988198018 shutdown_connections 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.815+0000 7f45d9789740 1 -- 172.21.15.28:0/2988198018 wait complete. 2024-03-22T02:19:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.816+0000 7f45d9789740 1 -- start start 2024-03-22T02:19:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.816+0000 7f45d9789740 1 Processor -- start 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.816+0000 7f45d9789740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6afdc0 0x55b66a6b0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b66a6b46e0 con 0x55b66a6abc30 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b66a5c8300 con 0x55b66a6b1790 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b66a6b4860 con 0x55b66a6afdc0 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45d9789740 10 client.0 _ll_register_callbacks cb 0x55b66a603e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34202/0 (socket says 172.21.15.28:34202) 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45cffff640 1 -- 172.21.15.28:0/4046315422 learned_addr learned my addr 172.21.15.28:0/4046315422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.817+0000 7f45cffff640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6afdc0 0x55b66a6b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.818+0000 7f45cffff640 1 -- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6afdc0 msgr2=0x55b66a6b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.818+0000 7f45cffff640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6afdc0 0x55b66a6b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.818+0000 7f45cffff640 1 -- 172.21.15.28:0/4046315422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 msgr2=0x55b66a6aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.818+0000 7f45cffff640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.818+0000 7f45cffff640 1 -- 172.21.15.28:0/4046315422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66a3ea7d0 con 0x55b66a6abc30 2024-03-22T02:19:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.819+0000 7f45cffff640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f45bc010e10 tx=0x7f45bc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3add5e284d728e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.819+0000 7f45affff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.819+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc004030 con 0x55b66a6abc30 2024-03-22T02:19:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.819+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45c4005040 con 0x55b66a6abc30 2024-03-22T02:19:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.819+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f45bc0041d0 con 0x55b66a6abc30 2024-03-22T02:19:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.820+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45bc029050 con 0x55b66a6abc30 2024-03-22T02:19:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.820+0000 7f45d9789740 10 client.17142 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.820+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b66a6b6bf0 con 0x55b66a6abc30 2024-03-22T02:19:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.820+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45c4005040 con 0x55b66a6abc30 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.820+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f45bc020040 con 0x55b66a6abc30 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 10 client.17142 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 10 client.17142 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 20 client.17142 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a6b82a0 0x55b66a6ba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 4 client.17142 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.821+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b66a619e10 con 0x55b66a6b82a0 2024-03-22T02:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.822+0000 7f45d493d640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a6b82a0 0x55b66a6ba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.823+0000 7f45d493d640 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a6b82a0 0x55b66a6ba760 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=63de6649aca2dc6e server_cookie=9a058e88dd8db822 in_seq=0 out_seq=0 2024-03-22T02:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.823+0000 7f45affff640 10 client.17142 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.824+0000 7f45affff640 1 -- 172.21.15.28:0/4046315422 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b66a619e10 con 0x55b66a6b82a0 2024-03-22T02:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.824+0000 7f45affff640 10 client.17142 handle_command_reply: tid=0 2024-03-22T02:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.824+0000 7f45d9789740 1 client.17142 shutdown 2024-03-22T02:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 msgr2=0x55b66a6ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f45bc010e10 tx=0x7f45bc005f90 comp rx=0 tx=0).stop 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 shutdown_connections 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a6b82a0 0x55b66a6ba760 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6afdc0 0x55b66a6b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 --2- 172.21.15.28:0/4046315422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6ae870 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 >> 172.21.15.28:0/4046315422 conn(0x55b66a6a2d90 msgr2=0x55b66a6a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45affff640 0 client.17142 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 shutdown_connections 2024-03-22T02:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.826+0000 7f45d9789740 1 -- 172.21.15.28:0/4046315422 wait complete. 2024-03-22T02:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.827+0000 7f45d9789740 20 client.17142 trim_cache size 0 max 16384 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 Processor -- start 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 -- start start 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01ab100 con 0x55b66a6afdc0 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01a89d0 con 0x55b66a6b1790 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.855+0000 7f45d9789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01a6820 con 0x55b66a6abc30 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45d493d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45d493d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34208/0 (socket says 172.21.15.28:34208) 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45d493d640 1 -- 172.21.15.28:0/162731958 learned_addr learned my addr 172.21.15.28:0/162731958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 -- 172.21.15.28:0/162731958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 msgr2=0x55b66a6a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 -- 172.21.15.28:0/162731958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 msgr2=0x55b66a6a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45cffff640 1 -- 172.21.15.28:0/162731958 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b66a6ab440 con 0x55b66a6abc30 2024-03-22T02:19:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.856+0000 7f45d493d640 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.857+0000 7f45cffff640 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6aeb90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fd0 tx=0x7f45bc005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85db4f456af7cf70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.857+0000 7f45ba7fc640 1 -- 172.21.15.28:0/162731958 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc010b70 con 0x55b66a6abc30 2024-03-22T02:19:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.858+0000 7f45ba7fc640 1 -- 172.21.15.28:0/162731958 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45bc004f50 con 0x55b66a6abc30 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.858+0000 7f45ba7fc640 1 -- 172.21.15.28:0/162731958 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc003850 con 0x55b66a6abc30 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 -- 172.21.15.28:0/162731958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 msgr2=0x55b66a6aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6aeb90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f45bc009fd0 tx=0x7f45bc005850 comp rx=0 tx=0).stop 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 -- 172.21.15.28:0/162731958 shutdown_connections 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6b1790 0x55b66a6a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6afdc0 0x55b66a6a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 --2- 172.21.15.28:0/162731958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6abc30 0x55b66a6aeb90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 -- 172.21.15.28:0/162731958 >> 172.21.15.28:0/162731958 conn(0x55b66a6a2d90 msgr2=0x55b66a6ae480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 -- 172.21.15.28:0/162731958 shutdown_connections 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.859+0000 7f45d9789740 1 -- 172.21.15.28:0/162731958 wait complete. 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 -- start start 2024-03-22T02:19:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 Processor -- start 2024-03-22T02:19:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6bd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6afdc0 0x55b66a6bd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 0x55b66a744b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01a6e90 con 0x55b66a6abc30 2024-03-22T02:19:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01a8300 con 0x55b66a6afdc0 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45d01a7560 con 0x55b66a6b1790 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.860+0000 7f45d9789740 10 client.0 _ll_register_callbacks cb 0x7f45bc009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.861+0000 7f45cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6bd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.861+0000 7f45cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6bd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34214/0 (socket says 172.21.15.28:34214) 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.861+0000 7f45cffff640 1 -- 172.21.15.28:0/3866358108 learned_addr learned my addr 172.21.15.28:0/3866358108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.861+0000 7f45cffff640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 0x55b66a744b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 -- 172.21.15.28:0/3866358108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6afdc0 msgr2=0x55b66a6bd650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6afdc0 0x55b66a6bd650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 -- 172.21.15.28:0/3866358108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 msgr2=0x55b66a6bd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6bd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 -- 172.21.15.28:0/3866358108 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b66a6c0720 con 0x55b66a6b1790 2024-03-22T02:19:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45cffff640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 0x55b66a744b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0058c0 tx=0x7f45bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f2cd02291778bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:20.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.862+0000 7f45affff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.863+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45bc004050 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.863+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45bc004f50 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.863+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f45bc0041f0 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.864+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f45bc040050 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.864+0000 7f45d9789740 10 client.26983 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.864+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55b66a7460e0 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.864+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45bc004f50 con 0x55b66a6b1790 2024-03-22T02:19:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.864+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f45bc039040 con 0x55b66a6b1790 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 10 client.26983 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 10 client.26983 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 20 client.26983 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a746b90 0x55b66a748fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 4 client.26983 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b66a614160 con 0x55b66a746b90 2024-03-22T02:19:20.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.865+0000 7f45d493d640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a746b90 0x55b66a748fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.867+0000 7f45d493d640 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a746b90 0x55b66a748fb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f60a4bcae3c69831 server_cookie=154ce9f083f20885 in_seq=0 out_seq=0 2024-03-22T02:19:20.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.867+0000 7f45affff640 10 client.26983 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.868+0000 7f45affff640 1 -- 172.21.15.28:0/3866358108 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b66a614160 con 0x55b66a746b90 2024-03-22T02:19:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.868+0000 7f45affff640 10 client.26983 handle_command_reply: tid=0 2024-03-22T02:19:20.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.869+0000 7f45d9789740 1 client.26983 shutdown 2024-03-22T02:19:20.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 msgr2=0x55b66a744b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 0x55b66a744b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0058c0 tx=0x7f45bc0046c0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 shutdown_connections 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55b66a746b90 0x55b66a748fb0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b66a6b1790 0x55b66a744b20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b66a6afdc0 0x55b66a6bd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 --2- 172.21.15.28:0/3866358108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b66a6abc30 0x55b66a6bd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.871+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 >> 172.21.15.28:0/3866358108 conn(0x55b66a6a2d90 msgr2=0x55b66a6a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.872+0000 7f45affff640 0 client.26983 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:20.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.872+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 shutdown_connections 2024-03-22T02:19:20.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.872+0000 7f45d9789740 1 -- 172.21.15.28:0/3866358108 wait complete. 2024-03-22T02:19:20.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.872+0000 7f45d9789740 20 client.26983 trim_cache size 0 max 16384 2024-03-22T02:19:20.873 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:20.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.873+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 msgr2=0x7f45c00a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.873+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 0x7f45c00a5460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f45c800dfd0 tx=0x7f45c8004020 comp rx=0 tx=0).stop 2024-03-22T02:19:20.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.873+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 msgr2=0x7f45d019f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:20.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.873+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009fd0 tx=0x7f45c800b260 comp rx=0 tx=0).stop 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 shutdown_connections 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f45c00a2fa0 0x7f45c00a5460 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45d0103770 0x7f45d019f4c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f45d0102db0 0x7f45d0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 --2- 172.21.15.28:0/2302207077 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45d0101bc0 0x7f45d0197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:20.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.874+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 >> 172.21.15.28:0/2302207077 conn(0x7f45d00fd130 msgr2=0x7f45d0106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:20.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.875+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 shutdown_connections 2024-03-22T02:19:20.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:20.875+0000 7f45d6ba1640 1 -- 172.21.15.28:0/2302207077 wait complete. 2024-03-22T02:19:20.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- 172.21.15.28:0/3353865057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8101c80 msgr2=0x7f4ef8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- 172.21.15.28:0/3353865057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8101c80 0x7f4ef8102100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009900 tx=0x7f4ee802bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- 172.21.15.28:0/3353865057 shutdown_connections 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- 172.21.15.28:0/3353865057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- 172.21.15.28:0/3353865057 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8101c80 0x7f4ef8102100 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- 172.21.15.28:0/3353865057 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8100a90 0x7f4ef8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- 172.21.15.28:0/3353865057 >> 172.21.15.28:0/3353865057 conn(0x7f4ef80fc020 msgr2=0x7f4ef80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- 172.21.15.28:0/3353865057 shutdown_connections 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- 172.21.15.28:0/3353865057 wait complete. 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 Processor -- start 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 -- start start 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 0x7f4ef8104a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.012+0000 7f4effc4e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4effc4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4effc4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef810c740 con 0x7f4ef8102640 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4effc4e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef810c5c0 con 0x7f4ef8100a90 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4effc4e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef810c8c0 con 0x7f4ef8101c80 2024-03-22T02:19:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 0x7f4ef8104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34238/0 (socket says 172.21.15.28:34238) 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 0x7f4ef8104a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46866/0 (socket says 172.21.15.28:46866) 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 -- 172.21.15.28:0/1415339151 learned_addr learned my addr 172.21.15.28:0/1415339151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45108/0 (socket says 172.21.15.28:45108) 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 -- 172.21.15.28:0/1415339151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 msgr2=0x7f4ef8103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 -- 172.21.15.28:0/1415339151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 msgr2=0x7f4ef8104a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 0x7f4ef8104a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 -- 172.21.15.28:0/1415339151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef8103e30 con 0x7f4ef8102640 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4efe1eb640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000efd0 tx=0x7f4ef000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.013+0000 7f4eeeffd640 1 -- 172.21.15.28:0/1415339151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0003b70 con 0x7f4ef8102640 2024-03-22T02:19:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.014+0000 7f4eeeffd640 1 -- 172.21.15.28:0/1415339151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef0003d10 con 0x7f4ef8102640 2024-03-22T02:19:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.014+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81a1fc0 con 0x7f4ef8102640 2024-03-22T02:19:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.014+0000 7f4eeeffd640 1 -- 172.21.15.28:0/1415339151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00109f0 con 0x7f4ef8102640 2024-03-22T02:19:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.014+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef81a2430 con 0x7f4ef8102640 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4eeeffd640 1 -- 172.21.15.28:0/1415339151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4ef0004020 con 0x7f4ef8102640 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 Processor -- start 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 -- start start 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637c3a0d8d0 con 0x5637c3aa9210 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5637c39b5e00 con 0x5637c3aae380 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4f02834740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5637c3a09880 con 0x5637c3aa8820 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34250/0 (socket says 172.21.15.28:34250) 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.017+0000 7f4efd9ea640 1 -- 172.21.15.28:0/4055471189 learned_addr learned my addr 172.21.15.28:0/4055471189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 -- 172.21.15.28:0/4055471189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 msgr2=0x5637c3aa8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 -- 172.21.15.28:0/4055471189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 msgr2=0x5637c3ab0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4eeeffd640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 0x7f4ee40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 -- 172.21.15.28:0/4055471189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637c37213f0 con 0x5637c3aa9210 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efe1eb640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 0x7f4ee40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009c80 tx=0x7f4ef4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f3550b10ddefa5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4eeeffd640 1 -- 172.21.15.28:0/1415339151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4ef00cfc70 con 0x7f4ef8102640 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4ee3fff640 1 -- 172.21.15.28:0/4055471189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef401e070 con 0x5637c3aa9210 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4ee3fff640 1 -- 172.21.15.28:0/4055471189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef4009e30 con 0x5637c3aa9210 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.018+0000 7f4ee3fff640 1 -- 172.21.15.28:0/4055471189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef400ea10 con 0x5637c3aa9210 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 -- 172.21.15.28:0/4055471189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 msgr2=0x5637c3aadd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4efe1eb640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 0x7f4ee40a53e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ef001a040 tx=0x7f4ef00bc040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009c80 tx=0x7f4ef4011810 comp rx=0 tx=0).stop 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 -- 172.21.15.28:0/4055471189 shutdown_connections 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3aadd10 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 --2- 172.21.15.28:0/4055471189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 -- 172.21.15.28:0/4055471189 >> 172.21.15.28:0/4055471189 conn(0x5637c3aa3d90 msgr2=0x5637c3aa4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 -- 172.21.15.28:0/4055471189 shutdown_connections 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.019+0000 7f4f02834740 1 -- 172.21.15.28:0/4055471189 wait complete. 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 -- start start 2024-03-22T02:19:21.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 Processor -- start 2024-03-22T02:19:21.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 0x5637c3ab2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 0x5637c3ab3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3aba2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5637c3ab1330 con 0x5637c3aae380 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5637c39c9300 con 0x5637c3aa8820 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5637c3ab14b0 con 0x5637c3aa9210 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4f02834740 10 client.0 _ll_register_callbacks cb 0x5637c3a04e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 0x5637c3ab2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 0x5637c3ab2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46878/0 (socket says 172.21.15.28:46878) 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4efd1e9640 1 -- 172.21.15.28:0/274410277 learned_addr learned my addr 172.21.15.28:0/274410277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 0x5637c3ab3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.020+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3aba2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 -- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 msgr2=0x5637c3ab2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 0x5637c3ab2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 -- 172.21.15.28:0/274410277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 msgr2=0x5637c3aba2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3aba2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 -- 172.21.15.28:0/274410277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637c37eb7d0 con 0x5637c3aa9210 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 0x5637c3ab3260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5637c3aa4d00 tx=0x7f4ef4005570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a972c924b9c7fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4028070 con 0x5637c3aa9210 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ef4009e30 con 0x5637c3aa9210 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4ef400eba0 con 0x5637c3aa9210 2024-03-22T02:19:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ef4034050 con 0x5637c3aa9210 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4f02834740 10 client.26995 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5637c3abd660 con 0x5637c3aa9210 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.021+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef4009e30 con 0x5637c3aa9210 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4ef4019430 con 0x5637c3aa9210 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 10 client.26995 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 10 client.26995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 20 client.26995 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3abea70 0x5637c3ac0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 4 client.26995 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637c3a1ae10 con 0x5637c3abea70 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3abea70 0x5637c3ac0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4ee17fa640 10 client.26995 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.022+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3abea70 0x5637c3ac0eb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f4c79330e43d6a server_cookie=925b3016f7fa0d51 in_seq=0 out_seq=0 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4ee17fa640 1 -- 172.21.15.28:0/274410277 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637c3a1ae10 con 0x5637c3abea70 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4ee17fa640 10 client.26995 handle_command_reply: tid=0 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 client.26995 shutdown 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 msgr2=0x5637c3ab3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 0x5637c3ab3260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5637c3aa4d00 tx=0x7f4ef4005570 comp rx=0 tx=0).stop 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 shutdown_connections 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3abea70 0x5637c3ac0eb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3aba2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa9210 0x5637c3ab3260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 --2- 172.21.15.28:0/274410277 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa8820 0x5637c3ab2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4ee17fa640 0 client.26995 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 >> 172.21.15.28:0/274410277 conn(0x5637c3aa3d90 msgr2=0x5637c3aa2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 shutdown_connections 2024-03-22T02:19:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 1 -- 172.21.15.28:0/274410277 wait complete. 2024-03-22T02:19:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.023+0000 7f4f02834740 20 client.26995 trim_cache size 0 max 16384 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.036+0000 7f4f02834740 1 Processor -- start 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 -- start start 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 0x5637c3aba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81af520 con 0x5637c3aae380 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81acdf0 con 0x5637c3aa9210 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4f02834740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81aac40 con 0x5637c3aa8820 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 0x5637c3aba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34254/0 (socket says 172.21.15.28:34254) 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd1e9640 1 -- 172.21.15.28:0/403166027 learned_addr learned my addr 172.21.15.28:0/403166027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 -- 172.21.15.28:0/403166027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 msgr2=0x5637c3aa7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 -- 172.21.15.28:0/403166027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 msgr2=0x5637c3ac5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 -- 172.21.15.28:0/403166027 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637c3aa4b00 con 0x5637c3aa9210 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 0x5637c3aba260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009250 tx=0x7f4ef4011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b43b3cb5e93374ac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.037+0000 7f4ee37fe640 1 -- 172.21.15.28:0/403166027 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4031070 con 0x5637c3aa9210 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4ee37fe640 1 -- 172.21.15.28:0/403166027 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ef400ee80 con 0x5637c3aa9210 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4ee37fe640 1 -- 172.21.15.28:0/403166027 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef4028050 con 0x5637c3aa9210 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- 172.21.15.28:0/403166027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 msgr2=0x5637c3aba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 0x5637c3aba260 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4ef4009250 tx=0x7f4ef4011dc0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- 172.21.15.28:0/403166027 shutdown_connections 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aae380 0x5637c3ac5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aa9210 0x5637c3aba260 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 --2- 172.21.15.28:0/403166027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3aa7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- 172.21.15.28:0/403166027 >> 172.21.15.28:0/403166027 conn(0x5637c3aa3d90 msgr2=0x5637c3aa6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- 172.21.15.28:0/403166027 shutdown_connections 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- 172.21.15.28:0/403166027 wait complete. 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.038+0000 7f4f02834740 1 -- start start 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 Processor -- start 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3ab77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3ab7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81ab2b0 con 0x5637c3aa9210 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81ac720 con 0x5637c3aae380 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef81ab980 con 0x5637c3aa8820 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4f02834740 10 client.0 _ll_register_callbacks cb 0x7f4ef401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:21.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd9ea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3ab7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46896/0 (socket says 172.21.15.28:46896) 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 -- 172.21.15.28:0/1946816933 learned_addr learned my addr 172.21.15.28:0/1946816933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3ab77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 -- 172.21.15.28:0/1946816933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 msgr2=0x5637c3ab77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3ab77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 -- 172.21.15.28:0/1946816933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 msgr2=0x5637c3ab7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3ab7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd1e9640 1 -- 172.21.15.28:0/1946816933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637c3aa4800 con 0x5637c3aae380 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.039+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3ab7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4efd1e9640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ee802c8e0 tx=0x7f4ee8045040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96a79068cb57df6e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:19:21.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8050700 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f4ee8004820 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5637c3a15160 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4ee8004a20 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4f02834740 10 client.26996 fetch_fsmap learned FSMap version 31 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5637c3aba4a0 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5637c3a15160 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.040+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 31) ==== 8346+0+0 (secure 0 0 0) 0x7f4ee80035f0 con 0x5637c3aae380 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 10 client.26996 fetch_fsmap finished waiting for FSMap version 31 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 10 client.26996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 20 client.26996 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3b253f0 0x5637c3b27810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 4 client.26996 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5637c39926a0 con 0x5637c3b253f0 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3b253f0 0x5637c3b27810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4ee17fa640 10 client.26996 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4efd9ea640 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3b253f0 0x5637c3b27810 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bd0716cd44c40c0 server_cookie=95257c87cb91fbd8 in_seq=0 out_seq=0 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4ee17fa640 1 -- 172.21.15.28:0/1946816933 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5637c39926a0 con 0x5637c3b253f0 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4ee17fa640 10 client.26996 handle_command_reply: tid=0 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 client.26996 shutdown 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 msgr2=0x5637c3ab8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ee802c8e0 tx=0x7f4ee8045040 comp rx=0 tx=0).stop 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 shutdown_connections 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5637c3b253f0 0x5637c3b27810 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5637c3aae380 0x5637c3ab8d10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5637c3aa9210 0x5637c3ab7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 --2- 172.21.15.28:0/1946816933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5637c3aa8820 0x5637c3ab77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4ee17fa640 0 client.26996 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.041+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 >> 172.21.15.28:0/1946816933 conn(0x5637c3aa3d90 msgr2=0x5637c3aa7880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 shutdown_connections 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4f02834740 1 -- 172.21.15.28:0/1946816933 wait complete. 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4f02834740 20 client.26996 trim_cache size 0 max 16384 2024-03-22T02:19:21.051 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 msgr2=0x7f4ee40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 0x7f4ee40a53e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f4ef001a040 tx=0x7f4ef00bc040 comp rx=0 tx=0).stop 2024-03-22T02:19:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 msgr2=0x7f4ef81035a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000efd0 tx=0x7f4ef000c6a0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 shutdown_connections 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ee40a2f20 0x7f4ee40a53e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ef8102640 0x7f4ef81035a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ef8101c80 0x7f4ef8103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 --2- 172.21.15.28:0/1415339151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ef8100a90 0x7f4ef8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 >> 172.21.15.28:0/1415339151 conn(0x7f4ef80fc020 msgr2=0x7f4ef80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 shutdown_connections 2024-03-22T02:19:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:21.042+0000 7f4effc4e640 1 -- 172.21.15.28:0/1415339151 wait complete. 2024-03-22T02:19:21.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 467150 12.01 MB/sec execute 123 sec latency 52.409 ms 2024-03-22T02:19:21.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:21 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/394363433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:21.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:21 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/394363433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:21.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:21 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/394363433' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:22.055 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:19:22.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:19:22.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:19:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 -- 172.21.15.28:0/3278405423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514105720 msgr2=0x7f2514105ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 --2- 172.21.15.28:0/3278405423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514105720 0x7f2514105ba0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f250c009be0 tx=0x7f250c02bef0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 -- 172.21.15.28:0/3278405423 shutdown_connections 2024-03-22T02:19:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 --2- 172.21.15.28:0/3278405423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514105720 0x7f2514105ba0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 --2- 172.21.15.28:0/3278405423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f25141051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 --2- 172.21.15.28:0/3278405423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514100400 0x7f2514102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.165+0000 7f25192df640 1 -- 172.21.15.28:0/3278405423 >> 172.21.15.28:0/3278405423 conn(0x7f25140fc020 msgr2=0x7f25140fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- 172.21.15.28:0/3278405423 shutdown_connections 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- 172.21.15.28:0/3278405423 wait complete. 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 Processor -- start 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- start start 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f251419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2514113e30 con 0x7f2514105720 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2514113cb0 con 0x7f2514100400 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25192df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2514113fb0 con 0x7f2514102d70 2024-03-22T02:19:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45508/0 (socket says 172.21.15.28:45508) 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44206/0 (socket says 172.21.15.28:44206) 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f2512ffd640 1 -- 172.21.15.28:0/3431205068 learned_addr learned my addr 172.21.15.28:0/3431205068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.166+0000 7f250bfff640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f251419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 -- 172.21.15.28:0/3431205068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 msgr2=0x7f251419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f251419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 -- 172.21.15.28:0/3431205068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 msgr2=0x7f251419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 -- 172.21.15.28:0/3431205068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25141a6070 con 0x7f2514105720 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f250bfff640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f251419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25137fe640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f250c02c430 tx=0x7f250c02cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f2510ff9640 1 -- 172.21.15.28:0/3431205068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c043020 con 0x7f2514105720 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25141a6300 con 0x7f2514105720 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f2510ff9640 1 -- 172.21.15.28:0/3431205068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250c003e50 con 0x7f2514105720 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f2510ff9640 1 -- 172.21.15.28:0/3431205068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c034690 con 0x7f2514105720 2024-03-22T02:19:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.167+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25141a68a0 con 0x7f2514105720 2024-03-22T02:19:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.169+0000 7f2510ff9640 1 -- 172.21.15.28:0/3431205068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f250c046020 con 0x7f2514105720 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 Processor -- start 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 -- start start 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f2510ff9640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 0x7f25000a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f1510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f0b0570 con 0x55bd7f151720 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f058e10 con 0x55bd7f14bbc0 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f251bf9e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f0ac520 con 0x55bd7f14c5b0 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f250bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 0x7f25000a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f250bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f250bfff640 1 -- 172.21.15.28:0/3091792912 learned_addr learned my addr 172.21.15.28:0/3091792912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44208/0 (socket says 172.21.15.28:44208) 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f250bfff640 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f1510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.170+0000 7f2510ff9640 1 -- 172.21.15.28:0/3431205068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f250c0f1d90 con 0x7f2514105720 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 -- 172.21.15.28:0/3091792912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 msgr2=0x55bd7f1510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f1510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 -- 172.21.15.28:0/3091792912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 msgr2=0x55bd7f153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 -- 172.21.15.28:0/3091792912 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd7ee8c7e0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f25137fe640 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 0x7f25000a53e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f25040096c0 tx=0x7f2504009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f250bfff640 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009fd0 tx=0x7f24fc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f67162bb424fc86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f25097fa640 1 -- 172.21.15.28:0/3091792912 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00fc20 con 0x55bd7f14bbc0 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f25097fa640 1 -- 172.21.15.28:0/3091792912 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24fc00fdc0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.171+0000 7f25097fa640 1 -- 172.21.15.28:0/3091792912 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24fc00e5d0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- 172.21.15.28:0/3091792912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 msgr2=0x55bd7f14bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009fd0 tx=0x7f24fc009b80 comp rx=0 tx=0).stop 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- 172.21.15.28:0/3091792912 shutdown_connections 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f1510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3091792912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f14bfe0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- 172.21.15.28:0/3091792912 >> 172.21.15.28:0/3091792912 conn(0x55bd7f147130 msgr2=0x55bd7f147530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- 172.21.15.28:0/3091792912 shutdown_connections 2024-03-22T02:19:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- 172.21.15.28:0/3091792912 wait complete. 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- start start 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 Processor -- start 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f1546d0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f06c110 con 0x55bd7f151720 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bd7f154850 con 0x55bd7f14c5b0 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f251bf9e740 10 client.0 _ll_register_callbacks cb 0x55bd7f0a8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.172+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44218/0 (socket says 172.21.15.28:44218) 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 -- 172.21.15.28:0/2502385491 learned_addr learned my addr 172.21.15.28:0/2502385491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f250bfff640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f25137fe640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 -- 172.21.15.28:0/2502385491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 msgr2=0x55bd7f156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 -- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 msgr2=0x55bd7f15d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 -- 172.21.15.28:0/2502385491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd7eeecf70 con 0x55bd7f14bbc0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f2512ffd640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f250400fbd0 tx=0x7f2504004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a08e5b15458357 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f24fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250402b060 con 0x55bd7f14bbc0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f250401cc90 con 0x55bd7f14bbc0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24fc00fdc0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2504032050 con 0x55bd7f14bbc0 2024-03-22T02:19:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f251bf9e740 10 client.17178 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd7ed84170 con 0x55bd7f14bbc0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.173+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24fc00fdc0 con 0x55bd7f14bbc0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f250401a550 con 0x55bd7f14bbc0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 10 client.17178 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 20 client.17178 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f164190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 4 client.17178 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd7f0bdab0 con 0x55bd7f161cf0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f250bfff640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f164190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f24fa7fc640 10 client.17178 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.174+0000 7f250bfff640 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f164190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fe05e2a670d56e9 server_cookie=64deeea53c911a68 in_seq=0 out_seq=0 2024-03-22T02:19:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f24fa7fc640 1 -- 172.21.15.28:0/2502385491 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bd7f0bdab0 con 0x55bd7f161cf0 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f24fa7fc640 10 client.17178 handle_command_reply: tid=0 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 client.17178 shutdown 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 msgr2=0x55bd7f155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f250400fbd0 tx=0x7f2504004730 comp rx=0 tx=0).stop 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 shutdown_connections 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f164190 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f24fa7fc640 0 client.17178 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 --2- 172.21.15.28:0/2502385491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f155e00 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.176+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 >> 172.21.15.28:0/2502385491 conn(0x55bd7f147130 msgr2=0x55bd7f1463f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.177+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 shutdown_connections 2024-03-22T02:19:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.177+0000 7f251bf9e740 1 -- 172.21.15.28:0/2502385491 wait complete. 2024-03-22T02:19:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.177+0000 7f251bf9e740 20 client.17178 trim_cache size 0 max 16384 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 Processor -- start 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 -- start start 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b3880 con 0x55bd7f151720 2024-03-22T02:19:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b1150 con 0x55bd7f14bbc0 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f251bf9e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25141aefa0 con 0x55bd7f14c5b0 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44232/0 (socket says 172.21.15.28:44232) 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 -- 172.21.15.28:0/1683114414 learned_addr learned my addr 172.21.15.28:0/1683114414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f250bfff640 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 -- 172.21.15.28:0/1683114414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 msgr2=0x55bd7f156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 -- 172.21.15.28:0/1683114414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 msgr2=0x55bd7f1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f1554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f25137fe640 1 -- 172.21.15.28:0/1683114414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd7f147b80 con 0x55bd7f151720 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.195+0000 7f2512ffd640 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f25137fe640 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f250c0ef420 tx=0x7f250c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932b08898fd136bf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f2508ff9640 1 -- 172.21.15.28:0/1683114414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c0cb660 con 0x55bd7f151720 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f2508ff9640 1 -- 172.21.15.28:0/1683114414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250c004040 con 0x55bd7f151720 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f2508ff9640 1 -- 172.21.15.28:0/1683114414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c033910 con 0x55bd7f151720 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f251bf9e740 1 -- 172.21.15.28:0/1683114414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 msgr2=0x55bd7f145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.196+0000 7f251bf9e740 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f250c0ef420 tx=0x7f250c004600 comp rx=0 tx=0).stop 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- 172.21.15.28:0/1683114414 shutdown_connections 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f151720 0x55bd7f145b60 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- 172.21.15.28:0/1683114414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f14bbc0 0x55bd7f1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- 172.21.15.28:0/1683114414 >> 172.21.15.28:0/1683114414 conn(0x55bd7f147130 msgr2=0x55bd7f155a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- 172.21.15.28:0/1683114414 shutdown_connections 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- 172.21.15.28:0/1683114414 wait complete. 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- start start 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 Processor -- start 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f15a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25141af610 con 0x55bd7f14bbc0 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25141b0a80 con 0x55bd7f151720 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25141afce0 con 0x55bd7f14c5b0 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f251bf9e740 10 client.0 _ll_register_callbacks cb 0x7f25040095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44236/0 (socket says 172.21.15.28:44236) 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 -- 172.21.15.28:0/3800865670 learned_addr learned my addr 172.21.15.28:0/3800865670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f25137fe640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f250bfff640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 -- 172.21.15.28:0/3800865670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 msgr2=0x55bd7f15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 -- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 msgr2=0x55bd7f15b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f2512ffd640 1 -- 172.21.15.28:0/3800865670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd7f15b950 con 0x55bd7f14bbc0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.197+0000 7f250bfff640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f2512ffd640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f250401a970 tx=0x7f250400ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3341372d3016b1e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250401a590 con 0x55bd7f14bbc0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2504038440 con 0x55bd7f14bbc0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f250c004040 con 0x55bd7f14bbc0 2024-03-22T02:19:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2504045050 con 0x55bd7f14bbc0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 10 client.17184 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bd7f1c8090 con 0x55bd7f14bbc0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f250c004040 con 0x55bd7f14bbc0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f2504040030 con 0x55bd7f14bbc0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 10 client.17184 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 20 client.17184 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f1caa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 4 client.17184 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.198+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bd7f0b7e00 con 0x55bd7f161cf0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.199+0000 7f250bfff640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f1caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.199+0000 7f250bfff640 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f1caa60 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38bf418dc5d55a8e server_cookie=b79c0a476c6058b2 in_seq=0 out_seq=0 2024-03-22T02:19:22.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.199+0000 7f24fa7fc640 10 client.17184 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.201+0000 7f24fa7fc640 1 -- 172.21.15.28:0/3800865670 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bd7f0b7e00 con 0x55bd7f161cf0 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.201+0000 7f24fa7fc640 10 client.17184 handle_command_reply: tid=0 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.201+0000 7f251bf9e740 1 client.17184 shutdown 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 msgr2=0x55bd7f159e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f250401a970 tx=0x7f250400ffd0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 shutdown_connections 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bd7f161cf0 0x55bd7f1caa60 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bd7f151720 0x55bd7f15b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f24fa7fc640 0 client.17184 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bd7f14c5b0 0x55bd7f15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 --2- 172.21.15.28:0/3800865670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bd7f14bbc0 0x55bd7f159e10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 >> 172.21.15.28:0/3800865670 conn(0x55bd7f147130 msgr2=0x55bd7f14a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 shutdown_connections 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 1 -- 172.21.15.28:0/3800865670 wait complete. 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f251bf9e740 20 client.17184 trim_cache size 0 max 16384 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "adc704b4-d327-439e-8da0-42e5f1dbbc25", 2024-03-22T02:19:22.209 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:19:22.210 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 msgr2=0x7f25000a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 0x7f25000a53e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f25040096c0 tx=0x7f2504009260 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 msgr2=0x7f25141a5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f250c02c430 tx=0x7f250c02cb90 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 shutdown_connections 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25000a2f20 0x7f25000a53e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2514105720 0x7f25141a5a00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2514102d70 0x7f251419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 --2- 172.21.15.28:0/3431205068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2514100400 0x7f251419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.202+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 >> 172.21.15.28:0/3431205068 conn(0x7f25140fc020 msgr2=0x7f2514103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.203+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 shutdown_connections 2024-03-22T02:19:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.203+0000 7f25192df640 1 -- 172.21.15.28:0/3431205068 wait complete. 2024-03-22T02:19:22.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:19:22.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:19:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- 172.21.15.28:0/772642892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0102640 msgr2=0x7f6ff0108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 --2- 172.21.15.28:0/772642892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0102640 0x7f6ff0108ed0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009ca0 tx=0x7f6fe802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- 172.21.15.28:0/772642892 shutdown_connections 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 --2- 172.21.15.28:0/772642892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0102640 0x7f6ff0108ed0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 --2- 172.21.15.28:0/772642892 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 --2- 172.21.15.28:0/772642892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0100a90 0x7f6ff0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- 172.21.15.28:0/772642892 >> 172.21.15.28:0/772642892 conn(0x7f6ff00fc020 msgr2=0x7f6ff00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- 172.21.15.28:0/772642892 shutdown_connections 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- 172.21.15.28:0/772642892 wait complete. 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 Processor -- start 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.320+0000 7f6ff6513640 1 -- start start 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0102640 0x7f6ff019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff010c6f0 con 0x7f6ff0100a90 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff010c570 con 0x7f6ff0101c80 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff6513640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff010c870 con 0x7f6ff0102640 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6ff4ab0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0102640 0x7f6ff019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44244/0 (socket says 172.21.15.28:44244) 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6feffff640 1 -- 172.21.15.28:0/37113741 learned_addr learned my addr 172.21.15.28:0/37113741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 -- 172.21.15.28:0/37113741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0102640 msgr2=0x7f6ff019f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0102640 0x7f6ff019f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 -- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 msgr2=0x7f6ff0197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 -- 172.21.15.28:0/37113741 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff019fa70 con 0x7f6ff0101c80 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6feffff640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.321+0000 7f6fef7fe640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0198390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400e9f0 tx=0x7f6fe400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.322+0000 7f6fed7fa640 1 -- 172.21.15.28:0/37113741 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe400ccc0 con 0x7f6ff0101c80 2024-03-22T02:19:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.322+0000 7f6fed7fa640 1 -- 172.21.15.28:0/37113741 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe4010910 con 0x7f6ff0101c80 2024-03-22T02:19:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.322+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff019fd60 con 0x7f6ff0101c80 2024-03-22T02:19:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.322+0000 7f6fed7fa640 1 -- 172.21.15.28:0/37113741 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4018960 con 0x7f6ff0101c80 2024-03-22T02:19:22.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.322+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff01a0250 con 0x7f6ff0101c80 2024-03-22T02:19:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6fed7fa640 1 -- 172.21.15.28:0/37113741 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6fe4010ac0 con 0x7f6ff0101c80 2024-03-22T02:19:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 Processor -- start 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 -- start start 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec45c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec4d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6fed7fa640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 0x7f6fd80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56053eba68c0 con 0x56053ec46230 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56053eb4ec20 con 0x56053ec4b3a0 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56053eba2870 con 0x56053ec45840 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6ff4ab0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec45c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6fef7fe640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 0x7f6fd80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44250/0 (socket says 172.21.15.28:44250) 2024-03-22T02:19:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.324+0000 7f6feffff640 1 -- 172.21.15.28:0/963154458 learned_addr learned my addr 172.21.15.28:0/963154458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec4d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6fed7fa640 1 -- 172.21.15.28:0/37113741 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f6fe40cfbc0 con 0x7f6ff0101c80 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 -- 172.21.15.28:0/963154458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 msgr2=0x56053ec45c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec45c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 -- 172.21.15.28:0/963154458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 msgr2=0x56053ec4d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec4d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 -- 172.21.15.28:0/963154458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56053eb73150 con 0x56053ec46230 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6fef7fe640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 0x7f6fd80a53e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6fe409bec0 tx=0x7f6fe40c6b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6feffff640 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00115c0 tx=0x7f6fe0011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbfd57b79b4d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6fd67fc640 1 -- 172.21.15.28:0/963154458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe0005020 con 0x56053ec46230 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6fd67fc640 1 -- 172.21.15.28:0/963154458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe00047a0 con 0x56053ec46230 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.325+0000 7f6fd67fc640 1 -- 172.21.15.28:0/963154458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe000f930 con 0x56053ec46230 2024-03-22T02:19:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- 172.21.15.28:0/963154458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 msgr2=0x56053ec4ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6fe00115c0 tx=0x7f6fe0011a80 comp rx=0 tx=0).stop 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- 172.21.15.28:0/963154458 shutdown_connections 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec4d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec4ad30 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/963154458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec45c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- 172.21.15.28:0/963154458 >> 172.21.15.28:0/963154458 conn(0x56053ec3c9a0 msgr2=0x56053ec3cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- 172.21.15.28:0/963154458 shutdown_connections 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- 172.21.15.28:0/963154458 wait complete. 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 -- start start 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.326+0000 7f6ff91d9740 1 Processor -- start 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec4fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 0x56053ec50290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56053ec4e350 con 0x56053ec4b3a0 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56053eb62130 con 0x56053ec46230 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56053ec4e4d0 con 0x56053ec45840 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 10 client.0 _ll_register_callbacks cb 0x56053eb61b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44264/0 (socket says 172.21.15.28:44264) 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 -- 172.21.15.28:0/9920041 learned_addr learned my addr 172.21.15.28:0/9920041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6feffff640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 0x56053ec50290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec4fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 -- 172.21.15.28:0/9920041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 msgr2=0x56053ec4fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec4fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 -- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 msgr2=0x56053ec50290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 0x56053ec50290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 -- 172.21.15.28:0/9920041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56053e9847d0 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6feffff640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 0x56053ec50290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009ca0 tx=0x7f6fe80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b09901b358217d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6fcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8039040 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f6fe8035a90 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.327+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe00047a0 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fe8057050 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 10 client.17196 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56053e8ba3f0 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe00047a0 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f6fe803f040 con 0x56053ec4b3a0 2024-03-22T02:19:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 10 client.17196 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 20 client.17196 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ec5bbb0 0x56053ec5e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 4 client.17196 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56053ebb3e00 con 0x56053ec5bbb0 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.328+0000 7f6feffff640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ec5bbb0 0x56053ec5e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.329+0000 7f6feffff640 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ec5bbb0 0x56053ec5e050 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd8c6ada255e136 server_cookie=b0e474bb89c87356 in_seq=0 out_seq=0 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.329+0000 7f6fcf7fe640 10 client.17196 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/9920041 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56053ebb3e00 con 0x56053ec5bbb0 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6fcf7fe640 10 client.17196 handle_command_reply: tid=0 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 client.17196 shutdown 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 msgr2=0x56053ec572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009ca0 tx=0x7f6fe80095c0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 shutdown_connections 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ec5bbb0 0x56053ec5e050 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec4b3a0 0x56053ec572d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec46230 0x56053ec50290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6fcf7fe640 0 client.17196 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/9920041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec4fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 >> 172.21.15.28:0/9920041 conn(0x56053ec3c9a0 msgr2=0x56053ec3bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 shutdown_connections 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 1 -- 172.21.15.28:0/9920041 wait complete. 2024-03-22T02:19:22.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.330+0000 7f6ff91d9740 20 client.17196 trim_cache size 0 max 16384 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 Processor -- start 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 -- start start 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 0x56053ec3b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec4b3a0 0x56053ec3b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01ab060 con 0x56053ec46230 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01a8930 con 0x56053ec45840 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01a6780 con 0x56053ec4b3a0 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.347+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6ff4ab0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 0x56053ec3b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44280/0 (socket says 172.21.15.28:44280) 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 -- 172.21.15.28:0/2667949960 learned_addr learned my addr 172.21.15.28:0/2667949960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec4b3a0 0x56053ec3b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 -- 172.21.15.28:0/2667949960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec4b3a0 msgr2=0x56053ec3b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec4b3a0 0x56053ec3b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 -- 172.21.15.28:0/2667949960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 msgr2=0x56053ec3b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 0x56053ec3b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 -- 172.21.15.28:0/2667949960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56053e87a0f0 con 0x56053ec46230 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 0x56053ec3b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009250 tx=0x7f6fe0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94927968e67a374b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6fd5ffb640 1 -- 172.21.15.28:0/2667949960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe001d040 con 0x56053ec46230 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6fd5ffb640 1 -- 172.21.15.28:0/2667949960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe002e030 con 0x56053ec46230 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.348+0000 7f6fd5ffb640 1 -- 172.21.15.28:0/2667949960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe002a5e0 con 0x56053ec46230 2024-03-22T02:19:22.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667949960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 msgr2=0x56053ec57290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0009250 tx=0x7f6fe0000bc0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667949960 shutdown_connections 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec4b3a0 0x56053ec3b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec57290 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667949960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec45840 0x56053ec3b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667949960 >> 172.21.15.28:0/2667949960 conn(0x56053ec3c9a0 msgr2=0x56053ec40490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667949960 shutdown_connections 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667949960 wait complete. 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- start start 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 Processor -- start 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec55ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01a6df0 con 0x56053ec46230 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01a8260 con 0x56053ec4b3a0 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.349+0000 7f6ff91d9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff01a74c0 con 0x56053ec45840 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff91d9740 10 client.0 _ll_register_callbacks cb 0x7f6fe8032eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 -- 172.21.15.28:0/2667141130 learned_addr learned my addr 172.21.15.28:0/2667141130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec55ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 -- 172.21.15.28:0/2667141130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 msgr2=0x56053ec545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 -- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 msgr2=0x56053ec55ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec55ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 -- 172.21.15.28:0/2667141130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56053ec560b0 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6feffff640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0011ac0 tx=0x7f6fe0009530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e841220477e2fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6fcf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe001d040 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f6fe00051e0 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6fe002e030 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6fe0044050 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff91d9740 10 client.17202 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56053ec573c0 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.350+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6fe002e030 con 0x56053ec46230 2024-03-22T02:19:22.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f6fe000f8e0 con 0x56053ec46230 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 10 client.17202 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 10 client.17202 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 20 client.17202 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ecc2360 0x56053ecc4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 4 client.17202 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56053ebae150 con 0x56053ecc2360 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ecc2360 0x56053ecc4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6fcf7fe640 10 client.17202 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.351+0000 7f6ff4ab0640 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ecc2360 0x56053ecc4780 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8e959c4dc66099 server_cookie=faa54d5a8a617ee5 in_seq=0 out_seq=0 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6fcf7fe640 1 -- 172.21.15.28:0/2667141130 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56053ebae150 con 0x56053ecc2360 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6fcf7fe640 10 client.17202 handle_command_reply: tid=0 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 client.17202 shutdown 2024-03-22T02:19:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 msgr2=0x56053ec54b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f6fe0011ac0 tx=0x7f6fe0009530 comp rx=0 tx=0).stop 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 shutdown_connections 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56053ecc2360 0x56053ecc4780 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56053ec4b3a0 0x56053ec55ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56053ec46230 0x56053ec54b00 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6fcf7fe640 0 client.17202 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 --2- 172.21.15.28:0/2667141130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56053ec45840 0x56053ec545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 >> 172.21.15.28:0/2667141130 conn(0x56053ec3c9a0 msgr2=0x56053ec4e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 shutdown_connections 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 1 -- 172.21.15.28:0/2667141130 wait complete. 2024-03-22T02:19:22.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff91d9740 20 client.17202 trim_cache size 0 max 16384 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "adc704b4-d327-439e-8da0-42e5f1dbbc25": { 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "adc704b4-d327-439e-8da0-42e5f1dbbc25", 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:19:22.361 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 msgr2=0x7f6fd80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 0x7f6fd80a53e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6fe409bec0 tx=0x7f6fe40c6b40 comp rx=0 tx=0).stop 2024-03-22T02:19:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 msgr2=0x7f6ff0198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0198390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6fe400e9f0 tx=0x7f6fe400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 shutdown_connections 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6fd80a2f20 0x7f6fd80a53e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ff0102640 0x7f6ff019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ff0101c80 0x7f6ff0198390 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.353+0000 7f6ff6513640 1 --2- 172.21.15.28:0/37113741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ff0100a90 0x7f6ff0197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.354+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 >> 172.21.15.28:0/37113741 conn(0x7f6ff00fc020 msgr2=0x7f6ff00fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.354+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 shutdown_connections 2024-03-22T02:19:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:22.354+0000 7f6ff6513640 1 -- 172.21.15.28:0/37113741 wait complete. 2024-03-22T02:19:22.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:22 smithi028 ceph-mon[31916]: pgmap v532: 129 pgs: 129 active+clean; 42 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 413 op/s 2024-03-22T02:19:22.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:22 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:22.364 INFO:tasks.cephfs.filesystem:scrub status for tag:adc704b4-d327-439e-8da0-42e5f1dbbc25 - {'path': '/', 'tag': 'adc704b4-d327-439e-8da0-42e5f1dbbc25', 'options': 'recursive,force'} 2024-03-22T02:19:22.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 469659 12.03 MB/sec execute 124 sec latency 14.233 ms 2024-03-22T02:19:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:22 smithi120 ceph-mon[36803]: pgmap v532: 129 pgs: 129 active+clean; 42 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 413 op/s 2024-03-22T02:19:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:22 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:22.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:22 smithi083 ceph-mon[36340]: pgmap v532: 129 pgs: 129 active+clean; 42 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.1 MiB/s wr, 413 op/s 2024-03-22T02:19:22.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:22 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:19:23.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 472681 12.07 MB/sec execute 125 sec latency 10.166 ms 2024-03-22T02:19:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:19:23.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:19:23.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:19:23.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:23 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:19:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:19:23.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:19:23.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:23 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:23.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:23 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:24.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 474941 12.04 MB/sec execute 126 sec latency 36.469 ms 2024-03-22T02:19:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:24 smithi028 ceph-mon[31916]: pgmap v533: 129 pgs: 129 active+clean; 39 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:19:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:24 smithi120 ceph-mon[36803]: pgmap v533: 129 pgs: 129 active+clean; 39 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:19:24.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:24 smithi083 ceph-mon[36340]: pgmap v533: 129 pgs: 129 active+clean; 39 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 439 op/s 2024-03-22T02:19:25.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 476670 12.04 MB/sec execute 127 sec latency 58.904 ms 2024-03-22T02:19:26.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 479148 12.05 MB/sec execute 128 sec latency 40.770 ms 2024-03-22T02:19:26.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:26 smithi028 ceph-mon[31916]: pgmap v534: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:19:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:26 smithi120 ceph-mon[36803]: pgmap v534: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:19:26.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:26 smithi083 ceph-mon[36340]: pgmap v534: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:19:27.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 481444 12.05 MB/sec execute 129 sec latency 31.495 ms 2024-03-22T02:19:28.417 INFO:tasks.workunit.client.0.smithi028.stdout: 1 483897 12.06 MB/sec execute 130 sec latency 35.528 ms 2024-03-22T02:19:28.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:28 smithi028 ceph-mon[31916]: pgmap v535: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 455 op/s 2024-03-22T02:19:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:28 smithi120 ceph-mon[36803]: pgmap v535: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 455 op/s 2024-03-22T02:19:28.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:28 smithi083 ceph-mon[36340]: pgmap v535: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.2 MiB/s wr, 455 op/s 2024-03-22T02:19:29.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 487011 12.11 MB/sec execute 131 sec latency 9.771 ms 2024-03-22T02:19:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/2698970403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4101be0 msgr2=0x7f7dd4102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/2698970403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4101be0 0x7f7dd4102000 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009900 tx=0x7f7dcc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/2698970403 shutdown_connections 2024-03-22T02:19:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/2698970403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4103790 0x7f7dd410a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/2698970403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4102dd0 0x7f7dd4103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/2698970403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4101be0 0x7f7dd4102000 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/2698970403 >> 172.21.15.28:0/2698970403 conn(0x7f7dd40fd150 msgr2=0x7f7dd40ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/2698970403 shutdown_connections 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.315+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/2698970403 wait complete. 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 Processor -- start 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 -- start start 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4101be0 0x7f7dd4075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4102dd0 0x7f7dd4073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d850 con 0x7f7dd4103790 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d6d0 con 0x7f7dd4102dd0 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd8fd0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd410d9d0 con 0x7f7dd4101be0 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4101be0 0x7f7dd4075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44304/0 (socket says 172.21.15.28:44304) 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/1378717005 learned_addr learned my addr 172.21.15.28:0/1378717005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4102dd0 0x7f7dd4073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/1378717005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4101be0 msgr2=0x7f7dd4075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4101be0 0x7f7dd4075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/1378717005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4102dd0 msgr2=0x7f7dd4073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4102dd0 0x7f7dd4073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.316+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/1378717005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd4074d50 con 0x7f7dd4103790 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400eb10 tx=0x7f7dc400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1378717005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4004b20 con 0x7f7dd4103790 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1378717005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dc4004cc0 con 0x7f7dd4103790 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd41a1f50 con 0x7f7dd4103790 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1378717005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4010760 con 0x7f7dd4103790 2024-03-22T02:19:30.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.317+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd41a2360 con 0x7f7dd4103790 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.319+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1378717005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7dc4005020 con 0x7f7dd4103790 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.319+0000 7f7ddbbb8740 1 Processor -- start 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.319+0000 7f7ddbbb8740 1 -- start start 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dcb7fe640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 0x7f7dc00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35de4e0 0x555ca35de900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35e39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ca3543040 con 0x555ca35deed0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555ca34eb9e0 con 0x555ca35de4e0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555ca353eff0 con 0x555ca35e4040 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 0x7f7dc00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1378717005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7dc40cfd70 con 0x7f7dd4103790 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49208/0 (socket says 172.21.15.28:49208) 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/433300783 learned_addr learned my addr 172.21.15.28:0/433300783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd259c640 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35de4e0 0x555ca35de900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35e39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/433300783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35de4e0 msgr2=0x555ca35de900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35de4e0 0x555ca35de900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/433300783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 msgr2=0x555ca35e39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35e39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/433300783 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ca3257210 con 0x555ca35e4040 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 0x7f7dc00a5480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009fd0 tx=0x7f7dbc009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.320+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40a4570 tx=0x7f7dc4098460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f32b9dcc9c5f0e2e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7dc8ff9640 1 -- 172.21.15.28:0/433300783 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc409c090 con 0x555ca35e4040 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7dc8ff9640 1 -- 172.21.15.28:0/433300783 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dc40a01e0 con 0x555ca35e4040 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7dc8ff9640 1 -- 172.21.15.28:0/433300783 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc409a420 con 0x555ca35e4040 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/433300783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 msgr2=0x555ca35e64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40a4570 tx=0x7f7dc4098460 comp rx=0 tx=0).stop 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/433300783 shutdown_connections 2024-03-22T02:19:30.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35e4040 0x555ca35e64d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35e39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/433300783 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35de4e0 0x555ca35de900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/433300783 >> 172.21.15.28:0/433300783 conn(0x555ca35d9a50 msgr2=0x555ca35d9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/433300783 shutdown_connections 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.321+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/433300783 wait complete. 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 -- start start 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 Processor -- start 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35e8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35deed0 0x555ca35e89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555ca35e6ff0 con 0x555ca35e4040 2024-03-22T02:19:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555ca34fedb0 con 0x555ca35deed0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555ca35e7170 con 0x555ca35de4e0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7ddbbb8740 10 client.0 _ll_register_callbacks cb 0x555ca3256a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35deed0 0x555ca35e89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35e8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49212/0 (socket says 172.21.15.28:49212) 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44310/0 (socket says 172.21.15.28:44310) 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd259c640 1 -- 172.21.15.28:0/4226352112 learned_addr learned my addr 172.21.15.28:0/4226352112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/4226352112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 msgr2=0x555ca35e8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35e8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35deed0 msgr2=0x555ca35e89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35deed0 0x555ca35e89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.322+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/4226352112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ca3323620 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40cd7d0 tx=0x7f7dc4009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc165631cccbb94 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40b4070 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7dc409c500 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dc40a01e0 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7dc409c090 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 10 client.17214 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555ca35f2ce0 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dc40a01e0 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f7dc4005310 con 0x555ca35e4040 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 10 client.17214 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 10 client.17214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:30.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 20 client.17214 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35f4030 0x555ca35f6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 4 client.17214 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.323+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ca3550580 con 0x555ca35f4030 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.324+0000 7f7dd259c640 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35f4030 0x555ca35f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.324+0000 7f7dd259c640 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35f4030 0x555ca35f6420 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84039cad823a659c server_cookie=72a167a1bab3644 in_seq=0 out_seq=0 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.324+0000 7f7db9ffb640 10 client.17214 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7db9ffb640 1 -- 172.21.15.28:0/4226352112 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555ca3550580 con 0x555ca35f4030 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7db9ffb640 10 client.17214 handle_command_reply: tid=0 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 client.17214 shutdown 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 msgr2=0x555ca35ef970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40cd7d0 tx=0x7f7dc4009220 comp rx=0 tx=0).stop 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 shutdown_connections 2024-03-22T02:19:30.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35f4030 0x555ca35f6420 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35e4040 0x555ca35ef970 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35deed0 0x555ca35e89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7db9ffb640 0 client.17214 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/4226352112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35e8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 >> 172.21.15.28:0/4226352112 conn(0x555ca35d9a50 msgr2=0x555ca35d8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 shutdown_connections 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/4226352112 wait complete. 2024-03-22T02:19:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.325+0000 7f7ddbbb8740 20 client.17214 trim_cache size 0 max 16384 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 Processor -- start 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 -- start start 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35d8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35deed0 0x555ca35fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41af3f0 con 0x555ca35de4e0 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41accc0 con 0x555ca35d9a50 2024-03-22T02:19:30.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41aab10 con 0x555ca35deed0 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7dd1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7dd259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35d8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7dd1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44326/0 (socket says 172.21.15.28:44326) 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7dd1d9b640 1 -- 172.21.15.28:0/2095498072 learned_addr learned my addr 172.21.15.28:0/2095498072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.340+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35deed0 0x555ca35fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 -- 172.21.15.28:0/2095498072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35deed0 msgr2=0x555ca35fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35deed0 0x555ca35fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 -- 172.21.15.28:0/2095498072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 msgr2=0x555ca35ef930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 -- 172.21.15.28:0/2095498072 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ca35f69b0 con 0x555ca35d9a50 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:30.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dd259c640 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35d8ad0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0096c0 tx=0x7f7dcc005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=275169f986c3fa40 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dbbfff640 1 -- 172.21.15.28:0/2095498072 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc04b070 con 0x555ca35d9a50 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dbbfff640 1 -- 172.21.15.28:0/2095498072 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dcc046440 con 0x555ca35d9a50 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7dbbfff640 1 -- 172.21.15.28:0/2095498072 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc005040 con 0x555ca35d9a50 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/2095498072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 msgr2=0x555ca35d8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.341+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35d8ad0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0096c0 tx=0x7f7dcc005fd0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/2095498072 shutdown_connections 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35deed0 0x555ca35fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35de4e0 0x555ca35ef930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/2095498072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35d8ad0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/2095498072 >> 172.21.15.28:0/2095498072 conn(0x555ca35e8020 msgr2=0x555ca35d83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/2095498072 shutdown_connections 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/2095498072 wait complete. 2024-03-22T02:19:30.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- start start 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 Processor -- start 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41ab180 con 0x555ca35deed0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41ac5f0 con 0x555ca35d9a50 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd41ab850 con 0x555ca35de4e0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7ddbbb8740 10 client.0 _ll_register_callbacks cb 0x7f7dc4098330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44334/0 (socket says 172.21.15.28:44334) 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/3309213018 learned_addr learned my addr 172.21.15.28:0/3309213018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd1d9b640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd259c640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/3309213018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 msgr2=0x555ca35edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35edbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 msgr2=0x555ca35ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35ed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.342+0000 7f7dd2d9d640 1 -- 172.21.15.28:0/3309213018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ca35ef1f0 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7dd2d9d640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4005680 tx=0x7f7dc400cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f293122860d1c42a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40b4070 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7dc409b790 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dcc046440 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7dc409bbd0 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 10 client.17220 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555ca365a8c0 con 0x555ca35deed0 2024-03-22T02:19:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dcc046440 con 0x555ca35deed0 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f7dc40a4610 con 0x555ca35deed0 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 10 client.17220 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 10 client.17220 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 20 client.17220 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35e4040 0x555ca365d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 4 client.17220 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ca354a8d0 con 0x555ca35e4040 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.343+0000 7f7dd259c640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35e4040 0x555ca365d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.344+0000 7f7dd259c640 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35e4040 0x555ca365d080 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e4682f7756734cf server_cookie=54b4a0056a1a6f93 in_seq=0 out_seq=0 2024-03-22T02:19:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.344+0000 7f7db9ffb640 10 client.17220 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:30.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 488456 12.06 MB/sec execute 132 sec latency 59.152 ms 2024-03-22T02:19:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:30 smithi028 ceph-mon[31916]: pgmap v536: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:19:30.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:30 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.676+0000 7f7db9ffb640 1 -- 172.21.15.28:0/3309213018 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ca354a8d0 con 0x555ca35e4040 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7db9ffb640 10 client.17220 handle_command_reply: tid=0 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 client.17220 shutdown 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 msgr2=0x555ca35eec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4005680 tx=0x7f7dc400cee0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 shutdown_connections 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555ca35e4040 0x555ca365d080 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555ca35deed0 0x555ca35eec00 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555ca35de4e0 0x555ca35edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7db9ffb640 0 client.17220 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 --2- 172.21.15.28:0/3309213018 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555ca35d9a50 0x555ca35ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 >> 172.21.15.28:0/3309213018 conn(0x555ca35e8020 msgr2=0x555ca35d8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 shutdown_connections 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 1 -- 172.21.15.28:0/3309213018 wait complete. 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.677+0000 7f7ddbbb8740 20 client.17220 trim_cache size 0 max 16384 2024-03-22T02:19:30.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:19:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:19:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:19:30.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 msgr2=0x7f7dc00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 0x7f7dc00a5480 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009fd0 tx=0x7f7dbc009b80 comp rx=0 tx=0).stop 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 msgr2=0x7f7dd40744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400eb10 tx=0x7f7dc400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 shutdown_connections 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7dc00a2fc0 0x7f7dc00a5480 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dd4103790 0x7f7dd40744c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7dd4102dd0 0x7f7dd4073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 --2- 172.21.15.28:0/1378717005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7dd4101be0 0x7f7dd4075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 >> 172.21.15.28:0/1378717005 conn(0x7f7dd40fd150 msgr2=0x7f7dd4106070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 shutdown_connections 2024-03-22T02:19:30.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:30.678+0000 7f7dd8fd0640 1 -- 172.21.15.28:0/1378717005 wait complete. 2024-03-22T02:19:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:30 smithi120 ceph-mon[36803]: pgmap v536: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:19:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:30 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:19:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:30 smithi083 ceph-mon[36340]: pgmap v536: 129 pgs: 129 active+clean; 33 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:19:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:30 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:19:31.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 490579 12.07 MB/sec execute 133 sec latency 118.237 ms 2024-03-22T02:19:31.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:31 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:19:31.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:31 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:19:31.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:31 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:19:32.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 493228 12.08 MB/sec execute 134 sec latency 34.195 ms 2024-03-22T02:19:32.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:32 smithi028 ceph-mon[31916]: pgmap v537: 129 pgs: 129 active+clean; 35 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:19:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:32 smithi120 ceph-mon[36803]: pgmap v537: 129 pgs: 129 active+clean; 35 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:19:32.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:32 smithi083 ceph-mon[36340]: pgmap v537: 129 pgs: 129 active+clean; 35 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-03-22T02:19:33.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 495277 12.08 MB/sec execute 135 sec latency 29.436 ms 2024-03-22T02:19:34.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 498002 12.09 MB/sec execute 136 sec latency 9.378 ms 2024-03-22T02:19:34.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:34 smithi028 ceph-mon[31916]: pgmap v538: 129 pgs: 129 active+clean; 34 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-22T02:19:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:34 smithi120 ceph-mon[36803]: pgmap v538: 129 pgs: 129 active+clean; 34 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-22T02:19:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:34 smithi083 ceph-mon[36340]: pgmap v538: 129 pgs: 129 active+clean; 34 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-03-22T02:19:35.418 INFO:tasks.workunit.client.0.smithi028.stdout: 1 500738 12.11 MB/sec execute 137 sec latency 80.411 ms 2024-03-22T02:19:36.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 502343 12.09 MB/sec execute 138 sec latency 96.763 ms 2024-03-22T02:19:36.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:36 smithi028 ceph-mon[31916]: pgmap v539: 129 pgs: 129 active+clean; 33 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.6 MiB/s wr, 450 op/s 2024-03-22T02:19:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:36 smithi120 ceph-mon[36803]: pgmap v539: 129 pgs: 129 active+clean; 33 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.6 MiB/s wr, 450 op/s 2024-03-22T02:19:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:36 smithi083 ceph-mon[36340]: pgmap v539: 129 pgs: 129 active+clean; 33 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.6 MiB/s wr, 450 op/s 2024-03-22T02:19:37.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 505017 12.11 MB/sec execute 139 sec latency 12.233 ms 2024-03-22T02:19:38.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 508016 12.14 MB/sec execute 140 sec latency 28.582 ms 2024-03-22T02:19:38.550 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:38 smithi120 ceph-mon[36803]: pgmap v540: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.7 MiB/s wr, 405 op/s 2024-03-22T02:19:38.566 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:38 smithi083 ceph-mon[36340]: pgmap v540: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.7 MiB/s wr, 405 op/s 2024-03-22T02:19:38.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:38 smithi028 ceph-mon[31916]: pgmap v540: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.7 MiB/s wr, 405 op/s 2024-03-22T02:19:39.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 510216 12.11 MB/sec execute 141 sec latency 27.812 ms 2024-03-22T02:19:40.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 512325 12.12 MB/sec execute 142 sec latency 30.233 ms 2024-03-22T02:19:40.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:40 smithi028 ceph-mon[31916]: pgmap v541: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:40 smithi120 ceph-mon[36803]: pgmap v541: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:40 smithi083 ceph-mon[36340]: pgmap v541: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 3.1 MiB/s wr, 416 op/s 2024-03-22T02:19:41.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 514513 12.12 MB/sec execute 143 sec latency 56.598 ms 2024-03-22T02:19:42.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 516408 12.10 MB/sec execute 144 sec latency 28.700 ms 2024-03-22T02:19:42.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:42 smithi028 ceph-mon[31916]: pgmap v542: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 3.3 MiB/s wr, 457 op/s 2024-03-22T02:19:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:42 smithi120 ceph-mon[36803]: pgmap v542: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 3.3 MiB/s wr, 457 op/s 2024-03-22T02:19:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:42 smithi083 ceph-mon[36340]: pgmap v542: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 3.3 MiB/s wr, 457 op/s 2024-03-22T02:19:43.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 519119 12.13 MB/sec execute 145 sec latency 17.070 ms 2024-03-22T02:19:43.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/1229804889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4100ab0 msgr2=0x7fa0c4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/1229804889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4100ed0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009900 tx=0x7fa0bc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/1229804889 shutdown_connections 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/1229804889 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c4108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/1229804889 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/1229804889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4100ed0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/1229804889 >> 172.21.15.28:0/1229804889 conn(0x7fa0c40fc020 msgr2=0x7fa0c40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/1229804889 shutdown_connections 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/1229804889 wait complete. 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 Processor -- start 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.773+0000 7fa0cbf8a640 1 -- start start 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c410c770 con 0x7fa0c4101ca0 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c410c5f0 con 0x7fa0c4102660 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c410c8f0 con 0x7fa0c4100ab0 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35042/0 (socket says 172.21.15.28:35042) 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 -- 172.21.15.28:0/2267296583 learned_addr learned my addr 172.21.15.28:0/2267296583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0ca527640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 -- 172.21.15.28:0/2267296583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4100ab0 msgr2=0x7fa0c4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 -- 172.21.15.28:0/2267296583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 msgr2=0x7fa0c419f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c419f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 -- 172.21.15.28:0/2267296583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0c419fab0 con 0x7fa0c4101ca0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0ca527640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c9525640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c2ffd640 1 -- 172.21.15.28:0/2267296583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800cd80 con 0x7fa0c4101ca0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c2ffd640 1 -- 172.21.15.28:0/2267296583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0b8005500 con 0x7fa0c4101ca0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0c419fda0 con 0x7fa0c4101ca0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.774+0000 7fa0c2ffd640 1 -- 172.21.15.28:0/2267296583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8010690 con 0x7fa0c4101ca0 2024-03-22T02:19:43.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.775+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0c41a0210 con 0x7fa0c4101ca0 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.777+0000 7fa0c2ffd640 1 -- 172.21.15.28:0/2267296583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa0b80046c0 con 0x7fa0c4101ca0 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.777+0000 7fa0c2ffd640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 0x7fa0ac0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.777+0000 7fa0ceb74740 1 Processor -- start 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.777+0000 7fa0c2ffd640 1 -- 172.21.15.28:0/2267296583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa0b80cfbc0 con 0x7fa0c4101ca0 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.777+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 0x7fa0ac0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 -- start start 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc4b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d36bbb0040 con 0x55d36bc51040 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d36bb589e0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d36bbabff0 con 0x55d36bc4bed0 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ca527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc4b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9525640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9525640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc509d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34434/0 (socket says 172.21.15.28:34434) 2024-03-22T02:19:43.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0ca527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc534d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35052/0 (socket says 172.21.15.28:35052) 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9525640 1 -- 172.21.15.28:0/1381896 learned_addr learned my addr 172.21.15.28:0/1381896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 0x7fa0ac0a5480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02c170 tx=0x7fa0bc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 -- 172.21.15.28:0/1381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 msgr2=0x55d36bc509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 -- 172.21.15.28:0/1381896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 msgr2=0x55d36bc534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 -- 172.21.15.28:0/1381896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36b8c4210 con 0x55d36bc4b4e0 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.778+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc4b900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009230 tx=0x7fa0bc044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafd1f3334257e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0abfff640 1 -- 172.21.15.28:0/1381896 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc004040 con 0x55d36bc4b4e0 2024-03-22T02:19:43.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0abfff640 1 -- 172.21.15.28:0/1381896 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc005530 con 0x55d36bc4b4e0 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0abfff640 1 -- 172.21.15.28:0/1381896 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc004970 con 0x55d36bc4b4e0 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 -- 172.21.15.28:0/1381896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 msgr2=0x55d36bc4b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc4b900 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc009230 tx=0x7fa0bc044000 comp rx=0 tx=0).stop 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 -- 172.21.15.28:0/1381896 shutdown_connections 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/1381896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc4b900 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 -- 172.21.15.28:0/1381896 >> 172.21.15.28:0/1381896 conn(0x55d36bc46a50 msgr2=0x55d36bc46e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 -- 172.21.15.28:0/1381896 shutdown_connections 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.779+0000 7fa0ceb74740 1 -- 172.21.15.28:0/1381896 wait complete. 2024-03-22T02:19:43.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 -- start start 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 Processor -- start 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc55a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc5c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d36bc53f90 con 0x55d36bc51040 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d36bb6bdb0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d36bc54110 con 0x55d36bc4bed0 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ceb74740 10 client.0 _ll_register_callbacks cb 0x55d36b8c3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0ca527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc5c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:58964/0 (socket says 172.21.15.28:58964) 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0c9525640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc55a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.780+0000 7fa0c9d26640 1 -- 172.21.15.28:0/3494916207 learned_addr learned my addr 172.21.15.28:0/3494916207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 -- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 msgr2=0x55d36bc554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 -- 172.21.15.28:0/3494916207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 msgr2=0x55d36bc5c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc5c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 -- 172.21.15.28:0/3494916207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36b990620 con 0x55d36bc4bed0 2024-03-22T02:19:43.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0ca527640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc5c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0c9525640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc55a20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8098340 tx=0x7fa0b809c5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e836040443aef57b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0b80b4020 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0bc005530 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa0b80a4460 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa0b80a48f0 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0ceb74740 10 client.27067 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.781+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d36bc5fe80 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc005530 con 0x55d36bc4bed0 2024-03-22T02:19:43.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fa0b809b3c0 con 0x55d36bc4bed0 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 10 client.27067 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 10 client.27067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 20 client.27067 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bc615c0 0x55d36bc639b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 4 client.27067 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d36bbbd580 con 0x55d36bc615c0 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bc615c0 0x55d36bc639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0a97fa640 10 client.27067 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.782+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bc615c0 0x55d36bc639b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a1851651caee4ac server_cookie=aeaefbe3897cfaa2 in_seq=0 out_seq=0 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.783+0000 7fa0a97fa640 1 -- 172.21.15.28:0/3494916207 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d36bbbd580 con 0x55d36bc615c0 2024-03-22T02:19:43.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.783+0000 7fa0a97fa640 10 client.27067 handle_command_reply: tid=0 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.783+0000 7fa0ceb74740 1 client.27067 shutdown 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 msgr2=0x55d36bc55a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc55a20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8098340 tx=0x7fa0b809c5e0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 shutdown_connections 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bc615c0 0x55d36bc639b0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc51040 0x55d36bc5c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc55a20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/3494916207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4b4e0 0x55d36bc554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0a97fa640 0 client.27067 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 >> 172.21.15.28:0/3494916207 conn(0x55d36bc46a50 msgr2=0x55d36bc45c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 shutdown_connections 2024-03-22T02:19:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 1 -- 172.21.15.28:0/3494916207 wait complete. 2024-03-22T02:19:43.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.784+0000 7fa0ceb74740 20 client.27067 trim_cache size 0 max 16384 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 Processor -- start 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 -- start start 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc5c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc51040 0x55d36bc4a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41ad150 con 0x55d36bc4b4e0 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41aaa20 con 0x55d36bc51040 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.799+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41a8870 con 0x55d36bc4bed0 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35070/0 (socket says 172.21.15.28:35070) 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0ca527640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc51040 0x55d36bc4a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 -- 172.21.15.28:0/4278013432 learned_addr learned my addr 172.21.15.28:0/4278013432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9525640 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc5c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 -- 172.21.15.28:0/4278013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 msgr2=0x55d36bc5c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc5c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 -- 172.21.15.28:0/4278013432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc51040 msgr2=0x55d36bc4a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc51040 0x55d36bc4a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 -- 172.21.15.28:0/4278013432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d36bc64030 con 0x55d36bc4b4e0 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02c240 tx=0x7fa0bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ec069435985949 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0ab7fe640 1 -- 172.21.15.28:0/4278013432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc05b070 con 0x55d36bc4b4e0 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0ab7fe640 1 -- 172.21.15.28:0/4278013432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc02cd00 con 0x55d36bc4b4e0 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.800+0000 7fa0ab7fe640 1 -- 172.21.15.28:0/4278013432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc004730 con 0x55d36bc4b4e0 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- 172.21.15.28:0/4278013432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 msgr2=0x55d36bc49cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02c240 tx=0x7fa0bc0046c0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- 172.21.15.28:0/4278013432 shutdown_connections 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc51040 0x55d36bc4a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc4bed0 0x55d36bc5c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/4278013432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc49cb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- 172.21.15.28:0/4278013432 >> 172.21.15.28:0/4278013432 conn(0x55d36bc46a50 msgr2=0x55d36bc554e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- 172.21.15.28:0/4278013432 shutdown_connections 2024-03-22T02:19:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- 172.21.15.28:0/4278013432 wait complete. 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 -- start start 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.801+0000 7fa0ceb74740 1 Processor -- start 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4bed0 0x55d36bc59c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 0x55d36bcc58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41a8ee0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41aa350 con 0x55d36bc4bed0 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa0c41a95b0 con 0x55d36bc51040 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ceb74740 10 client.0 _ll_register_callbacks cb 0x7fa0b809ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35076/0 (socket says 172.21.15.28:35076) 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 -- 172.21.15.28:0/659191672 learned_addr learned my addr 172.21.15.28:0/659191672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ca527640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 0x55d36bcc58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9525640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4bed0 0x55d36bc59c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 -- 172.21.15.28:0/659191672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 msgr2=0x55d36bcc58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 0x55d36bcc58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 -- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4bed0 msgr2=0x55d36bc59c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4bed0 0x55d36bc59c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 -- 172.21.15.28:0/659191672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36bc5a240 con 0x55d36bc4b4e0 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0ca527640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 0x55d36bcc58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0c9d26640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc005d70 tx=0x7fa0bc05c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6c543b666b8cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc05b070 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.802+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa0bc0054e0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0bc02cd00 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa0bc0040c0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 10 client.17238 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d36bcc6ea0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0bc02cd00 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fa0bc0035f0 con 0x55d36bc4b4e0 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 10 client.17238 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 10 client.17238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 20 client.17238 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bcc74b0 0x55d36bcc98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 4 client.17238 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d36bbb78d0 con 0x55d36bcc74b0 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.803+0000 7fa0c9525640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bcc74b0 0x55d36bcc98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.804+0000 7fa0a97fa640 10 client.17238 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:43.804+0000 7fa0c9525640 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bcc74b0 0x55d36bcc98d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35d6b54b6ed6c4d9 server_cookie=941c3de4c3d876aa in_seq=0 out_seq=0 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.072+0000 7fa0a97fa640 1 -- 172.21.15.28:0/659191672 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d36bbb78d0 con 0x55d36bcc74b0 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.072+0000 7fa0a97fa640 10 client.17238 handle_command_reply: tid=0 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.072+0000 7fa0ceb74740 1 client.17238 shutdown 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 msgr2=0x55d36bc59710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc005d70 tx=0x7fa0bc05c270 comp rx=0 tx=0).stop 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 shutdown_connections 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d36bcc74b0 0x55d36bcc98d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d36bc51040 0x55d36bcc58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0a97fa640 0 client.17238 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d36bc4bed0 0x55d36bc59c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 --2- 172.21.15.28:0/659191672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d36bc4b4e0 0x55d36bc59710 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 >> 172.21.15.28:0/659191672 conn(0x55d36bc46a50 msgr2=0x55d36bc50140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 shutdown_connections 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 1 -- 172.21.15.28:0/659191672 wait complete. 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.073+0000 7fa0ceb74740 20 client.17238 trim_cache size 0 max 16384 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:19:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:19:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:19:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 msgr2=0x7fa0ac0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 0x7fa0ac0a5480 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc02c170 tx=0x7fa0bc02df90 comp rx=0 tx=0).stop 2024-03-22T02:19:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 msgr2=0x7fa0c4198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fa0b800e9f0 tx=0x7fa0b800edf0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 shutdown_connections 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa0ac0a2fc0 0x7fa0ac0a5480 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa0c4102660 0x7fa0c419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0c4101ca0 0x7fa0c4198420 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 --2- 172.21.15.28:0/2267296583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa0c4100ab0 0x7fa0c4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 >> 172.21.15.28:0/2267296583 conn(0x7fa0c40fc020 msgr2=0x7fa0c40fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 shutdown_connections 2024-03-22T02:19:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:44.074+0000 7fa0cbf8a640 1 -- 172.21.15.28:0/2267296583 wait complete. 2024-03-22T02:19:44.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 521439 12.12 MB/sec execute 146 sec latency 113.613 ms 2024-03-22T02:19:44.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:44 smithi028 ceph-mon[31916]: pgmap v543: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:44 smithi120 ceph-mon[36803]: pgmap v543: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:44 smithi083 ceph-mon[36340]: pgmap v543: 129 pgs: 129 active+clean; 30 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-03-22T02:19:45.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 523368 12.11 MB/sec execute 147 sec latency 34.001 ms 2024-03-22T02:19:46.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 525869 12.14 MB/sec execute 148 sec latency 103.334 ms 2024-03-22T02:19:46.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:46 smithi028 ceph-mon[31916]: pgmap v544: 129 pgs: 129 active+clean; 32 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-22T02:19:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:46 smithi120 ceph-mon[36803]: pgmap v544: 129 pgs: 129 active+clean; 32 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-22T02:19:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:46 smithi083 ceph-mon[36340]: pgmap v544: 129 pgs: 129 active+clean; 32 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.7 MiB/s wr, 450 op/s 2024-03-22T02:19:47.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 528159 12.11 MB/sec execute 149 sec latency 44.064 ms 2024-03-22T02:19:48.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 530181 12.11 MB/sec execute 150 sec latency 30.930 ms 2024-03-22T02:19:48.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:48 smithi028 ceph-mon[31916]: pgmap v545: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-03-22T02:19:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:48 smithi120 ceph-mon[36803]: pgmap v545: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-03-22T02:19:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:48 smithi083 ceph-mon[36340]: pgmap v545: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 2.9 MiB/s wr, 315 op/s 2024-03-22T02:19:49.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 533113 12.15 MB/sec execute 151 sec latency 29.426 ms 2024-03-22T02:19:50.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 535385 12.13 MB/sec execute 152 sec latency 12.194 ms 2024-03-22T02:19:50.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:50 smithi028 ceph-mon[31916]: pgmap v546: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 3.8 MiB/s wr, 459 op/s 2024-03-22T02:19:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:50 smithi120 ceph-mon[36803]: pgmap v546: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 3.8 MiB/s wr, 459 op/s 2024-03-22T02:19:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:50 smithi083 ceph-mon[36340]: pgmap v546: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 3.8 MiB/s wr, 459 op/s 2024-03-22T02:19:51.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 536887 12.12 MB/sec execute 153 sec latency 103.316 ms 2024-03-22T02:19:52.365 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:19:52.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:19:52.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 539651 12.13 MB/sec execute 154 sec latency 29.831 ms 2024-03-22T02:19:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- 172.21.15.28:0/1434817556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 msgr2=0x7f2c18108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/1434817556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c18108ed0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f2c08009860 tx=0x7f2c0802bb70 comp rx=0 tx=0).stop 2024-03-22T02:19:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- 172.21.15.28:0/1434817556 shutdown_connections 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/1434817556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c18108ed0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/1434817556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18101c80 0x7f2c18102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/1434817556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18100a90 0x7f2c18100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- 172.21.15.28:0/1434817556 >> 172.21.15.28:0/1434817556 conn(0x7f2c180fc020 msgr2=0x7f2c180fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- 172.21.15.28:0/1434817556 shutdown_connections 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- 172.21.15.28:0/1434817556 wait complete. 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 Processor -- start 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.478+0000 7f2c1ec42640 1 -- start start 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18101c80 0x7f2c181983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c1819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1810c6f0 con 0x7f2c18102640 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1810c570 con 0x7f2c18100a90 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1810c870 con 0x7f2c18101c80 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c1819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46700/0 (socket says 172.21.15.28:46700) 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 -- 172.21.15.28:0/292953089 learned_addr learned my addr 172.21.15.28:0/292953089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c1819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41058/0 (socket says 172.21.15.28:41058) 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c17fff640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18101c80 0x7f2c181983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 -- 172.21.15.28:0/292953089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18101c80 msgr2=0x7f2c181983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18101c80 0x7f2c181983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 -- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 msgr2=0x7f2c1819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c1819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 -- 172.21.15.28:0/292953089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1819faa0 con 0x7f2c18100a90 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c3f0 tx=0x7f2c10009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c15ffb640 1 -- 172.21.15.28:0/292953089 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1001b070 con 0x7f2c18100a90 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.479+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1819fd90 con 0x7f2c18100a90 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.480+0000 7f2c15ffb640 1 -- 172.21.15.28:0/292953089 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c1000bb10 con 0x7f2c18100a90 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.480+0000 7f2c15ffb640 1 -- 172.21.15.28:0/292953089 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c10016d80 con 0x7f2c18100a90 2024-03-22T02:19:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.480+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c181a0250 con 0x7f2c18100a90 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c15ffb640 1 -- 172.21.15.28:0/292953089 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2c1001e070 con 0x7f2c18100a90 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c2181f740 1 Processor -- start 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c2181f740 1 -- start start 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c15ffb640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 0x7f2c000a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.482+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bb8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 0x7f2c000a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c2181f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648a5b188c0 con 0x5648a5bb8f90 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c2181f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648a5ac0c20 con 0x5648a5bb3e20 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c2181f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648a5b14870 con 0x5648a5bb3430 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c15ffb640 1 -- 172.21.15.28:0/292953089 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2c100cf5d0 con 0x7f2c18100a90 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40384/0 (socket says 172.21.15.28:40384) 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1c9de640 1 -- 172.21.15.28:0/1712849919 learned_addr learned my addr 172.21.15.28:0/1712849919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41070/0 (socket says 172.21.15.28:41070) 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bb8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 -- 172.21.15.28:0/1712849919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 msgr2=0x5648a5bb3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 -- 172.21.15.28:0/1712849919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 msgr2=0x5648a5bbb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 -- 172.21.15.28:0/1712849919 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a5ae5150 con 0x5648a5bb3e20 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 0x7f2c000a5460 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2c08005940 tx=0x7f2c08032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2c17fff640 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bb8920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009bf0 tx=0x7f2c0c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dad30005ba0484af server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2bfeffd640 1 -- 172.21.15.28:0/1712849919 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c004050 con 0x5648a5bb3e20 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2bfeffd640 1 -- 172.21.15.28:0/1712849919 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c005500 con 0x5648a5bb3e20 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.483+0000 7f2bfeffd640 1 -- 172.21.15.28:0/1712849919 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c004dd0 con 0x5648a5bb3e20 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 -- 172.21.15.28:0/1712849919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 msgr2=0x5648a5bb8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bb8920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009bf0 tx=0x7f2c0c011810 comp rx=0 tx=0).stop 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 -- 172.21.15.28:0/1712849919 shutdown_connections 2024-03-22T02:19:52.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bbb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bb8920 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1712849919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bb3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 -- 172.21.15.28:0/1712849919 >> 172.21.15.28:0/1712849919 conn(0x5648a5bae9a0 msgr2=0x5648a5baeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 -- 172.21.15.28:0/1712849919 shutdown_connections 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.484+0000 7f2c2181f740 1 -- 172.21.15.28:0/1712849919 wait complete. 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 -- start start 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 Processor -- start 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 0x5648a5bbd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 0x5648a5bbdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb8f90 0x5648a5bc4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648a5bbbf40 con 0x5648a5bb3e20 2024-03-22T02:19:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648a5ad4130 con 0x5648a5bb3430 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648a5bbc0c0 con 0x5648a5bb8f90 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c2181f740 10 client.0 _ll_register_callbacks cb 0x5648a5ad3b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 0x5648a5bbdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 0x5648a5bbd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 0x5648a5bbdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41082/0 (socket says 172.21.15.28:41082) 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c17fff640 1 -- 172.21.15.28:0/1301531904 learned_addr learned my addr 172.21.15.28:0/1301531904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb8f90 0x5648a5bc4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 -- 172.21.15.28:0/1301531904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb8f90 msgr2=0x5648a5bc4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb8f90 0x5648a5bc4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 -- 172.21.15.28:0/1301531904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 msgr2=0x5648a5bbdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 0x5648a5bbdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.485+0000 7f2c1c9de640 1 -- 172.21.15.28:0/1301531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a58f67d0 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 0x5648a5bbd960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c1009b930 tx=0x7f2c10005550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310ebece39e7472a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c100a4300 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c0c005500 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2c10009890 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2c10082050 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 10 client.27080 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5648a582c3f0 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c005500 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f2c1000b040 con 0x5648a5bb3430 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 10 client.27080 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:52.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 10 client.27080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 20 client.27080 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5bc9700 0x5648a5bcbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 4 client.27080 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648a5b25e00 con 0x5648a5bc9700 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.486+0000 7f2c17fff640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5bc9700 0x5648a5bcbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.487+0000 7f2bf7fff640 10 client.27080 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.487+0000 7f2c17fff640 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5bc9700 0x5648a5bcbb70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=470d4681d4ebf569 server_cookie=e6a5e6de111c14e5 in_seq=0 out_seq=0 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.487+0000 7f2bf7fff640 1 -- 172.21.15.28:0/1301531904 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648a5b25e00 con 0x5648a5bc9700 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.487+0000 7f2bf7fff640 10 client.27080 handle_command_reply: tid=0 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.487+0000 7f2c2181f740 1 client.27080 shutdown 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 msgr2=0x5648a5bbd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 0x5648a5bbd960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2c1009b930 tx=0x7f2c10005550 comp rx=0 tx=0).stop 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 shutdown_connections 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5bc9700 0x5648a5bcbb70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb8f90 0x5648a5bc4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2bf7fff640 0 client.27080 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3e20 0x5648a5bbdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 --2- 172.21.15.28:0/1301531904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3430 0x5648a5bbd960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 >> 172.21.15.28:0/1301531904 conn(0x5648a5bae9a0 msgr2=0x5648a5badc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 shutdown_connections 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 1 -- 172.21.15.28:0/1301531904 wait complete. 2024-03-22T02:19:52.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.488+0000 7f2c2181f740 20 client.27080 trim_cache size 0 max 16384 2024-03-22T02:19:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 Processor -- start 2024-03-22T02:19:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 -- start start 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 0x5648a5bad370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181ab090 con 0x5648a5bb3430 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181a8960 con 0x5648a5bb8f90 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c2181f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181a67b0 con 0x5648a5bb3e20 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40398/0 (socket says 172.21.15.28:40398) 2024-03-22T02:19:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 0x5648a5bad370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41096/0 (socket says 172.21.15.28:41096) 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 -- 172.21.15.28:0/338925355 learned_addr learned my addr 172.21.15.28:0/338925355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 -- 172.21.15.28:0/338925355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 msgr2=0x5648a5bad370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 0x5648a5bad370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 -- 172.21.15.28:0/338925355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 msgr2=0x5648a5bbdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c17fff640 1 -- 172.21.15.28:0/338925355 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a57ec0f0 con 0x5648a5bb3e20 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 0x5648a5bad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.506+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.507+0000 7f2c17fff640 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c004600 tx=0x7f2c0c004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90ec7346e382d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.507+0000 7f2bfe7fc640 1 -- 172.21.15.28:0/338925355 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c02ecb0 con 0x5648a5bb3e20 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.507+0000 7f2bfe7fc640 1 -- 172.21.15.28:0/338925355 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c02ee50 con 0x5648a5bb3e20 2024-03-22T02:19:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.507+0000 7f2bfe7fc640 1 -- 172.21.15.28:0/338925355 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c019a30 con 0x5648a5bb3e20 2024-03-22T02:19:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- 172.21.15.28:0/338925355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 msgr2=0x5648a5bc4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c004600 tx=0x7f2c0c004500 comp rx=0 tx=0).stop 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- 172.21.15.28:0/338925355 shutdown_connections 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb8f90 0x5648a5bad370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3e20 0x5648a5bc4e00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- 172.21.15.28:0/338925355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb3430 0x5648a5bbdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- 172.21.15.28:0/338925355 >> 172.21.15.28:0/338925355 conn(0x5648a5bae9a0 msgr2=0x5648a5bbd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- 172.21.15.28:0/338925355 shutdown_connections 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- 172.21.15.28:0/338925355 wait complete. 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- start start 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 Processor -- start 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bc2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181a6e20 con 0x5648a5bb8f90 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181a8290 con 0x5648a5bb3e20 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2c181a74f0 con 0x5648a5bb3430 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.508+0000 7f2c2181f740 10 client.0 _ll_register_callbacks cb 0x7f2c1009b3d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41112/0 (socket says 172.21.15.28:41112) 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 -- 172.21.15.28:0/519630552 learned_addr learned my addr 172.21.15.28:0/519630552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c17fff640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bc2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1c9de640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40414/0 (socket says 172.21.15.28:40414) 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 -- 172.21.15.28:0/519630552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 msgr2=0x5648a5bc2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 -- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 msgr2=0x5648a5bc2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bc2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 -- 172.21.15.28:0/519630552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a5bc4090 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c1d1df640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2c080349e0 tx=0x7f2c08046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130530b1276e50d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2bf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08057060 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2c08032640 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c0c02ee50 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2c0805e050 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c2181f740 10 client.17256 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.509+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5648a5c2fb40 con 0x5648a5bb8f90 2024-03-22T02:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c0c02ee50 con 0x5648a5bb8f90 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f2c0803f040 con 0x5648a5bb8f90 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 10 client.17256 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 20 client.17256 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5c30570 0x5648a5c32990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 4 client.17256 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5648a5b20150 con 0x5648a5c30570 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5c30570 0x5648a5c32990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2c1c9de640 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5c30570 0x5648a5c32990 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a680f25e202938 server_cookie=44a976a710b50e4d in_seq=0 out_seq=0 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.510+0000 7f2bf7fff640 10 client.17256 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2bf7fff640 1 -- 172.21.15.28:0/519630552 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5648a5b20150 con 0x5648a5c30570 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2bf7fff640 10 client.17256 handle_command_reply: tid=0 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 client.17256 shutdown 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 msgr2=0x5648a5bc3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f2c080349e0 tx=0x7f2c08046040 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 shutdown_connections 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648a5c30570 0x5648a5c32990 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648a5bb8f90 0x5648a5bc3aa0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648a5bb3e20 0x5648a5bc2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2bf7fff640 0 client.17256 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 --2- 172.21.15.28:0/519630552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648a5bb3430 0x5648a5bc2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 >> 172.21.15.28:0/519630552 conn(0x5648a5bae9a0 msgr2=0x5648a5bb1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 shutdown_connections 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 1 -- 172.21.15.28:0/519630552 wait complete. 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.511+0000 7f2c2181f740 20 client.17256 trim_cache size 0 max 16384 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:19:52.521 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 msgr2=0x7f2c000a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 0x7f2c000a5460 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2c08005940 tx=0x7f2c08032340 comp rx=0 tx=0).stop 2024-03-22T02:19:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 msgr2=0x7f2c18197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2c1000c3f0 tx=0x7f2c10009f90 comp rx=0 tx=0).stop 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 shutdown_connections 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2c000a2fa0 0x7f2c000a5460 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c18102640 0x7f2c1819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2c18101c80 0x7f2c181983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 --2- 172.21.15.28:0/292953089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2c18100a90 0x7f2c18197e80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 >> 172.21.15.28:0/292953089 conn(0x7f2c180fc020 msgr2=0x7f2c180fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 shutdown_connections 2024-03-22T02:19:52.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.512+0000 7f2c1ec42640 1 -- 172.21.15.28:0/292953089 wait complete. 2024-03-22T02:19:52.525 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:19:52.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:19:52.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:52 smithi028 ceph-mon[31916]: pgmap v547: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.4 MiB/s wr, 430 op/s 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 -- 172.21.15.28:0/3153383289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 msgr2=0x7fd444108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 --2- 172.21.15.28:0/3153383289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd444108ef0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009ca0 tx=0x7fd43c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 -- 172.21.15.28:0/3153383289 shutdown_connections 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 --2- 172.21.15.28:0/3153383289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd444108ef0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 --2- 172.21.15.28:0/3153383289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 0x7fd444102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 --2- 172.21.15.28:0/3153383289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 -- 172.21.15.28:0/3153383289 >> 172.21.15.28:0/3153383289 conn(0x7fd4440fc020 msgr2=0x7fd4440fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 -- 172.21.15.28:0/3153383289 shutdown_connections 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.631+0000 7fd449f81640 1 -- 172.21.15.28:0/3153383289 wait complete. 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 Processor -- start 2024-03-22T02:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 -- start start 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 0x7fd4441983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd44410c730 con 0x7fd444102660 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd44410c5b0 con 0x7fd444101ca0 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd449f81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd44410c8b0 con 0x7fd444100ab0 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd4437fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd443fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd442ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 0x7fd4441983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd443fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41130/0 (socket says 172.21.15.28:41130) 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.632+0000 7fd443fff640 1 -- 172.21.15.28:0/4271394753 learned_addr learned my addr 172.21.15.28:0/4271394753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 -- 172.21.15.28:0/4271394753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 msgr2=0x7fd444197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 -- 172.21.15.28:0/4271394753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 msgr2=0x7fd4441983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 0x7fd4441983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 -- 172.21.15.28:0/4271394753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44419fab0 con 0x7fd444102660 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd4437fe640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd443fff640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd43c037040 tx=0x7fd43c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c041020 con 0x7fd444102660 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44419fd40 con 0x7fd444102660 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd43c034cd0 con 0x7fd444102660 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43c039040 con 0x7fd444102660 2024-03-22T02:19:52.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.633+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4441a01d0 con 0x7fd444102660 2024-03-22T02:19:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.634+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4441a0400 con 0x7fd444102660 2024-03-22T02:19:52.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.635+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd43c044020 con 0x7fd444102660 2024-03-22T02:19:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.636+0000 7fd440ff9640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 0x7fd42c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.636+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fd43c0f1a00 con 0x7fd444102660 2024-03-22T02:19:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.636+0000 7fd4437fe640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 0x7fd42c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:52.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.636+0000 7fd4437fe640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 0x7fd42c0a53e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd444101b00 tx=0x7fd434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:52.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.640+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd43c04b020 con 0x7fd444102660 2024-03-22T02:19:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:52 smithi120 ceph-mon[36803]: pgmap v547: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.4 MiB/s wr, 430 op/s 2024-03-22T02:19:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:52 smithi083 ceph-mon[36340]: pgmap v547: 129 pgs: 129 active+clean; 28 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 3.4 MiB/s wr, 430 op/s 2024-03-22T02:19:52.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.869+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4441a0790 con 0x7fd444102660 2024-03-22T02:19:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.870+0000 7fd440ff9640 1 -- 172.21.15.28:0/4271394753 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11867 (secure 0 0 0) 0x7fd43c0be4b0 con 0x7fd444102660 2024-03-22T02:19:52.870 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:19:52.870 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:19:20.368528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26924":{"gid":26924,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2468455656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2468455656},{"type":"v1","addr":"172.21.15.83:6837","nonce":2468455656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:19:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-03-22T02:19:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.871+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 msgr2=0x7fd42c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.871+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 0x7fd42c0a53e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd444101b00 tx=0x7fd434005f90 comp rx=0 tx=0).stop 2024-03-22T02:19:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.871+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 msgr2=0x7fd44419f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fd43c037040 tx=0x7fd43c032f40 comp rx=0 tx=0).stop 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 shutdown_connections 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd42c0a2f20 0x7fd42c0a53e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd444102660 0x7fd44419f440 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd444101ca0 0x7fd4441983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 --2- 172.21.15.28:0/4271394753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd444100ab0 0x7fd444197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 >> 172.21.15.28:0/4271394753 conn(0x7fd4440fc020 msgr2=0x7fd4440fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 shutdown_connections 2024-03-22T02:19:52.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:52.872+0000 7fd449f81640 1 -- 172.21.15.28:0/4271394753 wait complete. 2024-03-22T02:19:52.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/2901298038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48100ab0 msgr2=0x7fdc48100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/2901298038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48100ab0 0x7fdc48100ed0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009ca0 tx=0x7fdc3802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/2901298038 shutdown_connections 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/2901298038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48102660 0x7fdc48108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/2901298038 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48101ca0 0x7fdc48102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/2901298038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48100ab0 0x7fdc48100ed0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/2901298038 >> 172.21.15.28:0/2901298038 conn(0x7fdc480fc040 msgr2=0x7fdc480fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/2901298038 shutdown_connections 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.008+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/2901298038 wait complete. 2024-03-22T02:19:53.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 Processor -- start 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 -- start start 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48101ca0 0x7fdc48198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810d850 con 0x7fdc48102660 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810d6d0 con 0x7fdc48100ab0 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc4d6a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810d9d0 con 0x7fdc48101ca0 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc3e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48101ca0 0x7fdc48198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41150/0 (socket says 172.21.15.28:41150) 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46784/0 (socket says 172.21.15.28:46784) 2024-03-22T02:19:53.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 -- 172.21.15.28:0/591840294 learned_addr learned my addr 172.21.15.28:0/591840294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 -- 172.21.15.28:0/591840294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48101ca0 msgr2=0x7fdc48198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48101ca0 0x7fdc48198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 -- 172.21.15.28:0/591840294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 msgr2=0x7fdc48197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc477fe640 1 -- 172.21.15.28:0/591840294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4819fb00 con 0x7fdc48102660 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.009+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc477fe640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000e970 tx=0x7fdc4000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc44ff9640 1 -- 172.21.15.28:0/591840294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4000ccc0 con 0x7fdc48102660 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc4819fdf0 con 0x7fdc48102660 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc44ff9640 1 -- 172.21.15.28:0/591840294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc4000ce60 con 0x7fdc48102660 2024-03-22T02:19:53.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc44ff9640 1 -- 172.21.15.28:0/591840294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc40010490 con 0x7fdc48102660 2024-03-22T02:19:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.010+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc481a02b0 con 0x7fdc48102660 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.012+0000 7fdc44ff9640 1 -- 172.21.15.28:0/591840294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdc40005020 con 0x7fdc48102660 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.012+0000 7fdc5036d740 1 Processor -- start 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.012+0000 7fdc5036d740 1 -- start start 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb87c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb8d380 0x55a01cb8f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a01caec8d0 con 0x55a01cb8d380 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a01ca94e00 con 0x55a01cb88210 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc5036d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a01cae8880 con 0x55a01cb87820 2024-03-22T02:19:53.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc44ff9640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 0x7fdc340a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc3e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb87c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46800/0 (socket says 172.21.15.28:46800) 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc3e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb87c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40464/0 (socket says 172.21.15.28:40464) 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 -- 172.21.15.28:0/1744341206 learned_addr learned my addr 172.21.15.28:0/1744341206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc477fe640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 0x7fdc340a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb8d380 0x55a01cb8f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc44ff9640 1 -- 172.21.15.28:0/591840294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fdc400cfc40 con 0x7fdc48102660 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 -- 172.21.15.28:0/1744341206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 msgr2=0x55a01cb87c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb87c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 -- 172.21.15.28:0/1744341206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb8d380 msgr2=0x55a01cb8f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb8d380 0x55a01cb8f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 -- 172.21.15.28:0/1744341206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a01c8003f0 con 0x55a01cb88210 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc477fe640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 0x7fdc340a53e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdc400983c0 tx=0x7fdc400c6c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.013+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009ca0 tx=0x7fdc38005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36448b10c2faa0e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc3d7fa640 1 -- 172.21.15.28:0/1744341206 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38043070 con 0x55a01cb88210 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc3d7fa640 1 -- 172.21.15.28:0/1744341206 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3802cca0 con 0x55a01cb88210 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc3d7fa640 1 -- 172.21.15.28:0/1744341206 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38039040 con 0x55a01cb88210 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 -- 172.21.15.28:0/1744341206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 msgr2=0x55a01cb8cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009ca0 tx=0x7fdc38005ef0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 -- 172.21.15.28:0/1744341206 shutdown_connections 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb8d380 0x55a01cb8f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb88210 0x55a01cb8cd10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 --2- 172.21.15.28:0/1744341206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb87c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 -- 172.21.15.28:0/1744341206 >> 172.21.15.28:0/1744341206 conn(0x55a01cb82d90 msgr2=0x55a01cb83190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 -- 172.21.15.28:0/1744341206 shutdown_connections 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.014+0000 7fdc5036d740 1 -- 172.21.15.28:0/1744341206 wait complete. 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 -- start start 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 Processor -- start 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 0x55a01cb92210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb99250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a01cb902d0 con 0x55a01cb87820 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a01caa8300 con 0x55a01cb8d380 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a01cb90450 con 0x55a01cb88210 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc5036d740 10 client.0 _ll_register_callbacks cb 0x55a01cae3e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 0x55a01cb92210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41162/0 (socket says 172.21.15.28:41162) 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 -- 172.21.15.28:0/2074663027 learned_addr learned my addr 172.21.15.28:0/2074663027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb99250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 -- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 msgr2=0x55a01cb92210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 0x55a01cb92210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 -- 172.21.15.28:0/2074663027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 msgr2=0x55a01cb99250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb99250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.015+0000 7fdc3e7fc640 1 -- 172.21.15.28:0/2074663027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a01c8ca7d0 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 0x55a01cb92210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdc300037d0 tx=0x7fdc30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41063e82fe044b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc30011950 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fdc30011af0 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc3802cca0 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdc30011cf0 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 10 client.17274 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a01cb9c630 con 0x55a01cb87820 2024-03-22T02:19:53.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc3802cca0 con 0x55a01cb87820 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fdc300213d0 con 0x55a01cb87820 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 10 client.17274 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 10 client.17274 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 20 client.17274 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cb9dd30 0x55a01cba01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 4 client.17274 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.016+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a01caf9e10 con 0x55a01cb9dd30 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.017+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cb9dd30 0x55a01cba01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.017+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cb9dd30 0x55a01cba01d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a47e071c0a24819 server_cookie=b3081024609a1ce2 in_seq=0 out_seq=0 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.017+0000 7fdc2e7fc640 10 client.17274 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2074663027 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a01caf9e10 con 0x55a01cb9dd30 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc2e7fc640 10 client.17274 handle_command_reply: tid=0 2024-03-22T02:19:53.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 client.17274 shutdown 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 msgr2=0x55a01cb91cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdc300037d0 tx=0x7fdc30004730 comp rx=0 tx=0).stop 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 shutdown_connections 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cb9dd30 0x55a01cba01d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb99250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb88210 0x55a01cb92210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc2e7fc640 0 client.17274 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2074663027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb87820 0x55a01cb91cd0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.018+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 >> 172.21.15.28:0/2074663027 conn(0x55a01cb82d90 msgr2=0x55a01cb81fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.019+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 shutdown_connections 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.019+0000 7fdc5036d740 1 -- 172.21.15.28:0/2074663027 wait complete. 2024-03-22T02:19:53.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.019+0000 7fdc5036d740 20 client.17274 trim_cache size 0 max 16384 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 Processor -- start 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 -- start start 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb91f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cba4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481ad310 con 0x55a01cb88210 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481aabe0 con 0x55a01cb8d380 2024-03-22T02:19:53.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.032+0000 7fdc5036d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481a8a30 con 0x55a01cb87820 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41178/0 (socket says 172.21.15.28:41178) 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 -- 172.21.15.28:0/3068380455 learned_addr learned my addr 172.21.15.28:0/3068380455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb91f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cba4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 -- 172.21.15.28:0/3068380455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 msgr2=0x55a01cb91f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb91f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 -- 172.21.15.28:0/3068380455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 msgr2=0x55a01cba4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cba4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 -- 172.21.15.28:0/3068380455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a01cb90d80 con 0x55a01cb88210 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005e30 tx=0x7fdc38032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24268a502c172bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc3cff9640 1 -- 172.21.15.28:0/3068380455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3802de60 con 0x55a01cb88210 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc3cff9640 1 -- 172.21.15.28:0/3068380455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc38032890 con 0x55a01cb88210 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.033+0000 7fdc3cff9640 1 -- 172.21.15.28:0/3068380455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38039040 con 0x55a01cb88210 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 -- 172.21.15.28:0/3068380455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 msgr2=0x55a01cb99210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fdc38005e30 tx=0x7fdc38032590 comp rx=0 tx=0).stop 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 -- 172.21.15.28:0/3068380455 shutdown_connections 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cba4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb99210 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 --2- 172.21.15.28:0/3068380455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb91f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 -- 172.21.15.28:0/3068380455 >> 172.21.15.28:0/3068380455 conn(0x55a01cb82d90 msgr2=0x55a01cb86880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 -- 172.21.15.28:0/3068380455 shutdown_connections 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.034+0000 7fdc5036d740 1 -- 172.21.15.28:0/3068380455 wait complete. 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 -- start start 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 Processor -- start 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb96fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481a90a0 con 0x55a01cb88210 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481aa510 con 0x55a01cb8d380 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc481a9770 con 0x55a01cb87820 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc5036d740 10 client.0 _ll_register_callbacks cb 0x7fdc300035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41190/0 (socket says 172.21.15.28:41190) 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 -- 172.21.15.28:0/2111272392 learned_addr learned my addr 172.21.15.28:0/2111272392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb96fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 -- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 msgr2=0x55a01cb96fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb96fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 -- 172.21.15.28:0/2111272392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 msgr2=0x55a01cb984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 -- 172.21.15.28:0/2111272392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a01cb98a90 con 0x55a01cb88210 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc46ffd640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdc38055040 tx=0x7fdc38005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ffc0b86f8ccb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.035+0000 7fdc2e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38039040 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc38032890 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fdc38052750 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdc38065050 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 10 client.17280 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a01cc046c0 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc38032890 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fdc3805c450 con 0x55a01cb88210 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 10 client.17280 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 10 client.17280 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 20 client.17280 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cc04cd0 0x55a01cc070c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 4 client.17280 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:53.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a01caf4160 con 0x55a01cc04cd0 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.036+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cc04cd0 0x55a01cc070c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc2e7fc640 10 client.17280 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc3e7fc640 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cc04cd0 0x55a01cc070c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9748c23c766f81b6 server_cookie=e6c77a61c94d038a in_seq=0 out_seq=0 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc2e7fc640 1 -- 172.21.15.28:0/2111272392 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a01caf4160 con 0x55a01cc04cd0 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc2e7fc640 10 client.17280 handle_command_reply: tid=0 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 client.17280 shutdown 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 msgr2=0x55a01cb974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fdc38055040 tx=0x7fdc38005bd0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 shutdown_connections 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a01cc04cd0 0x55a01cc070c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a01cb8d380 0x55a01cb984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc2e7fc640 0 client.17280 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a01cb88210 0x55a01cb974e0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 --2- 172.21.15.28:0/2111272392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a01cb87820 0x55a01cb96fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 >> 172.21.15.28:0/2111272392 conn(0x55a01cb82d90 msgr2=0x55a01cb86880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 shutdown_connections 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 1 -- 172.21.15.28:0/2111272392 wait complete. 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.037+0000 7fdc5036d740 20 client.17280 trim_cache size 0 max 16384 2024-03-22T02:19:53.046 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 msgr2=0x7fdc340a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 0x7fdc340a53e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdc400983c0 tx=0x7fdc400c6c20 comp rx=0 tx=0).stop 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 msgr2=0x7fdc4819f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000e970 tx=0x7fdc4000ed70 comp rx=0 tx=0).stop 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 shutdown_connections 2024-03-22T02:19:53.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc340a2f20 0x7fdc340a53e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc48102660 0x7fdc4819f490 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc48101ca0 0x7fdc48198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 --2- 172.21.15.28:0/591840294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc48100ab0 0x7fdc48197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 >> 172.21.15.28:0/591840294 conn(0x7fdc480fc040 msgr2=0x7fdc48105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 shutdown_connections 2024-03-22T02:19:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.038+0000 7fdc4d6a4640 1 -- 172.21.15.28:0/591840294 wait complete. 2024-03-22T02:19:53.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:19:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 -- 172.21.15.28:0/2826039488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc1078b0 msgr2=0x7ff8fc109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 --2- 172.21.15.28:0/2826039488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc109cc0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4009ca0 tx=0x7ff8f402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 -- 172.21.15.28:0/2826039488 shutdown_connections 2024-03-22T02:19:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 --2- 172.21.15.28:0/2826039488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc109cc0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 --2- 172.21.15.28:0/2826039488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 0x7ff8fc107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 --2- 172.21.15.28:0/2826039488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc102650 0x7ff8fc104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 -- 172.21.15.28:0/2826039488 >> 172.21.15.28:0/2826039488 conn(0x7ff8fc0fc020 msgr2=0x7ff8fc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 -- 172.21.15.28:0/2826039488 shutdown_connections 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.163+0000 7ff90431e640 1 -- 172.21.15.28:0/2826039488 wait complete. 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 Processor -- start 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 -- start start 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 0x7ff8fc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc10d810 con 0x7ff8fc102650 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc10d690 con 0x7ff8fc1078b0 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff90431e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc10d990 con 0x7ff8fc104f80 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41214/0 (socket says 172.21.15.28:41214) 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 -- 172.21.15.28:0/3338195035 learned_addr learned my addr 172.21.15.28:0/3338195035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9018b9640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 0x7ff8fc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9028bb640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 -- 172.21.15.28:0/3338195035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 msgr2=0x7ff8fc1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 0x7ff8fc1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 -- 172.21.15.28:0/3338195035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc1078b0 msgr2=0x7ff8fc19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.164+0000 7ff9020ba640 1 -- 172.21.15.28:0/3338195035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8fc19fad0 con 0x7ff8fc102650 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff9020ba640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800e9f0 tx=0x7ff8f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff8f37fe640 1 -- 172.21.15.28:0/3338195035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f800cd80 con 0x7ff8fc102650 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff8f37fe640 1 -- 172.21.15.28:0/3338195035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8f8010910 con 0x7ff8fc102650 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8fc19fdc0 con 0x7ff8fc102650 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff8f37fe640 1 -- 172.21.15.28:0/3338195035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f8018990 con 0x7ff8fc102650 2024-03-22T02:19:53.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.165+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8fc1a0230 con 0x7ff8fc102650 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 Processor -- start 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff8f37fe640 1 -- 172.21.15.28:0/3338195035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff8f8010ac0 con 0x7ff8fc102650 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 -- start start 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf243c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf248d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf24b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf1a88d0 con 0x55c0bf243820 2024-03-22T02:19:53.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf150e00 con 0x55c0bf249380 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.168+0000 7ff906fe5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf1a4880 con 0x55c0bf244210 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9028bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf243c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf248d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9028bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf243c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41222/0 (socket says 172.21.15.28:41222) 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9028bb640 1 -- 172.21.15.28:0/2796626841 learned_addr learned my addr 172.21.15.28:0/2796626841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf24b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff8f37fe640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 0x7ff8e80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 -- 172.21.15.28:0/2796626841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 msgr2=0x55c0bf248d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf248d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 -- 172.21.15.28:0/2796626841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 msgr2=0x55c0bf243c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf243c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 -- 172.21.15.28:0/2796626841 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0beebc3f0 con 0x55c0bf249380 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9028bb640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 0x7ff8e80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff8f37fe640 1 -- 172.21.15.28:0/3338195035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7ff8f80cfaf0 con 0x7ff8fc102650 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.169+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf24b810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc069810 tx=0x7ff8ec011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe481c59675dd556 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff8f0bfe640 1 -- 172.21.15.28:0/2796626841 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec004050 con 0x55c0bf249380 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff9028bb640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 0x7ff8e80a53e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006010 tx=0x7ff8f402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff8f0bfe640 1 -- 172.21.15.28:0/2796626841 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec005500 con 0x55c0bf249380 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff8f0bfe640 1 -- 172.21.15.28:0/2796626841 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec004d90 con 0x55c0bf249380 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 -- 172.21.15.28:0/2796626841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 msgr2=0x55c0bf24b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf24b810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc069810 tx=0x7ff8ec011810 comp rx=0 tx=0).stop 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 -- 172.21.15.28:0/2796626841 shutdown_connections 2024-03-22T02:19:53.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf24b810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf248d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2796626841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf243c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.170+0000 7ff906fe5740 1 -- 172.21.15.28:0/2796626841 >> 172.21.15.28:0/2796626841 conn(0x55c0bf23ed90 msgr2=0x55c0bf23f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- 172.21.15.28:0/2796626841 shutdown_connections 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- 172.21.15.28:0/2796626841 wait complete. 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- start start 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 Processor -- start 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf24e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf255250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf24c330 con 0x55c0bf243820 2024-03-22T02:19:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf164300 con 0x55c0bf249380 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c0bf24c4b0 con 0x55c0bf244210 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff906fe5740 10 client.0 _ll_register_callbacks cb 0x55c0bf19fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41224/0 (socket says 172.21.15.28:41224) 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 -- 172.21.15.28:0/1448710619 learned_addr learned my addr 172.21.15.28:0/1448710619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9018b9640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf24e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9028bb640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf255250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 -- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 msgr2=0x55c0bf24e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf24e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 -- 172.21.15.28:0/1448710619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 msgr2=0x55c0bf255250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.171+0000 7ff9020ba640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf255250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff9020ba640 1 -- 172.21.15.28:0/1448710619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0bef867d0 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff9018b9640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf24e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff9020ba640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800c6a0 tx=0x7ff8f8098220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40edcb1f73aedd28 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80d7040 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7ff8f80bab80 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ec005500 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8f809bec0 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 10 client.17292 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0bf258680 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec005500 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7ff8f8018bc0 con 0x55c0bf243820 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 10 client.17292 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 10 client.17292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 20 client.17292 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf259da0 0x55c0bf25c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 4 client.17292 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.172+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0bf1b5e10 con 0x55c0bf259da0 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.173+0000 7ff9018b9640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf259da0 0x55c0bf25c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.173+0000 7ff8e5ffb640 10 client.17292 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.173+0000 7ff9018b9640 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf259da0 0x55c0bf25c260 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96667469e7324a75 server_cookie=a6bf7972ae816ad4 in_seq=0 out_seq=0 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/1448710619 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c0bf1b5e10 con 0x55c0bf259da0 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff8e5ffb640 10 client.17292 handle_command_reply: tid=0 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 client.17292 shutdown 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 msgr2=0x55c0bf24dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800c6a0 tx=0x7ff8f8098220 comp rx=0 tx=0).stop 2024-03-22T02:19:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 shutdown_connections 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf259da0 0x55c0bf25c260 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf255250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff8e5ffb640 0 client.17292 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf24e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 --2- 172.21.15.28:0/1448710619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf24dcd0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 >> 172.21.15.28:0/1448710619 conn(0x55c0bf23ed90 msgr2=0x55c0bf23dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 shutdown_connections 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 1 -- 172.21.15.28:0/1448710619 wait complete. 2024-03-22T02:19:53.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.174+0000 7ff906fe5740 20 client.17292 trim_cache size 0 max 16384 2024-03-22T02:19:53.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.188+0000 7ff906fe5740 1 Processor -- start 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.188+0000 7ff906fe5740 1 -- start start 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf255210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1ad2a0 con 0x55c0bf249380 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1aab70 con 0x55c0bf243820 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff906fe5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1a89c0 con 0x55c0bf244210 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9028bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf255210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46868/0 (socket says 172.21.15.28:46868) 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9028bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41236/0 (socket says 172.21.15.28:41236) 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9020ba640 1 -- 172.21.15.28:0/2357293246 learned_addr learned my addr 172.21.15.28:0/2357293246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 -- 172.21.15.28:0/2357293246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 msgr2=0x55c0bf24dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 -- 172.21.15.28:0/2357293246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 msgr2=0x55c0bf23d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 -- 172.21.15.28:0/2357293246 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0bf23e930 con 0x55c0bf244210 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9020ba640 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9028bb640 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf255210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec01b040 tx=0x7ff8ec004500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14afabe25fbcedd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.189+0000 7ff8e7fff640 1 -- 172.21.15.28:0/2357293246 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec033070 con 0x55c0bf244210 2024-03-22T02:19:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff8e7fff640 1 -- 172.21.15.28:0/2357293246 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec009e70 con 0x55c0bf244210 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff8e7fff640 1 -- 172.21.15.28:0/2357293246 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec0035f0 con 0x55c0bf244210 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 -- 172.21.15.28:0/2357293246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 msgr2=0x55c0bf255210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf255210 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec01b040 tx=0x7ff8ec004500 comp rx=0 tx=0).stop 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 -- 172.21.15.28:0/2357293246 shutdown_connections 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf249380 0x55c0bf23d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf255210 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2357293246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf243820 0x55c0bf24dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 -- 172.21.15.28:0/2357293246 >> 172.21.15.28:0/2357293246 conn(0x55c0bf23ed90 msgr2=0x55c0bf24d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 -- 172.21.15.28:0/2357293246 shutdown_connections 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.190+0000 7ff906fe5740 1 -- 172.21.15.28:0/2357293246 wait complete. 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 -- start start 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 Processor -- start 2024-03-22T02:19:53.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf252b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf253b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1a9030 con 0x55c0bf243820 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1aa4a0 con 0x55c0bf249380 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff8fc1a9700 con 0x55c0bf244210 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff906fe5740 10 client.0 _ll_register_callbacks cb 0x7ff8f80d32c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41240/0 (socket says 172.21.15.28:41240) 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 -- 172.21.15.28:0/2401689428 learned_addr learned my addr 172.21.15.28:0/2401689428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf252b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9028bb640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf253b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 -- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 msgr2=0x55c0bf252b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf252b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 -- 172.21.15.28:0/2401689428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 msgr2=0x55c0bf253b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf253b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 -- 172.21.15.28:0/2401689428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0bf254110 con 0x55c0bf243820 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff9020ba640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80cd550 tx=0x7ff8f80d36b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65448a6ed8a83f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff8e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.191+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f80a4820 con 0x55c0bf243820 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7ff8f80bab80 con 0x55c0bf243820 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8ec009e70 con 0x55c0bf243820 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff8f809ff60 con 0x55c0bf243820 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 10 client.17298 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c0bf255480 con 0x55c0bf243820 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8ec009e70 con 0x55c0bf243820 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7ff8f809ad80 con 0x55c0bf243820 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 10 client.17298 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 10 client.17298 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 20 client.17298 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf2c0610 0x55c0bf2c2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 4 client.17298 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c0bf1b0160 con 0x55c0bf2c0610 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.192+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf2c0610 0x55c0bf2c2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff9018b9640 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf2c0610 0x55c0bf2c2a30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=617720862215a4f4 server_cookie=bf9df57238455c57 in_seq=0 out_seq=0 2024-03-22T02:19:53.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff8e5ffb640 10 client.17298 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff8e5ffb640 1 -- 172.21.15.28:0/2401689428 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c0bf1b0160 con 0x55c0bf2c0610 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff8e5ffb640 10 client.17298 handle_command_reply: tid=0 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 client.17298 shutdown 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 msgr2=0x55c0bf2525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7ff8f80cd550 tx=0x7ff8f80d36b0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 shutdown_connections 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c0bf2c0610 0x55c0bf2c2a30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c0bf249380 0x55c0bf253b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c0bf244210 0x55c0bf252b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 --2- 172.21.15.28:0/2401689428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c0bf243820 0x55c0bf2525d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 >> 172.21.15.28:0/2401689428 conn(0x55c0bf23ed90 msgr2=0x55c0bf241ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff8e5ffb640 0 client.17298 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 shutdown_connections 2024-03-22T02:19:53.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 1 -- 172.21.15.28:0/2401689428 wait complete. 2024-03-22T02:19:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.193+0000 7ff906fe5740 20 client.17298 trim_cache size 0 max 16384 2024-03-22T02:19:53.202 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 msgr2=0x7ff8e80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 0x7ff8e80a53e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4006010 tx=0x7ff8f402e040 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 msgr2=0x7ff8fc197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7ff8f800e9f0 tx=0x7ff8f800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 shutdown_connections 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff8e80a2f20 0x7ff8e80a53e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff8fc1078b0 0x7ff8fc19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff8fc104f80 0x7ff8fc1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 --2- 172.21.15.28:0/3338195035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8fc102650 0x7ff8fc197eb0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 >> 172.21.15.28:0/3338195035 conn(0x7ff8fc0fc020 msgr2=0x7ff8fc102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 shutdown_connections 2024-03-22T02:19:53.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.194+0000 7ff90431e640 1 -- 172.21.15.28:0/3338195035 wait complete. 2024-03-22T02:19:53.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.318+0000 7f00a1c5c640 1 -- 172.21.15.28:0/1658925750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 msgr2=0x7f009c103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.318+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/1658925750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c103230 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0090009900 tx=0x7f009002bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.318+0000 7f00a1c5c640 1 -- 172.21.15.28:0/1658925750 shutdown_connections 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/1658925750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 0x7f009c10a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/1658925750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c103230 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/1658925750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- 172.21.15.28:0/1658925750 >> 172.21.15.28:0/1658925750 conn(0x7f009c0fd130 msgr2=0x7f009c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- 172.21.15.28:0/1658925750 shutdown_connections 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- 172.21.15.28:0/1658925750 wait complete. 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 Processor -- start 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- start start 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 0x7f009c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f009c10d870 con 0x7f009c102db0 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f009c10d6f0 con 0x7f009c103770 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.319+0000 7f00a1c5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f009c10d9f0 con 0x7f009c101bc0 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41266/0 (socket says 172.21.15.28:41266) 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40564/0 (socket says 172.21.15.28:40564) 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 -- 172.21.15.28:0/532388329 learned_addr learned my addr 172.21.15.28:0/532388329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009bfff640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 0x7f009c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 -- 172.21.15.28:0/532388329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 msgr2=0x7f009c197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 -- 172.21.15.28:0/532388329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 msgr2=0x7f009c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 0x7f009c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 -- 172.21.15.28:0/532388329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c19fad0 con 0x7f009c102db0 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f009affd640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0090000c00 tx=0x7f009002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f0098ff9640 1 -- 172.21.15.28:0/532388329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090043020 con 0x7f009c102db0 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009c19fd60 con 0x7f009c102db0 2024-03-22T02:19:53.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f0098ff9640 1 -- 172.21.15.28:0/532388329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00900054e0 con 0x7f009c102db0 2024-03-22T02:19:53.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f0098ff9640 1 -- 172.21.15.28:0/532388329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090003c40 con 0x7f009c102db0 2024-03-22T02:19:53.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.320+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f009c1a0270 con 0x7f009c102db0 2024-03-22T02:19:53.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f0098ff9640 1 -- 172.21.15.28:0/532388329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0090046020 con 0x7f009c102db0 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 Processor -- start 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 -- start start 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336ceec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cf6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b336c538d0 con 0x55b336cf4380 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b336bfbe00 con 0x55b336cef210 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f00a4840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b336c4f880 con 0x55b336cee820 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f0098ff9640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 0x7f00840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cf6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46908/0 (socket says 172.21.15.28:46908) 2024-03-22T02:19:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009b7fe640 1 -- 172.21.15.28:0/3870316085 learned_addr learned my addr 172.21.15.28:0/3870316085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009bfff640 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336ceec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.323+0000 7f009bfff640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 0x7f00840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f0098ff9640 1 -- 172.21.15.28:0/532388329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f00900f1830 con 0x7f009c102db0 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 -- 172.21.15.28:0/3870316085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 msgr2=0x55b336ceec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336ceec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 -- 172.21.15.28:0/3870316085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 msgr2=0x55b336cf6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cf6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 -- 172.21.15.28:0/3870316085 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3369673f0 con 0x55b336cef210 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009bfff640 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336ceec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009b7fe640 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f008c011470 tx=0x7f008c011930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1793cc62724fd5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f009bfff640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 0x7f00840a53e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0094009710 tx=0x7f0094005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f0081ffb640 1 -- 172.21.15.28:0/3870316085 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c003970 con 0x55b336cef210 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f0081ffb640 1 -- 172.21.15.28:0/3870316085 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c004020 con 0x55b336cef210 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.324+0000 7f0081ffb640 1 -- 172.21.15.28:0/3870316085 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c00f930 con 0x55b336cef210 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- 172.21.15.28:0/3870316085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 msgr2=0x55b336cf3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f008c011470 tx=0x7f008c011930 comp rx=0 tx=0).stop 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- 172.21.15.28:0/3870316085 shutdown_connections 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cf6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf3d10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 --2- 172.21.15.28:0/3870316085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336ceec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- 172.21.15.28:0/3870316085 >> 172.21.15.28:0/3870316085 conn(0x55b336ce9d90 msgr2=0x55b336cea190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- 172.21.15.28:0/3870316085 shutdown_connections 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- 172.21.15.28:0/3870316085 wait complete. 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 -- start start 2024-03-22T02:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.325+0000 7f00a4840740 1 Processor -- start 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336cf8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b336cf7330 con 0x55b336cf4380 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b336c0f300 con 0x55b336cef210 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b336cf74b0 con 0x55b336cee820 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f00a4840740 10 client.0 _ll_register_callbacks cb 0x55b336c4ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41284/0 (socket says 172.21.15.28:41284) 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 -- 172.21.15.28:0/721083650 learned_addr learned my addr 172.21.15.28:0/721083650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009b7fe640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336cf8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009affd640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 -- 172.21.15.28:0/721083650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 msgr2=0x55b336cf8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336cf8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 -- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 msgr2=0x55b336cf9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 -- 172.21.15.28:0/721083650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b336a317d0 con 0x55b336cf4380 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f009bfff640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0094003870 tx=0x7f00940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66231bf97d7873a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f007affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009400f940 con 0x55b336cf4380 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.326+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f009400fae0 con 0x55b336cf4380 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f008c004020 con 0x55b336cf4380 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0094030050 con 0x55b336cf4380 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 10 client.17310 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b336d036f0 con 0x55b336cf4380 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c004020 con 0x55b336cf4380 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f009401c3b0 con 0x55b336cf4380 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 10 client.17310 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 20 client.17310 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d04e80 0x55b336d072e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 4 client.17310 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b336c60e10 con 0x55b336d04e80 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.327+0000 7f009b7fe640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d04e80 0x55b336d072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.328+0000 7f007affd640 10 client.17310 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.328+0000 7f009b7fe640 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d04e80 0x55b336d072e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd64c2c7ced07bde server_cookie=760fc13c30719256 in_seq=0 out_seq=0 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f007affd640 1 -- 172.21.15.28:0/721083650 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b336c60e10 con 0x55b336d04e80 2024-03-22T02:19:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f007affd640 10 client.17310 handle_command_reply: tid=0 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 client.17310 shutdown 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 msgr2=0x55b336d002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0094003870 tx=0x7f00940046c0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 shutdown_connections 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d04e80 0x55b336d072e0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336d002b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cef210 0x55b336cf9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f007affd640 0 client.17310 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 --2- 172.21.15.28:0/721083650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cee820 0x55b336cf8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 >> 172.21.15.28:0/721083650 conn(0x55b336ce9d90 msgr2=0x55b336ce8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 shutdown_connections 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 1 -- 172.21.15.28:0/721083650 wait complete. 2024-03-22T02:19:53.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.330+0000 7f00a4840740 20 client.17310 trim_cache size 0 max 16384 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 Processor -- start 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 -- start start 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cf78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 0x55b336ced700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1ad330 con 0x55b336cef210 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1aac00 con 0x55b336cee820 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f00a4840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1a8a50 con 0x55b336cf4380 2024-03-22T02:19:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41286/0 (socket says 172.21.15.28:41286) 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 0x55b336ced700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009affd640 1 -- 172.21.15.28:0/2526750592 learned_addr learned my addr 172.21.15.28:0/2526750592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cf78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 -- 172.21.15.28:0/2526750592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 msgr2=0x55b336ced700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 0x55b336ced700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 -- 172.21.15.28:0/2526750592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 msgr2=0x55b336d00270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009b7fe640 1 -- 172.21.15.28:0/2526750592 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b336cea800 con 0x55b336cee820 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009affd640 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.344+0000 7f009bfff640 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 0x55b336ced700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f009b7fe640 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cf78b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55b336cead00 tx=0x7f008c0048a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86e7cbf99c63f88b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00817fa640 1 -- 172.21.15.28:0/2526750592 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c004a10 con 0x55b336cee820 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00817fa640 1 -- 172.21.15.28:0/2526750592 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c004bb0 con 0x55b336cee820 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00817fa640 1 -- 172.21.15.28:0/2526750592 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c00f740 con 0x55b336cee820 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 -- 172.21.15.28:0/2526750592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 msgr2=0x55b336cf78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cf78b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x55b336cead00 tx=0x7f008c0048a0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 -- 172.21.15.28:0/2526750592 shutdown_connections 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cf4380 0x55b336ced700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cef210 0x55b336d00270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.345+0000 7f00a4840740 1 --2- 172.21.15.28:0/2526750592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cf78b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- 172.21.15.28:0/2526750592 >> 172.21.15.28:0/2526750592 conn(0x55b336ce9d90 msgr2=0x55b336cecff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- 172.21.15.28:0/2526750592 shutdown_connections 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- 172.21.15.28:0/2526750592 wait complete. 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- start start 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 Processor -- start 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cffa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cef210 0x55b336cfffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1a90c0 con 0x55b336cf4380 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1aa530 con 0x55b336cee820 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f009c1a9790 con 0x55b336cef210 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f00a4840740 10 client.0 _ll_register_callbacks cb 0x7f0094009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f009bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f009b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f009bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41294/0 (socket says 172.21.15.28:41294) 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f009bfff640 1 -- 172.21.15.28:0/1805112727 learned_addr learned my addr 172.21.15.28:0/1805112727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.346+0000 7f009affd640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cef210 0x55b336cfffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 -- 172.21.15.28:0/1805112727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cef210 msgr2=0x55b336cfffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cef210 0x55b336cfffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 -- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 msgr2=0x55b336cffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cffa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 -- 172.21.15.28:0/1805112727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b336cfe6b0 con 0x55b336cf4380 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009b7fe640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f009bfff640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0094004770 tx=0x7f009401cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa965d47413507b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0094039060 con 0x55b336cf4380 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f009401c5c0 con 0x55b336cf4380 2024-03-22T02:19:53.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f008c004bb0 con 0x55b336cf4380 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0094024040 con 0x55b336cf4380 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f00a4840740 10 client.17316 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b336d6bb50 con 0x55b336cf4380 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f008c004bb0 con 0x55b336cf4380 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f0094034700 con 0x55b336cf4380 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f00a4840740 10 client.17316 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.347+0000 7f00a4840740 10 client.17316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f00a4840740 20 client.17316 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d6c4a0 0x55b336d6e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f00a4840740 4 client.17316 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b336c5b160 con 0x55b336d6c4a0 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f009b7fe640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d6c4a0 0x55b336d6e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f009b7fe640 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d6c4a0 0x55b336d6e8c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=530e6c4ce398110d server_cookie=66f5d4a58ee1da4 in_seq=0 out_seq=0 2024-03-22T02:19:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.348+0000 7f007affd640 10 client.17316 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f007affd640 1 -- 172.21.15.28:0/1805112727 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b336c5b160 con 0x55b336d6c4a0 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f007affd640 10 client.17316 handle_command_reply: tid=0 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 client.17316 shutdown 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 msgr2=0x55b336cfe070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f0094004770 tx=0x7f009401cd40 comp rx=0 tx=0).stop 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 shutdown_connections 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b336d6c4a0 0x55b336d6e8c0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b336cf4380 0x55b336cfe070 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b336cef210 0x55b336cfffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f007affd640 0 client.17316 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 --2- 172.21.15.28:0/1805112727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b336cee820 0x55b336cffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 >> 172.21.15.28:0/1805112727 conn(0x55b336ce9d90 msgr2=0x55b336ced880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 shutdown_connections 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 1 -- 172.21.15.28:0/1805112727 wait complete. 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.350+0000 7f00a4840740 20 client.17316 trim_cache size 0 max 16384 2024-03-22T02:19:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 msgr2=0x7f00840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 0x7f00840a53e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0094009710 tx=0x7f0094005f90 comp rx=0 tx=0).stop 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 msgr2=0x7f009c1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0090000c00 tx=0x7f009002c650 comp rx=0 tx=0).stop 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 shutdown_connections 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00840a2f20 0x7f00840a53e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f009c103770 0x7f009c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f009c102db0 0x7f009c1983f0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 --2- 172.21.15.28:0/532388329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f009c101bc0 0x7f009c197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 >> 172.21.15.28:0/532388329 conn(0x7f009c0fd130 msgr2=0x7f009c1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 shutdown_connections 2024-03-22T02:19:53.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:53.351+0000 7f00a1c5c640 1 -- 172.21.15.28:0/532388329 wait complete. 2024-03-22T02:19:53.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 541566 12.12 MB/sec execute 155 sec latency 41.431 ms 2024-03-22T02:19:53.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:53 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4271394753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:53.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:53 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4271394753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:53.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:53 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4271394753' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:19:54.362 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:19:54.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:19:54.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:19:54.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 544002 12.13 MB/sec execute 156 sec latency 29.346 ms 2024-03-22T02:19:54.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- 172.21.15.28:0/406975661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954107890 msgr2=0x7f8954109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 --2- 172.21.15.28:0/406975661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954107890 0x7f8954109ce0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f8950009ca0 tx=0x7f895002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- 172.21.15.28:0/406975661 shutdown_connections 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 --2- 172.21.15.28:0/406975661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954107890 0x7f8954109ce0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 --2- 172.21.15.28:0/406975661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954104f60 0x7f8954107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 --2- 172.21.15.28:0/406975661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954102630 0x7f8954104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- 172.21.15.28:0/406975661 >> 172.21.15.28:0/406975661 conn(0x7f89540fc040 msgr2=0x7f89540fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- 172.21.15.28:0/406975661 shutdown_connections 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- 172.21.15.28:0/406975661 wait complete. 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 Processor -- start 2024-03-22T02:19:54.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.476+0000 7f895b871640 1 -- start start 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954102630 0x7f8954197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954107890 0x7f895419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f895410d850 con 0x7f8954104f60 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f895410d6d0 con 0x7f8954102630 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f895410d9d0 con 0x7f8954107890 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41314/0 (socket says 172.21.15.28:41314) 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 -- 172.21.15.28:0/3554071950 learned_addr learned my addr 172.21.15.28:0/3554071950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954102630 0x7f8954197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8959e0e640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954107890 0x7f895419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 -- 172.21.15.28:0/3554071950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954107890 msgr2=0x7f895419f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954107890 0x7f895419f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 -- 172.21.15.28:0/3554071950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954102630 msgr2=0x7f8954197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954102630 0x7f8954197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 -- 172.21.15.28:0/3554071950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895419fac0 con 0x7f8954104f60 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f8958e0c640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f894400d770 tx=0x7f894400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f894a7fc640 1 -- 172.21.15.28:0/3554071950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8944017070 con 0x7f8954104f60 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f894a7fc640 1 -- 172.21.15.28:0/3554071950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8944009c30 con 0x7f8954104f60 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895419fdb0 con 0x7f8954104f60 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f894a7fc640 1 -- 172.21.15.28:0/3554071950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894400f620 con 0x7f8954104f60 2024-03-22T02:19:54.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.477+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89541a0270 con 0x7f8954104f60 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.479+0000 7f894a7fc640 1 -- 172.21.15.28:0/3554071950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f894401a070 con 0x7f8954104f60 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f894a7fc640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 0x7f89400a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 Processor -- start 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 -- start start 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895960d640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 0x7f89400a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f894a7fc640 1 -- 172.21.15.28:0/3554071950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f89440cf5d0 con 0x7f8954104f60 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f448570 con 0x55fb2f4e9720 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f3f0e10 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.480+0000 7f895e456740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f444520 con 0x55fb2f4e45b0 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41318/0 (socket says 172.21.15.28:41318) 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 -- 172.21.15.28:0/2427572904 learned_addr learned my addr 172.21.15.28:0/2427572904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46978/0 (socket says 172.21.15.28:46978) 2024-03-22T02:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8958e0c640 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 -- 172.21.15.28:0/2427572904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 msgr2=0x55fb2f4e90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4e90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f895960d640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 0x7f89400a53e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f894c011550 tx=0x7f894c011950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 -- 172.21.15.28:0/2427572904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 msgr2=0x55fb2f4e3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 -- 172.21.15.28:0/2427572904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb2f2247e0 con 0x55fb2f4e9720 2024-03-22T02:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8958e0c640 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4e90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f8959e0e640 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8950009ca0 tx=0x7f89500323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d88e5e657393124 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f893f7fe640 1 -- 172.21.15.28:0/2427572904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950043070 con 0x55fb2f4e9720 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f893f7fe640 1 -- 172.21.15.28:0/2427572904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8950032c90 con 0x55fb2f4e9720 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.481+0000 7f893f7fe640 1 -- 172.21.15.28:0/2427572904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950039040 con 0x55fb2f4e9720 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- 172.21.15.28:0/2427572904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 msgr2=0x55fb2f4ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f8950009ca0 tx=0x7f89500323c0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- 172.21.15.28:0/2427572904 shutdown_connections 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4ebbb0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4e90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 --2- 172.21.15.28:0/2427572904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- 172.21.15.28:0/2427572904 >> 172.21.15.28:0/2427572904 conn(0x55fb2f4df130 msgr2=0x55fb2f4df530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- 172.21.15.28:0/2427572904 shutdown_connections 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- 172.21.15.28:0/2427572904 wait complete. 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 -- start start 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.482+0000 7f895e456740 1 Processor -- start 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4edc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4ee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f4ec670 con 0x55fb2f4e9720 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f404110 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fb2f4ec7f0 con 0x55fb2f4e45b0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895e456740 10 client.0 _ll_register_callbacks cb 0x55fb2f440be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41324/0 (socket says 172.21.15.28:41324) 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8958e0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4ee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 -- 172.21.15.28:0/1552199567 learned_addr learned my addr 172.21.15.28:0/1552199567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 -- 172.21.15.28:0/1552199567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 msgr2=0x55fb2f4ee1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4ee1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 -- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 msgr2=0x55fb2f4edc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4edc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 -- 172.21.15.28:0/1552199567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb2f284f70 con 0x55fb2f4e9720 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f8959e0e640 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f89500324c0 tx=0x7f8950032ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21aeb34e22416465 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f893cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.483+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895004c070 con 0x55fb2f4e9720 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8950048650 con 0x55fb2f4e9720 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8950032c90 con 0x55fb2f4e9720 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8950058050 con 0x55fb2f4e9720 2024-03-22T02:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 10 client.17328 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb2f11c170 con 0x55fb2f4e9720 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8950032c90 con 0x55fb2f4e9720 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f89500348b0 con 0x55fb2f4e9720 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 10 client.17328 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 10 client.17328 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 20 client.17328 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f4fbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 4 client.17328 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb2f455ab0 con 0x55fb2f4f9700 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.484+0000 7f895960d640 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f4fbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.485+0000 7f893cff9640 10 client.17328 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.485+0000 7f895960d640 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f4fbb50 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57e5b1d8726892d7 server_cookie=ec3263ff0fc02920 in_seq=0 out_seq=0 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f893cff9640 1 -- 172.21.15.28:0/1552199567 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fb2f455ab0 con 0x55fb2f4f9700 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f893cff9640 10 client.17328 handle_command_reply: tid=0 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 client.17328 shutdown 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 msgr2=0x55fb2f4f5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f89500324c0 tx=0x7f8950032ac0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 shutdown_connections 2024-03-22T02:19:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f4fbb50 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f5210 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4ee1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f893cff9640 0 client.17328 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 --2- 172.21.15.28:0/1552199567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4edc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 >> 172.21.15.28:0/1552199567 conn(0x55fb2f4df130 msgr2=0x55fb2f4de3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 shutdown_connections 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 1 -- 172.21.15.28:0/1552199567 wait complete. 2024-03-22T02:19:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.486+0000 7f895e456740 20 client.17328 trim_cache size 0 max 16384 2024-03-22T02:19:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 Processor -- start 2024-03-22T02:19:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 -- start start 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e9720 0x55fb2f4e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89541ad110 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89541aa9e0 con 0x55fb2f4e45b0 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895e456740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a8830 con 0x55fb2f4e9720 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41334/0 (socket says 172.21.15.28:41334) 2024-03-22T02:19:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 -- 172.21.15.28:0/1336524075 learned_addr learned my addr 172.21.15.28:0/1336524075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f8958e0c640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f8959e0e640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e9720 0x55fb2f4e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 -- 172.21.15.28:0/1336524075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e9720 msgr2=0x55fb2f4e4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e9720 0x55fb2f4e4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 -- 172.21.15.28:0/1336524075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 msgr2=0x55fb2f4f51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f895960d640 1 -- 172.21.15.28:0/1336524075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb2f4ee370 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.505+0000 7f8958e0c640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895960d640 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f894c011ef0 tx=0x7f894c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6944f799b8a4d318 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f893effd640 1 -- 172.21.15.28:0/1336524075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c01a040 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f893effd640 1 -- 172.21.15.28:0/1336524075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894c004490 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f893effd640 1 -- 172.21.15.28:0/1336524075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c004a00 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 -- 172.21.15.28:0/1336524075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 msgr2=0x55fb2f4e2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f894c011ef0 tx=0x7f894c020000 comp rx=0 tx=0).stop 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 -- 172.21.15.28:0/1336524075 shutdown_connections 2024-03-22T02:19:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e9720 0x55fb2f4e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 --2- 172.21.15.28:0/1336524075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4e2aa0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.506+0000 7f895e456740 1 -- 172.21.15.28:0/1336524075 >> 172.21.15.28:0/1336524075 conn(0x55fb2f4df130 msgr2=0x55fb2f4e2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- 172.21.15.28:0/1336524075 shutdown_connections 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- 172.21.15.28:0/1336524075 wait complete. 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- start start 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 Processor -- start 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a8ea0 con 0x55fb2f4e9720 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f89541aa310 con 0x55fb2f4e3bc0 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f89541a9570 con 0x55fb2f4e45b0 2024-03-22T02:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895e456740 10 client.0 _ll_register_callbacks cb 0x7f894c0099d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f8959e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f8958e0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895960d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:47006/0 (socket says 172.21.15.28:47006) 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.507+0000 7f895960d640 1 -- 172.21.15.28:0/3866722788 learned_addr learned my addr 172.21.15.28:0/3866722788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 -- 172.21.15.28:0/3866722788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 msgr2=0x55fb2f4f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 -- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 msgr2=0x55fb2f4f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 -- 172.21.15.28:0/3866722788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb2f4f4440 con 0x55fb2f4e9720 2024-03-22T02:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8958e0c640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f895960d640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f8959e0e640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f3e50 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f895002c630 tx=0x7f895002ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7882cd3ac4d647ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f893cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950061060 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8950034640 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f894c004490 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8950068050 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f895e456740 10 client.17334 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fb2f55fae0 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.508+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f894c004490 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f8950048a50 con 0x55fb2f4e9720 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 10 client.17334 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 10 client.17334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 20 client.17334 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f562430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 4 client.17334 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fb2f44fe00 con 0x55fb2f4f9700 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895960d640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f562430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f893cff9640 10 client.17334 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.509+0000 7f895960d640 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f562430 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d710330774e7606 server_cookie=339a07d2bc87f439 in_seq=0 out_seq=0 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.522+0000 7f893cff9640 1 -- 172.21.15.28:0/3866722788 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fb2f44fe00 con 0x55fb2f4f9700 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.522+0000 7f893cff9640 10 client.17334 handle_command_reply: tid=0 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.522+0000 7f895e456740 1 client.17334 shutdown 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 msgr2=0x55fb2f4f3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f3e50 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f895002c630 tx=0x7f895002ceb0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 shutdown_connections 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fb2f4f9700 0x55fb2f562430 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb2f4e9720 0x55fb2f4f3e50 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fb2f4e45b0 0x55fb2f4f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 --2- 172.21.15.28:0/3866722788 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fb2f4e3bc0 0x55fb2f4f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 >> 172.21.15.28:0/3866722788 conn(0x55fb2f4df130 msgr2=0x55fb2f4e2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f893cff9640 0 client.17334 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 shutdown_connections 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 1 -- 172.21.15.28:0/3866722788 wait complete. 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895e456740 20 client.17334 trim_cache size 0 max 16384 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e34ddba2-0d22-4f9a-a9fc-96bc52ed0d70", 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:19:54.525 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 msgr2=0x7f89400a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 0x7f89400a53e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f894c011550 tx=0x7f894c011950 comp rx=0 tx=0).stop 2024-03-22T02:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 msgr2=0x7f89541983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.523+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f894400d770 tx=0x7f894400dc30 comp rx=0 tx=0).stop 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 shutdown_connections 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f89400a2f20 0x7f89400a53e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8954107890 0x7f895419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8954104f60 0x7f89541983e0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 --2- 172.21.15.28:0/3554071950 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8954102630 0x7f8954197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 >> 172.21.15.28:0/3554071950 conn(0x7f89540fc040 msgr2=0x7f8954102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 shutdown_connections 2024-03-22T02:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.524+0000 7f895b871640 1 -- 172.21.15.28:0/3554071950 wait complete. 2024-03-22T02:19:54.536 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:19:54.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:19:54.565 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:54 smithi028 ceph-mon[31916]: pgmap v548: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.9 MiB/s wr, 367 op/s 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- 172.21.15.28:0/2287513687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101c80 msgr2=0x7f16a4102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2287513687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101c80 0x7f16a4102100 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f1690009900 tx=0x7f169002bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- 172.21.15.28:0/2287513687 shutdown_connections 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2287513687 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4102640 0x7f16a4108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2287513687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4101c80 0x7f16a4102100 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2287513687 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4100a90 0x7f16a4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- 172.21.15.28:0/2287513687 >> 172.21.15.28:0/2287513687 conn(0x7f16a40fc020 msgr2=0x7f16a40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- 172.21.15.28:0/2287513687 shutdown_connections 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- 172.21.15.28:0/2287513687 wait complete. 2024-03-22T02:19:54.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 Processor -- start 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.657+0000 7f16aadd0640 1 -- start start 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4101c80 0x7f16a4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4102640 0x7f16a419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410c710 con 0x7f16a4100a90 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410c590 con 0x7f16a4102640 2024-03-22T02:19:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16aadd0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410c890 con 0x7f16a4101c80 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41350/0 (socket says 172.21.15.28:41350) 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2104475453 learned_addr learned my addr 172.21.15.28:0/2104475453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a3fff640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4101c80 0x7f16a4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a936d640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4102640 0x7f16a419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2104475453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4101c80 msgr2=0x7f16a4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4101c80 0x7f16a4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2104475453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4102640 msgr2=0x7f16a419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4102640 0x7f16a419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2104475453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a419fb00 con 0x7f16a4100a90 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f169800e9c0 tx=0x7f169800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.658+0000 7f16a1ffb640 1 -- 172.21.15.28:0/2104475453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169800cde0 con 0x7f16a4100a90 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.659+0000 7f16a1ffb640 1 -- 172.21.15.28:0/2104475453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1698010910 con 0x7f16a4100a90 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.659+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a419fdf0 con 0x7f16a4100a90 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.659+0000 7f16a1ffb640 1 -- 172.21.15.28:0/2104475453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698018a00 con 0x7f16a4100a90 2024-03-22T02:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.659+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a41a0250 con 0x7f16a4100a90 2024-03-22T02:19:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.661+0000 7f16a1ffb640 1 -- 172.21.15.28:0/2104475453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f169801f020 con 0x7f16a4100a90 2024-03-22T02:19:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 Processor -- start 2024-03-22T02:19:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a1ffb640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 0x7f16940a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 -- start start 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a1ffb640 1 -- 172.21.15.28:0/2104475453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f16980cf9d0 con 0x7f16a4100a90 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a936d640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 0x7f16940a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27af840 0x55cbf27afc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 0x55cbf27b7830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbf27108c0 con 0x55cbf27b0230 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cbf26b8c20 con 0x55cbf27b53a0 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbf270c870 con 0x55cbf27af840 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41364/0 (socket says 172.21.15.28:41364) 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a3fff640 1 -- 172.21.15.28:0/731812734 learned_addr learned my addr 172.21.15.28:0/731812734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a936d640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 0x55cbf27b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.662+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27af840 0x55cbf27afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 -- 172.21.15.28:0/731812734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27af840 msgr2=0x55cbf27afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27af840 0x55cbf27afc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 -- 172.21.15.28:0/731812734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 msgr2=0x55cbf27b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 -- 172.21.15.28:0/731812734 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf26dd150 con 0x55cbf27b53a0 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a3fff640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 0x7f16940a53e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f169c005fd0 tx=0x7f169c011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f16a936d640 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 0x55cbf27b7830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f169c011d10 tx=0x7f169c020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eea4d0fed29aada3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f168effd640 1 -- 172.21.15.28:0/731812734 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169c017050 con 0x55cbf27b53a0 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f168effd640 1 -- 172.21.15.28:0/731812734 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f169c024030 con 0x55cbf27b53a0 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.663+0000 7f168effd640 1 -- 172.21.15.28:0/731812734 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169c005280 con 0x55cbf27b53a0 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- 172.21.15.28:0/731812734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 msgr2=0x55cbf27b7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 0x55cbf27b7830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f169c011d10 tx=0x7f169c020000 comp rx=0 tx=0).stop 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- 172.21.15.28:0/731812734 shutdown_connections 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b53a0 0x55cbf27b7830 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/731812734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27af840 0x55cbf27afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- 172.21.15.28:0/731812734 >> 172.21.15.28:0/731812734 conn(0x55cbf27a69a0 msgr2=0x55cbf27a6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- 172.21.15.28:0/731812734 shutdown_connections 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- 172.21.15.28:0/731812734 wait complete. 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.664+0000 7f16ad9bb740 1 -- start start 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 Processor -- start 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b0230 0x55cbf27ba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b53a0 0x55cbf27c1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbf27b8350 con 0x55cbf27b53a0 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cbf26cc130 con 0x55cbf27af840 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbf27b84d0 con 0x55cbf27b0230 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16ad9bb740 10 client.0 _ll_register_callbacks cb 0x55cbf26cbb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b0230 0x55cbf27ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:47050/0 (socket says 172.21.15.28:47050) 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2143027470 learned_addr learned my addr 172.21.15.28:0/2143027470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a936d640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b53a0 0x55cbf27c1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2143027470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b0230 msgr2=0x55cbf27ba310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b0230 0x55cbf27ba310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.665+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2143027470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b53a0 msgr2=0x55cbf27c1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b53a0 0x55cbf27c1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16a8b6c640 1 -- 172.21.15.28:0/2143027470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf24ee7d0 con 0x55cbf27af840 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f169809b9f0 tx=0x7f169809bcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25ca859c8eea5ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f1683fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:19:54.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16980a4270 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f169c024030 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f16980a4410 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1698081050 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16ad9bb740 10 client.27176 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cbf24243f0 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.666+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f169c024030 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f169809b140 con 0x55cbf27af840 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 10 client.27176 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 10 client.27176 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 20 client.27176 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf27c5b20 0x55cbf27c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 4 client.27176 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:54.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbf271de00 con 0x55cbf27c5b20 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16a3fff640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf27c5b20 0x55cbf27c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f1683fff640 10 client.27176 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.667+0000 7f16a3fff640 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf27c5b20 0x55cbf27c7f90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70296bb49e183af8 server_cookie=3a5ca47746f17035 in_seq=0 out_seq=0 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f1683fff640 1 -- 172.21.15.28:0/2143027470 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbf271de00 con 0x55cbf27c5b20 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f1683fff640 10 client.27176 handle_command_reply: tid=0 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 client.27176 shutdown 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 msgr2=0x55cbf27b9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f169809b9f0 tx=0x7f169809bcf0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 shutdown_connections 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf27c5b20 0x55cbf27c7f90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b53a0 0x55cbf27c1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b0230 0x55cbf27ba310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/2143027470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27b9dd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f1683fff640 0 client.27176 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 >> 172.21.15.28:0/2143027470 conn(0x55cbf27a69a0 msgr2=0x55cbf27a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 shutdown_connections 2024-03-22T02:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 1 -- 172.21.15.28:0/2143027470 wait complete. 2024-03-22T02:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.669+0000 7f16ad9bb740 20 client.27176 trim_cache size 0 max 16384 2024-03-22T02:19:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 Processor -- start 2024-03-22T02:19:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 -- start start 2024-03-22T02:19:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27ba270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41ab080 con 0x55cbf27b0230 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41a8950 con 0x55cbf27af840 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.688+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41a67a0 con 0x55cbf27b53a0 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27ba270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41384/0 (socket says 172.21.15.28:41384) 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 -- 172.21.15.28:0/3072933694 learned_addr learned my addr 172.21.15.28:0/3072933694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a936d640 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 -- 172.21.15.28:0/3072933694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 msgr2=0x55cbf27aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 -- 172.21.15.28:0/3072933694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 msgr2=0x55cbf27ba270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27ba270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 -- 172.21.15.28:0/3072933694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbf23e40f0 con 0x55cbf27b0230 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27ba270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f16a3fff640 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f169002cfd0 tx=0x7f169002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef0e2e695c2fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f168e7fc640 1 -- 172.21.15.28:0/3072933694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690046aa0 con 0x55cbf27b0230 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f168e7fc640 1 -- 172.21.15.28:0/3072933694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690046c40 con 0x55cbf27b0230 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.689+0000 7f168e7fc640 1 -- 172.21.15.28:0/3072933694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690005040 con 0x55cbf27b0230 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 -- 172.21.15.28:0/3072933694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 msgr2=0x55cbf27c1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f169002cfd0 tx=0x7f169002e040 comp rx=0 tx=0).stop 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 -- 172.21.15.28:0/3072933694 shutdown_connections 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27b0230 0x55cbf27c1310 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/3072933694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27af840 0x55cbf27ba270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 -- 172.21.15.28:0/3072933694 >> 172.21.15.28:0/3072933694 conn(0x55cbf27a69a0 msgr2=0x55cbf27a9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 -- 172.21.15.28:0/3072933694 shutdown_connections 2024-03-22T02:19:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.690+0000 7f16ad9bb740 1 -- 172.21.15.28:0/3072933694 wait complete. 2024-03-22T02:19:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 -- start start 2024-03-22T02:19:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 Processor -- start 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b0230 0x55cbf27bf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41a6e10 con 0x55cbf27af840 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41a8280 con 0x55cbf27b0230 2024-03-22T02:19:54.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16a41a74e0 con 0x55cbf27b53a0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16ad9bb740 10 client.0 _ll_register_callbacks cb 0x7f169809b740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16a8b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41390/0 (socket says 172.21.15.28:41390) 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.691+0000 7f16a8b6c640 1 -- 172.21.15.28:0/1443168452 learned_addr learned my addr 172.21.15.28:0/1443168452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a3fff640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b0230 0x55cbf27bf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a936d640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 -- 172.21.15.28:0/1443168452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 msgr2=0x55cbf27c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 -- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b0230 msgr2=0x55cbf27bf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b0230 0x55cbf27bf1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 -- 172.21.15.28:0/1443168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbf27c0740 con 0x55cbf27af840 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f16a8b6c640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f169809bc80 tx=0x7f1698005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c4d661429cf977 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:54.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.692+0000 7f1683fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1698097f90 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f16980a4270 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1690046c40 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f16980a4470 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f16ad9bb740 10 client.17352 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cbf282c040 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.693+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690046c40 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f169807ac60 con 0x55cbf27af840 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 10 client.17352 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 10 client.17352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 20 client.17352 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf282c9b0 0x55cbf282edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 4 client.17352 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:19:54.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbf2718150 con 0x55cbf282c9b0 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16a3fff640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf282c9b0 0x55cbf282edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f1683fff640 10 client.17352 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.694+0000 7f16a3fff640 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf282c9b0 0x55cbf282edd0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e5e0476a39c21c server_cookie=b8c60b2e40fad1aa in_seq=0 out_seq=0 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.695+0000 7f1683fff640 1 -- 172.21.15.28:0/1443168452 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55cbf2718150 con 0x55cbf282c9b0 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.695+0000 7f1683fff640 10 client.17352 handle_command_reply: tid=0 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.695+0000 7f16ad9bb740 1 client.17352 shutdown 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 msgr2=0x55cbf27bec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f169809bc80 tx=0x7f1698005b20 comp rx=0 tx=0).stop 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 shutdown_connections 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbf282c9b0 0x55cbf282edd0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbf27b53a0 0x55cbf27c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbf27b0230 0x55cbf27bf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 --2- 172.21.15.28:0/1443168452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbf27af840 0x55cbf27bec70 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f1683fff640 0 client.17352 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 >> 172.21.15.28:0/1443168452 conn(0x55cbf27a69a0 msgr2=0x55cbf27aa490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 shutdown_connections 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 1 -- 172.21.15.28:0/1443168452 wait complete. 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16ad9bb740 20 client.17352 trim_cache size 0 max 16384 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:19:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "e34ddba2-0d22-4f9a-a9fc-96bc52ed0d70": { 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e34ddba2-0d22-4f9a-a9fc-96bc52ed0d70", 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:19:54.703 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 msgr2=0x7f16940a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 0x7f16940a53e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f169c005fd0 tx=0x7f169c011810 comp rx=0 tx=0).stop 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 msgr2=0x7f16a4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f169800e9c0 tx=0x7f169800ee80 comp rx=0 tx=0).stop 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.696+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 shutdown_connections 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16940a2f20 0x7f16940a53e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16a4102640 0x7f16a419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16a4101c80 0x7f16a4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 --2- 172.21.15.28:0/2104475453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4100a90 0x7f16a4197ee0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:54.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 >> 172.21.15.28:0/2104475453 conn(0x7f16a40fc020 msgr2=0x7f16a40fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:54.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 shutdown_connections 2024-03-22T02:19:54.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:19:54.697+0000 7f16aadd0640 1 -- 172.21.15.28:0/2104475453 wait complete. 2024-03-22T02:19:54.708 INFO:tasks.cephfs.filesystem:scrub status for tag:e34ddba2-0d22-4f9a-a9fc-96bc52ed0d70 - {'path': '/', 'tag': 'e34ddba2-0d22-4f9a-a9fc-96bc52ed0d70', 'options': 'recursive,force'} 2024-03-22T02:19:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:54 smithi120 ceph-mon[36803]: pgmap v548: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.9 MiB/s wr, 367 op/s 2024-03-22T02:19:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:54 smithi083 ceph-mon[36340]: pgmap v548: 129 pgs: 129 active+clean; 29 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 2.9 MiB/s wr, 367 op/s 2024-03-22T02:19:55.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 546764 12.15 MB/sec execute 157 sec latency 30.898 ms 2024-03-22T02:19:55.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:55 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:19:55.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:55 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:55.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:55 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:19:55.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:55 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:55 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:19:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:55 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:55 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:19:55.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:55 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:55.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:55 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:19:55.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:55 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:55.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:55 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:19:55.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:55 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:56.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 548129 12.11 MB/sec execute 158 sec latency 53.044 ms 2024-03-22T02:19:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:56 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:19:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:56 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:56.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:56 smithi028 ceph-mon[31916]: pgmap v549: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 462 op/s 2024-03-22T02:19:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:56 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:19:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:56 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:56.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:56 smithi120 ceph-mon[36803]: pgmap v549: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 462 op/s 2024-03-22T02:19:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:56 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:19:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:56 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:19:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:56 smithi083 ceph-mon[36340]: pgmap v549: 129 pgs: 129 active+clean; 49 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.1 MiB/s wr, 462 op/s 2024-03-22T02:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 -- 172.21.15.28:0/858608566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 msgr2=0x7fd764100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/858608566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd764100ee0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fd760009900 tx=0x7fd76002bc30 comp rx=0 tx=0).stop 2024-03-22T02:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 -- 172.21.15.28:0/858608566 shutdown_connections 2024-03-22T02:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/858608566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd764108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/858608566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 0x7fd764102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/858608566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd764100ee0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.169+0000 7fd76bfeb640 1 -- 172.21.15.28:0/858608566 >> 172.21.15.28:0/858608566 conn(0x7fd7640fc030 msgr2=0x7fd7640fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- 172.21.15.28:0/858608566 shutdown_connections 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- 172.21.15.28:0/858608566 wait complete. 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 Processor -- start 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- start start 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 0x7fd76419cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd76419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c780 con 0x7fd764100ac0 2024-03-22T02:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c600 con 0x7fd764102670 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.170+0000 7fd76bfeb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c900 con 0x7fd764101cb0 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41404/0 (socket says 172.21.15.28:41404) 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 -- 172.21.15.28:0/3378788378 learned_addr learned my addr 172.21.15.28:0/3378788378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769586640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 0x7fd76419cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd76a588640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd76419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 -- 172.21.15.28:0/3378788378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 msgr2=0x7fd76419cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 0x7fd76419cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 -- 172.21.15.28:0/3378788378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 msgr2=0x7fd76419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd76419f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 -- 172.21.15.28:0/3378788378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76419fa80 con 0x7fd764100ac0 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd76a588640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd76419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd769d87640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd760000c00 tx=0x7fd76002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd75affd640 1 -- 172.21.15.28:0/3378788378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760043020 con 0x7fd764100ac0 2024-03-22T02:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd75affd640 1 -- 172.21.15.28:0/3378788378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7600054e0 con 0x7fd764100ac0 2024-03-22T02:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd764196920 con 0x7fd764100ac0 2024-03-22T02:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd75affd640 1 -- 172.21.15.28:0/3378788378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd760003c40 con 0x7fd764100ac0 2024-03-22T02:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.171+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd764196e70 con 0x7fd764100ac0 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.173+0000 7fd75affd640 1 -- 172.21.15.28:0/3378788378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd760046020 con 0x7fd764100ac0 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 Processor -- start 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 -- start start 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc59900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc5e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eadfbbe040 con 0x55eadfc5f040 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eadfb669e0 con 0x55eadfc594e0 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eadfbb9ff0 con 0x55eadfc59ed0 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd75affd640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 0x7fd7500a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd769586640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76a588640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc59900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd769d87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc5e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd769586640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41410/0 (socket says 172.21.15.28:41410) 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd769586640 1 -- 172.21.15.28:0/2907654132 learned_addr learned my addr 172.21.15.28:0/2907654132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.174+0000 7fd76a588640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 0x7fd7500a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd75affd640 1 -- 172.21.15.28:0/3378788378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fd7600f1820 con 0x7fd764100ac0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 -- 172.21.15.28:0/2907654132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 msgr2=0x55eadfc5e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc5e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 -- 172.21.15.28:0/2907654132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 msgr2=0x55eadfc614d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 -- 172.21.15.28:0/2907654132 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eadf8d2210 con 0x55eadfc594e0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd769586640 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc59900 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005fd0 tx=0x7fd75c00f8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb51106b00d94371 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd76a588640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 0x7fd7500a5480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd75c011fd0 tx=0x7fd75c01e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd74ffff640 1 -- 172.21.15.28:0/2907654132 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c022070 con 0x55eadfc594e0 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd74ffff640 1 -- 172.21.15.28:0/2907654132 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003e50 con 0x55eadfc594e0 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.175+0000 7fd74ffff640 1 -- 172.21.15.28:0/2907654132 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c004720 con 0x55eadfc594e0 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2907654132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 msgr2=0x55eadfc59900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc59900 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005fd0 tx=0x7fd75c00f8e0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2907654132 shutdown_connections 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc614d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc5e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2907654132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc59900 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2907654132 >> 172.21.15.28:0/2907654132 conn(0x55eadfc54a50 msgr2=0x55eadfc54e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2907654132 shutdown_connections 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2907654132 wait complete. 2024-03-22T02:19:57.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 -- start start 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.176+0000 7fd76ebcb740 1 Processor -- start 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc63a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eadfc61f30 con 0x55eadfc5f040 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eadfb79db0 con 0x55eadfc594e0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eadfc620b0 con 0x55eadfc59ed0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76ebcb740 10 client.0 _ll_register_callbacks cb 0x55eadf8d1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769d87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769586640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc63a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769d87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:47110/0 (socket says 172.21.15.28:47110) 2024-03-22T02:19:57.185 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769d87640 1 -- 172.21.15.28:0/4240772778 learned_addr learned my addr 172.21.15.28:0/4240772778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41426/0 (socket says 172.21.15.28:41426) 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 -- 172.21.15.28:0/4240772778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 msgr2=0x55eadfc63a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc63a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 -- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 msgr2=0x55eadfc634f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 -- 172.21.15.28:0/4240772778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eadf99e620 con 0x55eadfc5f040 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769d87640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd769586640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc63a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.177+0000 7fd76a588640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00ec70 tx=0x7fd75c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33644c90ae2ccb48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c033020 con 0x55eadfc5f040 2024-03-22T02:19:57.186 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fd75c0046b0 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd75c003e50 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd75c03e050 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 10 client.17364 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eadfc6dde0 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003e50 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fd75c0053a0 con 0x55eadfc5f040 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 10 client.17364 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 10 client.17364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 20 client.17364 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfc6f440 0x55eadfc71830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 4 client.17364 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eadfbcb580 con 0x55eadfc6f440 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.178+0000 7fd769d87640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfc6f440 0x55eadfc71830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.179+0000 7fd74d7fa640 10 client.17364 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.179+0000 7fd769d87640 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfc6f440 0x55eadfc71830 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef7955c2cff95b25 server_cookie=ec1c98d5627d328b in_seq=0 out_seq=0 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd74d7fa640 1 -- 172.21.15.28:0/4240772778 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eadfbcb580 con 0x55eadfc6f440 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd74d7fa640 10 client.17364 handle_command_reply: tid=0 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 client.17364 shutdown 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 msgr2=0x55eadfc6aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00ec70 tx=0x7fd75c0040c0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 shutdown_connections 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfc6f440 0x55eadfc71830 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc6aa00 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd74d7fa640 0 client.17364 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc63a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/4240772778 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc634f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 >> 172.21.15.28:0/4240772778 conn(0x55eadfc54a50 msgr2=0x55eadfc53c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 shutdown_connections 2024-03-22T02:19:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 1 -- 172.21.15.28:0/4240772778 wait complete. 2024-03-22T02:19:57.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.180+0000 7fd76ebcb740 20 client.17364 trim_cache size 0 max 16384 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.195+0000 7fd76ebcb740 1 Processor -- start 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 -- start start 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc594e0 0x55eadfc4fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 0x55eadfc583c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641ab000 con 0x55eadfc59ed0 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641a88d0 con 0x55eadfc5f040 2024-03-22T02:19:57.198 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641a6720 con 0x55eadfc594e0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41428/0 (socket says 172.21.15.28:41428) 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769d87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc594e0 0x55eadfc4fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 -- 172.21.15.28:0/2597064187 learned_addr learned my addr 172.21.15.28:0/2597064187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76a588640 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 0x55eadfc583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 -- 172.21.15.28:0/2597064187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc594e0 msgr2=0x55eadfc4fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc594e0 0x55eadfc4fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 -- 172.21.15.28:0/2597064187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 msgr2=0x55eadfc583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 0x55eadfc583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 -- 172.21.15.28:0/2597064187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eadfc59120 con 0x55eadfc59ed0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd76a588640 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 0x55eadfc583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.196+0000 7fd769586640 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd754009880 tx=0x7fd754005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36879003220aece server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd74f7fe640 1 -- 172.21.15.28:0/2597064187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd754009e60 con 0x55eadfc59ed0 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd74f7fe640 1 -- 172.21.15.28:0/2597064187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd754024420 con 0x55eadfc59ed0 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd74f7fe640 1 -- 172.21.15.28:0/2597064187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75401c650 con 0x55eadfc59ed0 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2597064187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 msgr2=0x55eadfc6a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fd754009880 tx=0x7fd754005f90 comp rx=0 tx=0).stop 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2597064187 shutdown_connections 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc5f040 0x55eadfc583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc59ed0 0x55eadfc6a9c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2597064187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc594e0 0x55eadfc4fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2597064187 >> 172.21.15.28:0/2597064187 conn(0x55eadfc54a50 msgr2=0x55eadfc57cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2597064187 shutdown_connections 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.197+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2597064187 wait complete. 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 -- start start 2024-03-22T02:19:57.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 Processor -- start 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc67ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc67ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641a6d90 con 0x55eadfc5f040 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641a8200 con 0x55eadfc594e0 2024-03-22T02:19:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7641a7460 con 0x55eadfc59ed0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76ebcb740 10 client.0 _ll_register_callbacks cb 0x7fd754010930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76a588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76a588640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41438/0 (socket says 172.21.15.28:41438) 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd769d87640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc67ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd769586640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc67ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76a588640 1 -- 172.21.15.28:0/2675088371 learned_addr learned my addr 172.21.15.28:0/2675088371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76a588640 1 -- 172.21.15.28:0/2675088371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 msgr2=0x55eadfc67ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.198+0000 7fd76a588640 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc67ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76a588640 1 -- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 msgr2=0x55eadfc67ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76a588640 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc67ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76a588640 1 -- 172.21.15.28:0/2675088371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eadfc695f0 con 0x55eadfc5f040 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76a588640 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005860 tx=0x7fd75c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ba523041ecf592 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:19:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c047060 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd754024420 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fd75c003cf0 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd75c04e050 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 10 client.17370 fetch_fsmap learned FSMap version 32 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eadfc6a900 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd754024420 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fd75c02faa0 con 0x55eadfc5f040 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 10 client.17370 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 10 client.17370 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 20 client.17370 populate_metadata read hostname 'smithi028' 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfcd5b10 0x55eadfcd7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 4 client.17370 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:19:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.199+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eadfbc58d0 con 0x55eadfcd5b10 2024-03-22T02:19:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.200+0000 7fd769d87640 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfcd5b10 0x55eadfcd7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:19:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.200+0000 7fd74d7fa640 10 client.17370 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.200+0000 7fd769d87640 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfcd5b10 0x55eadfcd7f30 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e7f415f6f3e2ba2 server_cookie=71e5f7fe9a62a824 in_seq=0 out_seq=0 2024-03-22T02:19:57.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 550577 12.14 MB/sec execute 159 sec latency 93.415 ms 2024-03-22T02:19:57.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd74d7fa640 1 -- 172.21.15.28:0/2675088371 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eadfbc58d0 con 0x55eadfcd5b10 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd74d7fa640 10 client.17370 handle_command_reply: tid=0 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 client.17370 shutdown 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 msgr2=0x55eadfc69000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd75c005860 tx=0x7fd75c009580 comp rx=0 tx=0).stop 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 shutdown_connections 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eadfcd5b10 0x55eadfcd7f30 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd74d7fa640 0 client.17370 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:19:57.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eadfc5f040 0x55eadfc69000 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eadfc59ed0 0x55eadfc67ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 --2- 172.21.15.28:0/2675088371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eadfc594e0 0x55eadfc67ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 >> 172.21.15.28:0/2675088371 conn(0x55eadfc54a50 msgr2=0x55eadfc57cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 shutdown_connections 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 1 -- 172.21.15.28:0/2675088371 wait complete. 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.494+0000 7fd76ebcb740 20 client.17370 trim_cache size 0 max 16384 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:19:57.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:19:57.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 msgr2=0x7fd7500a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 0x7fd7500a5480 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd75c011fd0 tx=0x7fd75c01e000 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 msgr2=0x7fd76406af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fd760000c00 tx=0x7fd76002c650 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 shutdown_connections 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7500a2fc0 0x7fd7500a5480 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102670 0x7fd76419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764101cb0 0x7fd76419cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 --2- 172.21.15.28:0/3378788378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ac0 0x7fd76406af80 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 >> 172.21.15.28:0/3378788378 conn(0x7fd7640fc030 msgr2=0x7fd764104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 shutdown_connections 2024-03-22T02:19:57.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:19:57.495+0000 7fd76bfeb640 1 -- 172.21.15.28:0/3378788378 wait complete. 2024-03-22T02:19:58.419 INFO:tasks.workunit.client.0.smithi028.stdout: 1 552023 12.10 MB/sec execute 160 sec latency 79.842 ms 2024-03-22T02:19:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:19:58 smithi120 ceph-mon[36803]: pgmap v550: 129 pgs: 129 active+clean; 52 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 392 op/s 2024-03-22T02:19:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:19:58 smithi083 ceph-mon[36340]: pgmap v550: 129 pgs: 129 active+clean; 52 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 392 op/s 2024-03-22T02:19:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:19:58 smithi028 ceph-mon[31916]: pgmap v550: 129 pgs: 129 active+clean; 52 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.5 MiB/s wr, 392 op/s 2024-03-22T02:19:59.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 554541 12.11 MB/sec execute 161 sec latency 10.096 ms 2024-03-22T02:20:00.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 556660 12.09 MB/sec execute 162 sec latency 89.252 ms 2024-03-22T02:20:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:00 smithi120 ceph-mon[36803]: pgmap v551: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-22T02:20:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:00 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:20:00.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:00 smithi120 ceph-mon[36803]: overall HEALTH_OK 2024-03-22T02:20:00.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:00 smithi083 ceph-mon[36340]: pgmap v551: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-22T02:20:00.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:00 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:20:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:00 smithi083 ceph-mon[36340]: overall HEALTH_OK 2024-03-22T02:20:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:00 smithi028 ceph-mon[31916]: pgmap v551: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-03-22T02:20:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:00 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:20:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:00 smithi028 ceph-mon[31916]: overall HEALTH_OK 2024-03-22T02:20:01.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 558308 12.08 MB/sec execute 163 sec latency 95.982 ms 2024-03-22T02:20:01.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:01 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:20:01.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:01 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:20:01.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:01 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:20:02.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 560726 12.09 MB/sec execute 164 sec latency 39.429 ms 2024-03-22T02:20:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:02 smithi120 ceph-mon[36803]: pgmap v552: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.5 MiB/s wr, 383 op/s 2024-03-22T02:20:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:02 smithi083 ceph-mon[36340]: pgmap v552: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.5 MiB/s wr, 383 op/s 2024-03-22T02:20:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:02 smithi028 ceph-mon[31916]: pgmap v552: 129 pgs: 129 active+clean; 48 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.5 MiB/s wr, 383 op/s 2024-03-22T02:20:03.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 562408 12.07 MB/sec execute 165 sec latency 28.399 ms 2024-03-22T02:20:04.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 565106 12.09 MB/sec execute 166 sec latency 20.411 ms 2024-03-22T02:20:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:04 smithi120 ceph-mon[36803]: pgmap v553: 129 pgs: 129 active+clean; 47 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 403 op/s 2024-03-22T02:20:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:04 smithi083 ceph-mon[36340]: pgmap v553: 129 pgs: 129 active+clean; 47 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 403 op/s 2024-03-22T02:20:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:04 smithi028 ceph-mon[31916]: pgmap v553: 129 pgs: 129 active+clean; 47 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 4.4 MiB/s wr, 403 op/s 2024-03-22T02:20:05.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 567236 12.07 MB/sec execute 167 sec latency 43.314 ms 2024-03-22T02:20:06.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 569011 12.07 MB/sec execute 168 sec latency 54.393 ms 2024-03-22T02:20:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:06 smithi120 ceph-mon[36803]: pgmap v554: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-03-22T02:20:06.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:06 smithi083 ceph-mon[36340]: pgmap v554: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-03-22T02:20:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:06 smithi028 ceph-mon[31916]: pgmap v554: 129 pgs: 129 active+clean; 51 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 478 op/s 2024-03-22T02:20:07.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 571614 12.08 MB/sec execute 169 sec latency 17.575 ms 2024-03-22T02:20:08.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 573297 12.07 MB/sec execute 170 sec latency 28.321 ms 2024-03-22T02:20:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:08 smithi120 ceph-mon[36803]: pgmap v555: 129 pgs: 129 active+clean; 52 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 391 op/s 2024-03-22T02:20:08.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:08 smithi083 ceph-mon[36340]: pgmap v555: 129 pgs: 129 active+clean; 52 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 391 op/s 2024-03-22T02:20:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:08 smithi028 ceph-mon[31916]: pgmap v555: 129 pgs: 129 active+clean; 52 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.0 MiB/s wr, 391 op/s 2024-03-22T02:20:09.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 575960 12.08 MB/sec execute 171 sec latency 13.186 ms 2024-03-22T02:20:10.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 578364 12.08 MB/sec execute 172 sec latency 29.179 ms 2024-03-22T02:20:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/3718438473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b84102650 msgr2=0x7f1b84104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/3718438473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b84102650 0x7f1b84104a40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009900 tx=0x7f1b8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/3718438473 shutdown_connections 2024-03-22T02:20:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/3718438473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b841078b0 0x7f1b84109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.597 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/3718438473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 0x7f1b84107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/3718438473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b84102650 0x7f1b84104a40 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/3718438473 >> 172.21.15.28:0/3718438473 conn(0x7f1b840fc020 msgr2=0x7f1b840fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/3718438473 shutdown_connections 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.594+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/3718438473 wait complete. 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 Processor -- start 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 -- start start 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b84102650 0x7f1b84101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 0x7f1b84100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410d850 con 0x7f1b841078b0 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410d6d0 con 0x7f1b84104f80 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b8b3bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b8410d9d0 con 0x7f1b84102650 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b89959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b89158640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b84102650 0x7f1b84101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b89959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40848/0 (socket says 172.21.15.28:40848) 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b89959640 1 -- 172.21.15.28:0/2500390603 learned_addr learned my addr 172.21.15.28:0/2500390603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.595+0000 7f1b88957640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 0x7f1b84100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 -- 172.21.15.28:0/2500390603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b84102650 msgr2=0x7f1b84101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b84102650 0x7f1b84101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 -- 172.21.15.28:0/2500390603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 msgr2=0x7f1b84100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 0x7f1b84100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 -- 172.21.15.28:0/2500390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b841013d0 con 0x7f1b841078b0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b89959640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00eaa0 tx=0x7f1b7c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b7a7fc640 1 -- 172.21.15.28:0/2500390603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00cdb0 con 0x7f1b841078b0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b7a7fc640 1 -- 172.21.15.28:0/2500390603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b7c005020 con 0x7f1b841078b0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b841a1f40 con 0x7f1b841078b0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b7a7fc640 1 -- 172.21.15.28:0/2500390603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c0116b0 con 0x7f1b841078b0 2024-03-22T02:20:10.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.596+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b841a23a0 con 0x7f1b841078b0 2024-03-22T02:20:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.599+0000 7f1b7a7fc640 1 -- 172.21.15.28:0/2500390603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1b7c005560 con 0x7f1b841078b0 2024-03-22T02:20:10.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.599+0000 7f1b8dfa6740 1 Processor -- start 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.599+0000 7f1b8dfa6740 1 -- start start 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.599+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88df900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.599+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88e74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4e8844040 con 0x55f4e88df4e0 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f4e87ec9e0 con 0x55f4e88dfed0 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f4e883fff0 con 0x55f4e88e5040 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42984/0 (socket says 172.21.15.28:42984) 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 -- 172.21.15.28:0/1578628564 learned_addr learned my addr 172.21.15.28:0/1578628564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b88957640 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88df900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b88957640 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88e74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b7a7fc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 0x7f1b700a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 -- 172.21.15.28:0/1578628564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 msgr2=0x55f4e88e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88e74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 -- 172.21.15.28:0/1578628564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 msgr2=0x55f4e88df900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88df900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 -- 172.21.15.28:0/1578628564 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e8558210 con 0x55f4e88dfed0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89959640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 0x7f1b700a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b7a7fc640 1 -- 172.21.15.28:0/2500390603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f1b7c0cf890 con 0x7f1b841078b0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.600+0000 7f1b89158640 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009900 tx=0x7f1b80005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b62f8585fb12f2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b6f7fe640 1 -- 172.21.15.28:0/1578628564 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8002cb80 con 0x55f4e88dfed0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b89959640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 0x7f1b700a5500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c09b960 tx=0x7f1b7c01d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b6f7fe640 1 -- 172.21.15.28:0/1578628564 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b80005040 con 0x55f4e88dfed0 2024-03-22T02:20:10.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b6f7fe640 1 -- 172.21.15.28:0/1578628564 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80004900 con 0x55f4e88dfed0 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1578628564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 msgr2=0x55f4e88e49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b80009900 tx=0x7f1b80005c00 comp rx=0 tx=0).stop 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1578628564 shutdown_connections 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88e74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.601+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88e49d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1578628564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88df900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1578628564 >> 172.21.15.28:0/1578628564 conn(0x55f4e88daa50 msgr2=0x55f4e88dae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1578628564 shutdown_connections 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1578628564 wait complete. 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- start start 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 Processor -- start 2024-03-22T02:20:10.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 0x55f4e88e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f4e88e7f90 con 0x55f4e88dfed0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f4e87ffdb0 con 0x55f4e88df4e0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f4e88e8110 con 0x55f4e88e5040 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b8dfa6740 10 client.0 _ll_register_callbacks cb 0x55f4e8557a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b88957640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.602+0000 7f1b89158640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 0x55f4e88e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b88957640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42992/0 (socket says 172.21.15.28:42992) 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b88957640 1 -- 172.21.15.28:0/3040502031 learned_addr learned my addr 172.21.15.28:0/3040502031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b88957640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 -- 172.21.15.28:0/3040502031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 msgr2=0x55f4e88f0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88f0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 -- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 msgr2=0x55f4e88e94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 -- 172.21.15.28:0/3040502031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e8624620 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b88957640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b89158640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 0x55f4e88e9a20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x55f4e88db9c0 tx=0x7f1b80005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df7fea330febd29 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8004d060 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f1b80004ad0 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b80005040 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1b80054050 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b8dfa6740 10 client.17382 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.603+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4e88f3dd0 con 0x55f4e88dfed0 2024-03-22T02:20:10.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b80005040 con 0x55f4e88dfed0 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f1b80034cc0 con 0x55f4e88dfed0 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 10 client.17382 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 10 client.17382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 20 client.17382 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e88f5120 0x55f4e88f7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 4 client.17382 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4e8851580 con 0x55f4e88f5120 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b88957640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e88f5120 0x55f4e88f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b88957640 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e88f5120 0x55f4e88f7510 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e11b4a0288c66543 server_cookie=d56cf70c73ed75a in_seq=0 out_seq=0 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.604+0000 7f1b6cff9640 10 client.17382 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.605+0000 7f1b6cff9640 1 -- 172.21.15.28:0/3040502031 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f4e8851580 con 0x55f4e88f5120 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.605+0000 7f1b6cff9640 10 client.17382 handle_command_reply: tid=0 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.605+0000 7f1b8dfa6740 1 client.17382 shutdown 2024-03-22T02:20:10.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 msgr2=0x55f4e88e9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 0x55f4e88e9a20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x55f4e88db9c0 tx=0x7f1b80005420 comp rx=0 tx=0).stop 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 shutdown_connections 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e88f5120 0x55f4e88f7510 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88e5040 0x55f4e88f0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88dfed0 0x55f4e88e9a20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b6cff9640 0 client.17382 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/3040502031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88df4e0 0x55f4e88e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 >> 172.21.15.28:0/3040502031 conn(0x55f4e88daa50 msgr2=0x55f4e88d9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 shutdown_connections 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/3040502031 wait complete. 2024-03-22T02:20:10.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.606+0000 7f1b8dfa6740 20 client.17382 trim_cache size 0 max 16384 2024-03-22T02:20:10.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.621+0000 7f1b8dfa6740 1 Processor -- start 2024-03-22T02:20:10.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 -- start start 2024-03-22T02:20:10.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.625 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88dfed0 0x55f4e88f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88e5040 0x55f4e88fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841af3c0 con 0x55f4e88df4e0 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841acc90 con 0x55f4e88e5040 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841aaae0 con 0x55f4e88dfed0 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b89158640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88dfed0 0x55f4e88f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40884/0 (socket says 172.21.15.28:40884) 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 -- 172.21.15.28:0/1029011136 learned_addr learned my addr 172.21.15.28:0/1029011136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:10.626 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88e5040 0x55f4e88fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 -- 172.21.15.28:0/1029011136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88dfed0 msgr2=0x55f4e88f0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88dfed0 0x55f4e88f0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 -- 172.21.15.28:0/1029011136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88e5040 msgr2=0x55f4e88fbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88e5040 0x55f4e88fbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.622+0000 7f1b88957640 1 -- 172.21.15.28:0/1029011136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4e88db7c0 con 0x55f4e88df4e0 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.623+0000 7f1b88957640 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400b6c0 tx=0x7f1b7400b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3ddabf46d30084 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.623+0000 7f1b6effd640 1 -- 172.21.15.28:0/1029011136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b74027590 con 0x55f4e88df4e0 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.623+0000 7f1b6effd640 1 -- 172.21.15.28:0/1029011136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b74027730 con 0x55f4e88df4e0 2024-03-22T02:20:10.627 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.623+0000 7f1b6effd640 1 -- 172.21.15.28:0/1029011136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b740047a0 con 0x55f4e88df4e0 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1029011136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 msgr2=0x55f4e88e1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1b7400b6c0 tx=0x7f1b7400b410 comp rx=0 tx=0).stop 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1029011136 shutdown_connections 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88e5040 0x55f4e88fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88dfed0 0x55f4e88f0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1029011136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88df4e0 0x55f4e88e1050 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1029011136 >> 172.21.15.28:0/1029011136 conn(0x55f4e88daa50 msgr2=0x55f4e88e0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1029011136 shutdown_connections 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.624+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1029011136 wait complete. 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 -- start start 2024-03-22T02:20:10.628 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 Processor -- start 2024-03-22T02:20:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 0x55f4e88ee860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88e5040 0x55f4e88efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841ab150 con 0x55f4e88e5040 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841ac5c0 con 0x55f4e88dfed0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1b841ab820 con 0x55f4e88df4e0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b8dfa6740 10 client.0 _ll_register_callbacks cb 0x7f1b7400bb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b88957640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88e5040 0x55f4e88efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b89158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b88957640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 0x55f4e88ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b89158640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43016/0 (socket says 172.21.15.28:43016) 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.625+0000 7f1b89158640 1 -- 172.21.15.28:0/1989660300 learned_addr learned my addr 172.21.15.28:0/1989660300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 -- 172.21.15.28:0/1989660300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 msgr2=0x55f4e88ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 0x55f4e88ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 -- 172.21.15.28:0/1989660300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88e5040 msgr2=0x55f4e88efdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88e5040 0x55f4e88efdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 -- 172.21.15.28:0/1989660300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4e88e8250 con 0x55f4e88dfed0 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b88957640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 0x55f4e88ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b89158640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b8002c8e0 tx=0x7f1b80005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b807ba7c26ac6773 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:10.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b80005d50 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b74027730 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f1b80004450 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1b80066050 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b8dfa6740 10 client.27218 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.626+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f4e895b850 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b74027730 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f1b800037b0 con 0x55f4e88dfed0 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 10 client.27218 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 10 client.27218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 20 client.27218 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:10.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e895c190 0x55f4e895e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 4 client.27218 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:20:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4e884b8d0 con 0x55f4e895c190 2024-03-22T02:20:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b88957640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e895c190 0x55f4e895e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b6cff9640 10 client.27218 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:10.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.627+0000 7f1b88957640 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e895c190 0x55f4e895e5b0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=448c8279b7633088 server_cookie=541b6aee57a65150 in_seq=0 out_seq=0 2024-03-22T02:20:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:10 smithi120 ceph-mon[36803]: pgmap v556: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 463 op/s 2024-03-22T02:20:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:10 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:20:10.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:10 smithi083 ceph-mon[36340]: pgmap v556: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 463 op/s 2024-03-22T02:20:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:10 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:20:10.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:10 smithi028 ceph-mon[31916]: pgmap v556: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 463 op/s 2024-03-22T02:20:10.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:10 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.921+0000 7f1b6cff9640 1 -- 172.21.15.28:0/1989660300 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4e884b8d0 con 0x55f4e895c190 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.921+0000 7f1b6cff9640 10 client.27218 handle_command_reply: tid=0 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.921+0000 7f1b8dfa6740 1 client.27218 shutdown 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 msgr2=0x55f4e88eeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1b8002c8e0 tx=0x7f1b80005740 comp rx=0 tx=0).stop 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 shutdown_connections 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f4e895c190 0x55f4e895e5b0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f4e88e5040 0x55f4e88efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f4e88dfed0 0x55f4e88eeda0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 --2- 172.21.15.28:0/1989660300 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4e88df4e0 0x55f4e88ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b6cff9640 0 client.27218 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 >> 172.21.15.28:0/1989660300 conn(0x55f4e88daa50 msgr2=0x55f4e88d9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 shutdown_connections 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 1 -- 172.21.15.28:0/1989660300 wait complete. 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.922+0000 7f1b8dfa6740 20 client.27218 trim_cache size 0 max 16384 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:20:10.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 msgr2=0x7f1b700a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 0x7f1b700a5500 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c09b960 tx=0x7f1b7c01d000 comp rx=0 tx=0).stop 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 msgr2=0x7f1b84100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00eaa0 tx=0x7f1b7c00ef60 comp rx=0 tx=0).stop 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 shutdown_connections 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1b700a3040 0x7f1b700a5500 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b841078b0 0x7f1b84100b40 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1b84104f80 0x7f1b84100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 --2- 172.21.15.28:0/2500390603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1b84102650 0x7f1b84101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 >> 172.21.15.28:0/2500390603 conn(0x7f1b840fc020 msgr2=0x7f1b84103190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 shutdown_connections 2024-03-22T02:20:10.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:10.923+0000 7f1b8b3bc640 1 -- 172.21.15.28:0/2500390603 wait complete. 2024-03-22T02:20:11.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 579347 12.05 MB/sec execute 173 sec latency 95.730 ms 2024-03-22T02:20:11.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:20:11.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:20:11.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:20:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:20:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:20:11.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:20:11.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:20:11.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:20:12.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 581427 12.02 MB/sec execute 174 sec latency 29.290 ms 2024-03-22T02:20:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:12 smithi120 ceph-mon[36803]: pgmap v557: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-03-22T02:20:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:12 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:20:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:12 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:20:12.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:12 smithi083 ceph-mon[36340]: pgmap v557: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-03-22T02:20:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:12 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:20:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:12 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:20:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:12 smithi028 ceph-mon[31916]: pgmap v557: 129 pgs: 129 active+clean; 55 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 320 op/s 2024-03-22T02:20:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:12 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:20:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:12 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:20:13.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 583249 12.02 MB/sec execute 175 sec latency 28.992 ms 2024-03-22T02:20:14.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 586083 12.05 MB/sec execute 176 sec latency 11.342 ms 2024-03-22T02:20:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:14 smithi120 ceph-mon[36803]: pgmap v558: 129 pgs: 129 active+clean; 54 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-22T02:20:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:14 smithi083 ceph-mon[36340]: pgmap v558: 129 pgs: 129 active+clean; 54 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-22T02:20:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:14 smithi028 ceph-mon[31916]: pgmap v558: 129 pgs: 129 active+clean; 54 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 300 op/s 2024-03-22T02:20:15.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 587763 12.02 MB/sec execute 177 sec latency 29.481 ms 2024-03-22T02:20:16.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 589996 12.03 MB/sec execute 178 sec latency 77.582 ms 2024-03-22T02:20:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:16 smithi120 ceph-mon[36803]: pgmap v559: 129 pgs: 129 active+clean; 46 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 443 op/s 2024-03-22T02:20:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:16 smithi083 ceph-mon[36340]: pgmap v559: 129 pgs: 129 active+clean; 46 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 443 op/s 2024-03-22T02:20:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:16 smithi028 ceph-mon[31916]: pgmap v559: 129 pgs: 129 active+clean; 46 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 443 op/s 2024-03-22T02:20:17.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 592192 12.01 MB/sec execute 179 sec latency 29.427 ms 2024-03-22T02:20:18.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 593991 12.01 MB/sec execute 180 sec latency 29.445 ms 2024-03-22T02:20:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:18 smithi120 ceph-mon[36803]: pgmap v560: 129 pgs: 129 active+clean; 45 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 398 op/s 2024-03-22T02:20:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:18 smithi083 ceph-mon[36340]: pgmap v560: 129 pgs: 129 active+clean; 45 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 398 op/s 2024-03-22T02:20:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:18 smithi028 ceph-mon[31916]: pgmap v560: 129 pgs: 129 active+clean; 45 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 398 op/s 2024-03-22T02:20:19.420 INFO:tasks.workunit.client.0.smithi028.stdout: 1 596869 12.04 MB/sec execute 181 sec latency 30.386 ms 2024-03-22T02:20:20.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 598690 12.01 MB/sec execute 182 sec latency 34.995 ms 2024-03-22T02:20:20.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:20 smithi028 ceph-mon[31916]: pgmap v561: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 444 op/s 2024-03-22T02:20:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:20 smithi120 ceph-mon[36803]: pgmap v561: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 444 op/s 2024-03-22T02:20:20.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:20 smithi083 ceph-mon[36340]: pgmap v561: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.4 MiB/s wr, 444 op/s 2024-03-22T02:20:21.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 600712 12.02 MB/sec execute 183 sec latency 23.346 ms 2024-03-22T02:20:22.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 602745 11.99 MB/sec execute 184 sec latency 68.683 ms 2024-03-22T02:20:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:22 smithi120 ceph-mon[36803]: pgmap v562: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-22T02:20:22.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:22 smithi083 ceph-mon[36340]: pgmap v562: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-22T02:20:22.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:22 smithi028 ceph-mon[31916]: pgmap v562: 129 pgs: 129 active+clean; 47 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 377 op/s 2024-03-22T02:20:23.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 604737 11.99 MB/sec execute 185 sec latency 28.658 ms 2024-03-22T02:20:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- 172.21.15.28:0/1603892806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50102660 msgr2=0x7f9e50108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 --2- 172.21.15.28:0/1603892806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50102660 0x7f9e50108ef0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f9e48009ca0 tx=0x7f9e4802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- 172.21.15.28:0/1603892806 shutdown_connections 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 --2- 172.21.15.28:0/1603892806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50102660 0x7f9e50108ef0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 --2- 172.21.15.28:0/1603892806 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 0x7f9e50102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 --2- 172.21.15.28:0/1603892806 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50100ab0 0x7f9e50100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- 172.21.15.28:0/1603892806 >> 172.21.15.28:0/1603892806 conn(0x7f9e500fc020 msgr2=0x7f9e500fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- 172.21.15.28:0/1603892806 shutdown_connections 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- 172.21.15.28:0/1603892806 wait complete. 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 Processor -- start 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.013+0000 7f9e57878640 1 -- start start 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 0x7f9e50109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50102660 0x7f9e50109fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5010d850 con 0x7f9e50100ab0 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5010d6d0 con 0x7f9e50101ca0 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5010d9d0 con 0x7f9e50102660 2024-03-22T02:20:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56792/0 (socket says 172.21.15.28:56792) 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 -- 172.21.15.28:0/3529342189 learned_addr learned my addr 172.21.15.28:0/3529342189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55e15640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50102660 0x7f9e50109fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e54e13640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 0x7f9e50109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 -- 172.21.15.28:0/3529342189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50102660 msgr2=0x7f9e50109fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50102660 0x7f9e50109fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 -- 172.21.15.28:0/3529342189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 msgr2=0x7f9e50109a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 0x7f9e50109a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 -- 172.21.15.28:0/3529342189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5010a870 con 0x7f9e50100ab0 2024-03-22T02:20:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e55614640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00e9c0 tx=0x7f9e4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e467fc640 1 -- 172.21.15.28:0/3529342189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c00cd80 con 0x7f9e50100ab0 2024-03-22T02:20:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e467fc640 1 -- 172.21.15.28:0/3529342189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4c0035f0 con 0x7f9e50100ab0 2024-03-22T02:20:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e501a1f70 con 0x7f9e50100ab0 2024-03-22T02:20:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e467fc640 1 -- 172.21.15.28:0/3529342189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c010690 con 0x7f9e50100ab0 2024-03-22T02:20:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.014+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e501a23d0 con 0x7f9e50100ab0 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.017+0000 7f9e467fc640 1 -- 172.21.15.28:0/3529342189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9e4c0037a0 con 0x7f9e50100ab0 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.017+0000 7f9e5a45a740 1 Processor -- start 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.017+0000 7f9e467fc640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 0x7f9e3c0a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.017+0000 7f9e5a45a740 1 -- start start 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e55e15640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 0x7f9e3c0a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568942544e0 0x556894254900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e467fc640 1 -- 172.21.15.28:0/3529342189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e4c0cfae0 con 0x7f9e50100ab0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689425c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5568941b9040 con 0x556894254ed0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568941619e0 con 0x55689425a040 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5568941b4ff0 con 0x5568942544e0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56800/0 (socket says 172.21.15.28:56800) 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 -- 172.21.15.28:0/1013153060 learned_addr learned my addr 172.21.15.28:0/1013153060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e55e15640 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689425c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e55614640 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568942544e0 0x556894254900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 -- 172.21.15.28:0/1013153060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568942544e0 msgr2=0x556894254900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568942544e0 0x556894254900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 -- 172.21.15.28:0/1013153060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 msgr2=0x55689425c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689425c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 -- 172.21.15.28:0/1013153060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556893ecd210 con 0x556894254ed0 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e55e15640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 0x7f9e3c0a5500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005e30 tx=0x7f9e4802ce40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009c90 tx=0x7f9e40011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f1b013aefa3cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.018+0000 7f9e3b7fe640 1 -- 172.21.15.28:0/1013153060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4001e070 con 0x556894254ed0 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e3b7fe640 1 -- 172.21.15.28:0/1013153060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e40009e40 con 0x556894254ed0 2024-03-22T02:20:24.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e3b7fe640 1 -- 172.21.15.28:0/1013153060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40018820 con 0x556894254ed0 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1013153060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 msgr2=0x5568942599d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f9e40009c90 tx=0x7f9e40011810 comp rx=0 tx=0).stop 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1013153060 shutdown_connections 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689425c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942599d0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1013153060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568942544e0 0x556894254900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1013153060 >> 172.21.15.28:0/1013153060 conn(0x55689424fa50 msgr2=0x55689424fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.033 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1013153060 shutdown_connections 2024-03-22T02:20:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.019+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1013153060 wait complete. 2024-03-22T02:20:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 -- start start 2024-03-22T02:20:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 Processor -- start 2024-03-22T02:20:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x556894265a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55689425cff0 con 0x5568942544e0 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556894174db0 con 0x556894254ed0 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55689425d170 con 0x55689425a040 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e5a45a740 10 client.0 _ll_register_callbacks cb 0x556893ecca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:24.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57426/0 (socket says 172.21.15.28:57426) 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e54e13640 1 -- 172.21.15.28:0/1671022768 learned_addr learned my addr 172.21.15.28:0/1671022768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56812/0 (socket says 172.21.15.28:56812) 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.020+0000 7f9e55e15640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x556894265a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 -- 172.21.15.28:0/1671022768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 msgr2=0x556894265a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x556894265a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 -- 172.21.15.28:0/1671022768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 msgr2=0x55689425e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 -- 172.21.15.28:0/1671022768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556893f99620 con 0x556894254ed0 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e55e15640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x556894265a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e55614640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000c00 tx=0x7f9e400056b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6e8f601153d6eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:20:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4001e040 con 0x556894254ed0 2024-03-22T02:20:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5568941c6580 con 0x556894254ed0 2024-03-22T02:20:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e40004030 con 0x556894254ed0 2024-03-22T02:20:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e40032050 con 0x556894254ed0 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e5a45a740 10 client.27230 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556894268de0 con 0x556894254ed0 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.021+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5568941c6580 con 0x556894254ed0 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f9e40021030 con 0x556894254ed0 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 10 client.27230 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 10 client.27230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 20 client.27230 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55689426a2a0 0x55689426c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 4 client.27230 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568941c08d0 con 0x55689426a2a0 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e55614640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55689426a2a0 0x55689426c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e38ff9640 10 client.27230 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.022+0000 7f9e55614640 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55689426a2a0 0x55689426c700 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e519ec8e7d91171 server_cookie=988ffcd2c8017607 in_seq=0 out_seq=0 2024-03-22T02:20:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.023+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1671022768 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568941c08d0 con 0x55689426a2a0 2024-03-22T02:20:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.023+0000 7f9e38ff9640 10 client.27230 handle_command_reply: tid=0 2024-03-22T02:20:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.023+0000 7f9e5a45a740 1 client.27230 shutdown 2024-03-22T02:20:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 msgr2=0x55689425ea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9e40000c00 tx=0x7f9e400056b0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 shutdown_connections 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55689426a2a0 0x55689426c700 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x556894265a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556894254ed0 0x55689425ea50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e38ff9640 0 client.27230 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1671022768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689425e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 >> 172.21.15.28:0/1671022768 conn(0x55689424fa50 msgr2=0x55689424ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 shutdown_connections 2024-03-22T02:20:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1671022768 wait complete. 2024-03-22T02:20:24.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.024+0000 7f9e5a45a740 20 client.27230 trim_cache size 0 max 16384 2024-03-22T02:20:24.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 Processor -- start 2024-03-22T02:20:24.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 -- start start 2024-03-22T02:20:24.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556894254ed0 0x5568942659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689424e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501af400 con 0x5568942544e0 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501accd0 con 0x55689425a040 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501aab20 con 0x556894254ed0 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56828/0 (socket says 172.21.15.28:56828) 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 -- 172.21.15.28:0/3122061204 learned_addr learned my addr 172.21.15.28:0/3122061204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e54e13640 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556894254ed0 0x5568942659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55e15640 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689424e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 -- 172.21.15.28:0/3122061204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556894254ed0 msgr2=0x5568942659e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556894254ed0 0x5568942659e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 -- 172.21.15.28:0/3122061204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 msgr2=0x55689424e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689424e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 -- 172.21.15.28:0/3122061204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55689425ced0 con 0x5568942544e0 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.040+0000 7f9e55614640 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0cd540 tx=0x7f9e4c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d7437181be5574 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e3affd640 1 -- 172.21.15.28:0/3122061204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c07f070 con 0x5568942544e0 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e3affd640 1 -- 172.21.15.28:0/3122061204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4c0a4810 con 0x5568942544e0 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e3affd640 1 -- 172.21.15.28:0/3122061204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0a45e0 con 0x5568942544e0 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 -- 172.21.15.28:0/3122061204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 msgr2=0x55689424e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0cd540 tx=0x7f9e4c003f10 comp rx=0 tx=0).stop 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 -- 172.21.15.28:0/3122061204 shutdown_connections 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55689425a040 0x55689424e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556894254ed0 0x5568942659e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/3122061204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568942544e0 0x55689424e3c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 -- 172.21.15.28:0/3122061204 >> 172.21.15.28:0/3122061204 conn(0x55689424fa50 msgr2=0x55689425e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 -- 172.21.15.28:0/3122061204 shutdown_connections 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.041+0000 7f9e5a45a740 1 -- 172.21.15.28:0/3122061204 wait complete. 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 -- start start 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 Processor -- start 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568942544e0 0x556894265700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x5568942626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ab190 con 0x556894254ed0 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ac600 con 0x5568942544e0 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e501ab860 con 0x55689425a040 2024-03-22T02:20:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e5a45a740 10 client.0 _ll_register_callbacks cb 0x7f9e4001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e54e13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56842/0 (socket says 172.21.15.28:56842) 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e54e13640 1 -- 172.21.15.28:0/1052669549 learned_addr learned my addr 172.21.15.28:0/1052669549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e55614640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568942544e0 0x556894265700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.042+0000 7f9e55e15640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x5568942626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 -- 172.21.15.28:0/1052669549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 msgr2=0x5568942626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x5568942626e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 -- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568942544e0 msgr2=0x556894265700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568942544e0 0x556894265700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 -- 172.21.15.28:0/1052669549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556894262ca0 con 0x556894254ed0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e54e13640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9e400038a0 tx=0x7f9e400099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee25b468223b247a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e40016020 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e40003f30 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e4c0a4810 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e4001e040 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e5a45a740 10 client.17406 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5568942cf620 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.043+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e4c0a4810 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f9e40021070 con 0x556894254ed0 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 10 client.17406 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 10 client.17406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 20 client.17406 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568942d0050 0x5568942d2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 4 client.17406 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:20:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55689413e770 con 0x5568942d0050 2024-03-22T02:20:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e55614640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568942d0050 0x5568942d2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e55614640 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568942d0050 0x5568942d2470 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c65aee92de6149cc server_cookie=7089d950920ebe73 in_seq=0 out_seq=0 2024-03-22T02:20:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.044+0000 7f9e38ff9640 10 client.17406 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e38ff9640 1 -- 172.21.15.28:0/1052669549 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55689413e770 con 0x5568942d0050 2024-03-22T02:20:24.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e38ff9640 10 client.17406 handle_command_reply: tid=0 2024-03-22T02:20:24.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 client.17406 shutdown 2024-03-22T02:20:24.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 msgr2=0x5568942621a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f9e400038a0 tx=0x7f9e400099d0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 shutdown_connections 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568942d0050 0x5568942d2470 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55689425a040 0x5568942626e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556894254ed0 0x5568942621a0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 --2- 172.21.15.28:0/1052669549 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568942544e0 0x556894265700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 >> 172.21.15.28:0/1052669549 conn(0x55689424fa50 msgr2=0x556894252cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e38ff9640 0 client.17406 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 shutdown_connections 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.331+0000 7f9e5a45a740 1 -- 172.21.15.28:0/1052669549 wait complete. 2024-03-22T02:20:24.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e5a45a740 20 client.17406 trim_cache size 0 max 16384 2024-03-22T02:20:24.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:20:24.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:20:24.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:20:24.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 msgr2=0x7f9e3c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 0x7f9e3c0a5500 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e48005e30 tx=0x7f9e4802ce40 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 msgr2=0x7f9e5010b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c00e9c0 tx=0x7f9e4c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 shutdown_connections 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e3c0a3040 0x7f9e3c0a5500 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e50102660 0x7f9e50109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e50101ca0 0x7f9e50109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 --2- 172.21.15.28:0/3529342189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e50100ab0 0x7f9e5010b3e0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 >> 172.21.15.28:0/3529342189 conn(0x7f9e500fc020 msgr2=0x7f9e50104ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.332+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 shutdown_connections 2024-03-22T02:20:24.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:24.333+0000 7f9e57878640 1 -- 172.21.15.28:0/3529342189 wait complete. 2024-03-22T02:20:24.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 607329 12.02 MB/sec execute 186 sec latency 139.194 ms 2024-03-22T02:20:24.709 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:20:24.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:20:24.733 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:24 smithi028 ceph-mon[31916]: pgmap v563: 129 pgs: 129 active+clean; 50 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-03-22T02:20:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:24 smithi120 ceph-mon[36803]: pgmap v563: 129 pgs: 129 active+clean; 50 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-03-22T02:20:24.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:24 smithi083 ceph-mon[36340]: pgmap v563: 129 pgs: 129 active+clean; 50 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 420 op/s 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/3993195425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08101c80 msgr2=0x7f5e08102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/3993195425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08101c80 0x7f5e08102100 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5df8009900 tx=0x7f5df802bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/3993195425 shutdown_connections 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/3993195425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08102640 0x7f5e08108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/3993195425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08101c80 0x7f5e08102100 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/3993195425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/3993195425 >> 172.21.15.28:0/3993195425 conn(0x7f5e080fc020 msgr2=0x7f5e080fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/3993195425 shutdown_connections 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/3993195425 wait complete. 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 Processor -- start 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.819+0000 7f5e0fd1e640 1 -- start start 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 0x7f5e081983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 0x7f5e0819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0810d870 con 0x7f5e08102640 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0810d6f0 con 0x7f5e08100a90 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0810d9f0 con 0x7f5e08101c80 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 0x7f5e0819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 0x7f5e081983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57468/0 (socket says 172.21.15.28:57468) 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0daba640 1 -- 172.21.15.28:0/2445563869 learned_addr learned my addr 172.21.15.28:0/2445563869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/2445563869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 msgr2=0x7f5e081983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 0x7f5e081983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/2445563869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 msgr2=0x7f5e08197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/2445563869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0819fac0 con 0x7f5e08102640 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 0x7f5e081983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 0x7f5e0819f450 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009fd0 tx=0x7f5e0400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5dfeffd640 1 -- 172.21.15.28:0/2445563869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0400ce20 con 0x7f5e08102640 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0819fdb0 con 0x7f5e08102640 2024-03-22T02:20:24.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5dfeffd640 1 -- 172.21.15.28:0/2445563869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e04005020 con 0x7f5e08102640 2024-03-22T02:20:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.820+0000 7f5dfeffd640 1 -- 172.21.15.28:0/2445563869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04010720 con 0x7f5e08102640 2024-03-22T02:20:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.821+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e081a0220 con 0x7f5e08102640 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.823+0000 7f5dfeffd640 1 -- 172.21.15.28:0/2445563869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5e04005540 con 0x7f5e08102640 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.823+0000 7f5e129e8740 1 Processor -- start 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.823+0000 7f5e129e8740 1 -- start start 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.823+0000 7f5dfeffd640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 0x7f5df40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665292920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 0x556665295420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5566651f28c0 con 0x55666528d430 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55666519ac20 con 0x55666528de20 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e129e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5566651ee870 con 0x556665292f90 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 0x7f5df40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5dfeffd640 1 -- 172.21.15.28:0/2445563869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5e040cfec0 con 0x7f5e08102640 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665292920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56868/0 (socket says 172.21.15.28:56868) 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0daba640 1 -- 172.21.15.28:0/1943304751 learned_addr learned my addr 172.21.15.28:0/1943304751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 0x556665295420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/1943304751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 msgr2=0x556665292920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665292920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/1943304751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 msgr2=0x55666528d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/1943304751 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566651bf150 con 0x556665292f90 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0daba640 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 0x7f5df40a53e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5df802c240 tx=0x7f5df8005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.824+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 0x556665295420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e040cd5b0 tx=0x7f5e0409c1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83035928959bde6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5df3fff640 1 -- 172.21.15.28:0/1943304751 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0409a4e0 con 0x556665292f90 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5df3fff640 1 -- 172.21.15.28:0/1943304751 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e040051d0 con 0x556665292f90 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5df3fff640 1 -- 172.21.15.28:0/1943304751 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e040bc0f0 con 0x556665292f90 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 -- 172.21.15.28:0/1943304751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 msgr2=0x556665295420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 0x556665295420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5e040cd5b0 tx=0x7f5e0409c1e0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 -- 172.21.15.28:0/1943304751 shutdown_connections 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556665292f90 0x556665295420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665292920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 --2- 172.21.15.28:0/1943304751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528d430 0x55666528d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.825+0000 7f5e129e8740 1 -- 172.21.15.28:0/1943304751 >> 172.21.15.28:0/1943304751 conn(0x5566652889a0 msgr2=0x556665288da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- 172.21.15.28:0/1943304751 shutdown_connections 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- 172.21.15.28:0/1943304751 wait complete. 2024-03-22T02:20:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- start start 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 Processor -- start 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556665295f40 con 0x556665292f90 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5566651ae130 con 0x55666528de20 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5566652960c0 con 0x55666528d430 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e129e8740 10 client.0 _ll_register_callbacks cb 0x5566651adb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56880/0 (socket says 172.21.15.28:56880) 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56566/0 (socket says 172.21.15.28:56566) 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/2557299286 learned_addr learned my addr 172.21.15.28:0/2557299286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.826+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57498/0 (socket says 172.21.15.28:57498) 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/2557299286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 msgr2=0x556665297970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/2557299286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 msgr2=0x55666529eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/2557299286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556664fd07d0 con 0x55666528de20 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0daba640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5df8036040 tx=0x7f5df802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47bd133e7d81dbc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8052020 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5df803a040 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e040051d0 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5df805d050 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e129e8740 10 client.27248 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556664f063f0 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.827+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e040051d0 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f5df8034b20 con 0x55666528de20 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 10 client.27248 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 10 client.27248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 20 client.27248 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5566652a3810 0x5566652a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 4 client.27248 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566651ffe00 con 0x5566652a3810 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e0daba640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5566652a3810 0x5566652a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5df17fa640 10 client.27248 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.828+0000 7f5e0daba640 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5566652a3810 0x5566652a5c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=275da5494ca6cdc7 server_cookie=523adb63e348f1da in_seq=0 out_seq=0 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5df17fa640 1 -- 172.21.15.28:0/2557299286 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566651ffe00 con 0x5566652a3810 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5df17fa640 10 client.27248 handle_command_reply: tid=0 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 client.27248 shutdown 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 msgr2=0x556665297eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5df8036040 tx=0x7f5df802bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 shutdown_connections 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5566652a3810 0x5566652a5c70 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528de20 0x556665297eb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 --2- 172.21.15.28:0/2557299286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665297970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5df17fa640 0 client.27248 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.829+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 >> 172.21.15.28:0/2557299286 conn(0x5566652889a0 msgr2=0x556665287c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.830+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 shutdown_connections 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.830+0000 7f5e129e8740 1 -- 172.21.15.28:0/2557299286 wait complete. 2024-03-22T02:20:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.830+0000 7f5e129e8740 20 client.27248 trim_cache size 0 max 16384 2024-03-22T02:20:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 Processor -- start 2024-03-22T02:20:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 -- start start 2024-03-22T02:20:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665296f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 0x556665287370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081ad1d0 con 0x55666528de20 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081aaaa0 con 0x556665292f90 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e129e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081a88f0 con 0x55666528d430 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e0d2b9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56884/0 (socket says 172.21.15.28:56884) 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/3560358611 learned_addr learned my addr 172.21.15.28:0/3560358611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e0daba640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665296f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.848+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 0x556665287370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/3560358611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 msgr2=0x556665296f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665296f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/3560358611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 msgr2=0x556665287370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 0x556665287370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 -- 172.21.15.28:0/3560358611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556664ec60f0 con 0x55666528de20 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 0x556665287370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5df802c170 tx=0x7f5df8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f542b122060533b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5df37fe640 1 -- 172.21.15.28:0/3560358611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8066070 con 0x55666528de20 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5df37fe640 1 -- 172.21.15.28:0/3560358611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df8004320 con 0x55666528de20 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.849+0000 7f5df37fe640 1 -- 172.21.15.28:0/3560358611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df8052070 con 0x55666528de20 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- 172.21.15.28:0/3560358611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 msgr2=0x55666529eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5df802c170 tx=0x7f5df8005c90 comp rx=0 tx=0).stop 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- 172.21.15.28:0/3560358611 shutdown_connections 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556665292f90 0x556665287370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666528de20 0x55666529eeb0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- 172.21.15.28:0/3560358611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528d430 0x556665296f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- 172.21.15.28:0/3560358611 >> 172.21.15.28:0/3560358611 conn(0x5566652889a0 msgr2=0x55666528f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- 172.21.15.28:0/3560358611 shutdown_connections 2024-03-22T02:20:24.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- 172.21.15.28:0/3560358611 wait complete. 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- start start 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 Processor -- start 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 0x55666529c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528de20 0x55666529cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081a8f60 con 0x556665292f90 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081aa3d0 con 0x55666528d430 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5e081a9630 con 0x55666528de20 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e129e8740 10 client.0 _ll_register_callbacks cb 0x7f5df80046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.850+0000 7f5e0e2bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 0x55666529c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0daba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 0x55666529c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57518/0 (socket says 172.21.15.28:57518) 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0daba640 1 -- 172.21.15.28:0/702768554 learned_addr learned my addr 172.21.15.28:0/702768554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:24.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0d2b9640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528de20 0x55666529cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/702768554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528de20 msgr2=0x55666529cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528de20 0x55666529cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 msgr2=0x55666529c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 0x55666529c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 -- 172.21.15.28:0/702768554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55666529e220 con 0x556665292f90 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e0e2bb640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529dc30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005fd0 tx=0x7f5e0400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=808cf4ff4a1f9e1d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5df17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e04080060 con 0x556665292f90 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5e040051d0 con 0x556665292f90 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5df8004320 con 0x556665292f90 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5e0409b940 con 0x556665292f90 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e129e8740 10 client.17424 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:24.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556665309d20 con 0x556665292f90 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.851+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5df8004320 con 0x556665292f90 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f5e04097f30 con 0x556665292f90 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 10 client.17424 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 10 client.17424 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 20 client.17424 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666530a2a0 0x55666530c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 4 client.17424 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556665297760 con 0x55666530a2a0 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e0daba640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666530a2a0 0x55666530c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5e0daba640 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666530a2a0 0x55666530c690 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dad68db60c8c8992 server_cookie=5358a5307428ed9c in_seq=0 out_seq=0 2024-03-22T02:20:24.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.852+0000 7f5df17fa640 10 client.17424 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.853+0000 7f5df17fa640 1 -- 172.21.15.28:0/702768554 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556665297760 con 0x55666530a2a0 2024-03-22T02:20:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.853+0000 7f5df17fa640 10 client.17424 handle_command_reply: tid=0 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.853+0000 7f5e129e8740 1 client.17424 shutdown 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.853+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 msgr2=0x55666529dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.853+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529dc30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5e04005fd0 tx=0x7f5e0400c890 comp rx=0 tx=0).stop 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 shutdown_connections 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666530a2a0 0x55666530c690 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556665292f90 0x55666529dc30 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666528de20 0x55666529cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5df17fa640 0 client.17424 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 --2- 172.21.15.28:0/702768554 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666528d430 0x55666529c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 >> 172.21.15.28:0/702768554 conn(0x5566652889a0 msgr2=0x55666528c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 shutdown_connections 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 1 -- 172.21.15.28:0/702768554 wait complete. 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e129e8740 20 client.17424 trim_cache size 0 max 16384 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:20:24.862 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 msgr2=0x7f5df40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 0x7f5df40a53e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5df802c240 tx=0x7f5df8005dd0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 msgr2=0x7f5e0819f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 0x7f5e0819f450 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009fd0 tx=0x7f5e0400ef10 comp rx=0 tx=0).stop 2024-03-22T02:20:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 shutdown_connections 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5df40a2f20 0x7f5df40a53e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e08102640 0x7f5e0819f450 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5e08101c80 0x7f5e081983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 --2- 172.21.15.28:0/2445563869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5e08100a90 0x7f5e08197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 >> 172.21.15.28:0/2445563869 conn(0x7f5e080fc020 msgr2=0x7f5e08105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 shutdown_connections 2024-03-22T02:20:24.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:24.854+0000 7f5e0fd1e640 1 -- 172.21.15.28:0/2445563869 wait complete. 2024-03-22T02:20:24.866 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:20:24.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 -- 172.21.15.28:0/3702950050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 msgr2=0x7f23e010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 --2- 172.21.15.28:0/3702950050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e010d5e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009860 tx=0x7f23d402bb70 comp rx=0 tx=0).stop 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 -- 172.21.15.28:0/3702950050 shutdown_connections 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 --2- 172.21.15.28:0/3702950050 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 0x7f23e0118ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 --2- 172.21.15.28:0/3702950050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e010d5e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 --2- 172.21.15.28:0/3702950050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 0x7f23e010aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.035+0000 7f23e5e0d640 1 -- 172.21.15.28:0/3702950050 >> 172.21.15.28:0/3702950050 conn(0x7f23e006e8c0 msgr2=0x7f23e0070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.036+0000 7f23e5e0d640 1 -- 172.21.15.28:0/3702950050 shutdown_connections 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.036+0000 7f23e5e0d640 1 -- 172.21.15.28:0/3702950050 wait complete. 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 Processor -- start 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 -- start start 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 0x7f23e01a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 0x7f23e01ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23e011c430 con 0x7f23e010b170 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23e011c2b0 con 0x7f23e010dc50 2024-03-22T02:20:25.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.037+0000 7f23e5e0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23e011c5b0 con 0x7f23e0073990 2024-03-22T02:20:25.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.038+0000 7f23dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.038+0000 7f23dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56916/0 (socket says 172.21.15.28:56916) 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.038+0000 7f23dffff640 1 -- 172.21.15.28:0/2473076509 learned_addr learned my addr 172.21.15.28:0/2473076509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.038+0000 7f23e4e0b640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 0x7f23e01a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.038+0000 7f23e560c640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 0x7f23e01ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.039+0000 7f23dffff640 1 -- 172.21.15.28:0/2473076509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 msgr2=0x7f23e01a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.039+0000 7f23dffff640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 0x7f23e01a6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.039+0000 7f23dffff640 1 -- 172.21.15.28:0/2473076509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 msgr2=0x7f23e01ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.039+0000 7f23dffff640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 0x7f23e01ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.039+0000 7f23dffff640 1 -- 172.21.15.28:0/2473076509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e01ae670 con 0x7f23e010b170 2024-03-22T02:20:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.040+0000 7f23dffff640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009780 tx=0x7f23d4005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.041+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4041020 con 0x7f23e010b170 2024-03-22T02:20:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.041+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d402cda0 con 0x7f23e010b170 2024-03-22T02:20:25.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.041+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4039040 con 0x7f23e010b170 2024-03-22T02:20:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.041+0000 7f23e5e0d640 1 -- 172.21.15.28:0/2473076509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e01ae900 con 0x7f23e010b170 2024-03-22T02:20:25.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.041+0000 7f23e5e0d640 1 -- 172.21.15.28:0/2473076509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e01aeda0 con 0x7f23e010b170 2024-03-22T02:20:25.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.043+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e0062790 con 0x7f23e010b170 2024-03-22T02:20:25.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.050+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f23d4044020 con 0x7f23e010b170 2024-03-22T02:20:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.050+0000 7f23ddffb640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 0x7f23c80a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.051+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f23d40f1130 con 0x7f23e010b170 2024-03-22T02:20:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.051+0000 7f23e4e0b640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 0x7f23c80a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.052+0000 7f23e4e0b640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 0x7f23c80a55b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f23d8005fd0 tx=0x7f23d80076d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.056+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f23d404b020 con 0x7f23e010b170 2024-03-22T02:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.296+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23e00629b0 con 0x7f23e010b170 2024-03-22T02:20:25.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.297+0000 7f23ddffb640 1 -- 172.21.15.28:0/2473076509 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11867 (secure 0 0 0) 0x7f23d40bdb70 con 0x7f23e010b170 2024-03-22T02:20:25.299 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:20:25.299 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:19:20.368528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26924":{"gid":26924,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2468455656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2468455656},{"type":"v1","addr":"172.21.15.83:6837","nonce":2468455656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:20:25.299 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 msgr2=0x7f23c80a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 0x7f23c80a55b0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f23d8005fd0 tx=0x7f23d80076d0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 msgr2=0x7f23e01a6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009780 tx=0x7f23d4005d60 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 shutdown_connections 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23c80a30f0 0x7f23c80a55b0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f23e010dc50 0x7f23e01ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e010b170 0x7f23e01a6f90 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 --2- 172.21.15.28:0/2473076509 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23e0073990 0x7f23e01a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 >> 172.21.15.28:0/2473076509 conn(0x7f23e006e8c0 msgr2=0x7f23e0109d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 shutdown_connections 2024-03-22T02:20:25.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.299+0000 7f23c77fe640 1 -- 172.21.15.28:0/2473076509 wait complete. 2024-03-22T02:20:25.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:20:25.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 609039 11.99 MB/sec execute 187 sec latency 33.828 ms 2024-03-22T02:20:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 -- 172.21.15.28:0/1252958884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 msgr2=0x7f324c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 --2- 172.21.15.28:0/1252958884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c101fc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f323c009900 tx=0x7f323c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 -- 172.21.15.28:0/1252958884 shutdown_connections 2024-03-22T02:20:25.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 --2- 172.21.15.28:0/1252958884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 0x7f324c109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 --2- 172.21.15.28:0/1252958884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 0x7f324c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 --2- 172.21.15.28:0/1252958884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c101fc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 -- 172.21.15.28:0/1252958884 >> 172.21.15.28:0/1252958884 conn(0x7f324c0fd130 msgr2=0x7f324c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 -- 172.21.15.28:0/1252958884 shutdown_connections 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.424+0000 7f32522fb640 1 -- 172.21.15.28:0/1252958884 wait complete. 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 Processor -- start 2024-03-22T02:20:25.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 -- start start 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 0x7f324c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 0x7f324c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10d810 con 0x7f324c101ba0 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10d690 con 0x7f324c103750 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f32522fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f324c10d990 con 0x7f324c102d90 2024-03-22T02:20:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56932/0 (socket says 172.21.15.28:56932) 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f324bfff640 1 -- 172.21.15.28:0/4193417111 learned_addr learned my addr 172.21.15.28:0/4193417111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f324b7fe640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 0x7f324c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.425+0000 7f3250898640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 0x7f324c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 -- 172.21.15.28:0/4193417111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 msgr2=0x7f324c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 0x7f324c19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 -- 172.21.15.28:0/4193417111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 msgr2=0x7f324c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 0x7f324c1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 -- 172.21.15.28:0/4193417111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f324c1a1d40 con 0x7f324c101ba0 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f324bfff640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f32497fa640 1 -- 172.21.15.28:0/4193417111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c041070 con 0x7f324c101ba0 2024-03-22T02:20:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f32497fa640 1 -- 172.21.15.28:0/4193417111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f323c004450 con 0x7f324c101ba0 2024-03-22T02:20:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f324c1a1fd0 con 0x7f324c101ba0 2024-03-22T02:20:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f32497fa640 1 -- 172.21.15.28:0/4193417111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c034440 con 0x7f324c101ba0 2024-03-22T02:20:25.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.426+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f324c1a23e0 con 0x7f324c101ba0 2024-03-22T02:20:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.428+0000 7f32497fa640 1 -- 172.21.15.28:0/4193417111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f323c044070 con 0x7f324c101ba0 2024-03-22T02:20:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 Processor -- start 2024-03-22T02:20:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 -- start start 2024-03-22T02:20:25.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4b210 0x5565a2f4fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f52810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565a2eaf8d0 con 0x5565a2f4b210 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565a2e57e00 con 0x5565a2f4a820 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3254fc2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565a2eab880 con 0x5565a2f50380 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4b210 0x5565a2f4fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57550/0 (socket says 172.21.15.28:57550) 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 -- 172.21.15.28:0/2154969504 learned_addr learned my addr 172.21.15.28:0/2154969504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f324b7fe640 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f52810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f32497fa640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 0x7f32340a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 0x7f32340a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 -- 172.21.15.28:0/2154969504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 msgr2=0x5565a2f52810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f52810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 -- 172.21.15.28:0/2154969504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4b210 msgr2=0x5565a2f4fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4b210 0x5565a2f4fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f3250898640 1 -- 172.21.15.28:0/2154969504 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a2bc33f0 con 0x5565a2f4a820 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f324b7fe640 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f52810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.429+0000 7f32497fa640 1 -- 172.21.15.28:0/4193417111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f323c0f12d0 con 0x7f324c101ba0 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.430+0000 7f3250898640 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3244005c40 tx=0x7f3244013f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a1586032de6993 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.430+0000 7f3250898640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 0x7f32340a53e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3244013000 tx=0x7f324401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.430+0000 7f32327fc640 1 -- 172.21.15.28:0/2154969504 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244012280 con 0x5565a2f4a820 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.430+0000 7f32327fc640 1 -- 172.21.15.28:0/2154969504 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3244022070 con 0x5565a2f4a820 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.430+0000 7f32327fc640 1 -- 172.21.15.28:0/2154969504 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3244004050 con 0x5565a2f4a820 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- 172.21.15.28:0/2154969504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 msgr2=0x5565a2f4ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3244005c40 tx=0x7f3244013f90 comp rx=0 tx=0).stop 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- 172.21.15.28:0/2154969504 shutdown_connections 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f52810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4b210 0x5565a2f4fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 --2- 172.21.15.28:0/2154969504 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4a820 0x5565a2f4ac40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- 172.21.15.28:0/2154969504 >> 172.21.15.28:0/2154969504 conn(0x5565a2f45d90 msgr2=0x5565a2f46190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- 172.21.15.28:0/2154969504 shutdown_connections 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- 172.21.15.28:0/2154969504 wait complete. 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 -- start start 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.431+0000 7f3254fc2740 1 Processor -- start 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f55240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f5c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565a2f532d0 con 0x5565a2f4a820 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565a2e6b300 con 0x5565a2f50380 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565a2f53450 con 0x5565a2f4b210 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 10 client.0 _ll_register_callbacks cb 0x5565a2ea6e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56950/0 (socket says 172.21.15.28:56950) 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f55240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 -- 172.21.15.28:0/120927193 learned_addr learned my addr 172.21.15.28:0/120927193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3250898640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f5c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 -- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 msgr2=0x5565a2f55240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f55240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 -- 172.21.15.28:0/120927193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 msgr2=0x5565a2f5c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f5c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 -- 172.21.15.28:0/120927193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a2c8d7d0 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f324bfff640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f323c0eed30 tx=0x7f323c003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b79fd927e05f3be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f322b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c04b050 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f323c0b9da0 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.432+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3244022070 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f323c0bd270 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 10 client.17442 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5565a2f5f5f0 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3244022070 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f323c0035f0 con 0x5565a2f4a820 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 10 client.17442 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 10 client.17442 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:20:25.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 20 client.17442 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2f60c80 0x5565a2f630d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 4 client.17442 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565a2ebce10 con 0x5565a2f60c80 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.433+0000 7f324b7fe640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2f60c80 0x5565a2f630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f322b7fe640 10 client.17442 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f324b7fe640 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2f60c80 0x5565a2f630d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d53d715614c53014 server_cookie=7355b9ee34ee88fe in_seq=0 out_seq=0 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f322b7fe640 1 -- 172.21.15.28:0/120927193 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565a2ebce10 con 0x5565a2f60c80 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f322b7fe640 10 client.17442 handle_command_reply: tid=0 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f3254fc2740 1 client.17442 shutdown 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 msgr2=0x5565a2f54d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f323c0eed30 tx=0x7f323c003f40 comp rx=0 tx=0).stop 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 shutdown_connections 2024-03-22T02:20:25.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.434+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2f60c80 0x5565a2f630d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f5c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f55240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 --2- 172.21.15.28:0/120927193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54d00 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f322b7fe640 0 client.17442 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 >> 172.21.15.28:0/120927193 conn(0x5565a2f45d90 msgr2=0x5565a2f44fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 shutdown_connections 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 1 -- 172.21.15.28:0/120927193 wait complete. 2024-03-22T02:20:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.435+0000 7f3254fc2740 20 client.17442 trim_cache size 0 max 16384 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 Processor -- start 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 -- start start 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f5c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f54860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1af460 con 0x5565a2f4a820 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1acd30 con 0x5565a2f50380 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3254fc2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1aab80 con 0x5565a2f4b210 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56954/0 (socket says 172.21.15.28:56954) 2024-03-22T02:20:25.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 -- 172.21.15.28:0/1875398860 learned_addr learned my addr 172.21.15.28:0/1875398860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324b7fe640 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f5c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f3250898640 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f54860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 -- 172.21.15.28:0/1875398860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 msgr2=0x5565a2f5c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f5c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 -- 172.21.15.28:0/1875398860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 msgr2=0x5565a2f54860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f54860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 -- 172.21.15.28:0/1875398860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565a2f63750 con 0x5565a2f4a820 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.449+0000 7f324bfff640 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f323c0ba100 tx=0x7f323c0c17d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7637b9b83026da57 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3231ffb640 1 -- 172.21.15.28:0/1875398860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c04b070 con 0x5565a2f4a820 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3231ffb640 1 -- 172.21.15.28:0/1875398860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f323c004c80 con 0x5565a2f4a820 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3231ffb640 1 -- 172.21.15.28:0/1875398860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c0bd090 con 0x5565a2f4a820 2024-03-22T02:20:25.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 -- 172.21.15.28:0/1875398860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 msgr2=0x5565a2f54320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f323c0ba100 tx=0x7f323c0c17d0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 -- 172.21.15.28:0/1875398860 shutdown_connections 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f50380 0x5565a2f54860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f4b210 0x5565a2f5c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 --2- 172.21.15.28:0/1875398860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f54320 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 -- 172.21.15.28:0/1875398860 >> 172.21.15.28:0/1875398860 conn(0x5565a2f45d90 msgr2=0x5565a2f49880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 -- 172.21.15.28:0/1875398860 shutdown_connections 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.450+0000 7f3254fc2740 1 -- 172.21.15.28:0/1875398860 wait complete. 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 -- start start 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 Processor -- start 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4b210 0x5565a2f59f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f5aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1ab1f0 con 0x5565a2f4a820 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1ac660 con 0x5565a2f4b210 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f324c1ab8c0 con 0x5565a2f50380 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3254fc2740 10 client.0 _ll_register_callbacks cb 0x7f323c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56968/0 (socket says 172.21.15.28:56968) 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 -- 172.21.15.28:0/3361371880 learned_addr learned my addr 172.21.15.28:0/3361371880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324b7fe640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4b210 0x5565a2f59f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f3250898640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f5aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 -- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 msgr2=0x5565a2f5aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f5aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 -- 172.21.15.28:0/3361371880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4b210 msgr2=0x5565a2f59f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4b210 0x5565a2f59f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.451+0000 7f324bfff640 1 -- 172.21.15.28:0/3361371880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565a2f5b4a0 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f324bfff640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f323c005ab0 tx=0x7f323c0bd630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f6dd6200a6e7a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f323c0da070 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f323c0bcd80 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f323c004c80 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f323c04b070 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 10 client.17448 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5565a2f5b4a0 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f323c004c80 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f323c0bc5f0 con 0x5565a2f4a820 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 10 client.17448 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 10 client.17448 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:20:25.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 20 client.17448 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2fc7510 0x5565a2fc9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 4 client.17448 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5565a2eb7160 con 0x5565a2fc7510 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.452+0000 7f324b7fe640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2fc7510 0x5565a2fc9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f322b7fe640 10 client.17448 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f324b7fe640 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2fc7510 0x5565a2fc9930 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1262803238133a9c server_cookie=7f05550972103bc8 in_seq=0 out_seq=0 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f322b7fe640 1 -- 172.21.15.28:0/3361371880 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5565a2eb7160 con 0x5565a2fc7510 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f322b7fe640 10 client.17448 handle_command_reply: tid=0 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 client.17448 shutdown 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 msgr2=0x5565a2f59a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f323c005ab0 tx=0x7f323c0bd630 comp rx=0 tx=0).stop 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 shutdown_connections 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565a2f50380 0x5565a2f5aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5565a2fc7510 0x5565a2fc9930 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565a2f4b210 0x5565a2f59f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 --2- 172.21.15.28:0/3361371880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565a2f4a820 0x5565a2f59a00 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f322b7fe640 0 client.17448 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 >> 172.21.15.28:0/3361371880 conn(0x5565a2f45d90 msgr2=0x5565a2f44740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 shutdown_connections 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 1 -- 172.21.15.28:0/3361371880 wait complete. 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.453+0000 7f3254fc2740 20 client.17448 trim_cache size 0 max 16384 2024-03-22T02:20:25.462 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 msgr2=0x7f32340a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 0x7f32340a53e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3244013000 tx=0x7f324401f000 comp rx=0 tx=0).stop 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 msgr2=0x7f324c19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f323c000c00 tx=0x7f323c02c640 comp rx=0 tx=0).stop 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 shutdown_connections 2024-03-22T02:20:25.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32340a2f20 0x7f32340a53e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f324c103750 0x7f324c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f324c102d90 0x7f324c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 --2- 172.21.15.28:0/4193417111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f324c101ba0 0x7f324c19a120 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 >> 172.21.15.28:0/4193417111 conn(0x7f324c0fd130 msgr2=0x7f324c073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 shutdown_connections 2024-03-22T02:20:25.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.454+0000 7f32522fb640 1 -- 172.21.15.28:0/4193417111 wait complete. 2024-03-22T02:20:25.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:20:25.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:25 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2473076509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 -- 172.21.15.28:0/3768995860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c80ffd50 msgr2=0x7f62c8100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 --2- 172.21.15.28:0/3768995860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c80ffd50 0x7f62c8100170 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009930 tx=0x7f62c402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 -- 172.21.15.28:0/3768995860 shutdown_connections 2024-03-22T02:20:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 --2- 172.21.15.28:0/3768995860 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c8103780 0x7f62c810a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 --2- 172.21.15.28:0/3768995860 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8102dc0 0x7f62c8103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 --2- 172.21.15.28:0/3768995860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c80ffd50 0x7f62c8100170 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 -- 172.21.15.28:0/3768995860 >> 172.21.15.28:0/3768995860 conn(0x7f62c80fb680 msgr2=0x7f62c80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 -- 172.21.15.28:0/3768995860 shutdown_connections 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.572+0000 7f62cf55b640 1 -- 172.21.15.28:0/3768995860 wait complete. 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 Processor -- start 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 -- start start 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c80ffd50 0x7f62c8199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8103780 0x7f62c81a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c810d6e0 con 0x7f62c8102dc0 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62c810d560 con 0x7f62c8103780 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cf55b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62c810d860 con 0x7f62c80ffd50 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56980/0 (socket says 172.21.15.28:56980) 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 -- 172.21.15.28:0/2415113930 learned_addr learned my addr 172.21.15.28:0/2415113930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8103780 0x7f62c81a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62ce559640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c80ffd50 0x7f62c8199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 -- 172.21.15.28:0/2415113930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c80ffd50 msgr2=0x7f62c8199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c80ffd50 0x7f62c8199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 -- 172.21.15.28:0/2415113930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8103780 msgr2=0x7f62c81a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8103780 0x7f62c81a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.573+0000 7f62cdd58640 1 -- 172.21.15.28:0/2415113930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62c81a1bc0 con 0x7f62c8102dc0 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62cdd58640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62b800ea60 tx=0x7f62b800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62bf7fe640 1 -- 172.21.15.28:0/2415113930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b800cde0 con 0x7f62c8102dc0 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62bf7fe640 1 -- 172.21.15.28:0/2415113930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62b80035f0 con 0x7f62c8102dc0 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62c81a1eb0 con 0x7f62c8102dc0 2024-03-22T02:20:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62bf7fe640 1 -- 172.21.15.28:0/2415113930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b8010670 con 0x7f62c8102dc0 2024-03-22T02:20:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.574+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62c81a2370 con 0x7f62c8102dc0 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.576+0000 7f62bf7fe640 1 -- 172.21.15.28:0/2415113930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f62b80037a0 con 0x7f62c8102dc0 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.576+0000 7f62d339d740 1 Processor -- start 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 -- start start 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73deef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62bf7fe640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 0x7f62b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73df1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f73d4e8d0 con 0x562f73de9a10 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562f73cf6e00 con 0x562f73def570 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62d339d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f73d4a880 con 0x562f73dea400 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56990/0 (socket says 172.21.15.28:56990) 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62cdd58640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 0x7f62b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 -- 172.21.15.28:0/2798508377 learned_addr learned my addr 172.21.15.28:0/2798508377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ce559640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73deef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62bf7fe640 1 -- 172.21.15.28:0/2415113930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f62b80cfb10 con 0x7f62c8102dc0 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73df1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 -- 172.21.15.28:0/2798508377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 msgr2=0x562f73deef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73deef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 -- 172.21.15.28:0/2798508377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 msgr2=0x562f73df1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73df1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ced5a640 1 -- 172.21.15.28:0/2798508377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f73a623f0 con 0x562f73de9a10 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.577+0000 7f62ce559640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73deef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62cdd58640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 0x7f62b40a53e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f62b801f000 tx=0x7f62b80912b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f62c0013b50 tx=0x7f62c0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac40b8e2e98c3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62bcff9640 1 -- 172.21.15.28:0/2798508377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c000fd10 con 0x562f73de9a10 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62bcff9640 1 -- 172.21.15.28:0/2798508377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c0013130 con 0x562f73de9a10 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62bcff9640 1 -- 172.21.15.28:0/2798508377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0003bb0 con 0x562f73de9a10 2024-03-22T02:20:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62d339d740 1 -- 172.21.15.28:0/2798508377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 msgr2=0x562f73de9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.578+0000 7f62d339d740 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f62c0013b50 tx=0x7f62c0009b30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- 172.21.15.28:0/2798508377 shutdown_connections 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73df1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73deef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- 172.21.15.28:0/2798508377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73de9e30 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- 172.21.15.28:0/2798508377 >> 172.21.15.28:0/2798508377 conn(0x562f73de4f80 msgr2=0x562f73de5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- 172.21.15.28:0/2798508377 shutdown_connections 2024-03-22T02:20:25.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- 172.21.15.28:0/2798508377 wait complete. 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- start start 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 Processor -- start 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73df3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73dfb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562f73df24c0 con 0x562f73de9a10 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562f73d0a300 con 0x562f73def570 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562f73df2640 con 0x562f73dea400 2024-03-22T02:20:25.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62d339d740 10 client.0 _ll_register_callbacks cb 0x562f73d45e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62ced5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73df3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62ce559640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62ce559640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56674/0 (socket says 172.21.15.28:56674) 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.579+0000 7f62ce559640 1 -- 172.21.15.28:0/1147840071 learned_addr learned my addr 172.21.15.28:0/1147840071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ced5a640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73dfb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 -- 172.21.15.28:0/1147840071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 msgr2=0x562f73dfb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73dfb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 -- 172.21.15.28:0/1147840071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 msgr2=0x562f73df3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73df3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 -- 172.21.15.28:0/1147840071 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f73b2c7d0 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62ce559640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009900 tx=0x7f62c4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33348b75308de6a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4005d50 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c0013130 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f62c4005230 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f62c4005430 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62d339d740 10 client.27292 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562f73dfe7f0 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.580+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c0013130 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f62c4003a00 con 0x562f73dea400 2024-03-22T02:20:25.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 10 client.27292 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 10 client.27292 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 20 client.27292 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73dffe10 0x562f73e022b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 4 client.27292 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f73d5be10 con 0x562f73dffe10 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62ced5a640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73dffe10 0x562f73e022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62b1ffb640 10 client.27292 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.581+0000 7f62ced5a640 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73dffe10 0x562f73e022b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f7bf1938672f2bb server_cookie=864f9e1e10ef63fc in_seq=0 out_seq=0 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62b1ffb640 1 -- 172.21.15.28:0/1147840071 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562f73d5be10 con 0x562f73dffe10 2024-03-22T02:20:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62b1ffb640 10 client.27292 handle_command_reply: tid=0 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 client.27292 shutdown 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 msgr2=0x562f73df4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009900 tx=0x7f62c4004270 comp rx=0 tx=0).stop 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 shutdown_connections 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73dffe10 0x562f73e022b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73dfb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73df4380 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62b1ffb640 0 client.27292 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 --2- 172.21.15.28:0/1147840071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73de9a10 0x562f73df3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 >> 172.21.15.28:0/1147840071 conn(0x562f73de4f80 msgr2=0x562f73de4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 shutdown_connections 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 1 -- 172.21.15.28:0/1147840071 wait complete. 2024-03-22T02:20:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.582+0000 7f62d339d740 20 client.27292 trim_cache size 0 max 16384 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 Processor -- start 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 -- start start 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73de9a10 0x562f73e06b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73e070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81af2a0 con 0x562f73dea400 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81acb70 con 0x562f73def570 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62d339d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81aa9c0 con 0x562f73de9a10 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57008/0 (socket says 172.21.15.28:57008) 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 -- 172.21.15.28:0/4028415003 learned_addr learned my addr 172.21.15.28:0/4028415003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ced5a640 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73de9a10 0x562f73e06b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ced5a640 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73e070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 -- 172.21.15.28:0/4028415003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73de9a10 msgr2=0x562f73e06b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73de9a10 0x562f73e06b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 -- 172.21.15.28:0/4028415003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 msgr2=0x562f73e070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73e070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 -- 172.21.15.28:0/4028415003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f73de4b20 con 0x562f73dea400 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.596+0000 7f62ce559640 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f62c402e000 tx=0x7f62c402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5a72c52ee1b562 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62b3fff640 1 -- 172.21.15.28:0/4028415003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4000e60 con 0x562f73dea400 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62b3fff640 1 -- 172.21.15.28:0/4028415003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4041070 con 0x562f73dea400 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62b3fff640 1 -- 172.21.15.28:0/4028415003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4034d00 con 0x562f73dea400 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- 172.21.15.28:0/4028415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 msgr2=0x562f73dfb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f62c402e000 tx=0x7f62c402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- 172.21.15.28:0/4028415003 shutdown_connections 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73def570 0x562f73e070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73dea400 0x562f73dfb380 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 --2- 172.21.15.28:0/4028415003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73de9a10 0x562f73e06b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- 172.21.15.28:0/4028415003 >> 172.21.15.28:0/4028415003 conn(0x562f73de4f80 msgr2=0x562f73de8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- 172.21.15.28:0/4028415003 shutdown_connections 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- 172.21.15.28:0/4028415003 wait complete. 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.597+0000 7f62d339d740 1 -- start start 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 Processor -- start 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73de9a10 0x562f73dfac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73dfb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81ab030 con 0x562f73def570 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81ac4a0 con 0x562f73de9a10 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f62c81ab700 con 0x562f73dea400 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 10 client.0 _ll_register_callbacks cb 0x7f62c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57024/0 (socket says 172.21.15.28:57024) 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 -- 172.21.15.28:0/2169613003 learned_addr learned my addr 172.21.15.28:0/2169613003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ce559640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73dfb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73de9a10 0x562f73dfac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 -- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 msgr2=0x562f73dfb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73dfb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 -- 172.21.15.28:0/2169613003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73de9a10 msgr2=0x562f73dfac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73de9a10 0x562f73dfac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 -- 172.21.15.28:0/2169613003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f73df9a00 con 0x562f73def570 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62ced5a640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009270 tx=0x7f62c402cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81335ca4a143451f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c002c4d0 con 0x562f73def570 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f62c002c670 con 0x562f73def570 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.598+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c4041070 con 0x562f73def570 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f62c0021450 con 0x562f73def570 2024-03-22T02:20:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 10 client.17466 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562f73e66c10 con 0x562f73def570 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c4041070 con 0x562f73def570 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f62c0020440 con 0x562f73def570 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 10 client.17466 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 10 client.17466 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 20 client.17466 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73e67400 0x562f73e69820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 4 client.17466 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562f73d56160 con 0x562f73e67400 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.599+0000 7f62ce559640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73e67400 0x562f73e69820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62b1ffb640 10 client.17466 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62ce559640 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73e67400 0x562f73e69820 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=973e40cf132b0f84 server_cookie=572a7e2f70dc283d in_seq=0 out_seq=0 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62b1ffb640 1 -- 172.21.15.28:0/2169613003 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562f73d56160 con 0x562f73e67400 2024-03-22T02:20:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62b1ffb640 10 client.17466 handle_command_reply: tid=0 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 client.17466 shutdown 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 msgr2=0x562f73df9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009270 tx=0x7f62c402cdd0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 shutdown_connections 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x562f73e67400 0x562f73e69820 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562f73def570 0x562f73df9290 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62b1ffb640 0 client.17466 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:25.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562f73dea400 0x562f73dfb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 --2- 172.21.15.28:0/2169613003 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562f73de9a10 0x562f73dfac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 >> 172.21.15.28:0/2169613003 conn(0x562f73de4f80 msgr2=0x562f73de3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 shutdown_connections 2024-03-22T02:20:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 1 -- 172.21.15.28:0/2169613003 wait complete. 2024-03-22T02:20:25.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.600+0000 7f62d339d740 20 client.17466 trim_cache size 0 max 16384 2024-03-22T02:20:25.618 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:25.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 msgr2=0x7f62b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 0x7f62b40a53e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f62b801f000 tx=0x7f62b80912b0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 msgr2=0x7f62c819a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f62b800ea60 tx=0x7f62b800ef20 comp rx=0 tx=0).stop 2024-03-22T02:20:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 shutdown_connections 2024-03-22T02:20:25.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f62b40a2f20 0x7f62b40a53e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f62c8103780 0x7f62c81a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c8102dc0 0x7f62c819a4e0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 --2- 172.21.15.28:0/2415113930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f62c80ffd50 0x7f62c8199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 >> 172.21.15.28:0/2415113930 conn(0x7f62c80fb680 msgr2=0x7f62c8100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 shutdown_connections 2024-03-22T02:20:25.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.601+0000 7f62cf55b640 1 -- 172.21.15.28:0/2415113930 wait complete. 2024-03-22T02:20:25.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:20:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/1338612153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100400 msgr2=0x7f5d24100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/1338612153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100400 0x7f5d24100820 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009900 tx=0x7f5d2002bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/1338612153 shutdown_connections 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/1338612153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d24101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/1338612153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100d60 0x7f5d241011c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/1338612153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100400 0x7f5d24100820 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/1338612153 >> 172.21.15.28:0/1338612153 conn(0x7f5d240fc020 msgr2=0x7f5d240fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/1338612153 shutdown_connections 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/1338612153 wait complete. 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 Processor -- start 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- start start 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100400 0x7f5d2419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d241a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410fa40 con 0x7f5d24100d60 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410f8c0 con 0x7f5d24100400 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.729+0000 7f5d2bf3e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2410fbc0 con 0x7f5d24101700 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57032/0 (socket says 172.21.15.28:57032) 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 -- 172.21.15.28:0/4028985237 learned_addr learned my addr 172.21.15.28:0/4028985237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d29cda640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100400 0x7f5d2419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d241a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 -- 172.21.15.28:0/4028985237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 msgr2=0x7f5d241a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d241a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 -- 172.21.15.28:0/4028985237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100400 msgr2=0x7f5d2419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100400 0x7f5d2419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 -- 172.21.15.28:0/4028985237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d241a1cd0 con 0x7f5d24100d60 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d241a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d294d9640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9f0 tx=0x7f5d1400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d1affd640 1 -- 172.21.15.28:0/4028985237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400cd80 con 0x7f5d24100d60 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d1affd640 1 -- 172.21.15.28:0/4028985237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d14010910 con 0x7f5d24100d60 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d241a1fc0 con 0x7f5d24100d60 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d1affd640 1 -- 172.21.15.28:0/4028985237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14018990 con 0x7f5d24100d60 2024-03-22T02:20:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.730+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d241a2480 con 0x7f5d24100d60 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.732+0000 7f5d1affd640 1 -- 172.21.15.28:0/4028985237 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5d14010ac0 con 0x7f5d24100d60 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 Processor -- start 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 -- start start 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d1affd640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 0x7f5d100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb15c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb1ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb1d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55912ba7a8d0 con 0x55912bb1b380 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55912ba22e00 con 0x55912bb16210 2024-03-22T02:20:25.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55912ba76880 con 0x55912bb15820 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2a4db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb1d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d1affd640 1 -- 172.21.15.28:0/4028985237 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5d140cfb90 con 0x7f5d24100d60 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2a4db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb1d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57036/0 (socket says 172.21.15.28:57036) 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d2a4db640 1 -- 172.21.15.28:0/2386781966 learned_addr learned my addr 172.21.15.28:0/2386781966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.733+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb1ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 0x7f5d100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d29cda640 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb15c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 -- 172.21.15.28:0/2386781966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 msgr2=0x55912bb15c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb15c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 -- 172.21.15.28:0/2386781966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 msgr2=0x55912bb1d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb1d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 -- 172.21.15.28:0/2386781966 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55912b78e3f0 con 0x55912bb16210 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb1ad10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5d140cd2d0 tx=0x7f5d1409bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1586c0a3ad87f324 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 0x7f5d100a53e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c012d90 tx=0x7f5d1c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d0ffff640 1 -- 172.21.15.28:0/2386781966 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14070060 con 0x55912bb16210 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d0ffff640 1 -- 172.21.15.28:0/2386781966 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d1409beb0 con 0x55912bb16210 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.734+0000 7f5d0ffff640 1 -- 172.21.15.28:0/2386781966 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14076950 con 0x55912bb16210 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2386781966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 msgr2=0x55912bb1ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb1ad10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5d140cd2d0 tx=0x7f5d1409bb20 comp rx=0 tx=0).stop 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2386781966 shutdown_connections 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb1d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb1ad10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2386781966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb15c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2386781966 >> 172.21.15.28:0/2386781966 conn(0x55912bb10d90 msgr2=0x55912bb11190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2386781966 shutdown_connections 2024-03-22T02:20:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.735+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2386781966 wait complete. 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 -- start start 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 Processor -- start 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb1b380 0x55912bb272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55912bb1e330 con 0x55912bb15820 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55912ba36300 con 0x55912bb1b380 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55912bb1e4b0 con 0x55912bb16210 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2ec03740 10 client.0 _ll_register_callbacks cb 0x55912ba71e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d29cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d294d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d294d9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56744/0 (socket says 172.21.15.28:56744) 2024-03-22T02:20:25.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d29cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57052/0 (socket says 172.21.15.28:57052) 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d2a4db640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb1b380 0x55912bb272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.736+0000 7f5d294d9640 1 -- 172.21.15.28:0/2398881608 learned_addr learned my addr 172.21.15.28:0/2398881608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 -- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb1b380 msgr2=0x55912bb272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb1b380 0x55912bb272f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 -- 172.21.15.28:0/2398881608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 msgr2=0x55912bb1fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 -- 172.21.15.28:0/2398881608 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55912b8587d0 con 0x55912bb16210 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d29cda640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d294d9640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5d1409bbb0 tx=0x7f5d140bc190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97e436bf2ffe3934 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1409a1b0 con 0x55912bb16210 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d1409beb0 con 0x55912bb16210 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5d14076170 con 0x55912bb16210 2024-03-22T02:20:25.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d14076370 con 0x55912bb16210 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d2ec03740 10 client.27310 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.737+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55912bb1e330 con 0x55912bb16210 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d1409beb0 con 0x55912bb16210 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f5d14097c90 con 0x55912bb16210 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 10 client.27310 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 20 client.27310 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb2b8f0 0x55912bb2dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 4 client.27310 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55912ba87e10 con 0x55912bb2b8f0 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d29cda640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb2b8f0 0x55912bb2dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d0d7fa640 10 client.27310 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:25.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.738+0000 7f5d29cda640 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb2b8f0 0x55912bb2dce0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad93ca22cd2b165b server_cookie=7b358afa0399e0e4 in_seq=0 out_seq=0 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/2398881608 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55912ba87e10 con 0x55912bb2b8f0 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d0d7fa640 10 client.27310 handle_command_reply: tid=0 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 client.27310 shutdown 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 msgr2=0x55912bb202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5d1409bbb0 tx=0x7f5d140bc190 comp rx=0 tx=0).stop 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 shutdown_connections 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb2b8f0 0x55912bb2dce0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb1b380 0x55912bb272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb16210 0x55912bb202b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/2398881608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb1fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d0d7fa640 0 client.27310 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 >> 172.21.15.28:0/2398881608 conn(0x55912bb10d90 msgr2=0x55912bb0ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 shutdown_connections 2024-03-22T02:20:25.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 1 -- 172.21.15.28:0/2398881608 wait complete. 2024-03-22T02:20:25.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.740+0000 7f5d2ec03740 20 client.27310 trim_cache size 0 max 16384 2024-03-22T02:20:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:25 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2473076509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.753+0000 7f5d2ec03740 1 Processor -- start 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 -- start start 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb272b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb1b380 0x55912bb325a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241af520 con 0x55912bb15820 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241acdf0 con 0x55912bb16210 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241aac40 con 0x55912bb1b380 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57056/0 (socket says 172.21.15.28:57056) 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 -- 172.21.15.28:0/3885589817 learned_addr learned my addr 172.21.15.28:0/3885589817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d294d9640 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb272b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb1b380 0x55912bb325a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 -- 172.21.15.28:0/3885589817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb1b380 msgr2=0x55912bb325a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb1b380 0x55912bb325a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 -- 172.21.15.28:0/3885589817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 msgr2=0x55912bb272b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb272b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 -- 172.21.15.28:0/3885589817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55912bb11800 con 0x55912bb15820 2024-03-22T02:20:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d29cda640 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ab0 tx=0x7f5d2002df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f76d576c796f182 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d0f7fe640 1 -- 172.21.15.28:0/3885589817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2002cd30 con 0x55912bb15820 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d0f7fe640 1 -- 172.21.15.28:0/3885589817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d20050030 con 0x55912bb15820 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.754+0000 7f5d0f7fe640 1 -- 172.21.15.28:0/3885589817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d200046c0 con 0x55912bb15820 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3885589817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 msgr2=0x55912bb14700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ab0 tx=0x7f5d2002df90 comp rx=0 tx=0).stop 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3885589817 shutdown_connections 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb1b380 0x55912bb325a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb272b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3885589817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb15820 0x55912bb14700 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3885589817 >> 172.21.15.28:0/3885589817 conn(0x55912bb10d90 msgr2=0x55912bb13ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3885589817 shutdown_connections 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3885589817 wait complete. 2024-03-22T02:20:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 -- start start 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.755+0000 7f5d2ec03740 1 Processor -- start 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb24780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb24cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241ab2b0 con 0x55912bb1b380 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241ac720 con 0x55912bb16210 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241ab980 con 0x55912bb15820 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2ec03740 10 client.0 _ll_register_callbacks cb 0x7f5d1c012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d29cda640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb24780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d294d9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb24cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57062/0 (socket says 172.21.15.28:57062) 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 -- 172.21.15.28:0/3322549386 learned_addr learned my addr 172.21.15.28:0/3322549386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 -- 172.21.15.28:0/3322549386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 msgr2=0x55912bb24780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb24780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 -- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 msgr2=0x55912bb24cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb24cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 -- 172.21.15.28:0/3322549386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55912bb262c0 con 0x55912bb1b380 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d2a4db640 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00f6c0 tx=0x7f5d1c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa80d26834eac625 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.756+0000 7f5d0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c004b20 con 0x55912bb1b380 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5d1c004cc0 con 0x55912bb1b380 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d20050030 con 0x55912bb1b380 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5d1c01e6c0 con 0x55912bb1b380 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d2ec03740 10 client.17484 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55912bb91be0 con 0x55912bb1b380 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d20050030 con 0x55912bb1b380 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f5d1c005580 con 0x55912bb1b380 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d2ec03740 10 client.17484 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.757+0000 7f5d2ec03740 10 client.17484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d2ec03740 20 client.17484 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb924f0 0x55912bb948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d2ec03740 4 client.17484 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55912bb1fb60 con 0x55912bb924f0 2024-03-22T02:20:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d29cda640 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb924f0 0x55912bb948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d29cda640 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb924f0 0x55912bb948e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cde66c486e29e57 server_cookie=915ceb2f9137aae1 in_seq=0 out_seq=0 2024-03-22T02:20:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d0d7fa640 10 client.17484 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.758+0000 7f5d0d7fa640 1 -- 172.21.15.28:0/3322549386 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55912bb1fb60 con 0x55912bb924f0 2024-03-22T02:20:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d0d7fa640 10 client.17484 handle_command_reply: tid=0 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 client.17484 shutdown 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 msgr2=0x55912bb25cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00f6c0 tx=0x7f5d1c004ab0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 shutdown_connections 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55912bb924f0 0x55912bb948e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55912bb1b380 0x55912bb25cd0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55912bb16210 0x55912bb24cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 --2- 172.21.15.28:0/3322549386 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55912bb15820 0x55912bb24780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 >> 172.21.15.28:0/3322549386 conn(0x55912bb10d90 msgr2=0x55912bb14880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d0d7fa640 0 client.17484 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 shutdown_connections 2024-03-22T02:20:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 1 -- 172.21.15.28:0/3322549386 wait complete. 2024-03-22T02:20:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.759+0000 7f5d2ec03740 20 client.17484 trim_cache size 0 max 16384 2024-03-22T02:20:25.767 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 msgr2=0x7f5d100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 0x7f5d100a53e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c012d90 tx=0x7f5d1c009500 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 msgr2=0x7f5d2419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9f0 tx=0x7f5d1400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 shutdown_connections 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d100a2f20 0x7f5d100a53e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d24101700 0x7f5d241a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d24100d60 0x7f5d2419a5f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 --2- 172.21.15.28:0/4028985237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d24100400 0x7f5d2419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 >> 172.21.15.28:0/4028985237 conn(0x7f5d240fc020 msgr2=0x7f5d240fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 shutdown_connections 2024-03-22T02:20:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:25.760+0000 7f5d2bf3e640 1 -- 172.21.15.28:0/4028985237 wait complete. 2024-03-22T02:20:25.769 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:25 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2473076509' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:26.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 611442 12.00 MB/sec execute 188 sec latency 15.898 ms 2024-03-22T02:20:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:26 smithi120 ceph-mon[36803]: pgmap v564: 129 pgs: 129 active+clean; 53 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-22T02:20:26.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:26 smithi083 ceph-mon[36340]: pgmap v564: 129 pgs: 129 active+clean; 53 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-22T02:20:26.773 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:20:26.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:20:26.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:20:26.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:26 smithi028 ceph-mon[31916]: pgmap v564: 129 pgs: 129 active+clean; 53 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 448 op/s 2024-03-22T02:20:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1710673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c100a90 msgr2=0x7fa49c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1710673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c100a90 0x7fa49c100eb0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fa48c009900 tx=0x7fa48c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1710673683 shutdown_connections 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1710673683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c102640 0x7fa49c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1710673683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 0x7fa49c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1710673683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c100a90 0x7fa49c100eb0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.885+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1710673683 >> 172.21.15.28:0/1710673683 conn(0x7fa49c0fc020 msgr2=0x7fa49c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1710673683 shutdown_connections 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1710673683 wait complete. 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 Processor -- start 2024-03-22T02:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- start start 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c100a90 0x7fa49c069880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 0x7fa49c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c10c710 con 0x7fa49c102640 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c10c590 con 0x7fa49c101c80 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.886+0000 7fa4a29d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c10c890 con 0x7fa49c100a90 2024-03-22T02:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57072/0 (socket says 172.21.15.28:57072) 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 -- 172.21.15.28:0/1841952374 learned_addr learned my addr 172.21.15.28:0/1841952374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa49bfff640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c100a90 0x7fa49c069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 0x7fa49c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 -- 172.21.15.28:0/1841952374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c100a90 msgr2=0x7fa49c069880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c100a90 0x7fa49c069880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 -- 172.21.15.28:0/1841952374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 msgr2=0x7fa49c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 0x7fa49c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 -- 172.21.15.28:0/1841952374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49c19faf0 con 0x7fa49c102640 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a0f6d640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa49400eb10 tx=0x7fa49400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4997fa640 1 -- 172.21.15.28:0/1841952374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494004b20 con 0x7fa49c102640 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49c19fde0 con 0x7fa49c102640 2024-03-22T02:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4997fa640 1 -- 172.21.15.28:0/1841952374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa494004cc0 con 0x7fa49c102640 2024-03-22T02:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49c1a0350 con 0x7fa49c102640 2024-03-22T02:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.887+0000 7fa4997fa640 1 -- 172.21.15.28:0/1841952374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa494010760 con 0x7fa49c102640 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.889+0000 7fa4997fa640 1 -- 172.21.15.28:0/1841952374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa494010900 con 0x7fa49c102640 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 Processor -- start 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 -- start start 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e75720 0x55a258e77bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a258dd4570 con 0x55a258e6fbc0 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a258d7ce10 con 0x55a258e75720 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a258dd0520 con 0x55a258e705b0 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57084/0 (socket says 172.21.15.28:57084) 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49b7fe640 1 -- 172.21.15.28:0/2098311857 learned_addr learned my addr 172.21.15.28:0/2098311857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e75720 0x55a258e77bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4997fa640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 0x7fa4840a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 -- 172.21.15.28:0/2098311857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e75720 msgr2=0x55a258e77bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e75720 0x55a258e77bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 -- 172.21.15.28:0/2098311857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 msgr2=0x55a258e6ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa49bfff640 1 -- 172.21.15.28:0/2098311857 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a258bb07e0 con 0x55a258e705b0 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4a0f6d640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 0x7fa4840a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.890+0000 7fa4997fa640 1 -- 172.21.15.28:0/1841952374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa4940cfed0 con 0x7fa49c102640 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa49bfff640 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e750b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02c170 tx=0x7fa48c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ca4c141568522f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4827fc640 1 -- 172.21.15.28:0/2098311857 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c02cb80 con 0x55a258e705b0 2024-03-22T02:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4a0f6d640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 0x7fa4840a53e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa49409bef0 tx=0x7fa494066d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4827fc640 1 -- 172.21.15.28:0/2098311857 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c005040 con 0x55a258e705b0 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4827fc640 1 -- 172.21.15.28:0/2098311857 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c004920 con 0x55a258e705b0 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4a55b0740 1 -- 172.21.15.28:0/2098311857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 msgr2=0x55a258e750b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.891+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e750b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa48c02c170 tx=0x7fa48c02c660 comp rx=0 tx=0).stop 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- 172.21.15.28:0/2098311857 shutdown_connections 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e75720 0x55a258e77bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e750b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/2098311857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- 172.21.15.28:0/2098311857 >> 172.21.15.28:0/2098311857 conn(0x55a258e6b130 msgr2=0x55a258e6b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- 172.21.15.28:0/2098311857 shutdown_connections 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- 172.21.15.28:0/2098311857 wait complete. 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- start start 2024-03-22T02:20:26.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 Processor -- start 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e6fbc0 0x55a258e79e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e7a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a258e78670 con 0x55a258e75720 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a258d90110 con 0x55a258e6fbc0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a258e787f0 con 0x55a258e705b0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.892+0000 7fa4a55b0740 10 client.0 _ll_register_callbacks cb 0x55a258dccbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e7a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57092/0 (socket says 172.21.15.28:57092) 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 -- 172.21.15.28:0/993178866 learned_addr learned my addr 172.21.15.28:0/993178866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e6fbc0 0x55a258e79e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 -- 172.21.15.28:0/993178866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 msgr2=0x55a258e7a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e7a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 -- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e6fbc0 msgr2=0x55a258e79e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e6fbc0 0x55a258e79e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 -- 172.21.15.28:0/993178866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a258c10f70 con 0x55a258e75720 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa490013a20 tx=0x7fa490009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e0e9651e7253340 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa47b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490004030 con 0x55a258e75720 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa4900041d0 con 0x55a258e75720 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa48c005040 con 0x55a258e75720 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa4900043d0 con 0x55a258e75720 2024-03-22T02:20:26.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa4a55b0740 10 client.17496 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a258aa8170 con 0x55a258e75720 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.893+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa48c005040 con 0x55a258e75720 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fa4900047a0 con 0x55a258e75720 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 10 client.17496 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 10 client.17496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 20 client.17496 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258e85b40 0x55a258e87fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 4 client.17496 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a258de1ab0 con 0x55a258e85b40 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa49bfff640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258e85b40 0x55a258e87fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa47b7fe640 10 client.17496 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.894+0000 7fa49bfff640 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258e85b40 0x55a258e87fd0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881fe696ac3a022d server_cookie=1514e9c4cbabb939 in_seq=0 out_seq=0 2024-03-22T02:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa47b7fe640 1 -- 172.21.15.28:0/993178866 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa48c0509c0 con 0x55a258e85b40 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa47b7fe640 10 client.17496 handle_command_reply: tid=0 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 client.17496 shutdown 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 msgr2=0x55a258e81310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa490013a20 tx=0x7fa490009c30 comp rx=0 tx=0).stop 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 shutdown_connections 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258e85b40 0x55a258e87fd0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa47b7fe640 0 client.17496 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e75720 0x55a258e81310 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e705b0 0x55a258e7a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/993178866 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e6fbc0 0x55a258e79e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 >> 172.21.15.28:0/993178866 conn(0x55a258e6b130 msgr2=0x55a258e6a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 shutdown_connections 2024-03-22T02:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 1 -- 172.21.15.28:0/993178866 wait complete. 2024-03-22T02:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.896+0000 7fa4a55b0740 20 client.17496 trim_cache size 0 max 16384 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 Processor -- start 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 -- start start 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e705b0 0x55a258e812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e85b40 0x55a258e8c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1ad2f0 con 0x55a258e6fbc0 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1aabc0 con 0x55a258e705b0 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1a8a10 con 0x55a258e85b40 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e705b0 0x55a258e812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57106/0 (socket says 172.21.15.28:57106) 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 -- 172.21.15.28:0/3141238627 learned_addr learned my addr 172.21.15.28:0/3141238627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e85b40 0x55a258e8c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 -- 172.21.15.28:0/3141238627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e85b40 msgr2=0x55a258e8c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e85b40 0x55a258e8c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 -- 172.21.15.28:0/3141238627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e705b0 msgr2=0x55a258e812d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e705b0 0x55a258e812d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.914+0000 7fa49b7fe640 1 -- 172.21.15.28:0/3141238627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a258e70ca0 con 0x55a258e6fbc0 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa490004db0 tx=0x7fa490001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ebfd440cd3589f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa481ffb640 1 -- 172.21.15.28:0/3141238627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490004570 con 0x55a258e6fbc0 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa481ffb640 1 -- 172.21.15.28:0/3141238627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa490004710 con 0x55a258e6fbc0 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa481ffb640 1 -- 172.21.15.28:0/3141238627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4900053f0 con 0x55a258e6fbc0 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 -- 172.21.15.28:0/3141238627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 msgr2=0x55a258e6a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fa490004db0 tx=0x7fa490001e50 comp rx=0 tx=0).stop 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 -- 172.21.15.28:0/3141238627 shutdown_connections 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e85b40 0x55a258e8c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e705b0 0x55a258e812d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/3141238627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e6fbc0 0x55a258e6a270 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.915+0000 7fa4a55b0740 1 -- 172.21.15.28:0/3141238627 >> 172.21.15.28:0/3141238627 conn(0x55a258e6b130 msgr2=0x55a258e69b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- 172.21.15.28:0/3141238627 shutdown_connections 2024-03-22T02:20:26.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- 172.21.15.28:0/3141238627 wait complete. 2024-03-22T02:20:26.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- start start 2024-03-22T02:20:26.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 Processor -- start 2024-03-22T02:20:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e6fbc0 0x55a258e80f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e85b40 0x55a258e7fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1a9080 con 0x55a258e705b0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1aa4f0 con 0x55a258e85b40 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c1a9750 con 0x55a258e6fbc0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa4a55b0740 10 client.0 _ll_register_callbacks cb 0x7fa490009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa49b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e6fbc0 0x55a258e80f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa49bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57112/0 (socket says 172.21.15.28:57112) 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa49bfff640 1 -- 172.21.15.28:0/670416207 learned_addr learned my addr 172.21.15.28:0/670416207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.916+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e85b40 0x55a258e7fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 -- 172.21.15.28:0/670416207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e6fbc0 msgr2=0x55a258e80f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e6fbc0 0x55a258e80f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 -- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e85b40 msgr2=0x55a258e7fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e85b40 0x55a258e7fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 -- 172.21.15.28:0/670416207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a258e80020 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa49bfff640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0099f0 tx=0x7fa48c02c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7855ae0d20d6fc15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c004410 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa48c02c980 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa490004710 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa48c05d050 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa4a55b0740 10 client.17502 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a258eeccd0 con 0x55a258e705b0 2024-03-22T02:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa490004710 con 0x55a258e705b0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.917+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fa48c050070 con 0x55a258e705b0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 10 client.17502 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 10 client.17502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 20 client.17502 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258eed2e0 0x55a258eef700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 4 client.17502 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a258ddbe00 con 0x55a258eed2e0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258eed2e0 0x55a258eef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa49b7fe640 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258eed2e0 0x55a258eef700 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bc94cfe4a984449 server_cookie=a7e74a1663c7a76c in_seq=0 out_seq=0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.918+0000 7fa47b7fe640 10 client.17502 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa47b7fe640 1 -- 172.21.15.28:0/670416207 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a258ddbe00 con 0x55a258eed2e0 2024-03-22T02:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa47b7fe640 10 client.17502 handle_command_reply: tid=0 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 client.17502 shutdown 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 msgr2=0x55a258e7f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0099f0 tx=0x7fa48c02c6d0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 shutdown_connections 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a258eed2e0 0x55a258eef700 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a258e85b40 0x55a258e7fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a258e705b0 0x55a258e7f550 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 --2- 172.21.15.28:0/670416207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a258e6fbc0 0x55a258e80f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa47b7fe640 0 client.17502 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 >> 172.21.15.28:0/670416207 conn(0x55a258e6b130 msgr2=0x55a258e6a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 shutdown_connections 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 1 -- 172.21.15.28:0/670416207 wait complete. 2024-03-22T02:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.921+0000 7fa4a55b0740 20 client.17502 trim_cache size 0 max 16384 2024-03-22T02:20:26.928 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:20:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "487e99f3-89dc-4f20-8c25-0ea31ee7b1e2", 2024-03-22T02:20:26.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:20:26.928 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 msgr2=0x7fa4840a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 0x7fa4840a53e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa49409bef0 tx=0x7fa494066d40 comp rx=0 tx=0).stop 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 msgr2=0x7fa49c19f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fa49400eb10 tx=0x7fa49400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 shutdown_connections 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa4840a2f20 0x7fa4840a53e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa49c102640 0x7fa49c19f230 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa49c101c80 0x7fa49c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 --2- 172.21.15.28:0/1841952374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa49c100a90 0x7fa49c069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:26.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 >> 172.21.15.28:0/1841952374 conn(0x7fa49c0fc020 msgr2=0x7fa49c104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:26.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 shutdown_connections 2024-03-22T02:20:26.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:26.922+0000 7fa4a29d0640 1 -- 172.21.15.28:0/1841952374 wait complete. 2024-03-22T02:20:26.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:20:26.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:20:27.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/3124057004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc01091c0 msgr2=0x7fcbc0109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/3124057004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc01091c0 0x7fcbc0109640 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009900 tx=0x7fcbb402bc40 comp rx=0 tx=0).stop 2024-03-22T02:20:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/3124057004 shutdown_connections 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/3124057004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc0109b80 0x7fcbc0110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/3124057004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc01091c0 0x7fcbc0109640 secure :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009900 tx=0x7fcbb402bc40 comp rx=0 tx=0).stop 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/3124057004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbc0107fd0 0x7fcbc01083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.118+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/3124057004 >> 172.21.15.28:0/3124057004 conn(0x7fcbc0076f70 msgr2=0x7fcbc00793b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.119+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/3124057004 shutdown_connections 2024-03-22T02:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.119+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/3124057004 wait complete. 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 Processor -- start 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 -- start start 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbc0107fd0 0x7fcbc019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc01a3a50 0x7fcbc01a5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc0113cb0 con 0x7fcbc0109b80 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc0113b30 con 0x7fcbc0107fd0 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbc4ab7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc0113e30 con 0x7fcbc01a3a50 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57136/0 (socket says 172.21.15.28:57136) 2024-03-22T02:20:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.120+0000 7fcbbeffd640 1 -- 172.21.15.28:0/2742508825 learned_addr learned my addr 172.21.15.28:0/2742508825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 -- 172.21.15.28:0/2742508825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc01a3a50 msgr2=0x7fcbc01a5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc01a3a50 0x7fcbc01a5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 -- 172.21.15.28:0/2742508825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbc0107fd0 msgr2=0x7fcbc019e4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbc0107fd0 0x7fcbc019e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 -- 172.21.15.28:0/2742508825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc01a64b0 con 0x7fcbc0109b80 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcbb402c250 tx=0x7fcbb4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.121+0000 7fcbbcff9640 1 -- 172.21.15.28:0/2742508825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4041070 con 0x7fcbc0109b80 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.122+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc01a6740 con 0x7fcbc0109b80 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.122+0000 7fcbbcff9640 1 -- 172.21.15.28:0/2742508825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb40054e0 con 0x7fcbc0109b80 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.122+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc01a6c00 con 0x7fcbc0109b80 2024-03-22T02:20:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.122+0000 7fcbbcff9640 1 -- 172.21.15.28:0/2742508825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4034440 con 0x7fcbc0109b80 2024-03-22T02:20:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.125+0000 7fcbbcff9640 1 -- 172.21.15.28:0/2742508825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fcbb4044070 con 0x7fcbc0109b80 2024-03-22T02:20:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.126+0000 7fcbbcff9640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 0x7fcba80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.126+0000 7fcbbcff9640 1 -- 172.21.15.28:0/2742508825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fcbb40f1270 con 0x7fcbc0109b80 2024-03-22T02:20:27.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.126+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 0x7fcba80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.126+0000 7fcbc89d6740 1 Processor -- start 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.126+0000 7fcbc89d6740 1 -- start start 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 0x562338b51a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 0x562338b59630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562338ab68c0 con 0x562338b51640 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562338a5ec20 con 0x562338b52030 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562338ab2870 con 0x562338b571a0 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 0x562338b59630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57784/0 (socket says 172.21.15.28:57784) 2024-03-22T02:20:27.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbbeffd640 1 -- 172.21.15.28:0/2107667230 learned_addr learned my addr 172.21.15.28:0/2107667230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:27.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.127+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 0x562338b51a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 0x7fcba80a5460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0109020 tx=0x7fcbb8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/2107667230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 msgr2=0x562338b59630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 0x562338b59630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/2107667230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 msgr2=0x562338b56b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/2107667230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562338a83150 con 0x562338b51640 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbffff640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 0x562338b59630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.128+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 0x562338b51a60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8005fd0 tx=0x7fcbb8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df7a83884ac3c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.129+0000 7fcba5ffb640 1 -- 172.21.15.28:0/2107667230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8021070 con 0x562338b51640 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.129+0000 7fcba5ffb640 1 -- 172.21.15.28:0/2107667230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb801d9f0 con 0x562338b51640 2024-03-22T02:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.129+0000 7fcba5ffb640 1 -- 172.21.15.28:0/2107667230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb801ca40 con 0x562338b51640 2024-03-22T02:20:27.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.129+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:27.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2107667230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 msgr2=0x562338b51a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 0x562338b51a60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcbb8005fd0 tx=0x7fcbb8011000 comp rx=0 tx=0).stop 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2107667230 shutdown_connections 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b571a0 0x562338b59630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b56b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2107667230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b51640 0x562338b51a60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.130+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2107667230 >> 172.21.15.28:0/2107667230 conn(0x562338b4cbb0 msgr2=0x562338b4cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.131+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2107667230 shutdown_connections 2024-03-22T02:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.132+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2107667230 wait complete. 2024-03-22T02:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.132+0000 7fcbc89d6740 1 -- start start 2024-03-22T02:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbc89d6740 1 Processor -- start 2024-03-22T02:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 0x562338b6d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b571a0 0x562338b74170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 0x562338b6d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.133+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56876/0 (socket says 172.21.15.28:56876) 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 0x562338b6d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57150/0 (socket says 172.21.15.28:57150) 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/294736206 learned_addr learned my addr 172.21.15.28:0/294736206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562338b6b160 con 0x562338b52030 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562338a72130 con 0x562338b571a0 2024-03-22T02:20:27.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562338b6b2e0 con 0x562338b51640 2024-03-22T02:20:27.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbc89d6740 10 client.0 _ll_register_callbacks cb 0x562338a71b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:27.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbffff640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b571a0 0x562338b74170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b571a0 msgr2=0x562338b74170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b571a0 0x562338b74170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/294736206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 msgr2=0x562338b6d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 0x562338b6d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.134+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623388947d0 con 0x562338b51640 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.135+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800a010 tx=0x7fcbb801d720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d42606df4885c535 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.135+0000 7fcb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:20:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb8021070 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fcbb8009040 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb801d9f0 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcbb8004c50 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcbc89d6740 10 client.27343 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5623387ca3f0 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb801d9f0 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.136+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fcbb8005070 con 0x562338b51640 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 10 client.27343 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 10 client.27343 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 20 client.27343 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338b78cc0 0x562338b7b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 4 client.27343 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562338ac3e00 con 0x562338b78cc0 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.137+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338b78cc0 0x562338b7b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.140+0000 7fcb9effd640 10 client.27343 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.140+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338b78cc0 0x562338b7b140 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=898f5b3d1f22ef47 server_cookie=e0a7856ba9b64c26 in_seq=0 out_seq=0 2024-03-22T02:20:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.142+0000 7fcb9effd640 1 -- 172.21.15.28:0/294736206 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562338ac3e00 con 0x562338b78cc0 2024-03-22T02:20:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.142+0000 7fcb9effd640 10 client.27343 handle_command_reply: tid=0 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.142+0000 7fcbc89d6740 1 client.27343 shutdown 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 msgr2=0x562338b6cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800a010 tx=0x7fcbb801d720 comp rx=0 tx=0).stop 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 shutdown_connections 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338b78cc0 0x562338b7b140 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b571a0 0x562338b74170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b52030 0x562338b6d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/294736206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b51640 0x562338b6cbf0 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcbb800a010 tx=0x7fcbb801d720 comp rx=0 tx=0).stop 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 >> 172.21.15.28:0/294736206 conn(0x562338b4cbb0 msgr2=0x562338b4be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcb9effd640 0 client.27343 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 shutdown_connections 2024-03-22T02:20:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.143+0000 7fcbc89d6740 1 -- 172.21.15.28:0/294736206 wait complete. 2024-03-22T02:20:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.144+0000 7fcbc89d6740 20 client.27343 trim_cache size 0 max 16384 2024-03-22T02:20:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.174+0000 7fcbc89d6740 1 Processor -- start 2024-03-22T02:20:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.174+0000 7fcbc89d6740 1 -- start start 2024-03-22T02:20:27.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.174+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.174+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b74130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.174+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 0x562338b6cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc01ab490 con 0x562338b571a0 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc01a8d60 con 0x562338b53cd0 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbbf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56884/0 (socket says 172.21.15.28:56884) 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbbf7fe640 1 -- 172.21.15.28:0/159563410 learned_addr learned my addr 172.21.15.28:0/159563410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbbeffd640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b74130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc0082880 con 0x562338b52030 2024-03-22T02:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.175+0000 7fcbbffff640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 0x562338b6cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbffff640 1 -- 172.21.15.28:0/159563410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 msgr2=0x562338b6c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbffff640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbffff640 1 -- 172.21.15.28:0/159563410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 msgr2=0x562338b74130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbffff640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b74130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbffff640 1 -- 172.21.15.28:0/159563410 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56233878a0f0 con 0x562338b53cd0 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.176+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:27.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.177+0000 7fcbbffff640 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 0x562338b6cd40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00045a0 tx=0x7fcbb00039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f4855d34ed4cdd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.178+0000 7fcba57fa640 1 -- 172.21.15.28:0/159563410 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0005220 con 0x562338b53cd0 2024-03-22T02:20:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.178+0000 7fcba57fa640 1 -- 172.21.15.28:0/159563410 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0004c80 con 0x562338b53cd0 2024-03-22T02:20:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.178+0000 7fcba57fa640 1 -- 172.21.15.28:0/159563410 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000bc40 con 0x562338b53cd0 2024-03-22T02:20:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.179+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 msgr2=0x562338b6cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.179+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 0x562338b6cd40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00045a0 tx=0x7fcbb00039c0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 shutdown_connections 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b53cd0 0x562338b6cd40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b74130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/159563410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b52030 0x562338b6c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 >> 172.21.15.28:0/159563410 conn(0x562338b4cbb0 msgr2=0x562338b535c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 shutdown_connections 2024-03-22T02:20:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 -- 172.21.15.28:0/159563410 wait complete. 2024-03-22T02:20:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.180+0000 7fcbc89d6740 1 -- start start 2024-03-22T02:20:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 Processor -- start 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b71860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b53cd0 0x562338b71da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc01a7220 con 0x562338b571a0 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc01a8690 con 0x562338b52030 2024-03-22T02:20:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc01a78f0 con 0x562338b53cd0 2024-03-22T02:20:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.181+0000 7fcbc89d6740 10 client.0 _ll_register_callbacks cb 0x7fcbb8010430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57162/0 (socket says 172.21.15.28:57162) 2024-03-22T02:20:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 -- 172.21.15.28:0/2602301814 learned_addr learned my addr 172.21.15.28:0/2602301814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b71860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 -- 172.21.15.28:0/2602301814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b53cd0 msgr2=0x562338b71da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b53cd0 0x562338b71da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 -- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 msgr2=0x562338b71860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b71860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.182+0000 7fcbbffff640 1 -- 172.21.15.28:0/2602301814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562338b733a0 con 0x562338b571a0 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.183+0000 7fcbbffff640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0003ec0 tx=0x7fcbb00049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58703955fdc8f62c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.183+0000 7fcb9effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.183+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0004e30 con 0x562338b571a0 2024-03-22T02:20:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.183+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fcbb000bbd0 con 0x562338b571a0 2024-03-22T02:20:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.183+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb0004c80 con 0x562338b571a0 2024-03-22T02:20:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.184+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fcbb002f050 con 0x562338b571a0 2024-03-22T02:20:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.184+0000 7fcbc89d6740 10 client.17520 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.184+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562338bdf0f0 con 0x562338b571a0 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.185+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0004c80 con 0x562338b571a0 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.185+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fcbb0019030 con 0x562338b571a0 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 10 client.17520 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 20 client.17520 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338bdf980 0x562338be1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 4 client.17520 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:20:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562338abe150 con 0x562338bdf980 2024-03-22T02:20:27.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.186+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338bdf980 0x562338be1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcb9effd640 10 client.17520 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcbbf7fe640 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338bdf980 0x562338be1da0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9164886cf8e19730 server_cookie=e92f0bb67a9b8794 in_seq=0 out_seq=0 2024-03-22T02:20:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcb9effd640 1 -- 172.21.15.28:0/2602301814 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562338abe150 con 0x562338bdf980 2024-03-22T02:20:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcb9effd640 10 client.17520 handle_command_reply: tid=0 2024-03-22T02:20:27.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcbc89d6740 1 client.17520 shutdown 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 msgr2=0x562338b72db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.188+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0003ec0 tx=0x7fcbb00049d0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 shutdown_connections 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562338bdf980 0x562338be1da0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562338b571a0 0x562338b72db0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcb9effd640 0 client.17520 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562338b53cd0 0x562338b71da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 --2- 172.21.15.28:0/2602301814 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562338b52030 0x562338b71860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 >> 172.21.15.28:0/2602301814 conn(0x562338b4cbb0 msgr2=0x562338b6c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 shutdown_connections 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 1 -- 172.21.15.28:0/2602301814 wait complete. 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.189+0000 7fcbc89d6740 20 client.17520 trim_cache size 0 max 16384 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "487e99f3-89dc-4f20-8c25-0ea31ee7b1e2": { 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "487e99f3-89dc-4f20-8c25-0ea31ee7b1e2", 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:20:27.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:20:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:20:27.193 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 msgr2=0x7fcba80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 0x7fcba80a5460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0109020 tx=0x7fcbb8009500 comp rx=0 tx=0).stop 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 msgr2=0x7fcbc019e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcbb402c250 tx=0x7fcbb4005dd0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 shutdown_connections 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcba80a2fa0 0x7fcba80a5460 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbc01a3a50 0x7fcbc01a5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbc0109b80 0x7fcbc019e9f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 --2- 172.21.15.28:0/2742508825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbc0107fd0 0x7fcbc019e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:27.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.190+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 >> 172.21.15.28:0/2742508825 conn(0x7fcbc0076f70 msgr2=0x7fcbc010c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:27.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.191+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 shutdown_connections 2024-03-22T02:20:27.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:27.191+0000 7fcbc4ab7640 1 -- 172.21.15.28:0/2742508825 wait complete. 2024-03-22T02:20:27.205 INFO:tasks.cephfs.filesystem:scrub status for tag:487e99f3-89dc-4f20-8c25-0ea31ee7b1e2 - {'path': '/', 'tag': '487e99f3-89dc-4f20-8c25-0ea31ee7b1e2', 'options': 'recursive,force'} 2024-03-22T02:20:27.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 613311 11.98 MB/sec execute 189 sec latency 73.385 ms 2024-03-22T02:20:28.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 615226 11.98 MB/sec execute 190 sec latency 32.091 ms 2024-03-22T02:20:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: pgmap v565: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-03-22T02:20:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:20:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:20:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:20:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:28 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: pgmap v565: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:28 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: pgmap v565: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.0 MiB/s wr, 317 op/s 2024-03-22T02:20:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:20:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:20:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:20:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:28 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:20:29.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 618090 12.01 MB/sec execute 191 sec latency 8.585 ms 2024-03-22T02:20:30.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 619827 11.98 MB/sec execute 192 sec latency 37.517 ms 2024-03-22T02:20:30.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:30 smithi120 ceph-mon[36803]: pgmap v566: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-03-22T02:20:30.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:30 smithi083 ceph-mon[36340]: pgmap v566: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-03-22T02:20:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:30 smithi028 ceph-mon[31916]: pgmap v566: 129 pgs: 129 active+clean; 44 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 404 op/s 2024-03-22T02:20:31.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 622112 11.99 MB/sec execute 193 sec latency 14.857 ms 2024-03-22T02:20:32.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 624108 11.97 MB/sec execute 194 sec latency 61.355 ms 2024-03-22T02:20:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:32 smithi120 ceph-mon[36803]: pgmap v567: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-03-22T02:20:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:32 smithi083 ceph-mon[36340]: pgmap v567: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-03-22T02:20:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:32 smithi028 ceph-mon[31916]: pgmap v567: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 431 op/s 2024-03-22T02:20:33.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 625886 11.97 MB/sec execute 195 sec latency 44.671 ms 2024-03-22T02:20:34.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 628846 12.00 MB/sec execute 196 sec latency 14.908 ms 2024-03-22T02:20:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:34 smithi120 ceph-mon[36803]: pgmap v568: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 423 op/s 2024-03-22T02:20:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:34 smithi083 ceph-mon[36340]: pgmap v568: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 423 op/s 2024-03-22T02:20:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:34 smithi028 ceph-mon[31916]: pgmap v568: 129 pgs: 129 active+clean; 49 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 423 op/s 2024-03-22T02:20:35.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 631047 11.98 MB/sec execute 197 sec latency 27.044 ms 2024-03-22T02:20:35.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:35 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:20:35.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:35 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:20:35.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:35 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:20:36.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 632976 11.98 MB/sec execute 198 sec latency 28.875 ms 2024-03-22T02:20:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:36 smithi120 ceph-mon[36803]: pgmap v569: 129 pgs: 129 active+clean; 51 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 438 op/s 2024-03-22T02:20:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:36 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:20:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:36 smithi083 ceph-mon[36340]: pgmap v569: 129 pgs: 129 active+clean; 51 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 438 op/s 2024-03-22T02:20:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:36 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:20:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:36 smithi028 ceph-mon[31916]: pgmap v569: 129 pgs: 129 active+clean; 51 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 438 op/s 2024-03-22T02:20:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:36 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:20:37.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 635360 11.98 MB/sec execute 199 sec latency 69.167 ms 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 -- 172.21.15.28:0/4290243942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101cd0 msgr2=0x7f2404102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 --2- 172.21.15.28:0/4290243942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101cd0 0x7f2404102150 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009900 tx=0x7f23ec02bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 -- 172.21.15.28:0/4290243942 shutdown_connections 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 --2- 172.21.15.28:0/4290243942 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 0x7f2404108f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 --2- 172.21.15.28:0/4290243942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404101cd0 0x7f2404102150 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 --2- 172.21.15.28:0/4290243942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404100ae0 0x7f2404100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 -- 172.21.15.28:0/4290243942 >> 172.21.15.28:0/4290243942 conn(0x7f24040fc030 msgr2=0x7f24040fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 -- 172.21.15.28:0/4290243942 shutdown_connections 2024-03-22T02:20:37.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.428+0000 7f2408df7640 1 -- 172.21.15.28:0/4290243942 wait complete. 2024-03-22T02:20:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 Processor -- start 2024-03-22T02:20:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 -- start start 2024-03-22T02:20:37.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404101cd0 0x7f24041983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 0x7f240419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c760 con 0x7f2404100ae0 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c5e0 con 0x7f2404102690 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2408df7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f240410c8e0 con 0x7f2404101cd0 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41532/0 (socket says 172.21.15.28:41532) 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f2402d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 0x7f240419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 -- 172.21.15.28:0/1104439 learned_addr learned my addr 172.21.15.28:0/1104439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f23fbfff640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404101cd0 0x7f24041983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 -- 172.21.15.28:0/1104439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404101cd0 msgr2=0x7f24041983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404101cd0 0x7f24041983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 -- 172.21.15.28:0/1104439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 msgr2=0x7f240419f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 0x7f240419f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.429+0000 7f240259c640 1 -- 172.21.15.28:0/1104439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240419fab0 con 0x7f2404100ae0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f240259c640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e9c0 tx=0x7f23f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f23fb7fe640 1 -- 172.21.15.28:0/1104439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f400cd80 con 0x7f2404100ae0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240419fda0 con 0x7f2404100ae0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f23fb7fe640 1 -- 172.21.15.28:0/1104439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23f40035f0 con 0x7f2404100ae0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f23fb7fe640 1 -- 172.21.15.28:0/1104439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4010690 con 0x7f2404100ae0 2024-03-22T02:20:37.433 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.430+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24041a01b0 con 0x7f2404100ae0 2024-03-22T02:20:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.432+0000 7f23fb7fe640 1 -- 172.21.15.28:0/1104439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f23f40037a0 con 0x7f2404100ae0 2024-03-22T02:20:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.432+0000 7f240b9dd740 1 Processor -- start 2024-03-22T02:20:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 -- start start 2024-03-22T02:20:37.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb379d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb3a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4eda97040 con 0x55e4edb324e0 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e4eda3f9e0 con 0x55e4edb38040 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240b9dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e4eda92ff0 con 0x55e4edb32ed0 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41534/0 (socket says 172.21.15.28:41534) 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 -- 172.21.15.28:0/3869629028 learned_addr learned my addr 172.21.15.28:0/3869629028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f240259c640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb379d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f23fbfff640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb3a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f23fb7fe640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 0x7f23f00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 -- 172.21.15.28:0/3869629028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 msgr2=0x55e4edb379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb379d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 -- 172.21.15.28:0/3869629028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 msgr2=0x55e4edb3a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb3a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 -- 172.21.15.28:0/3869629028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4ed7ab210 con 0x55e4edb324e0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f23fbfff640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb3a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f23fb7fe640 1 -- 172.21.15.28:0/1104439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f23f40cfb80 con 0x7f2404100ae0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.433+0000 7f2402d9d640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 0x7f23f00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.434+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f23fc011590 tx=0x7f23fc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf2017c0ab95dd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.434+0000 7f23f8ff9640 1 -- 172.21.15.28:0/3869629028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc017050 con 0x55e4edb324e0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.434+0000 7f23f8ff9640 1 -- 172.21.15.28:0/3869629028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23fc00fd20 con 0x55e4edb324e0 2024-03-22T02:20:37.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.434+0000 7f23f8ff9640 1 -- 172.21.15.28:0/3869629028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc01bb30 con 0x55e4edb324e0 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.434+0000 7f2402d9d640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 0x7f23f00a5480 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f23fc022000 tx=0x7f23fc023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- 172.21.15.28:0/3869629028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 msgr2=0x55e4edb32900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f23fc011590 tx=0x7f23fc011a50 comp rx=0 tx=0).stop 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- 172.21.15.28:0/3869629028 shutdown_connections 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb3a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb379d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3869629028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb32900 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- 172.21.15.28:0/3869629028 >> 172.21.15.28:0/3869629028 conn(0x55e4edb2da50 msgr2=0x55e4edb2de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- 172.21.15.28:0/3869629028 shutdown_connections 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- 172.21.15.28:0/3869629028 wait complete. 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- start start 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 Processor -- start 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb408a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb40de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e4edb3f400 con 0x55e4edb38040 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e4eda52db0 con 0x55e4edb324e0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e4edb3f580 con 0x55e4edb32ed0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240b9dd740 10 client.0 _ll_register_callbacks cb 0x55e4ed7aaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f2402d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f23fbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb40de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f240259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f2402d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41536/0 (socket says 172.21.15.28:41536) 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f23fbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb40de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52946/0 (socket says 172.21.15.28:52946) 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.435+0000 7f2402d9d640 1 -- 172.21.15.28:0/3075949668 learned_addr learned my addr 172.21.15.28:0/3075949668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 -- 172.21.15.28:0/3075949668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 msgr2=0x55e4edb40de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb40de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 -- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 msgr2=0x55e4edb408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb408a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 -- 172.21.15.28:0/3075949668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4ed877620 con 0x55e4edb38040 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f240259c640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb408a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f23fc004c10 tx=0x7f23fc004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ab5c252bb1e23f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f23e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc035060 con 0x55e4edb38040 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f23fc00f640 con 0x55e4edb38040 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23fc00fd20 con 0x55e4edb38040 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23fc017020 con 0x55e4edb38040 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f240b9dd740 10 client.17532 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:37.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4edb4b200 con 0x55e4edb38040 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.436+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23fc00fd20 con 0x55e4edb38040 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f23fc01a070 con 0x55e4edb38040 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 10 client.17532 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 10 client.17532 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 20 client.17532 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edb4c750 0x55e4edb4eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 4 client.17532 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e4edaa4580 con 0x55e4edb4c750 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240259c640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edb4c750 0x55e4edb4eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.447 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f23e9ffb640 10 client.17532 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.437+0000 7f240259c640 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edb4c750 0x55e4edb4eb40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e47af4da76da5329 server_cookie=9859274aa8e3247f in_seq=0 out_seq=0 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f23e9ffb640 1 -- 172.21.15.28:0/3075949668 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e4edaa4580 con 0x55e4edb4c750 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f23e9ffb640 10 client.17532 handle_command_reply: tid=0 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 client.17532 shutdown 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 msgr2=0x55e4edb47e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f23fc004c10 tx=0x7f23fc004c40 comp rx=0 tx=0).stop 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 shutdown_connections 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edb4c750 0x55e4edb4eb40 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb47e20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb40de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3075949668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb408a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f23e9ffb640 0 client.17532 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:37.448 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 >> 172.21.15.28:0/3075949668 conn(0x55e4edb2da50 msgr2=0x55e4edb2cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 shutdown_connections 2024-03-22T02:20:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 1 -- 172.21.15.28:0/3075949668 wait complete. 2024-03-22T02:20:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.439+0000 7f240b9dd740 20 client.17532 trim_cache size 0 max 16384 2024-03-22T02:20:37.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 Processor -- start 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 -- start start 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb2cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb2d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb47de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24041ab000 con 0x55e4edb324e0 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f24041a88d0 con 0x55e4edb38040 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240b9dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f24041a6720 con 0x55e4edb32ed0 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb2cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb2cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41540/0 (socket says 172.21.15.28:41540) 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f240259c640 1 -- 172.21.15.28:0/3459092907 learned_addr learned my addr 172.21.15.28:0/3459092907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:37.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb47de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.456+0000 7f23fbfff640 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb2d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 -- 172.21.15.28:0/3459092907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 msgr2=0x55e4edb2d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb2d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 -- 172.21.15.28:0/3459092907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 msgr2=0x55e4edb2cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb2cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 -- 172.21.15.28:0/3459092907 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e4edb32120 con 0x55e4edb38040 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f2402d9d640 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb47de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006010 tx=0x7f23fc009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db1ea76c0691bec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f23ebfff640 1 -- 172.21.15.28:0/3459092907 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc035040 con 0x55e4edb38040 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f23ebfff640 1 -- 172.21.15.28:0/3459092907 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23fc01be10 con 0x55e4edb38040 2024-03-22T02:20:37.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.457+0000 7f23ebfff640 1 -- 172.21.15.28:0/3459092907 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00fd00 con 0x55e4edb38040 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 -- 172.21.15.28:0/3459092907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 msgr2=0x55e4edb47de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb47de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f23fc006010 tx=0x7f23fc009b30 comp rx=0 tx=0).stop 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 -- 172.21.15.28:0/3459092907 shutdown_connections 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb38040 0x55e4edb47de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb2d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 --2- 172.21.15.28:0/3459092907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb324e0 0x55e4edb2cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 -- 172.21.15.28:0/3459092907 >> 172.21.15.28:0/3459092907 conn(0x55e4edb2da50 msgr2=0x55e4edb2c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 -- 172.21.15.28:0/3459092907 shutdown_connections 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.458+0000 7f240b9dd740 1 -- 172.21.15.28:0/3459092907 wait complete. 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 -- start start 2024-03-22T02:20:37.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 Processor -- start 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb452a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24041a6d90 con 0x55e4edb38040 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f24041a8200 con 0x55e4edb324e0 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f24041a7460 con 0x55e4edb32ed0 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240b9dd740 10 client.0 _ll_register_callbacks cb 0x7f23f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f23fbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb452a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41378/0 (socket says 172.21.15.28:41378) 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f240259c640 1 -- 172.21.15.28:0/444640466 learned_addr learned my addr 172.21.15.28:0/444640466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:37.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.459+0000 7f2402d9d640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 -- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 msgr2=0x55e4edb452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 -- 172.21.15.28:0/444640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 msgr2=0x55e4edb46780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb46780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 -- 172.21.15.28:0/444640466 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e4edb46d70 con 0x55e4edb32ed0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23fbfff640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb457e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009230 tx=0x7f23ec004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1cb21db170e41fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec005030 con 0x55e4edb32ed0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f23ec0051d0 con 0x55e4edb32ed0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.460+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23fc01be10 con 0x55e4edb32ed0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f23ec005410 con 0x55e4edb32ed0 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 10 client.27364 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:37.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e4edb480f0 con 0x55e4edb32ed0 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23fc01be10 con 0x55e4edb32ed0 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f23ec003710 con 0x55e4edb32ed0 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 10 client.27364 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 10 client.27364 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 20 client.27364 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edbb3230 0x55e4edbb5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 4 client.27364 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.461+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e4eda9e8d0 con 0x55e4edbb3230 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.462+0000 7f240259c640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edbb3230 0x55e4edbb5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.462+0000 7f240259c640 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edbb3230 0x55e4edbb5650 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23dd30da0df0551a server_cookie=9901cca0263fbda in_seq=0 out_seq=0 2024-03-22T02:20:37.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.462+0000 7f23e9ffb640 10 client.27364 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f23e9ffb640 1 -- 172.21.15.28:0/444640466 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e4eda9e8d0 con 0x55e4edbb3230 2024-03-22T02:20:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f23e9ffb640 10 client.27364 handle_command_reply: tid=0 2024-03-22T02:20:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 client.27364 shutdown 2024-03-22T02:20:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 msgr2=0x55e4edb457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb457e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23ec009230 tx=0x7f23ec004270 comp rx=0 tx=0).stop 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 shutdown_connections 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e4edbb3230 0x55e4edbb5650 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e4edb38040 0x55e4edb46780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f23e9ffb640 0 client.27364 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e4edb32ed0 0x55e4edb457e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 --2- 172.21.15.28:0/444640466 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e4edb324e0 0x55e4edb452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 >> 172.21.15.28:0/444640466 conn(0x55e4edb2da50 msgr2=0x55e4edb2cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 shutdown_connections 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 1 -- 172.21.15.28:0/444640466 wait complete. 2024-03-22T02:20:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f240b9dd740 20 client.27364 trim_cache size 0 max 16384 2024-03-22T02:20:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:20:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:20:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:20:37.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 msgr2=0x7f23f00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.709+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 0x7f23f00a5480 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f23fc022000 tx=0x7f23fc023000 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 msgr2=0x7f2404197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f23f400e9c0 tx=0x7f23f400ee80 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 shutdown_connections 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23f00a2fc0 0x7f23f00a5480 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2404102690 0x7f240419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2404101cd0 0x7f24041983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 --2- 172.21.15.28:0/1104439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2404100ae0 0x7f2404197e90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 >> 172.21.15.28:0/1104439 conn(0x7f24040fc030 msgr2=0x7f24040fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:37.713 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 shutdown_connections 2024-03-22T02:20:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:37.710+0000 7f2408df7640 1 -- 172.21.15.28:0/1104439 wait complete. 2024-03-22T02:20:38.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 636448 11.95 MB/sec execute 200 sec latency 173.640 ms 2024-03-22T02:20:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:38 smithi120 ceph-mon[36803]: pgmap v570: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 378 op/s 2024-03-22T02:20:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:38 smithi083 ceph-mon[36340]: pgmap v570: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 378 op/s 2024-03-22T02:20:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:38 smithi028 ceph-mon[31916]: pgmap v570: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 378 op/s 2024-03-22T02:20:39.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 639252 11.98 MB/sec execute 201 sec latency 12.542 ms 2024-03-22T02:20:40.421 INFO:tasks.workunit.client.0.smithi028.stdout: 1 640815 11.95 MB/sec execute 202 sec latency 70.323 ms 2024-03-22T02:20:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:40 smithi120 ceph-mon[36803]: pgmap v571: 129 pgs: 129 active+clean; 55 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2024-03-22T02:20:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:40 smithi083 ceph-mon[36340]: pgmap v571: 129 pgs: 129 active+clean; 55 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2024-03-22T02:20:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:40 smithi028 ceph-mon[31916]: pgmap v571: 129 pgs: 129 active+clean; 55 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 476 op/s 2024-03-22T02:20:41.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 643241 11.96 MB/sec execute 203 sec latency 23.009 ms 2024-03-22T02:20:42.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 644779 11.93 MB/sec execute 204 sec latency 52.174 ms 2024-03-22T02:20:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:42 smithi120 ceph-mon[36803]: pgmap v572: 129 pgs: 129 active+clean; 47 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:20:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:42 smithi083 ceph-mon[36340]: pgmap v572: 129 pgs: 129 active+clean; 47 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:20:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:42 smithi028 ceph-mon[31916]: pgmap v572: 129 pgs: 129 active+clean; 47 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 402 op/s 2024-03-22T02:20:43.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 646952 11.94 MB/sec execute 205 sec latency 16.106 ms 2024-03-22T02:20:44.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 649544 11.95 MB/sec execute 206 sec latency 40.320 ms 2024-03-22T02:20:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:44 smithi120 ceph-mon[36803]: pgmap v573: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 346 op/s 2024-03-22T02:20:44.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:44 smithi083 ceph-mon[36340]: pgmap v573: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 346 op/s 2024-03-22T02:20:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:44 smithi028 ceph-mon[31916]: pgmap v573: 129 pgs: 129 active+clean; 48 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 346 op/s 2024-03-22T02:20:45.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 651045 11.93 MB/sec execute 207 sec latency 54.346 ms 2024-03-22T02:20:46.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 653756 11.95 MB/sec execute 208 sec latency 9.061 ms 2024-03-22T02:20:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:46 smithi120 ceph-mon[36803]: pgmap v574: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-03-22T02:20:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:46 smithi083 ceph-mon[36340]: pgmap v574: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-03-22T02:20:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:46 smithi028 ceph-mon[31916]: pgmap v574: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 432 op/s 2024-03-22T02:20:47.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 655419 11.92 MB/sec execute 209 sec latency 71.265 ms 2024-03-22T02:20:48.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 657430 11.93 MB/sec execute 210 sec latency 25.927 ms 2024-03-22T02:20:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:48 smithi120 ceph-mon[36803]: pgmap v575: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 371 op/s 2024-03-22T02:20:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:48 smithi083 ceph-mon[36340]: pgmap v575: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 371 op/s 2024-03-22T02:20:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:48 smithi028 ceph-mon[31916]: pgmap v575: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 371 op/s 2024-03-22T02:20:49.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 659964 11.93 MB/sec execute 211 sec latency 29.503 ms 2024-03-22T02:20:50.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 661456 11.91 MB/sec execute 212 sec latency 34.150 ms 2024-03-22T02:20:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:50 smithi120 ceph-mon[36803]: pgmap v576: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2024-03-22T02:20:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:50 smithi083 ceph-mon[36340]: pgmap v576: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2024-03-22T02:20:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 -- 172.21.15.28:0/3331806364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0101bc0 msgr2=0x7f8ab0101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 --2- 172.21.15.28:0/3331806364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0101fe0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009900 tx=0x7f8aa002bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 -- 172.21.15.28:0/3331806364 shutdown_connections 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 --2- 172.21.15.28:0/3331806364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 0x7f8ab010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 --2- 172.21.15.28:0/3331806364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0102db0 0x7f8ab0103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 --2- 172.21.15.28:0/3331806364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0101fe0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 -- 172.21.15.28:0/3331806364 >> 172.21.15.28:0/3331806364 conn(0x7f8ab00fd150 msgr2=0x7f8ab00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 -- 172.21.15.28:0/3331806364 shutdown_connections 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.800+0000 7f8ab64af640 1 -- 172.21.15.28:0/3331806364 wait complete. 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 Processor -- start 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 -- start start 2024-03-22T02:20:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 0x7f8ab01a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab010d810 con 0x7f8ab0102db0 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab010d690 con 0x7f8ab0103770 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab64af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab010d990 con 0x7f8ab0101bc0 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33274/0 (socket says 172.21.15.28:33274) 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/20391112 learned_addr learned my addr 172.21.15.28:0/20391112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8aaffff640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.801+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 0x7f8ab01a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/20391112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0101bc0 msgr2=0x7f8ab0199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/20391112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 msgr2=0x7f8ab01a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 0x7f8ab01a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/20391112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab01a1c00 con 0x7f8ab0102db0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400e9f0 tx=0x7f8aa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aad7fa640 1 -- 172.21.15.28:0/20391112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa400cd80 con 0x7f8ab0102db0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aad7fa640 1 -- 172.21.15.28:0/20391112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa4010910 con 0x7f8ab0102db0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab01a1ef0 con 0x7f8ab0102db0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8aad7fa640 1 -- 172.21.15.28:0/20391112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4018990 con 0x7f8ab0102db0 2024-03-22T02:20:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.802+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab01a23b0 con 0x7f8ab0102db0 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.804+0000 7f8aad7fa640 1 -- 172.21.15.28:0/20391112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8aa4010ac0 con 0x7f8ab0102db0 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.804+0000 7f8ab9174740 1 Processor -- start 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.804+0000 7f8ab9174740 1 -- start start 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.804+0000 7f8aad7fa640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 0x7f8a980a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c76ded0 0x55d24c7729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c773040 0x55d24c7754d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d24c6d2040 con 0x55d24c773040 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d24c67a9e0 con 0x55d24c76ded0 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab9174740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d24c6cdff0 con 0x55d24c76d4e0 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aad7fa640 1 -- 172.21.15.28:0/20391112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8aa40cfb40 con 0x7f8ab0102db0 2024-03-22T02:20:50.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 0x7f8a980a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60766/0 (socket says 172.21.15.28:60766) 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 -- 172.21.15.28:0/1475659289 learned_addr learned my addr 172.21.15.28:0/1475659289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c76ded0 0x55d24c7729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c773040 0x55d24c7754d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 -- 172.21.15.28:0/1475659289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c76ded0 msgr2=0x55d24c7729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c76ded0 0x55d24c7729d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 -- 172.21.15.28:0/1475659289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c773040 msgr2=0x55d24c7754d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c773040 0x55d24c7754d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 -- 172.21.15.28:0/1475659289 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24c3e6210 con 0x55d24c76d4e0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 0x7f8a980a5480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8005fd0 tx=0x7f8aa8013b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.805+0000 7f8aaffff640 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009900 tx=0x7f8aa0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=749a57ccaf8d081c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8a967fc640 1 -- 172.21.15.28:0/1475659289 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa002cb80 con 0x55d24c76d4e0 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8a967fc640 1 -- 172.21.15.28:0/1475659289 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa0005040 con 0x55d24c76d4e0 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8a967fc640 1 -- 172.21.15.28:0/1475659289 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa0004920 con 0x55d24c76d4e0 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 -- 172.21.15.28:0/1475659289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 msgr2=0x55d24c76d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009900 tx=0x7f8aa0009560 comp rx=0 tx=0).stop 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 -- 172.21.15.28:0/1475659289 shutdown_connections 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c773040 0x55d24c7754d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c76ded0 0x55d24c7729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 --2- 172.21.15.28:0/1475659289 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76d900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 -- 172.21.15.28:0/1475659289 >> 172.21.15.28:0/1475659289 conn(0x55d24c768a50 msgr2=0x55d24c768e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 -- 172.21.15.28:0/1475659289 shutdown_connections 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.806+0000 7f8ab9174740 1 -- 172.21.15.28:0/1475659289 wait complete. 2024-03-22T02:20:50.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 -- start start 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 Processor -- start 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c777990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c77e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d24c775f90 con 0x55d24c76d4e0 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d24c68ddb0 con 0x55d24c773040 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d24c776110 con 0x55d24c76ded0 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c777990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33300/0 (socket says 172.21.15.28:33300) 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab9174740 10 client.0 _ll_register_callbacks cb 0x55d24c3e5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c777990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:60782/0 (socket says 172.21.15.28:60782) 2024-03-22T02:20:50.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8ab4a4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c77e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.807+0000 7f8aaffff640 1 -- 172.21.15.28:0/532839413 learned_addr learned my addr 172.21.15.28:0/532839413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 -- 172.21.15.28:0/532839413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 msgr2=0x55d24c777990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c777990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 -- 172.21.15.28:0/532839413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 msgr2=0x55d24c777450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 -- 172.21.15.28:0/532839413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24c4b2620 con 0x55d24c773040 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8aaffff640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c77e9d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800f790 tx=0x7f8aa8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e133c75697fc9ba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8a8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8003970 con 0x55d24c773040 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aa0005040 con 0x55d24c773040 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8aa8003b10 con 0x55d24c773040 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8aa8033050 con 0x55d24c773040 2024-03-22T02:20:50.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab9174740 10 client.27374 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.808+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d24c781e00 con 0x55d24c773040 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa0005040 con 0x55d24c773040 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f8aa801f070 con 0x55d24c773040 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 10 client.27374 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 10 client.27374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 20 client.27374 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c785960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 4 client.27374 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d24c6df580 con 0x55d24c783530 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8aaffff640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c785960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8aaffff640 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c785960 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eff439935628f53 server_cookie=ba7bc42bbd2c0532 in_seq=0 out_seq=0 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.809+0000 7f8a8f7fe640 10 client.27374 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:50.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.811+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/532839413 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d24c6df580 con 0x55d24c783530 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.811+0000 7f8a8f7fe640 10 client.27374 handle_command_reply: tid=0 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 client.27374 shutdown 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 msgr2=0x55d24c77e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c77e9d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800f790 tx=0x7f8aa8004ab0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 shutdown_connections 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c785960 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c77e9d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c777990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8a8f7fe640 0 client.27374 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 --2- 172.21.15.28:0/532839413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c777450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 >> 172.21.15.28:0/532839413 conn(0x55d24c768a50 msgr2=0x55d24c767c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 shutdown_connections 2024-03-22T02:20:50.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 1 -- 172.21.15.28:0/532839413 wait complete. 2024-03-22T02:20:50.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.812+0000 7f8ab9174740 20 client.27374 trim_cache size 0 max 16384 2024-03-22T02:20:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:50 smithi028 ceph-mon[31916]: pgmap v576: 129 pgs: 129 active+clean; 52 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.2 MiB/s wr, 389 op/s 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 Processor -- start 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 -- start start 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c76e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01af320 con 0x55d24c76ded0 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01acbf0 con 0x55d24c773040 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab9174740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01aaa40 con 0x55d24c76d4e0 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33316/0 (socket says 172.21.15.28:33316) 2024-03-22T02:20:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/825698232 learned_addr learned my addr 172.21.15.28:0/825698232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaffff640 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8ab4a4c640 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c76e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/825698232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 msgr2=0x55d24c76c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/825698232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 msgr2=0x55d24c76e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c76e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/825698232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d24c777ce0 con 0x55d24c76ded0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8aa409be40 tx=0x7f8aa40cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bdbc8668514bf5b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.828+0000 7f8a95ffb640 1 -- 172.21.15.28:0/825698232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40a4810 con 0x55d24c76ded0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8a95ffb640 1 -- 172.21.15.28:0/825698232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa409a1b0 con 0x55d24c76ded0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8a95ffb640 1 -- 172.21.15.28:0/825698232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4018d90 con 0x55d24c76ded0 2024-03-22T02:20:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 -- 172.21.15.28:0/825698232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 msgr2=0x55d24c77e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8aa409be40 tx=0x7f8aa40cf040 comp rx=0 tx=0).stop 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 -- 172.21.15.28:0/825698232 shutdown_connections 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c76e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76ded0 0x55d24c77e990 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 --2- 172.21.15.28:0/825698232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76d4e0 0x55d24c76c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 -- 172.21.15.28:0/825698232 >> 172.21.15.28:0/825698232 conn(0x55d24c768a50 msgr2=0x55d24c76bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 -- 172.21.15.28:0/825698232 shutdown_connections 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.829+0000 7f8ab9174740 1 -- 172.21.15.28:0/825698232 wait complete. 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 -- start start 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 Processor -- start 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c77bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c7e7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01ab0b0 con 0x55d24c76d4e0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01ac520 con 0x55d24c773040 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab01ab780 con 0x55d24c76ded0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab9174740 10 client.0 _ll_register_callbacks cb 0x7f8aa8012050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8aaffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33320/0 (socket says 172.21.15.28:33320) 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8aaf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c77bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8aaffff640 1 -- 172.21.15.28:0/2866710703 learned_addr learned my addr 172.21.15.28:0/2866710703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.830+0000 7f8ab4a4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c7e7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 msgr2=0x55d24c7e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c7e7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/2866710703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 msgr2=0x55d24c77b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 -- 172.21.15.28:0/2866710703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d24c77c220 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaffff640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8aaf7fe640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c77bbe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400e9c0 tx=0x7f8aa4005cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d42e9d1d2febe6e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40a4810 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8aa409a1b0 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8aa409b300 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8aa40b00d0 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 10 client.27382 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d24c7e8ed0 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aa409a1b0 con 0x55d24c76ded0 2024-03-22T02:20:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f8aa4097c00 con 0x55d24c76ded0 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 10 client.27382 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 10 client.27382 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.831+0000 7f8ab9174740 20 client.27382 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c7eb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8ab9174740 4 client.27382 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d24c6d98d0 con 0x55d24c783530 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8aaffff640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c7eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8a8f7fe640 10 client.27382 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:50.832+0000 7f8aaffff640 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c7eb910 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f392130c2821a08 server_cookie=c5f1f8938a28b0ab in_seq=0 out_seq=0 2024-03-22T02:20:51.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8a8f7fe640 1 -- 172.21.15.28:0/2866710703 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d24c6d98d0 con 0x55d24c783530 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8a8f7fe640 10 client.27382 handle_command_reply: tid=0 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 client.27382 shutdown 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 msgr2=0x55d24c77bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c77bbe0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400e9c0 tx=0x7f8aa4005cc0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 shutdown_connections 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d24c783530 0x55d24c7eb910 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d24c773040 0x55d24c7e7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d24c76ded0 0x55d24c77bbe0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8a8f7fe640 0 client.27382 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.123+0000 7f8ab9174740 1 --2- 172.21.15.28:0/2866710703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d24c76d4e0 0x55d24c77b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 >> 172.21.15.28:0/2866710703 conn(0x55d24c768a50 msgr2=0x55d24c76c540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:51.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 shutdown_connections 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab9174740 1 -- 172.21.15.28:0/2866710703 wait complete. 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab9174740 20 client.27382 trim_cache size 0 max 16384 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:20:51.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:20:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 msgr2=0x7f8a980a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 0x7f8a980a5480 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8005fd0 tx=0x7f8aa8013b60 comp rx=0 tx=0).stop 2024-03-22T02:20:51.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 msgr2=0x7f8ab019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.124+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8aa400e9f0 tx=0x7f8aa400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 shutdown_connections 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a980a2fc0 0x7f8a980a5480 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8ab0103770 0x7f8ab01a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab0102db0 0x7f8ab019a520 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 --2- 172.21.15.28:0/20391112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8ab0101bc0 0x7f8ab0199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 >> 172.21.15.28:0/20391112 conn(0x7f8ab00fd150 msgr2=0x7f8ab0073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 shutdown_connections 2024-03-22T02:20:51.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:20:51.125+0000 7f8ab64af640 1 -- 172.21.15.28:0/20391112 wait complete. 2024-03-22T02:20:51.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 663869 11.92 MB/sec execute 213 sec latency 126.966 ms 2024-03-22T02:20:52.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 665145 11.89 MB/sec execute 214 sec latency 117.608 ms 2024-03-22T02:20:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:52 smithi120 ceph-mon[36803]: pgmap v577: 129 pgs: 129 active+clean; 56 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 308 op/s 2024-03-22T02:20:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:52 smithi083 ceph-mon[36340]: pgmap v577: 129 pgs: 129 active+clean; 56 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 308 op/s 2024-03-22T02:20:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:52 smithi028 ceph-mon[31916]: pgmap v577: 129 pgs: 129 active+clean; 56 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 308 op/s 2024-03-22T02:20:53.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 667778 11.91 MB/sec execute 215 sec latency 35.359 ms 2024-03-22T02:20:54.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 670012 11.89 MB/sec execute 216 sec latency 29.110 ms 2024-03-22T02:20:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:54 smithi120 ceph-mon[36803]: pgmap v578: 129 pgs: 129 active+clean; 57 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2024-03-22T02:20:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:54 smithi083 ceph-mon[36340]: pgmap v578: 129 pgs: 129 active+clean; 57 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2024-03-22T02:20:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:54 smithi028 ceph-mon[31916]: pgmap v578: 129 pgs: 129 active+clean; 57 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.8 MiB/s wr, 282 op/s 2024-03-22T02:20:55.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 671765 11.90 MB/sec execute 217 sec latency 19.777 ms 2024-03-22T02:20:56.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 674557 11.91 MB/sec execute 218 sec latency 47.264 ms 2024-03-22T02:20:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:56 smithi120 ceph-mon[36803]: pgmap v579: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 409 op/s 2024-03-22T02:20:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:56 smithi083 ceph-mon[36340]: pgmap v579: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 409 op/s 2024-03-22T02:20:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:56 smithi028 ceph-mon[31916]: pgmap v579: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.5 MiB/s wr, 409 op/s 2024-03-22T02:20:57.206 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:20:57.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:20:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 -- 172.21.15.28:0/2795371309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 msgr2=0x7f48541002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 --2- 172.21.15.28:0/2795371309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f48541002b0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4844009930 tx=0x7f484402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 -- 172.21.15.28:0/2795371309 shutdown_connections 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 --2- 172.21.15.28:0/2795371309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854103820 0x7f485410a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 --2- 172.21.15.28:0/2795371309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854102e60 0x7f48541032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 --2- 172.21.15.28:0/2795371309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f48541002b0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 -- 172.21.15.28:0/2795371309 >> 172.21.15.28:0/2795371309 conn(0x7f48540fb7c0 msgr2=0x7f48540fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 -- 172.21.15.28:0/2795371309 shutdown_connections 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.347+0000 7f485a40e640 1 -- 172.21.15.28:0/2795371309 wait complete. 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 Processor -- start 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 -- start start 2024-03-22T02:20:57.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854102e60 0x7f4854198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854103820 0x7f485419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f485410d860 con 0x7f48540ffe90 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f485410d6e0 con 0x7f4854103820 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f485a40e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f485410d9e0 con 0x7f4854102e60 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56608/0 (socket says 172.21.15.28:56608) 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f4853fff640 1 -- 172.21.15.28:0/124693328 learned_addr learned my addr 172.21.15.28:0/124693328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f48537fe640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854102e60 0x7f4854198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.348+0000 7f48589ab640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854103820 0x7f485419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 -- 172.21.15.28:0/124693328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854102e60 msgr2=0x7f4854198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854102e60 0x7f4854198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 -- 172.21.15.28:0/124693328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854103820 msgr2=0x7f485419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854103820 0x7f485419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 -- 172.21.15.28:0/124693328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485419fb20 con 0x7f48540ffe90 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f4853fff640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f484402c7b0 tx=0x7f484402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f48517fa640 1 -- 172.21.15.28:0/124693328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844041070 con 0x7f48540ffe90 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f48517fa640 1 -- 172.21.15.28:0/124693328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48440054e0 con 0x7f48540ffe90 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485419fdb0 con 0x7f48540ffe90 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f48517fa640 1 -- 172.21.15.28:0/124693328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844034440 con 0x7f48540ffe90 2024-03-22T02:20:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.349+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48541a0210 con 0x7f48540ffe90 2024-03-22T02:20:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.351+0000 7f48517fa640 1 -- 172.21.15.28:0/124693328 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4844044070 con 0x7f48540ffe90 2024-03-22T02:20:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 Processor -- start 2024-03-22T02:20:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 -- start start 2024-03-22T02:20:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 0x55a4c0159850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48517fa640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 0x7f483c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0161420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a4c00be8c0 con 0x55a4c0159e20 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a4c0066c20 con 0x55a4c015ef90 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f485cffa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a4c00ba870 con 0x55a4c0159430 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48589ab640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 0x7f483c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f4853fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 0x55a4c0159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56614/0 (socket says 172.21.15.28:56614) 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 -- 172.21.15.28:0/1663619634 learned_addr learned my addr 172.21.15.28:0/1663619634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48517fa640 1 -- 172.21.15.28:0/124693328 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f48440f1200 con 0x7f48540ffe90 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48589ab640 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 -- 172.21.15.28:0/1663619634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 msgr2=0x55a4c0159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 0x55a4c0159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 -- 172.21.15.28:0/1663619634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 msgr2=0x55a4c0161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0161420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48537fe640 1 -- 172.21.15.28:0/1663619634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4c008b150 con 0x55a4c0159e20 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f4853fff640 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 0x55a4c0159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.352+0000 7f48589ab640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 0x7f483c0a5460 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f484c005fd0 tx=0x7f484c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f48537fe640 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4848009c90 tx=0x7f4848011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4653b77e2b19d328 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f483a7fc640 1 -- 172.21.15.28:0/1663619634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484801e070 con 0x55a4c0159e20 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f483a7fc640 1 -- 172.21.15.28:0/1663619634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4848019b10 con 0x55a4c0159e20 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f483a7fc640 1 -- 172.21.15.28:0/1663619634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484800e9f0 con 0x55a4c0159e20 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 -- 172.21.15.28:0/1663619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 msgr2=0x55a4c015e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4848009c90 tx=0x7f4848011810 comp rx=0 tx=0).stop 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 -- 172.21.15.28:0/1663619634 shutdown_connections 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0161420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159e20 0x55a4c015e920 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 --2- 172.21.15.28:0/1663619634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159430 0x55a4c0159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 -- 172.21.15.28:0/1663619634 >> 172.21.15.28:0/1663619634 conn(0x55a4c01549a0 msgr2=0x55a4c0154da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 -- 172.21.15.28:0/1663619634 shutdown_connections 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.353+0000 7f485cffa740 1 -- 172.21.15.28:0/1663619634 wait complete. 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 -- start start 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 Processor -- start 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c0163e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c016aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56624/0 (socket says 172.21.15.28:56624) 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f4853fff640 1 -- 172.21.15.28:0/482053010 learned_addr learned my addr 172.21.15.28:0/482053010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a4c0161f40 con 0x55a4c0159430 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a4c007a130 con 0x55a4c0159e20 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a4c01620c0 con 0x55a4c015ef90 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f48537fe640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c0163e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48589ab640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c016aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.354+0000 7f485cffa740 10 client.0 _ll_register_callbacks cb 0x55a4c0079b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 -- 172.21.15.28:0/482053010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 msgr2=0x55a4c016aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c016aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 -- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 msgr2=0x55a4c0163e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c0163e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4bfe9c7d0 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f4853fff640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f48440b9f40 tx=0x7f484403aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711d4d73b11c3a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48337fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484404b040 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4848019b10 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f48440035f0 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48440bced0 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f485cffa740 10 client.17568 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4bfdd23f0 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.355+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4844004c80 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f48440e6700 con 0x55a4c0159430 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 10 client.17568 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:57.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 10 client.17568 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 20 client.17568 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c0171bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 4 client.17568 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4848019b10 con 0x55a4c016f730 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f48537fe640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c0171bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f48337fe640 10 client.17568 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.356+0000 7f48537fe640 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c0171bc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=581070425e33dba7 server_cookie=97b734f19b0f4979 in_seq=0 out_seq=0 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.358+0000 7f48337fe640 1 -- 172.21.15.28:0/482053010 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4848019b10 con 0x55a4c016f730 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.358+0000 7f48337fe640 10 client.17568 handle_command_reply: tid=0 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.358+0000 7f485cffa740 1 client.17568 shutdown 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 msgr2=0x55a4c0163950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f48440b9f40 tx=0x7f484403aa90 comp rx=0 tx=0).stop 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 shutdown_connections 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c0171bc0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f48337fe640 0 client.17568 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c016aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c0163e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 --2- 172.21.15.28:0/482053010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163950 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 >> 172.21.15.28:0/482053010 conn(0x55a4c01549a0 msgr2=0x55a4c0153c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 shutdown_connections 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 1 -- 172.21.15.28:0/482053010 wait complete. 2024-03-22T02:20:57.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.359+0000 7f485cffa740 20 client.17568 trim_cache size 0 max 16384 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 Processor -- start 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 -- start start 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c016ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c0158310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48541ad300 con 0x55a4c0159430 2024-03-22T02:20:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f48541aabd0 con 0x55a4c0159e20 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.376+0000 7f485cffa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48541a8a20 con 0x55a4c015ef90 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56632/0 (socket says 172.21.15.28:56632) 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 -- 172.21.15.28:0/2438870720 learned_addr learned my addr 172.21.15.28:0/2438870720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f48589ab640 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c0158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f48537fe640 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c016ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 -- 172.21.15.28:0/2438870720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 msgr2=0x55a4c0158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c0158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 -- 172.21.15.28:0/2438870720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 msgr2=0x55a4c016ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c016ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 -- 172.21.15.28:0/2438870720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4bfd920f0 con 0x55a4c0159430 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4853fff640 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f484402c170 tx=0x7f484402cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246377d76b1451c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4839ffb640 1 -- 172.21.15.28:0/2438870720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484404b040 con 0x55a4c0159430 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4839ffb640 1 -- 172.21.15.28:0/2438870720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4844004c80 con 0x55a4c0159430 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.377+0000 7f4839ffb640 1 -- 172.21.15.28:0/2438870720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440bd570 con 0x55a4c0159430 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- 172.21.15.28:0/2438870720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 msgr2=0x55a4c0163df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f484402c170 tx=0x7f484402cf90 comp rx=0 tx=0).stop 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- 172.21.15.28:0/2438870720 shutdown_connections 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c015ef90 0x55a4c0158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c0159e20 0x55a4c016ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- 172.21.15.28:0/2438870720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c0163df0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- 172.21.15.28:0/2438870720 >> 172.21.15.28:0/2438870720 conn(0x55a4c01549a0 msgr2=0x55a4c0157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- 172.21.15.28:0/2438870720 shutdown_connections 2024-03-22T02:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- 172.21.15.28:0/2438870720 wait complete. 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- start start 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 Processor -- start 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159e20 0x55a4c0168e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0169e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48541a9090 con 0x55a4c0159430 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f48541aa500 con 0x55a4c015ef90 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f48541a9760 con 0x55a4c0159e20 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.378+0000 7f485cffa740 10 client.0 _ll_register_callbacks cb 0x7f48440b9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56648/0 (socket says 172.21.15.28:56648) 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 -- 172.21.15.28:0/4007148930 learned_addr learned my addr 172.21.15.28:0/4007148930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48589ab640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0169e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48537fe640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159e20 0x55a4c0168e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 -- 172.21.15.28:0/4007148930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159e20 msgr2=0x55a4c0168e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159e20 0x55a4c0168e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 -- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 msgr2=0x55a4c0169e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0169e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 -- 172.21.15.28:0/4007148930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4c016a420 con 0x55a4c0159430 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f4853fff640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f48440bb820 tx=0x7f484402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d3719f8c443224 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48337fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48440ee060 con 0x55a4c0159430 2024-03-22T02:20:57.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4844004c80 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f48440c1380 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48440e4050 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f485cffa740 10 client.17574 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a4c016a420 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.379+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4844004c80 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f48440bc9c0 con 0x55a4c0159430 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 10 client.17574 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 20 client.17574 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c01d8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 4 client.17574 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:20:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a4c00cbe00 con 0x55a4c016f730 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48537fe640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c01d8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48537fe640 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c01d8310 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c1d09025c2abb6 server_cookie=114a2c86f43a4769 in_seq=0 out_seq=0 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48337fe640 10 client.17574 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48337fe640 1 -- 172.21.15.28:0/4007148930 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a4c00cbe00 con 0x55a4c016f730 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f48337fe640 10 client.17574 handle_command_reply: tid=0 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.380+0000 7f485cffa740 1 client.17574 shutdown 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 msgr2=0x55a4c01688e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f48440bb820 tx=0x7f484402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 shutdown_connections 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a4c016f730 0x55a4c01d8310 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a4c015ef90 0x55a4c0169e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a4c0159e20 0x55a4c0168e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f48337fe640 0 client.17574 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 --2- 172.21.15.28:0/4007148930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a4c0159430 0x55a4c01688e0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 >> 172.21.15.28:0/4007148930 conn(0x55a4c01549a0 msgr2=0x55a4c0158490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 shutdown_connections 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 1 -- 172.21.15.28:0/4007148930 wait complete. 2024-03-22T02:20:57.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485cffa740 20 client.17574 trim_cache size 0 max 16384 2024-03-22T02:20:57.393 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:20:57.393 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:20:57.393 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 msgr2=0x7f483c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 0x7f483c0a5460 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f484c005fd0 tx=0x7f484c009f10 comp rx=0 tx=0).stop 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 msgr2=0x7f4854197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f484402c7b0 tx=0x7f484402c650 comp rx=0 tx=0).stop 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 shutdown_connections 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f483c0a2fa0 0x7f483c0a5460 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4854103820 0x7f485419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4854102e60 0x7f4854198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 --2- 172.21.15.28:0/124693328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48540ffe90 0x7f4854197f00 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.381+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 >> 172.21.15.28:0/124693328 conn(0x7f48540fb7c0 msgr2=0x7f48541061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.382+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 shutdown_connections 2024-03-22T02:20:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.382+0000 7f485a40e640 1 -- 172.21.15.28:0/124693328 wait complete. 2024-03-22T02:20:57.396 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:20:57.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:20:57.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 675703 11.88 MB/sec execute 219 sec latency 73.704 ms 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 -- 172.21.15.28:0/2938447324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08104f60 msgr2=0x7f8f08107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/2938447324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08104f60 0x7f8f08107350 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009900 tx=0x7f8efc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 -- 172.21.15.28:0/2938447324 shutdown_connections 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/2938447324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08107890 0x7f8f08109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/2938447324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08104f60 0x7f8f08107350 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/2938447324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 0x7f8f08104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 -- 172.21.15.28:0/2938447324 >> 172.21.15.28:0/2938447324 conn(0x7f8f080fc020 msgr2=0x7f8f080fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 -- 172.21.15.28:0/2938447324 shutdown_connections 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.500+0000 7f8f0f18c640 1 -- 172.21.15.28:0/2938447324 wait complete. 2024-03-22T02:20:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 Processor -- start 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 -- start start 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 0x7f8f08197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08104f60 0x7f8f08198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810d820 con 0x7f8f08107890 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810d6a0 con 0x7f8f08102630 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0f18c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8f0810d9a0 con 0x7f8f08104f60 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0d729640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0d729640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56662/0 (socket says 172.21.15.28:56662) 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f07fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08104f60 0x7f8f08198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0d729640 1 -- 172.21.15.28:0/4255736872 learned_addr learned my addr 172.21.15.28:0/4255736872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.501+0000 7f8f0cf28640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 0x7f8f08197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 -- 172.21.15.28:0/4255736872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08104f60 msgr2=0x7f8f08198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08104f60 0x7f8f08198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 -- 172.21.15.28:0/4255736872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 msgr2=0x7f8f08197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 0x7f8f08197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 -- 172.21.15.28:0/4255736872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f0819fae0 con 0x7f8f08107890 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0d729640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800e970 tx=0x7f8ef800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef800ccc0 con 0x7f8f08107890 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ef800ce60 con 0x7f8f08107890 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f0819fdd0 con 0x7f8f08107890 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8010490 con 0x7f8f08107890 2024-03-22T02:20:57.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.502+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f081a0240 con 0x7f8f08107890 2024-03-22T02:20:57.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.503+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f080fdb10 con 0x7f8f08107890 2024-03-22T02:20:57.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.504+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8ef8005020 con 0x7f8f08107890 2024-03-22T02:20:57.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.505+0000 7f8f05ffb640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 0x7f8ef00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.505+0000 7f8f0cf28640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 0x7f8ef00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.505+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8ef80cfc90 con 0x7f8f08107890 2024-03-22T02:20:57.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.505+0000 7f8f0cf28640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 0x7f8ef00a53e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005fd0 tx=0x7f8f00009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.509+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ef809c650 con 0x7f8f08107890 2024-03-22T02:20:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.739+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f08062830 con 0x7f8f08107890 2024-03-22T02:20:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.740+0000 7f8f05ffb640 1 -- 172.21.15.28:0/4255736872 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11867 (secure 0 0 0) 0x7f8ef809bf50 con 0x7f8f08107890 2024-03-22T02:20:57.741 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:20:57.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":32,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:19:20.368528+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26924":{"gid":26924,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2468455656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2468455656},{"type":"v1","addr":"172.21.15.83:6837","nonce":2468455656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:20:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 msgr2=0x7f8ef00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 0x7f8ef00a53e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8f00005fd0 tx=0x7f8f00009500 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 msgr2=0x7f8f0819f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800e970 tx=0x7f8ef800ed70 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 shutdown_connections 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8ef00a2f20 0x7f8ef00a53e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f08107890 0x7f8f0819f470 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f08104f60 0x7f8f08198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 --2- 172.21.15.28:0/4255736872 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8f08102630 0x7f8f08197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 >> 172.21.15.28:0/4255736872 conn(0x7f8f080fc020 msgr2=0x7f8f08103170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 shutdown_connections 2024-03-22T02:20:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.742+0000 7f8f0f18c640 1 -- 172.21.15.28:0/4255736872 wait complete. 2024-03-22T02:20:57.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:20:57.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- 172.21.15.28:0/3504832295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080104f60 msgr2=0x7f8080107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 --2- 172.21.15.28:0/3504832295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080104f60 0x7f8080107350 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f8074009900 tx=0x7f807402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- 172.21.15.28:0/3504832295 shutdown_connections 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 --2- 172.21.15.28:0/3504832295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f8080109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 --2- 172.21.15.28:0/3504832295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080104f60 0x7f8080107350 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 --2- 172.21.15.28:0/3504832295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080102630 0x7f8080104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- 172.21.15.28:0/3504832295 >> 172.21.15.28:0/3504832295 conn(0x7f80800fc040 msgr2=0x7f80800fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- 172.21.15.28:0/3504832295 shutdown_connections 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- 172.21.15.28:0/3504832295 wait complete. 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 Processor -- start 2024-03-22T02:20:57.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.901+0000 7f8086681640 1 -- start start 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080104f60 0x7f8080198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f808019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f808010d840 con 0x7f8080102630 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f808010d6c0 con 0x7f8080107890 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8086681640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f808010d9c0 con 0x7f8080104f60 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56670/0 (socket says 172.21.15.28:56670) 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 -- 172.21.15.28:0/459122315 learned_addr learned my addr 172.21.15.28:0/459122315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8084c1e640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f808019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807f7fe640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080104f60 0x7f8080198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 -- 172.21.15.28:0/459122315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080104f60 msgr2=0x7f8080198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080104f60 0x7f8080198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 -- 172.21.15.28:0/459122315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 msgr2=0x7f808019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f808019f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 -- 172.21.15.28:0/459122315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808019fb60 con 0x7f8080102630 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f8084c1e640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f808019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807ffff640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f807000e9f0 tx=0x7f807000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807d7fa640 1 -- 172.21.15.28:0/459122315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807000cd80 con 0x7f8080102630 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807d7fa640 1 -- 172.21.15.28:0/459122315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8070005500 con 0x7f8080102630 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.902+0000 7f807d7fa640 1 -- 172.21.15.28:0/459122315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8070010690 con 0x7f8080102630 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.903+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808019fe50 con 0x7f8080102630 2024-03-22T02:20:57.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.903+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a0260 con 0x7f8080102630 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f807d7fa640 1 -- 172.21.15.28:0/459122315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f80700046c0 con 0x7f8080102630 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 Processor -- start 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 -- start start 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db3210 0x556566db7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556566d178d0 con 0x556566db8380 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556566cbfe00 con 0x556566db2820 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8089267740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556566d13880 con 0x556566db3210 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.905+0000 7f807ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db3210 0x556566db7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44256/0 (socket says 172.21.15.28:44256) 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 -- 172.21.15.28:0/2336744206 learned_addr learned my addr 172.21.15.28:0/2336744206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f807f7fe640 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56674/0 (socket says 172.21.15.28:56674) 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f807d7fa640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 0x7f80680a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 -- 172.21.15.28:0/2336744206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db3210 msgr2=0x556566db7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db3210 0x556566db7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 -- 172.21.15.28:0/2336744206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 msgr2=0x556566dba810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 -- 172.21.15.28:0/2336744206 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556566a2b3f0 con 0x556566db2820 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f807f7fe640 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 0x7f80680a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f807d7fa640 1 -- 172.21.15.28:0/459122315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f80700cfc20 con 0x7f8080102630 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8078011940 tx=0x7f8078011e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e64e11c485d0558 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f8084c1e640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 0x7f80680a53e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8078023000 tx=0x7f8078024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f80667fc640 1 -- 172.21.15.28:0/2336744206 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078017070 con 0x556566db2820 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f80667fc640 1 -- 172.21.15.28:0/2336744206 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8078011250 con 0x556566db2820 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.906+0000 7f80667fc640 1 -- 172.21.15.28:0/2336744206 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078010c00 con 0x556566db2820 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- 172.21.15.28:0/2336744206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 msgr2=0x556566db2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8078011940 tx=0x7f8078011e00 comp rx=0 tx=0).stop 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- 172.21.15.28:0/2336744206 shutdown_connections 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dba810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db3210 0x556566db7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 --2- 172.21.15.28:0/2336744206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566db2c40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- 172.21.15.28:0/2336744206 >> 172.21.15.28:0/2336744206 conn(0x556566dadd90 msgr2=0x556566dae190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- 172.21.15.28:0/2336744206 shutdown_connections 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- 172.21.15.28:0/2336744206 wait complete. 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.907+0000 7f8089267740 1 -- start start 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 Processor -- start 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dbce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 0x556566dc4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556566dbb330 con 0x556566db3210 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556566cd3300 con 0x556566db8380 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556566dbb4b0 con 0x556566db2820 2024-03-22T02:20:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8089267740 10 client.0 _ll_register_callbacks cb 0x556566d0ee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 0x556566dc4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 0x556566dc4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44258/0 (socket says 172.21.15.28:44258) 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56682/0 (socket says 172.21.15.28:56682) 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f8084c1e640 1 -- 172.21.15.28:0/704218448 learned_addr learned my addr 172.21.15.28:0/704218448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807ffff640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dbce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 -- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 msgr2=0x556566dbce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dbce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 -- 172.21.15.28:0/704218448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 msgr2=0x556566dc4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 0x556566dc4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 -- 172.21.15.28:0/704218448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556566af57d0 con 0x556566db3210 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f807f7fe640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f807402c170 tx=0x7f8074004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d172d6c8cf1e586f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.908+0000 7f805f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074005d50 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f8074005230 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8078011250 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f807404e050 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 10 client.17592 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556566dc7720 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8078011250 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f80740038c0 con 0x556566db3210 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 10 client.17592 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 10 client.17592 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 20 client.17592 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566dc8e90 0x556566dcb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 4 client.17592 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:57.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556566d24e10 con 0x556566dc8e90 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.909+0000 7f807ffff640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566dc8e90 0x556566dcb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.910+0000 7f805f7fe640 10 client.17592 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.910+0000 7f807ffff640 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566dc8e90 0x556566dcb2a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81ea7600882cbbb server_cookie=ec376c14a4577fdc in_seq=0 out_seq=0 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.910+0000 7f805f7fe640 1 -- 172.21.15.28:0/704218448 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556566d24e10 con 0x556566dc8e90 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.910+0000 7f805f7fe640 10 client.17592 handle_command_reply: tid=0 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.910+0000 7f8089267740 1 client.17592 shutdown 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 msgr2=0x556566dbd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f807402c170 tx=0x7f8074004270 comp rx=0 tx=0).stop 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 shutdown_connections 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566dc8e90 0x556566dcb2a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db8380 0x556566dc4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dbd370 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f805f7fe640 0 client.17592 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 --2- 172.21.15.28:0/704218448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dbce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 >> 172.21.15.28:0/704218448 conn(0x556566dadd90 msgr2=0x556566dacfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 shutdown_connections 2024-03-22T02:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 1 -- 172.21.15.28:0/704218448 wait complete. 2024-03-22T02:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.911+0000 7f8089267740 20 client.17592 trim_cache size 0 max 16384 2024-03-22T02:20:57.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 Processor -- start 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 -- start start 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dac740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db3210 0x556566dc4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.924+0000 7f8089267740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80801ad270 con 0x556566db8380 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8089267740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80801aab40 con 0x556566db3210 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8089267740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80801a8990 con 0x556566db2820 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56688/0 (socket says 172.21.15.28:56688) 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f807ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 -- 172.21.15.28:0/3689917524 learned_addr learned my addr 172.21.15.28:0/3689917524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f807f7fe640 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db3210 0x556566dc4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 -- 172.21.15.28:0/3689917524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 msgr2=0x556566dac740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dac740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 -- 172.21.15.28:0/3689917524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db3210 msgr2=0x556566dc4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db3210 0x556566dc4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 -- 172.21.15.28:0/3689917524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556566db2460 con 0x556566db8380 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f807ffff640 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8084c1e640 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f80780095f0 tx=0x7f8078009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd61ab61b90810e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8065ffb640 1 -- 172.21.15.28:0/3689917524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8078017040 con 0x556566db8380 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8065ffb640 1 -- 172.21.15.28:0/3689917524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8078011510 con 0x556566db8380 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.925+0000 7f8065ffb640 1 -- 172.21.15.28:0/3689917524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807801cc90 con 0x556566db8380 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- 172.21.15.28:0/3689917524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 msgr2=0x556566dacc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f80780095f0 tx=0x7f8078009ee0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- 172.21.15.28:0/3689917524 shutdown_connections 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db8380 0x556566dacc80 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db3210 0x556566dc4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 --2- 172.21.15.28:0/3689917524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db2820 0x556566dac740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- 172.21.15.28:0/3689917524 >> 172.21.15.28:0/3689917524 conn(0x556566dadd90 msgr2=0x556566db47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- 172.21.15.28:0/3689917524 shutdown_connections 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- 172.21.15.28:0/3689917524 wait complete. 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.926+0000 7f8089267740 1 -- start start 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 Processor -- start 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566dc1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db8380 0x556566dc31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80801a9000 con 0x556566db3210 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f80801aa470 con 0x556566db2820 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f80801a96d0 con 0x556566db8380 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 10 client.0 _ll_register_callbacks cb 0x7f8074009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566dc1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56692/0 (socket says 172.21.15.28:56692) 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 -- 172.21.15.28:0/656876593 learned_addr learned my addr 172.21.15.28:0/656876593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8084c1e640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db8380 0x556566dc31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 -- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db8380 msgr2=0x556566dc31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db8380 0x556566dc31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 -- 172.21.15.28:0/656876593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 msgr2=0x556566dc1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566dc1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 -- 172.21.15.28:0/656876593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556566dc37a0 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f807f7fe640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f807402bc70 tx=0x7f807402cd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c95a54036dbd36 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f805f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074005d50 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f807404b050 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.927+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8078011510 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8074060050 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 10 client.17598 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556566e2f3c0 con 0x556566db3210 2024-03-22T02:20:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8078011510 con 0x556566db3210 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f8074004880 con 0x556566db3210 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 10 client.17598 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 10 client.17598 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 20 client.17598 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566e2fe10 0x556566e32230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 4 client.17598 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556566d1f160 con 0x556566e2fe10 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f807ffff640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566e2fe10 0x556566e32230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f807ffff640 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566e2fe10 0x556566e32230 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=237cd850a4c86078 server_cookie=30aea154ef09d6ce in_seq=0 out_seq=0 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.928+0000 7f805f7fe640 10 client.17598 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f805f7fe640 1 -- 172.21.15.28:0/656876593 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556566d1f160 con 0x556566e2fe10 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f805f7fe640 10 client.17598 handle_command_reply: tid=0 2024-03-22T02:20:57.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 client.17598 shutdown 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 msgr2=0x556566dc2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f807402bc70 tx=0x7f807402cd90 comp rx=0 tx=0).stop 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 shutdown_connections 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556566db8380 0x556566dc31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556566db3210 0x556566dc2240 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x556566e2fe10 0x556566e32230 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 --2- 172.21.15.28:0/656876593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556566db2820 0x556566dc1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f805f7fe640 0 client.17598 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 >> 172.21.15.28:0/656876593 conn(0x556566dadd90 msgr2=0x556566db0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 shutdown_connections 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 1 -- 172.21.15.28:0/656876593 wait complete. 2024-03-22T02:20:57.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8089267740 20 client.17598 trim_cache size 0 max 16384 2024-03-22T02:20:57.939 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 msgr2=0x7f80680a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 0x7f80680a53e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8078023000 tx=0x7f8078024000 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 msgr2=0x7f8080197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.929+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f807000e9f0 tx=0x7f807000edf0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 shutdown_connections 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f80680a2f20 0x7f80680a53e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8080107890 0x7f808019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8080104f60 0x7f8080198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 --2- 172.21.15.28:0/459122315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080102630 0x7f8080197f40 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 >> 172.21.15.28:0/459122315 conn(0x7f80800fc040 msgr2=0x7f80801030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 shutdown_connections 2024-03-22T02:20:57.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:57.930+0000 7f8086681640 1 -- 172.21.15.28:0/459122315 wait complete. 2024-03-22T02:20:57.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 -- 172.21.15.28:0/2010434185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 msgr2=0x7f0ebc100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/2010434185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc100ed0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009900 tx=0x7f0eb402bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 -- 172.21.15.28:0/2010434185 shutdown_connections 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/2010434185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc102660 0x7f0ebc108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/2010434185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc101ca0 0x7f0ebc102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/2010434185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc100ed0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 -- 172.21.15.28:0/2010434185 >> 172.21.15.28:0/2010434185 conn(0x7f0ebc0fc040 msgr2=0x7f0ebc0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 -- 172.21.15.28:0/2010434185 shutdown_connections 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.052+0000 7f0ec3e39640 1 -- 172.21.15.28:0/2010434185 wait complete. 2024-03-22T02:20:58.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 Processor -- start 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 -- start start 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc101ca0 0x7f0ebc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc102660 0x7f0ebc19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10c730 con 0x7f0ebc100ab0 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10c5b0 con 0x7f0ebc102660 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec3e39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc10c8b0 con 0x7f0ebc101ca0 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56720/0 (socket says 172.21.15.28:56720) 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/1711921350 learned_addr learned my addr 172.21.15.28:0/1711921350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec13d4640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc101ca0 0x7f0ebc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc102660 0x7f0ebc19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/1711921350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc101ca0 msgr2=0x7f0ebc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc101ca0 0x7f0ebc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/1711921350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc102660 msgr2=0x7f0ebc19f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc102660 0x7f0ebc19f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/1711921350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ebc19fb70 con 0x7f0ebc100ab0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.053+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4000c00 tx=0x7f0eb402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.054+0000 7f0eb2ffd640 1 -- 172.21.15.28:0/1711921350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4041070 con 0x7f0ebc100ab0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.054+0000 7f0eb2ffd640 1 -- 172.21.15.28:0/1711921350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eb4004450 con 0x7f0ebc100ab0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.054+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ebc19fda0 con 0x7f0ebc100ab0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.054+0000 7f0eb2ffd640 1 -- 172.21.15.28:0/1711921350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4034440 con 0x7f0ebc100ab0 2024-03-22T02:20:58.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.054+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ebc1a0220 con 0x7f0ebc100ab0 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.056+0000 7f0eb2ffd640 1 -- 172.21.15.28:0/1711921350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0eb4044070 con 0x7f0ebc100ab0 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.056+0000 7f0ec6a23740 1 Processor -- start 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 -- start start 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb96cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96d210 0x5557cb971d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5557cb8d18d0 con 0x5557cb96d210 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5557cb879e00 con 0x5557cb96c820 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5557cb8cd880 con 0x5557cb972380 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0eb2ffd640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 0x7f0ea80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec23d6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb96cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96d210 0x5557cb971d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40334/0 (socket says 172.21.15.28:40334) 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 -- 172.21.15.28:0/1510728870 learned_addr learned my addr 172.21.15.28:0/1510728870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0eb2ffd640 1 -- 172.21.15.28:0/1711921350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0eb40f1290 con 0x7f0ebc100ab0 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 0x7f0ea80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 -- 172.21.15.28:0/1510728870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 msgr2=0x5557cb96cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb96cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 -- 172.21.15.28:0/1510728870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96d210 msgr2=0x5557cb971d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96d210 0x5557cb971d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.057+0000 7f0ec13d4640 1 -- 172.21.15.28:0/1510728870 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557cb5e53f0 con 0x5557cb972380 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec13d4640 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c90 tx=0x7f0eac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd2bd9189ec8620e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 0x7f0ea80a5460 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x5557cb96da70 tx=0x7f0eb8013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ea7fff640 1 -- 172.21.15.28:0/1510728870 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac01e070 con 0x5557cb972380 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ea7fff640 1 -- 172.21.15.28:0/1510728870 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eac019b10 con 0x5557cb972380 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ea7fff640 1 -- 172.21.15.28:0/1510728870 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eac00ea60 con 0x5557cb972380 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 -- 172.21.15.28:0/1510728870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 msgr2=0x5557cb974810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0eac009c90 tx=0x7f0eac011810 comp rx=0 tx=0).stop 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 -- 172.21.15.28:0/1510728870 shutdown_connections 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb974810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96d210 0x5557cb971d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.058+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/1510728870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb96cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- 172.21.15.28:0/1510728870 >> 172.21.15.28:0/1510728870 conn(0x5557cb967d90 msgr2=0x5557cb968190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- 172.21.15.28:0/1510728870 shutdown_connections 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- 172.21.15.28:0/1510728870 wait complete. 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- start start 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 Processor -- start 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96d210 0x5557cb977240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5557cb975330 con 0x5557cb972380 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5557cb88d300 con 0x5557cb96c820 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.059+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5557cb9754b0 con 0x5557cb96d210 2024-03-22T02:20:58.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec6a23740 10 client.0 _ll_register_callbacks cb 0x5557cb8c8e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec13d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96d210 0x5557cb977240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44296/0 (socket says 172.21.15.28:44296) 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/4267509489 learned_addr learned my addr 172.21.15.28:0/4267509489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 -- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96d210 msgr2=0x5557cb977240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96d210 0x5557cb977240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 -- 172.21.15.28:0/4267509489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 msgr2=0x5557cb976d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 -- 172.21.15.28:0/4267509489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557cb6af7d0 con 0x5557cb972380 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97e280 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005c40 tx=0x7f0eb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed32beed88650f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ea57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80047b0 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0eb8011020 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eac019b10 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0eb8011220 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec6a23740 10 client.17610 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.060+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557cb9816c0 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eac019b10 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f0eb801f420 con 0x5557cb972380 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 10 client.17610 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 10 client.17610 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 20 client.17610 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb982e10 0x5557cb9852d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 4 client.17610 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557cb8dee10 con 0x5557cb982e10 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb982e10 0x5557cb9852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ea57fa640 10 client.17610 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.061+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb982e10 0x5557cb9852d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61a5277e3cf490d3 server_cookie=770dad0ce9836f1f in_seq=0 out_seq=0 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.062+0000 7f0ea57fa640 1 -- 172.21.15.28:0/4267509489 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557cb8dee10 con 0x5557cb982e10 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.062+0000 7f0ea57fa640 10 client.17610 handle_command_reply: tid=0 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.062+0000 7f0ec6a23740 1 client.17610 shutdown 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.062+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 msgr2=0x5557cb97e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.062+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97e280 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8005c40 tx=0x7f0eb8004a40 comp rx=0 tx=0).stop 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 shutdown_connections 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb982e10 0x5557cb9852d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97e280 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96d210 0x5557cb977240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/4267509489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96c820 0x5557cb976d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ea57fa640 0 client.17610 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 >> 172.21.15.28:0/4267509489 conn(0x5557cb967d90 msgr2=0x5557cb966fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 shutdown_connections 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 1 -- 172.21.15.28:0/4267509489 wait complete. 2024-03-22T02:20:58.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.063+0000 7f0ec6a23740 20 client.17610 trim_cache size 0 max 16384 2024-03-22T02:20:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 Processor -- start 2024-03-22T02:20:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 -- start start 2024-03-22T02:20:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb96ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb97e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1ad310 con 0x5557cb96c820 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1aabe0 con 0x5557cb96d210 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1a8a30 con 0x5557cb972380 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56736/0 (socket says 172.21.15.28:56736) 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/154891066 learned_addr learned my addr 172.21.15.28:0/154891066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec13d4640 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb96ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.076+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb97e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/154891066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 msgr2=0x5557cb97e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb97e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/154891066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 msgr2=0x5557cb96ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb96ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 -- 172.21.15.28:0/154891066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557cb96d5f0 con 0x5557cb96c820 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40bd620 tx=0x7f0eb40903c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0ea4aa247f9ab7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ea77fe640 1 -- 172.21.15.28:0/154891066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb404b040 con 0x5557cb96c820 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ea77fe640 1 -- 172.21.15.28:0/154891066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eb4004c80 con 0x5557cb96c820 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.077+0000 7f0ea77fe640 1 -- 172.21.15.28:0/154891066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb40c5f60 con 0x5557cb96c820 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- 172.21.15.28:0/154891066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 msgr2=0x5557cb96e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0eb40bd620 tx=0x7f0eb40903c0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- 172.21.15.28:0/154891066 shutdown_connections 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb972380 0x5557cb97e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb96ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/154891066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb96c820 0x5557cb96e710 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- 172.21.15.28:0/154891066 >> 172.21.15.28:0/154891066 conn(0x5557cb967d90 msgr2=0x5557cb976d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- 172.21.15.28:0/154891066 shutdown_connections 2024-03-22T02:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- 172.21.15.28:0/154891066 wait complete. 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- start start 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 Processor -- start 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 0x5557cb97b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb97b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1a90a0 con 0x5557cb972380 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1aa510 con 0x5557cb96d210 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ebc1a9770 con 0x5557cb96c820 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec6a23740 10 client.0 _ll_register_callbacks cb 0x7f0eb8009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec23d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec1bd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 0x5557cb97b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec13d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb97b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.078+0000 7f0ec23d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56738/0 (socket says 172.21.15.28:56738) 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 -- 172.21.15.28:0/2307378484 learned_addr learned my addr 172.21.15.28:0/2307378484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 -- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 msgr2=0x5557cb97b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 0x5557cb97b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 -- 172.21.15.28:0/2307378484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 msgr2=0x5557cb97b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb97b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 -- 172.21.15.28:0/2307378484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557cb97cde0 con 0x5557cb972380 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 0x5557cb97b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec23d6640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009ed0 tx=0x7f0eb8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9be040b2d027daa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ea57fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8003c80 con 0x5557cb972380 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0eb801fcc0 con 0x5557cb972380 2024-03-22T02:20:58.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb4004c80 con 0x5557cb972380 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0eb8044050 con 0x5557cb972380 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec6a23740 10 client.17616 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.079+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5557cb97e150 con 0x5557cb972380 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0eb4004c80 con 0x5557cb972380 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f0eb8028420 con 0x5557cb972380 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 10 client.17616 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 10 client.17616 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 20 client.17616 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb9e95a0 0x5557cb9eb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 4 client.17616 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5557cb8d9160 con 0x5557cb9e95a0 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb9e95a0 0x5557cb9eb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ea57fa640 10 client.17616 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec1bd5640 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb9e95a0 0x5557cb9eb990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac2e013dcff1d3ba server_cookie=bae7b43b98379a41 in_seq=0 out_seq=0 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ea57fa640 1 -- 172.21.15.28:0/2307378484 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5557cb8d9160 con 0x5557cb9e95a0 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ea57fa640 10 client.17616 handle_command_reply: tid=0 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.080+0000 7f0ec6a23740 1 client.17616 shutdown 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 msgr2=0x5557cb97c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0eb8009ed0 tx=0x7f0eb8009580 comp rx=0 tx=0).stop 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 shutdown_connections 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5557cb9e95a0 0x5557cb9eb990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557cb972380 0x5557cb97c7f0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557cb96d210 0x5557cb97b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 --2- 172.21.15.28:0/2307378484 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557cb96c820 0x5557cb97b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ea57fa640 0 client.17616 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:20:58.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 >> 172.21.15.28:0/2307378484 conn(0x5557cb967d90 msgr2=0x5557cb976ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 shutdown_connections 2024-03-22T02:20:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 1 -- 172.21.15.28:0/2307378484 wait complete. 2024-03-22T02:20:58.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec6a23740 20 client.17616 trim_cache size 0 max 16384 2024-03-22T02:20:58.090 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 msgr2=0x7f0ea80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 0x7f0ea80a5460 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x5557cb96da70 tx=0x7f0eb8013040 comp rx=0 tx=0).stop 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 msgr2=0x7f0ebc06acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4000c00 tx=0x7f0eb402c640 comp rx=0 tx=0).stop 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 shutdown_connections 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ea80a2fa0 0x7f0ea80a5460 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ebc102660 0x7f0ebc19f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ebc101ca0 0x7f0ebc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 --2- 172.21.15.28:0/1711921350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ebc100ab0 0x7f0ebc06acf0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 >> 172.21.15.28:0/1711921350 conn(0x7f0ebc0fc040 msgr2=0x7f0ebc105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 shutdown_connections 2024-03-22T02:20:58.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.081+0000 7f0ec3e39640 1 -- 172.21.15.28:0/1711921350 wait complete. 2024-03-22T02:20:58.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:20:58.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.253+0000 7fce25fe9640 1 -- 172.21.15.28:0/1229245148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce20101ba0 msgr2=0x7fce20101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 --2- 172.21.15.28:0/1229245148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce20101ba0 0x7fce20101fc0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fce10009900 tx=0x7fce1002bc40 comp rx=0 tx=0).stop 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 -- 172.21.15.28:0/1229245148 shutdown_connections 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 --2- 172.21.15.28:0/1229245148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 0x7fce20109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 --2- 172.21.15.28:0/1229245148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 --2- 172.21.15.28:0/1229245148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce20101ba0 0x7fce20101fc0 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fce10009900 tx=0x7fce1002bc40 comp rx=0 tx=0).stop 2024-03-22T02:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.254+0000 7fce25fe9640 1 -- 172.21.15.28:0/1229245148 >> 172.21.15.28:0/1229245148 conn(0x7fce200fd0f0 msgr2=0x7fce200ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.255+0000 7fce25fe9640 1 -- 172.21.15.28:0/1229245148 shutdown_connections 2024-03-22T02:20:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.255+0000 7fce25fe9640 1 -- 172.21.15.28:0/1229245148 wait complete. 2024-03-22T02:20:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.256+0000 7fce25fe9640 1 Processor -- start 2024-03-22T02:20:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 -- start start 2024-03-22T02:20:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 0x7fce20198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce2019d5b0 0x7fce2019f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce2010d840 con 0x7fce20102d90 2024-03-22T02:20:58.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce2010d6c0 con 0x7fce2019d5b0 2024-03-22T02:20:58.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce25fe9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce2010d9c0 con 0x7fce20103750 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56754/0 (socket says 172.21.15.28:56754) 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2911163160 learned_addr learned my addr 172.21.15.28:0/2911163160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.257+0000 7fce1ffff640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce2019d5b0 0x7fce2019f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1effd640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 0x7fce20198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2911163160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 msgr2=0x7fce20198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 0x7fce20198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2911163160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce2019d5b0 msgr2=0x7fce2019f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce2019d5b0 0x7fce2019f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.258+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2911163160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce20073ec0 con 0x7fce20102d90 2024-03-22T02:20:58.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.259+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fce10036040 tx=0x7fce1002ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.260+0000 7fce1cff9640 1 -- 172.21.15.28:0/2911163160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10041070 con 0x7fce20102d90 2024-03-22T02:20:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.260+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce20074150 con 0x7fce20102d90 2024-03-22T02:20:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.260+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce20074660 con 0x7fce20102d90 2024-03-22T02:20:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.260+0000 7fce1cff9640 1 -- 172.21.15.28:0/2911163160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce100054e0 con 0x7fce20102d90 2024-03-22T02:20:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.260+0000 7fce1cff9640 1 -- 172.21.15.28:0/2911163160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10034440 con 0x7fce20102d90 2024-03-22T02:20:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.262+0000 7fce1cff9640 1 -- 172.21.15.28:0/2911163160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fce10044070 con 0x7fce20102d90 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce1cff9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 0x7fce080a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce1cff9640 1 -- 172.21.15.28:0/2911163160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fce100f1260 con 0x7fce20102d90 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce1effd640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 0x7fce080a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 Processor -- start 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 -- start start 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdc24378d0 con 0x55cdc24d8380 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cdc23dfe00 con 0x55cdc24d2820 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.263+0000 7fce28bd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cdc2433880 con 0x55cdc24d3210 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56758/0 (socket says 172.21.15.28:56758) 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24d2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44346/0 (socket says 172.21.15.28:44346) 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 -- 172.21.15.28:0/935518032 learned_addr learned my addr 172.21.15.28:0/935518032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1effd640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 0x7fce080a54b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fce14005ef0 tx=0x7fce14005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1effd640 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 -- 172.21.15.28:0/935518032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 msgr2=0x55cdc24d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24d7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 -- 172.21.15.28:0/935518032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 msgr2=0x55cdc24d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 -- 172.21.15.28:0/935518032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdc214b3f0 con 0x55cdc24d8380 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1effd640 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24d7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce1ffff640 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fce18007f40 tx=0x7fce1800f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c225ae47671f488 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce05ffb640 1 -- 172.21.15.28:0/935518032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18010900 con 0x55cdc24d8380 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce05ffb640 1 -- 172.21.15.28:0/935518032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce180183f0 con 0x55cdc24d8380 2024-03-22T02:20:58.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.264+0000 7fce05ffb640 1 -- 172.21.15.28:0/935518032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18017600 con 0x55cdc24d8380 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 -- 172.21.15.28:0/935518032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 msgr2=0x55cdc24da810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fce18007f40 tx=0x7fce1800f850 comp rx=0 tx=0).stop 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 -- 172.21.15.28:0/935518032 shutdown_connections 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24da810 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24d7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 --2- 172.21.15.28:0/935518032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 -- 172.21.15.28:0/935518032 >> 172.21.15.28:0/935518032 conn(0x55cdc24cdd90 msgr2=0x55cdc24ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 -- 172.21.15.28:0/935518032 shutdown_connections 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.265+0000 7fce28bd5740 1 -- 172.21.15.28:0/935518032 wait complete. 2024-03-22T02:20:58.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 -- start start 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 Processor -- start 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24dd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdc24db330 con 0x55cdc24d2820 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cdc23f3300 con 0x55cdc24d3210 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cdc24db4b0 con 0x55cdc24d8380 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce28bd5740 10 client.0 _ll_register_callbacks cb 0x55cdc242ee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56762/0 (socket says 172.21.15.28:56762) 2024-03-22T02:20:58.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2427165378 learned_addr learned my addr 172.21.15.28:0/2427165378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.266+0000 7fce1effd640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24dd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2427165378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 msgr2=0x55cdc24e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 msgr2=0x55cdc24dd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24dd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2427165378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdc22157d0 con 0x55cdc24d2820 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fce100bd280 tx=0x7fce1002bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684c60673b6a4600 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1004b040 con 0x55cdc24d2820 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fce100d65a0 con 0x55cdc24d2820 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce180183f0 con 0x55cdc24d2820 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce100bdc80 con 0x55cdc24d2820 2024-03-22T02:20:58.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 10 client.17625 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cdc24e7530 con 0x55cdc24d2820 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce180183f0 con 0x55cdc24d2820 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fce100dd050 con 0x55cdc24d2820 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 10 client.17625 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 10 client.17625 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 20 client.17625 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.267+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc24e8cc0 0x55cdc24eb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.268+0000 7fce28bd5740 4 client.17625 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.268+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdc2444e10 con 0x55cdc24e8cc0 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.268+0000 7fce1effd640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc24e8cc0 0x55cdc24eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.268+0000 7fcdfeffd640 10 client.17625 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.268+0000 7fce1effd640 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc24e8cc0 0x55cdc24eb180 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=312fb93fa397c09b server_cookie=2fb7b9a44f1b8b4c in_seq=0 out_seq=0 2024-03-22T02:20:58.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2427165378 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cdc2444e10 con 0x55cdc24e8cc0 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fcdfeffd640 10 client.17625 handle_command_reply: tid=0 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 client.17625 shutdown 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 msgr2=0x55cdc24dccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fce100bd280 tx=0x7fce1002bc40 comp rx=0 tx=0).stop 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 shutdown_connections 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc24e8cc0 0x55cdc24eb180 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24dd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2427165378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24dccd0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fcdfeffd640 0 client.17625 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 >> 172.21.15.28:0/2427165378 conn(0x55cdc24cdd90 msgr2=0x55cdc24ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 shutdown_connections 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 1 -- 172.21.15.28:0/2427165378 wait complete. 2024-03-22T02:20:58.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.269+0000 7fce28bd5740 20 client.17625 trim_cache size 0 max 16384 2024-03-22T02:20:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.282+0000 7fce28bd5740 1 Processor -- start 2024-03-22T02:20:58.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.282+0000 7fce28bd5740 1 -- start start 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24e41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c012640 con 0x55cdc24d8380 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c00ff10 con 0x55cdc24d2820 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce28bd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c00dd60 con 0x55cdc24d3210 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56772/0 (socket says 172.21.15.28:56772) 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44358/0 (socket says 172.21.15.28:44358) 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 -- 172.21.15.28:0/279687802 learned_addr learned my addr 172.21.15.28:0/279687802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24e41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 -- 172.21.15.28:0/279687802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 msgr2=0x55cdc24e41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24e41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 -- 172.21.15.28:0/279687802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 msgr2=0x55cdc24dd170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 -- 172.21.15.28:0/279687802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdc24eb800 con 0x55cdc24d8380 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.283+0000 7fce1ffff640 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fce18007f40 tx=0x7fce18007350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55fc108ad78d2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.284+0000 7fce057fa640 1 -- 172.21.15.28:0/279687802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18007ac0 con 0x55cdc24d8380 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.284+0000 7fce057fa640 1 -- 172.21.15.28:0/279687802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce18003e50 con 0x55cdc24d8380 2024-03-22T02:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.284+0000 7fce057fa640 1 -- 172.21.15.28:0/279687802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18005320 con 0x55cdc24d8380 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- 172.21.15.28:0/279687802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 msgr2=0x55cdc24cc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fce18007f40 tx=0x7fce18007350 comp rx=0 tx=0).stop 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- 172.21.15.28:0/279687802 shutdown_connections 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d8380 0x55cdc24cc740 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d3210 0x55cdc24e41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- 172.21.15.28:0/279687802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d2820 0x55cdc24dd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- 172.21.15.28:0/279687802 >> 172.21.15.28:0/279687802 conn(0x55cdc24cdd90 msgr2=0x55cdc24ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- 172.21.15.28:0/279687802 shutdown_connections 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- 172.21.15.28:0/279687802 wait complete. 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- start start 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 Processor -- start 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24e4230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c00e3d0 con 0x55cdc24d2820 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c00f840 con 0x55cdc24d3210 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fce0c00eaa0 con 0x55cdc24d8380 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce28bd5740 10 client.0 _ll_register_callbacks cb 0x7fce10009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:58.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.285+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56778/0 (socket says 172.21.15.28:56778) 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2074907399 learned_addr learned my addr 172.21.15.28:0/2074907399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1ffff640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2074907399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 msgr2=0x55cdc24e2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 msgr2=0x55cdc24e4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24e4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 -- 172.21.15.28:0/2074907399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdc24cd930 con 0x55cdc24d2820 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1effd640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24e4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce1f7fe640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fce100c5dd0 tx=0x7fce1002cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48abb1f2dddfbfd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fcdfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:58.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1004b070 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce18003e50 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fce100bcc80 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce100bd350 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce28bd5740 10 client.17631 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.286+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55cdc254fab0 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce18003e50 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7fce100dd040 con 0x55cdc24d2820 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 10 client.17631 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 10 client.17631 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 20 client.17631 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc25501f0 0x55cdc2552610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 4 client.17631 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:20:58.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cdc243f160 con 0x55cdc25501f0 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce1effd640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc25501f0 0x55cdc2552610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fcdfeffd640 10 client.17631 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce1effd640 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc25501f0 0x55cdc2552610 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a13c3e08ef2afa88 server_cookie=8cd4539c5c2b9cbd in_seq=0 out_seq=0 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fcdfeffd640 1 -- 172.21.15.28:0/2074907399 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cdc243f160 con 0x55cdc25501f0 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fcdfeffd640 10 client.17631 handle_command_reply: tid=0 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.287+0000 7fce28bd5740 1 client.17631 shutdown 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 msgr2=0x55cdc24e3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fce100c5dd0 tx=0x7fce1002cf90 comp rx=0 tx=0).stop 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 shutdown_connections 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cdc25501f0 0x55cdc2552610 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cdc24d8380 0x55cdc24e2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cdc24d3210 0x55cdc24e4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fcdfeffd640 0 client.17631 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 --2- 172.21.15.28:0/2074907399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdc24d2820 0x55cdc24e3cf0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 >> 172.21.15.28:0/2074907399 conn(0x55cdc24cdd90 msgr2=0x55cdc24d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 shutdown_connections 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 1 -- 172.21.15.28:0/2074907399 wait complete. 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce28bd5740 20 client.17631 trim_cache size 0 max 16384 2024-03-22T02:20:58.297 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 msgr2=0x7fce080a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 0x7fce080a54b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fce14005ef0 tx=0x7fce14005e80 comp rx=0 tx=0).stop 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 msgr2=0x7fce20198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fce10036040 tx=0x7fce1002ce50 comp rx=0 tx=0).stop 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 shutdown_connections 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fce080a2ff0 0x7fce080a54b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fce2019d5b0 0x7fce2019f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fce20103750 0x7fce20198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 --2- 172.21.15.28:0/2911163160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce20102d90 0x7fce20198010 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.288+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 >> 172.21.15.28:0/2911163160 conn(0x7fce200fd0f0 msgr2=0x7fce201060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.289+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 shutdown_connections 2024-03-22T02:20:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:58.289+0000 7fce25fe9640 1 -- 172.21.15.28:0/2911163160 wait complete. 2024-03-22T02:20:58.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 678476 11.90 MB/sec execute 220 sec latency 30.884 ms 2024-03-22T02:20:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:58 smithi120 ceph-mon[36803]: pgmap v580: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-03-22T02:20:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:20:58 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4255736872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:58 smithi083 ceph-mon[36340]: pgmap v580: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-03-22T02:20:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:20:58 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4255736872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:58 smithi028 ceph-mon[31916]: pgmap v580: 129 pgs: 129 active+clean; 53 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-03-22T02:20:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:20:58 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4255736872' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:20:59.301 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:20:59.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:20:59.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:20:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 -- 172.21.15.28:0/4014315104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 msgr2=0x7f08ec101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 --2- 172.21.15.28:0/4014315104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec101c00 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009ca0 tx=0x7f08dc02c180 comp rx=0 tx=0).stop 2024-03-22T02:20:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 -- 172.21.15.28:0/4014315104 shutdown_connections 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 --2- 172.21.15.28:0/4014315104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec101c00 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 --2- 172.21.15.28:0/4014315104 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 0x7f08ec101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 --2- 172.21.15.28:0/4014315104 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 -- 172.21.15.28:0/4014315104 >> 172.21.15.28:0/4014315104 conn(0x7f08ec0fc020 msgr2=0x7f08ec0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 -- 172.21.15.28:0/4014315104 shutdown_connections 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.412+0000 7f08f09c9640 1 -- 172.21.15.28:0/4014315104 wait complete. 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 Processor -- start 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 -- start start 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 0x7f08ec19a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec1a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.413+0000 7f08f09c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10fa10 con 0x7f08ec101780 2024-03-22T02:20:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08f09c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10f890 con 0x7f08ec100480 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08f09c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec10fb90 con 0x7f08ec100de0 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44378/0 (socket says 172.21.15.28:44378) 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 -- 172.21.15.28:0/1085479412 learned_addr learned my addr 172.21.15.28:0/1085479412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ead9d640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec1a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 0x7f08ec19a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 -- 172.21.15.28:0/1085479412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 msgr2=0x7f08ec19a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 0x7f08ec19a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 -- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 msgr2=0x7f08ec1a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec1a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 -- 172.21.15.28:0/1085479412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ec1a1c80 con 0x7f08ec100480 2024-03-22T02:20:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ead9d640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08ea59c640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f08e400e9f0 tx=0x7f08e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08db7fe640 1 -- 172.21.15.28:0/1085479412 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e400cd80 con 0x7f08ec100480 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08db7fe640 1 -- 172.21.15.28:0/1085479412 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e4010910 con 0x7f08ec100480 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ec1a1f70 con 0x7f08ec100480 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08db7fe640 1 -- 172.21.15.28:0/1085479412 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4018990 con 0x7f08ec100480 2024-03-22T02:20:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.414+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ec1a23d0 con 0x7f08ec100480 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.417+0000 7f08db7fe640 1 -- 172.21.15.28:0/1085479412 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f08e4010ac0 con 0x7f08ec100480 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.417+0000 7f08f35a8740 1 Processor -- start 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.417+0000 7f08f35a8740 1 -- start start 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56507d228570 con 0x56507d2c3bc0 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56507d1d0e10 con 0x56507d2c9720 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08f35a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56507d224520 con 0x56507d2c45b0 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ead9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56794/0 (socket says 172.21.15.28:56794) 2024-03-22T02:20:59.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ead9d640 1 -- 172.21.15.28:0/1422639681 learned_addr learned my addr 172.21.15.28:0/1422639681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ea59c640 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08db7fe640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 0x7f08d40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 -- 172.21.15.28:0/1422639681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 msgr2=0x56507d2c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 -- 172.21.15.28:0/1422639681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 msgr2=0x56507d2c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 -- 172.21.15.28:0/1422639681 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507d0047e0 con 0x56507d2c9720 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ead9d640 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08ead9d640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 0x7f08d40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08db7fe640 1 -- 172.21.15.28:0/1085479412 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f08e40cfae0 con 0x7f08ec100480 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.418+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2cbbb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009bd0 tx=0x7f08e0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ded9c0b3d895e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08d8ff9640 1 -- 172.21.15.28:0/1422639681 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0004230 con 0x56507d2c9720 2024-03-22T02:20:59.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08d8ff9640 1 -- 172.21.15.28:0/1422639681 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e0009e40 con 0x56507d2c9720 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08d8ff9640 1 -- 172.21.15.28:0/1422639681 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0004d90 con 0x56507d2c9720 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 -- 172.21.15.28:0/1422639681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 msgr2=0x56507d2cbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08ead9d640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 0x7f08d40a53e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f08dc032500 tx=0x7f08dc02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2cbbb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f08e0009bd0 tx=0x7f08e0011810 comp rx=0 tx=0).stop 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 -- 172.21.15.28:0/1422639681 shutdown_connections 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2cbbb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 --2- 172.21.15.28:0/1422639681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 -- 172.21.15.28:0/1422639681 >> 172.21.15.28:0/1422639681 conn(0x56507d2bf130 msgr2=0x56507d2bf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 -- 172.21.15.28:0/1422639681 shutdown_connections 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.419+0000 7f08f35a8740 1 -- 172.21.15.28:0/1422639681 wait complete. 2024-03-22T02:20:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 -- start start 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 Processor -- start 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56507d2cc6d0 con 0x56507d2c45b0 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56507d1e4110 con 0x56507d2c9720 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56507d2cc850 con 0x56507d2c3bc0 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08f35a8740 10 client.0 _ll_register_callbacks cb 0x56507d220be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56798/0 (socket says 172.21.15.28:56798) 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 -- 172.21.15.28:0/4189675705 learned_addr learned my addr 172.21.15.28:0/4189675705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08ead9d640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 -- 172.21.15.28:0/4189675705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 msgr2=0x56507d2cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2cdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 -- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 msgr2=0x56507d2d52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08e9d9b640 1 -- 172.21.15.28:0/4189675705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507d064f70 con 0x56507d2c45b0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.420+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f08e0005e00 tx=0x7f08e00038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4a7229e3f34f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0027020 con 0x56507d2c45b0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f08e0003c90 con 0x56507d2c45b0 2024-03-22T02:20:59.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56507d235ab0 con 0x56507d2c45b0 2024-03-22T02:20:59.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 680451 11.88 MB/sec execute 221 sec latency 28.923 ms 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08e0032050 con 0x56507d2c45b0 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 10 client.17643 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56507cefc170 con 0x56507d2c45b0 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56507d235ab0 con 0x56507d2c45b0 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f08e0018070 con 0x56507d2c45b0 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 10 client.17643 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 10 client.17643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:59.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 20 client.17643 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d2d9970 0x56507d2dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 4 client.17643 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.421+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56507d22fe00 con 0x56507d2d9970 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.422+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d2d9970 0x56507d2dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.422+0000 7f08d1ffb640 10 client.17643 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.422+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d2d9970 0x56507d2dbde0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e8d62450e2a14b6 server_cookie=f9f3b70cf7226348 in_seq=0 out_seq=0 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.423+0000 7f08d1ffb640 1 -- 172.21.15.28:0/4189675705 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56507d22fe00 con 0x56507d2d9970 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.423+0000 7f08d1ffb640 10 client.17643 handle_command_reply: tid=0 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 client.17643 shutdown 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 msgr2=0x56507d2ce280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f08e0005e00 tx=0x7f08e00038a0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 shutdown_connections 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d2d9970 0x56507d2dbde0 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2ce280 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08d1ffb640 0 client.17643 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4189675705 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2cdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 >> 172.21.15.28:0/4189675705 conn(0x56507d2bf130 msgr2=0x56507d2be3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 shutdown_connections 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 1 -- 172.21.15.28:0/4189675705 wait complete. 2024-03-22T02:20:59.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.424+0000 7f08f35a8740 20 client.17643 trim_cache size 0 max 16384 2024-03-22T02:20:59.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 Processor -- start 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 -- start start 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2d5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2c5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1af420 con 0x56507d2c3bc0 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1accf0 con 0x56507d2c9720 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08f35a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1aab40 con 0x56507d2c45b0 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2d5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56814/0 (socket says 172.21.15.28:56814) 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 -- 172.21.15.28:0/4132586206 learned_addr learned my addr 172.21.15.28:0/4132586206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ead9d640 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 -- 172.21.15.28:0/4132586206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 msgr2=0x56507d2d5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2d5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 -- 172.21.15.28:0/4132586206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 msgr2=0x56507d2c5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2c5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08ea59c640 1 -- 172.21.15.28:0/4132586206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56507d2cc5b0 con 0x56507d2c3bc0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.442+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2d5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08ea59c640 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f08e40cd1d0 tx=0x7f08e40bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5ca28857f42cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08d3fff640 1 -- 172.21.15.28:0/4132586206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e409bc80 con 0x56507d2c3bc0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08d3fff640 1 -- 172.21.15.28:0/4132586206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e409beb0 con 0x56507d2c3bc0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08d3fff640 1 -- 172.21.15.28:0/4132586206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e409b810 con 0x56507d2c3bc0 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 -- 172.21.15.28:0/4132586206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 msgr2=0x56507d2c2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f08e40cd1d0 tx=0x7f08e40bc040 comp rx=0 tx=0).stop 2024-03-22T02:20:59.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 -- 172.21.15.28:0/4132586206 shutdown_connections 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2c5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c45b0 0x56507d2d5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 --2- 172.21.15.28:0/4132586206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c3bc0 0x56507d2c2aa0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 -- 172.21.15.28:0/4132586206 >> 172.21.15.28:0/4132586206 conn(0x56507d2bf130 msgr2=0x56507d2c2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 -- 172.21.15.28:0/4132586206 shutdown_connections 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.443+0000 7f08f35a8740 1 -- 172.21.15.28:0/4132586206 wait complete. 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 -- start start 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 Processor -- start 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2d2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1ab1b0 con 0x56507d2c45b0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1ac620 con 0x56507d2c9720 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f08ec1ab880 con 0x56507d2c3bc0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08f35a8740 10 client.0 _ll_register_callbacks cb 0x7f08e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56830/0 (socket says 172.21.15.28:56830) 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08ea59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2d2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 -- 172.21.15.28:0/3422433531 learned_addr learned my addr 172.21.15.28:0/3422433531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08ead9d640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 -- 172.21.15.28:0/3422433531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 msgr2=0x56507d2d2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2d2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 -- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 msgr2=0x56507d2d3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 -- 172.21.15.28:0/3422433531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56507d2d4310 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08ea59c640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08e9d9b640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f08e0006010 tx=0x7f08e0019890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130dc6b78eabacbd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.444+0000 7f08d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e0015020 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f08e0019dc0 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08e409beb0 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f08e0027040 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 10 client.17649 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56507d2d5650 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e409beb0 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f08e0018050 con 0x56507d2c45b0 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 10 client.17649 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 10 client.17649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 20 client.17649 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d340400 0x56507d342820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 4 client.17649 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56507d1adbe0 con 0x56507d340400 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.445+0000 7f08ea59c640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d340400 0x56507d342820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.446+0000 7f08d1ffb640 10 client.17649 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.446+0000 7f08ea59c640 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d340400 0x56507d342820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=696870bc69e2c05b server_cookie=69ad91f15ddc78f4 in_seq=0 out_seq=0 2024-03-22T02:20:59.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08d1ffb640 1 -- 172.21.15.28:0/3422433531 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56507d1adbe0 con 0x56507d340400 2024-03-22T02:20:59.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08d1ffb640 10 client.17649 handle_command_reply: tid=0 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 client.17649 shutdown 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 msgr2=0x56507d2d2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f08e0006010 tx=0x7f08e0019890 comp rx=0 tx=0).stop 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 shutdown_connections 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56507d340400 0x56507d342820 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56507d2c9720 0x56507d2d3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56507d2c45b0 0x56507d2d2d60 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 --2- 172.21.15.28:0/3422433531 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56507d2c3bc0 0x56507d2d2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08d1ffb640 0 client.17649 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 >> 172.21.15.28:0/3422433531 conn(0x56507d2bf130 msgr2=0x56507d2c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 shutdown_connections 2024-03-22T02:20:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 1 -- 172.21.15.28:0/3422433531 wait complete. 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.447+0000 7f08f35a8740 20 client.17649 trim_cache size 0 max 16384 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "f80e2abf-4188-492b-b577-533ffb842781", 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:20:59.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 msgr2=0x7f08d40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 0x7f08d40a53e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f08dc032500 tx=0x7f08dc02e040 comp rx=0 tx=0).stop 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 msgr2=0x7f08ec19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f08e400e9f0 tx=0x7f08e400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 shutdown_connections 2024-03-22T02:20:59.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08d40a2f20 0x7f08d40a53e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec101780 0x7f08ec1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f08ec100de0 0x7f08ec19a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 --2- 172.21.15.28:0/1085479412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f08ec100480 0x7f08ec19a060 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 >> 172.21.15.28:0/1085479412 conn(0x7f08ec0fc020 msgr2=0x7f08ec0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 shutdown_connections 2024-03-22T02:20:59.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.448+0000 7f08f09c9640 1 -- 172.21.15.28:0/1085479412 wait complete. 2024-03-22T02:20:59.461 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:20:59.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:20:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.566+0000 7f9e2135e640 1 -- 172.21.15.28:0/4029239955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c100a90 msgr2=0x7f9e1c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.566+0000 7f9e2135e640 1 --2- 172.21.15.28:0/4029239955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c100a90 0x7f9e1c100eb0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009900 tx=0x7f9e0c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 -- 172.21.15.28:0/4029239955 shutdown_connections 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 --2- 172.21.15.28:0/4029239955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c102640 0x7f9e1c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 --2- 172.21.15.28:0/4029239955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c101c80 0x7f9e1c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 --2- 172.21.15.28:0/4029239955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c100a90 0x7f9e1c100eb0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 -- 172.21.15.28:0/4029239955 >> 172.21.15.28:0/4029239955 conn(0x7f9e1c0fc020 msgr2=0x7f9e1c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 -- 172.21.15.28:0/4029239955 shutdown_connections 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 -- 172.21.15.28:0/4029239955 wait complete. 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 Processor -- start 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.567+0000 7f9e2135e640 1 -- start start 2024-03-22T02:20:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c100a90 0x7f9e1c197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c101c80 0x7f9e1c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10d830 con 0x7f9e1c102640 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10d6b0 con 0x7f9e1c100a90 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c10d9b0 con 0x7f9e1c101c80 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56858/0 (socket says 172.21.15.28:56858) 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c100a90 0x7f9e1c197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/235447816 learned_addr learned my addr 172.21.15.28:0/235447816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c101c80 0x7f9e1c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/235447816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c101c80 msgr2=0x7f9e1c198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c101c80 0x7f9e1c198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/235447816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c100a90 msgr2=0x7f9e1c197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c100a90 0x7f9e1c197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/235447816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c19fb30 con 0x7f9e1c102640 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400eb10 tx=0x7f9e1400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e03fff640 1 -- 172.21.15.28:0/235447816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14004b20 con 0x7f9e1c102640 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e03fff640 1 -- 172.21.15.28:0/235447816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e14004cc0 con 0x7f9e1c102640 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c19fe20 con 0x7f9e1c102640 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e03fff640 1 -- 172.21.15.28:0/235447816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14010760 con 0x7f9e1c102640 2024-03-22T02:20:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.568+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c1a02e0 con 0x7f9e1c102640 2024-03-22T02:20:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.570+0000 7f9e03fff640 1 -- 172.21.15.28:0/235447816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9e14005020 con 0x7f9e1c102640 2024-03-22T02:20:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e03fff640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 0x7f9e040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 Processor -- start 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e1affd640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 0x7f9e040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e03fff640 1 -- 172.21.15.28:0/235447816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e140cff00 con 0x7f9e1c102640 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 -- start start 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39430 0x562bccf39850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 0x562bccf41420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bcce9e8c0 con 0x562bccf39430 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562bcce46c20 con 0x562bccf39e20 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.571+0000 7f9e24029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562bcce9a870 con 0x562bccf3ef90 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 0x562bccf41420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44448/0 (socket says 172.21.15.28:44448) 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/2729602255 learned_addr learned my addr 172.21.15.28:0/2729602255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39430 0x562bccf39850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1affd640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 0x7f9e040a53e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c000c00 tx=0x7f9e0c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/2729602255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 msgr2=0x562bccf3e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/2729602255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39430 msgr2=0x562bccf39850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39430 0x562bccf39850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/2729602255 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bcce6b150 con 0x562bccf3ef90 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 0x562bccf41420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e140049d0 tx=0x7f9e1409bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7391a6b9706cdce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e017fa640 1 -- 172.21.15.28:0/2729602255 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1409c1c0 con 0x562bccf3ef90 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e017fa640 1 -- 172.21.15.28:0/2729602255 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e140a0310 con 0x562bccf3ef90 2024-03-22T02:20:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.572+0000 7f9e017fa640 1 -- 172.21.15.28:0/2729602255 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e14005570 con 0x562bccf3ef90 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- 172.21.15.28:0/2729602255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 msgr2=0x562bccf41420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 0x562bccf41420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9e140049d0 tx=0x7f9e1409bd00 comp rx=0 tx=0).stop 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- 172.21.15.28:0/2729602255 shutdown_connections 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf3ef90 0x562bccf41420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39e20 0x562bccf3e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 --2- 172.21.15.28:0/2729602255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39430 0x562bccf39850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- 172.21.15.28:0/2729602255 >> 172.21.15.28:0/2729602255 conn(0x562bccf349a0 msgr2=0x562bccf34da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- 172.21.15.28:0/2729602255 shutdown_connections 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- 172.21.15.28:0/2729602255 wait complete. 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.573+0000 7f9e24029740 1 -- start start 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 Processor -- start 2024-03-22T02:20:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccf438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccf4ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bccf41ee0 con 0x562bccf39e20 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562bcce5a130 con 0x562bccf3ef90 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562bccf42060 con 0x562bccf39430 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e24029740 10 client.0 _ll_register_callbacks cb 0x562bcce59b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccf438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56880/0 (socket says 172.21.15.28:56880) 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3811976191 learned_addr learned my addr 172.21.15.28:0/3811976191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccf4ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3811976191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 msgr2=0x562bccf438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccf438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 msgr2=0x562bccf4ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccf4ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3811976191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bccc7c7d0 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccf4ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f9e100126e0 tx=0x7f9e10012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dd7951f11c6dff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9dfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.574+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e100121d0 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e10004050 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e140a0310 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e10025050 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 10 client.17661 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bccbb23f0 con 0x562bccf39e20 2024-03-22T02:20:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e140a0310 con 0x562bccf39e20 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f9e100038f0 con 0x562bccf39e20 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 10 client.17661 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 10 client.17661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 20 client.17661 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccf4f790 0x562bccf51bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 4 client.17661 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bcceabe00 con 0x562bccf4f790 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.575+0000 7f9e1affd640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccf4f790 0x562bccf51bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e1affd640 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccf4f790 0x562bccf51bc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b41cfb6de50cbdbd server_cookie=97767a21e8c11abc in_seq=0 out_seq=0 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9dfeff5640 10 client.17661 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3811976191 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562bcceabe00 con 0x562bccf4f790 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9dfeff5640 10 client.17661 handle_command_reply: tid=0 2024-03-22T02:20:59.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 client.17661 shutdown 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 msgr2=0x562bccf43e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f9e100126e0 tx=0x7f9e10012ba0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 shutdown_connections 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccf4f790 0x562bccf51bc0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccf4ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccf43e20 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9dfeff5640 0 client.17661 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 --2- 172.21.15.28:0/3811976191 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccf438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.576+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 >> 172.21.15.28:0/3811976191 conn(0x562bccf349a0 msgr2=0x562bccf33c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.577+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 shutdown_connections 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.577+0000 7f9e24029740 1 -- 172.21.15.28:0/3811976191 wait complete. 2024-03-22T02:20:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.577+0000 7f9e24029740 20 client.17661 trim_cache size 0 max 16384 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 Processor -- start 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 -- start start 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39430 0x562bccf42630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39e20 0x562bccf3b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1ad380 con 0x562bccf3ef90 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1aac50 con 0x562bccf39430 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e24029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1a8aa0 con 0x562bccf39e20 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e1b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56888/0 (socket says 172.21.15.28:56888) 2024-03-22T02:20:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/1478954771 learned_addr learned my addr 172.21.15.28:0/1478954771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e1affd640 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39430 0x562bccf42630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.594+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39e20 0x562bccf3b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/1478954771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39e20 msgr2=0x562bccf3b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39e20 0x562bccf3b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/1478954771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39430 msgr2=0x562bccf42630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39430 0x562bccf42630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 -- 172.21.15.28:0/1478954771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bccb720f0 con 0x562bccf3ef90 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009ae0 tx=0x7f9e14098830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fcf7ee90027169 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e00ff9640 1 -- 172.21.15.28:0/1478954771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e140ba070 con 0x562bccf3ef90 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e00ff9640 1 -- 172.21.15.28:0/1478954771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e1409c1c0 con 0x562bccf3ef90 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e00ff9640 1 -- 172.21.15.28:0/1478954771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1409a550 con 0x562bccf3ef90 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 -- 172.21.15.28:0/1478954771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 msgr2=0x562bccf3b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f9e14009ae0 tx=0x7f9e14098830 comp rx=0 tx=0).stop 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 -- 172.21.15.28:0/1478954771 shutdown_connections 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf3ef90 0x562bccf3b860 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39e20 0x562bccf3b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 --2- 172.21.15.28:0/1478954771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf39430 0x562bccf42630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.595+0000 7f9e24029740 1 -- 172.21.15.28:0/1478954771 >> 172.21.15.28:0/1478954771 conn(0x562bccf349a0 msgr2=0x562bccf438e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- 172.21.15.28:0/1478954771 shutdown_connections 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- 172.21.15.28:0/1478954771 wait complete. 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- start start 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 Processor -- start 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccfb0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccfb82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1a9110 con 0x562bccf39e20 2024-03-22T02:20:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1aa580 con 0x562bccf3ef90 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1a97e0 con 0x562bccf39430 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e24029740 10 client.0 _ll_register_callbacks cb 0x7f9e10009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccfb0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56890/0 (socket says 172.21.15.28:56890) 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3596738924 learned_addr learned my addr 172.21.15.28:0/3596738924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccfb82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3596738924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 msgr2=0x562bccfb0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccfb0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 msgr2=0x562bccfb82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccfb82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1a7fc640 1 -- 172.21.15.28:0/3596738924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bccfb88a0 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.596+0000 7f9e1b7fe640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccfb82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e1a7fc640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9e100126b0 tx=0x7f9e10009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e16189f6129284 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e10003800 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e10018040 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e1409c1c0 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f9e10038050 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 10 client.17667 fetch_fsmap learned FSMap version 32 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562bccfb9c20 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e1409c1c0 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f9e1000fa90 con 0x562bccf39e20 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 10 client.17667 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:20:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 10 client.17667 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 20 client.17667 populate_metadata read hostname 'smithi028' 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccfba5c0 0x562bccfbc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 4 client.17667 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562bccea6150 con 0x562bccfba5c0 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.597+0000 7f9e1affd640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccfba5c0 0x562bccfbc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.598+0000 7f9dfeff5640 10 client.17667 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.598+0000 7f9e1affd640 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccfba5c0 0x562bccfbc9e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=995fdfa0e5287068 server_cookie=35a1aef13d45b2d in_seq=0 out_seq=0 2024-03-22T02:20:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9dfeff5640 1 -- 172.21.15.28:0/3596738924 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562bccea6150 con 0x562bccfba5c0 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9dfeff5640 10 client.17667 handle_command_reply: tid=0 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 client.17667 shutdown 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 msgr2=0x562bccfb1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f9e100126b0 tx=0x7f9e10009fc0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 shutdown_connections 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562bccfba5c0 0x562bccfbc9e0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562bccf3ef90 0x562bccfb82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bccf39e20 0x562bccfb1270 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9dfeff5640 0 client.17667 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 --2- 172.21.15.28:0/3596738924 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562bccf39430 0x562bccfb0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 >> 172.21.15.28:0/3596738924 conn(0x562bccf349a0 msgr2=0x562bccf43810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 shutdown_connections 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 1 -- 172.21.15.28:0/3596738924 wait complete. 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.599+0000 7f9e24029740 20 client.17667 trim_cache size 0 max 16384 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "f80e2abf-4188-492b-b577-533ffb842781": { 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "f80e2abf-4188-492b-b577-533ffb842781", 2024-03-22T02:20:59.608 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:20:59.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:20:59.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:20:59.609 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 msgr2=0x7f9e040a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 0x7f9e040a53e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c000c00 tx=0x7f9e0c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 msgr2=0x7f9e1c19f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f9e1400eb10 tx=0x7f9e1400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 shutdown_connections 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e040a2f20 0x7f9e040a53e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e1c102640 0x7f9e1c19f4c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e1c101c80 0x7f9e1c198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 --2- 172.21.15.28:0/235447816 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e1c100a90 0x7f9e1c197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 >> 172.21.15.28:0/235447816 conn(0x7f9e1c0fc020 msgr2=0x7f9e1c105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 shutdown_connections 2024-03-22T02:20:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:20:59.600+0000 7f9e2135e640 1 -- 172.21.15.28:0/235447816 wait complete. 2024-03-22T02:20:59.612 INFO:tasks.cephfs.filesystem:scrub status for tag:f80e2abf-4188-492b-b577-533ffb842781 - {'path': '/', 'tag': 'f80e2abf-4188-492b-b577-533ffb842781', 'options': 'recursive,force'} 2024-03-22T02:21:00.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 682422 11.89 MB/sec execute 222 sec latency 23.867 ms 2024-03-22T02:21:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:00 smithi120 ceph-mon[36803]: pgmap v581: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:21:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:00 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:21:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:00 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:00.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:00 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:21:00.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:00 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:00 smithi083 ceph-mon[36340]: pgmap v581: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:21:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:00 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:21:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:00 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:00 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:21:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:00 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:00 smithi028 ceph-mon[31916]: pgmap v581: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 427 op/s 2024-03-22T02:21:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:00 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:21:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:00 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:00 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:21:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:00 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:01.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 685045 11.89 MB/sec execute 223 sec latency 26.624 ms 2024-03-22T02:21:01.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:01 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:21:01.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:01 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:01 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:21:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:01 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:01.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:01 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:21:01.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:01 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:02.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 686323 11.87 MB/sec execute 224 sec latency 71.915 ms 2024-03-22T02:21:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:02 smithi120 ceph-mon[36803]: pgmap v582: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 414 op/s 2024-03-22T02:21:02.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:02 smithi083 ceph-mon[36340]: pgmap v582: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 414 op/s 2024-03-22T02:21:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:02 smithi028 ceph-mon[31916]: pgmap v582: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 414 op/s 2024-03-22T02:21:03.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 688909 11.89 MB/sec execute 225 sec latency 114.150 ms 2024-03-22T02:21:04.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 -- 172.21.15.28:0/662112236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074100ac0 msgr2=0x7f0074100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 --2- 172.21.15.28:0/662112236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074100ac0 0x7f0074100ee0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0068009900 tx=0x7f006802bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 -- 172.21.15.28:0/662112236 shutdown_connections 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 --2- 172.21.15.28:0/662112236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074102670 0x7f0074108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 --2- 172.21.15.28:0/662112236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074101cb0 0x7f0074102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 --2- 172.21.15.28:0/662112236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074100ac0 0x7f0074100ee0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.262+0000 7f007ae2a640 1 -- 172.21.15.28:0/662112236 >> 172.21.15.28:0/662112236 conn(0x7f00740fc050 msgr2=0x7f00740fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.263+0000 7f007ae2a640 1 -- 172.21.15.28:0/662112236 shutdown_connections 2024-03-22T02:21:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.263+0000 7f007ae2a640 1 -- 172.21.15.28:0/662112236 wait complete. 2024-03-22T02:21:04.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.264+0000 7f007ae2a640 1 Processor -- start 2024-03-22T02:21:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.265+0000 7f007ae2a640 1 -- start start 2024-03-22T02:21:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.265+0000 7f007ae2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 0x7f0074195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f007ae2a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074101cb0 0x7f007419aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f007ae2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 0x7f0074195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f0078bc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 0x7f0074195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f0078bc6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 0x7f0074195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34048/0 (socket says 172.21.15.28:34048) 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f0078bc6640 1 -- 172.21.15.28:0/2459379690 learned_addr learned my addr 172.21.15.28:0/2459379690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f007410d880 con 0x7f0074100ac0 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f007410d700 con 0x7f0074102670 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f007410da00 con 0x7f0074101cb0 2024-03-22T02:21:04.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f0073fff640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074101cb0 0x7f007419aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.266+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 0x7f0074195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 -- 172.21.15.28:0/2459379690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074101cb0 msgr2=0x7f007419aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074101cb0 0x7f007419aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 -- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 msgr2=0x7f0074195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 0x7f0074195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00741ae770 con 0x7f0074102670 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 0x7f0074195f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f006c009fd0 tx=0x7f006c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.267+0000 7f0071ffb640 1 -- 172.21.15.28:0/2459379690 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c00ce20 con 0x7f0074102670 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.268+0000 7f0071ffb640 1 -- 172.21.15.28:0/2459379690 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f006c005020 con 0x7f0074102670 2024-03-22T02:21:04.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.268+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00741ae9a0 con 0x7f0074102670 2024-03-22T02:21:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.268+0000 7f0071ffb640 1 -- 172.21.15.28:0/2459379690 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006c010720 con 0x7f0074102670 2024-03-22T02:21:04.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.268+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00741aeea0 con 0x7f0074102670 2024-03-22T02:21:04.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f0071ffb640 1 -- 172.21.15.28:0/2459379690 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f006c004b20 con 0x7f0074102670 2024-03-22T02:21:04.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 Processor -- start 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 -- start start 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d04e0 0x564e9a9d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9d59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 0x564e9a9d84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e9a935040 con 0x564e9a9d6040 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e9a8dd9e0 con 0x564e9a9d04e0 2024-03-22T02:21:04.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.271+0000 7f007da10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e9a930ff0 con 0x564e9a9d0ed0 2024-03-22T02:21:04.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0078bc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9d59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0078bc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9d59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58900/0 (socket says 172.21.15.28:58900) 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0078bc6640 1 -- 172.21.15.28:0/2636971255 learned_addr learned my addr 172.21.15.28:0/2636971255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0071ffb640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 0x7f00640a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0073fff640 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 0x564e9a9d84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0073fff640 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d04e0 0x564e9a9d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.272+0000 7f0071ffb640 1 -- 172.21.15.28:0/2459379690 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f006c0cff30 con 0x7f0074102670 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 -- 172.21.15.28:0/2636971255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 msgr2=0x564e9a9d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9d59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 -- 172.21.15.28:0/2636971255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d04e0 msgr2=0x564e9a9d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d04e0 0x564e9a9d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 -- 172.21.15.28:0/2636971255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9a649210 con 0x564e9a9d6040 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 0x7f00640a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f0073fff640 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 0x564e9a9d84d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x564e9a9d1730 tx=0x7f0060005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7f7ba637231550 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f005effd640 1 -- 172.21.15.28:0/2636971255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060012d20 con 0x564e9a9d6040 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f00793c7640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 0x7f00640a5550 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f006c01a040 tx=0x7f006c005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f005effd640 1 -- 172.21.15.28:0/2636971255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0060005020 con 0x564e9a9d6040 2024-03-22T02:21:04.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.273+0000 7f005effd640 1 -- 172.21.15.28:0/2636971255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060004a00 con 0x564e9a9d6040 2024-03-22T02:21:04.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.274+0000 7f007da10740 1 -- 172.21.15.28:0/2636971255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 msgr2=0x564e9a9d84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.274+0000 7f007da10740 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 0x564e9a9d84d0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x564e9a9d1730 tx=0x7f0060005c90 comp rx=0 tx=0).stop 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 -- 172.21.15.28:0/2636971255 shutdown_connections 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d6040 0x564e9a9d84d0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9d59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 --2- 172.21.15.28:0/2636971255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d04e0 0x564e9a9d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 -- 172.21.15.28:0/2636971255 >> 172.21.15.28:0/2636971255 conn(0x564e9a9cba50 msgr2=0x564e9a9cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 -- 172.21.15.28:0/2636971255 shutdown_connections 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.275+0000 7f007da10740 1 -- 172.21.15.28:0/2636971255 wait complete. 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.276+0000 7f007da10740 1 -- start start 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.276+0000 7f007da10740 1 Processor -- start 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9daac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e9a9d8ff0 con 0x564e9a9d04e0 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564e9a8f0db0 con 0x564e9a9d0ed0 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564e9a9d9170 con 0x564e9a9d6040 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f007da10740 10 client.0 _ll_register_callbacks cb 0x564e9a648a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34066/0 (socket says 172.21.15.28:34066) 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f0073fff640 1 -- 172.21.15.28:0/2571102518 learned_addr learned my addr 172.21.15.28:0/2571102518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:04.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.277+0000 7f0073fff640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 -- 172.21.15.28:0/2571102518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 msgr2=0x564e9a9e1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9e1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 -- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 msgr2=0x564e9a9daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 -- 172.21.15.28:0/2571102518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9a715620 con 0x564e9a9d04e0 2024-03-22T02:21:04.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0073fff640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0060004650 tx=0x7f0060001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906464229c44f920 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0053fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060012d20 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f00600055c0 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.278+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0060005020 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0060010950 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 10 client.17679 fetch_fsmap learned FSMap version 32 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e9a9e4e70 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0060005020 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f0060029020 con 0x564e9a9d04e0 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 10 client.17679 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 10 client.17679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 20 client.17679 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:04.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9a9e62f0 0x564e9a9e86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 4 client.17679 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e9a942580 con 0x564e9a9e62f0 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.279+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9a9e62f0 0x564e9a9e86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.280+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9a9e62f0 0x564e9a9e86e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fd5db8a232abe70 server_cookie=7536a28cd6fae43c in_seq=0 out_seq=0 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.280+0000 7f0053fff640 10 client.17679 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f0053fff640 1 -- 172.21.15.28:0/2571102518 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e9a942580 con 0x564e9a9e62f0 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f0053fff640 10 client.17679 handle_command_reply: tid=0 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 client.17679 shutdown 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 msgr2=0x564e9a9da580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f0060004650 tx=0x7f0060001e70 comp rx=0 tx=0).stop 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 shutdown_connections 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9a9e62f0 0x564e9a9e86e0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9e1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9daac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f0053fff640 0 client.17679 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 --2- 172.21.15.28:0/2571102518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9da580 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 >> 172.21.15.28:0/2571102518 conn(0x564e9a9cba50 msgr2=0x564e9a9cac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 shutdown_connections 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 1 -- 172.21.15.28:0/2571102518 wait complete. 2024-03-22T02:21:04.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.281+0000 7f007da10740 20 client.17679 trim_cache size 0 max 16384 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.296+0000 7f007da10740 1 Processor -- start 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.296+0000 7f007da10740 1 -- start start 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9e1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9d2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00741bbe30 con 0x564e9a9d04e0 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00741b9700 con 0x564e9a9d0ed0 2024-03-22T02:21:04.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f007da10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00741b7550 con 0x564e9a9d6040 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34076/0 (socket says 172.21.15.28:34076) 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0073fff640 1 -- 172.21.15.28:0/2524949831 learned_addr learned my addr 172.21.15.28:0/2524949831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9e1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0073fff640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9d2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 -- 172.21.15.28:0/2524949831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 msgr2=0x564e9a9d2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9d2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 -- 172.21.15.28:0/2524949831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 msgr2=0x564e9a9daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 -- 172.21.15.28:0/2524949831 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9a9d5200 con 0x564e9a9d0ed0 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0073fff640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.297+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9e1ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5450149c59b367d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f005e7fc640 1 -- 172.21.15.28:0/2524949831 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006804ac70 con 0x564e9a9d0ed0 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f005e7fc640 1 -- 172.21.15.28:0/2524949831 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f006804ae10 con 0x564e9a9d0ed0 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f005e7fc640 1 -- 172.21.15.28:0/2524949831 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006804a510 con 0x564e9a9d0ed0 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 -- 172.21.15.28:0/2524949831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 msgr2=0x564e9a9e1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9e1ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0068036040 tx=0x7f0068005fd0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 -- 172.21.15.28:0/2524949831 shutdown_connections 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d6040 0x564e9a9d2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d0ed0 0x564e9a9e1ac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 --2- 172.21.15.28:0/2524949831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9daa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 -- 172.21.15.28:0/2524949831 >> 172.21.15.28:0/2524949831 conn(0x564e9a9cba50 msgr2=0x564e9a9d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 -- 172.21.15.28:0/2524949831 shutdown_connections 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.298+0000 7f007da10740 1 -- 172.21.15.28:0/2524949831 wait complete. 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 -- start start 2024-03-22T02:21:04.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 Processor -- start 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9df7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d6040 0x564e9a9e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00741b7bc0 con 0x564e9a9d04e0 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00741b9030 con 0x564e9a9d6040 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00741b8290 con 0x564e9a9d0ed0 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f007da10740 10 client.0 _ll_register_callbacks cb 0x7f006000be00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f0073fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34078/0 (socket says 172.21.15.28:34078) 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f0073fff640 1 -- 172.21.15.28:0/2196868202 learned_addr learned my addr 172.21.15.28:0/2196868202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9df7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.299+0000 7f0073fff640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d6040 0x564e9a9e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 -- 172.21.15.28:0/2196868202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 msgr2=0x564e9a9df7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9df7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 -- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d6040 msgr2=0x564e9a9e07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d6040 0x564e9a9e07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 -- 172.21.15.28:0/2196868202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9a9e0db0 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0073fff640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f006000bbb0 tx=0x7f006002d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54aa164582915e08 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0053fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060041060 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f006804ae10 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f006002bab0 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0060048050 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f007da10740 10 client.17685 fetch_fsmap learned FSMap version 32 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564e9aa4c620 con 0x564e9a9d04e0 2024-03-22T02:21:04.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.300+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f006804ae10 con 0x564e9a9d04e0 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 32) ==== 8350+0+0 (secure 0 0 0) 0x7f0060029040 con 0x564e9a9d04e0 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 10 client.17685 fetch_fsmap finished waiting for FSMap version 32 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 10 client.17685 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 20 client.17685 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9aa4cff0 0x564e9aa4f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 4 client.17685 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e9a93c8d0 con 0x564e9aa4cff0 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9aa4cff0 0x564e9aa4f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f0078bc6640 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9aa4cff0 0x564e9aa4f410 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48fd1e919fd09c06 server_cookie=adade1ef2ee35b6d in_seq=0 out_seq=0 2024-03-22T02:21:04.308 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.301+0000 7f0053fff640 10 client.17685 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:04.422 INFO:tasks.workunit.client.0.smithi028.stdout: 1 689928 11.85 MB/sec execute 226 sec latency 91.275 ms 2024-03-22T02:21:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.533+0000 7f0053fff640 1 -- 172.21.15.28:0/2196868202 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e9a93c8d0 con 0x564e9aa4cff0 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.533+0000 7f0053fff640 10 client.17685 handle_command_reply: tid=0 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.533+0000 7f007da10740 1 client.17685 shutdown 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.533+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 msgr2=0x564e9a9df270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.533+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f006000bbb0 tx=0x7f006002d040 comp rx=0 tx=0).stop 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 shutdown_connections 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564e9aa4cff0 0x564e9aa4f410 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564e9a9d6040 0x564e9a9e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564e9a9d0ed0 0x564e9a9df7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f0053fff640 0 client.17685 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 --2- 172.21.15.28:0/2196868202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e9a9d04e0 0x564e9a9df270 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 >> 172.21.15.28:0/2196868202 conn(0x564e9a9cba50 msgr2=0x564e9a9da8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 shutdown_connections 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 1 -- 172.21.15.28:0/2196868202 wait complete. 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.534+0000 7f007da10740 20 client.17685 trim_cache size 0 max 16384 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:21:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:21:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 msgr2=0x7f00640a5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 0x7f00640a5550 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f006c01a040 tx=0x7f006c005940 comp rx=0 tx=0).stop 2024-03-22T02:21:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 msgr2=0x7f0074195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 0x7f0074195f90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f006c009fd0 tx=0x7f006c00ef10 comp rx=0 tx=0).stop 2024-03-22T02:21:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 shutdown_connections 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f00640a3090 0x7f00640a5550 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0074102670 0x7f0074195f90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0074101cb0 0x7f007419aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 --2- 172.21.15.28:0/2459379690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0074100ac0 0x7f0074195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 >> 172.21.15.28:0/2459379690 conn(0x7f00740fc050 msgr2=0x7f00740fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 shutdown_connections 2024-03-22T02:21:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:04.535+0000 7f007ae2a640 1 -- 172.21.15.28:0/2459379690 wait complete. 2024-03-22T02:21:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:04 smithi120 ceph-mon[36803]: pgmap v583: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 361 op/s 2024-03-22T02:21:04.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:04 smithi083 ceph-mon[36340]: pgmap v583: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 361 op/s 2024-03-22T02:21:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:04 smithi028 ceph-mon[31916]: pgmap v583: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 361 op/s 2024-03-22T02:21:05.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 692600 11.87 MB/sec execute 227 sec latency 152.610 ms 2024-03-22T02:21:05.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:05 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:21:05.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:05 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:21:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:05 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:21:06.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 694485 11.85 MB/sec execute 228 sec latency 46.704 ms 2024-03-22T02:21:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:06 smithi120 ceph-mon[36803]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:21:06.774 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:06 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:21:06.774 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:06 smithi083 ceph-mon[36340]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:21:06.774 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:06 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:21:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:06 smithi028 ceph-mon[31916]: pgmap v584: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 426 op/s 2024-03-22T02:21:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:06 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:21:07.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 696643 11.86 MB/sec execute 229 sec latency 22.806 ms 2024-03-22T02:21:08.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 698643 11.84 MB/sec execute 230 sec latency 67.825 ms 2024-03-22T02:21:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:08 smithi120 ceph-mon[36803]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-03-22T02:21:08.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:08 smithi083 ceph-mon[36340]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-03-22T02:21:08.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:08 smithi028 ceph-mon[31916]: pgmap v585: 129 pgs: 129 active+clean; 60 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 306 op/s 2024-03-22T02:21:09.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 700402 11.84 MB/sec execute 231 sec latency 40.122 ms 2024-03-22T02:21:10.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 703249 11.86 MB/sec execute 232 sec latency 15.746 ms 2024-03-22T02:21:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:10 smithi120 ceph-mon[36803]: pgmap v586: 129 pgs: 129 active+clean; 57 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 411 op/s 2024-03-22T02:21:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:10 smithi083 ceph-mon[36340]: pgmap v586: 129 pgs: 129 active+clean; 57 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 411 op/s 2024-03-22T02:21:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:10 smithi028 ceph-mon[31916]: pgmap v586: 129 pgs: 129 active+clean; 57 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 411 op/s 2024-03-22T02:21:11.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 705045 11.84 MB/sec execute 233 sec latency 28.951 ms 2024-03-22T02:21:11.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:11 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:21:11.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:11 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:21:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:11 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:21:11.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:11 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:21:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:11 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:21:11.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:11 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:21:12.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 707108 11.85 MB/sec execute 234 sec latency 63.157 ms 2024-03-22T02:21:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: pgmap v587: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-22T02:21:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.750 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.751 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:12.751 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:12.751 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:21:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: pgmap v587: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-22T02:21:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:12.767 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: pgmap v587: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:21:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:21:12.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:21:12.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:21:13.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 709090 11.83 MB/sec execute 235 sec latency 49.425 ms 2024-03-22T02:21:13.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:13 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:21:13.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:13 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:21:13.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:13 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:21:13.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:13 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:21:13.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:13 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:21:13.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:13 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:21:13.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:13 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:21:13.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:13 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:21:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:13 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:21:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:13 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:21:13.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:13 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:21:13.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:13 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:21:14.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 710961 11.83 MB/sec execute 236 sec latency 30.103 ms 2024-03-22T02:21:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:14 smithi120 ceph-mon[36803]: pgmap v588: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-03-22T02:21:14.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:14 smithi083 ceph-mon[36340]: pgmap v588: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-03-22T02:21:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:14 smithi028 ceph-mon[31916]: pgmap v588: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-03-22T02:21:15.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 713778 11.85 MB/sec execute 237 sec latency 11.530 ms 2024-03-22T02:21:16.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 715258 11.83 MB/sec execute 238 sec latency 29.267 ms 2024-03-22T02:21:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:16 smithi120 ceph-mon[36803]: pgmap v589: 129 pgs: 129 active+clean; 54 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-03-22T02:21:16.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:16 smithi083 ceph-mon[36340]: pgmap v589: 129 pgs: 129 active+clean; 54 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-03-22T02:21:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:16 smithi028 ceph-mon[31916]: pgmap v589: 129 pgs: 129 active+clean; 54 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 431 op/s 2024-03-22T02:21:17.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 717674 11.84 MB/sec execute 239 sec latency 66.869 ms 2024-03-22T02:21:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 -- 172.21.15.28:0/915249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 msgr2=0x7f958c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 --2- 172.21.15.28:0/915249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c100ed0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f957c009900 tx=0x7f957c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 -- 172.21.15.28:0/915249900 shutdown_connections 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 --2- 172.21.15.28:0/915249900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 --2- 172.21.15.28:0/915249900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 --2- 172.21.15.28:0/915249900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c100ed0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 -- 172.21.15.28:0/915249900 >> 172.21.15.28:0/915249900 conn(0x7f958c0fc020 msgr2=0x7f958c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 -- 172.21.15.28:0/915249900 shutdown_connections 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.625+0000 7f95921bc640 1 -- 172.21.15.28:0/915249900 wait complete. 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 Processor -- start 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 -- start start 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.630 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c1035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c730 con 0x7f958c100ab0 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c5b0 con 0x7f958c101ca0 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f95921bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f958c10c8b0 con 0x7f958c102660 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37402/0 (socket says 172.21.15.28:37402) 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 -- 172.21.15.28:0/144442550 learned_addr learned my addr 172.21.15.28:0/144442550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958bfff640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c1035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958affd640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 -- 172.21.15.28:0/144442550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 msgr2=0x7f958c1035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c1035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.631 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 -- 172.21.15.28:0/144442550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 msgr2=0x7f958c103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958b7fe640 1 -- 172.21.15.28:0/144442550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c103e50 con 0x7f958c100ab0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958bfff640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.626+0000 7f958affd640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f958b7fe640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f957c000c00 tx=0x7f957c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f9588ff9640 1 -- 172.21.15.28:0/144442550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c041070 con 0x7f958c100ab0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f9588ff9640 1 -- 172.21.15.28:0/144442550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f957c004450 con 0x7f958c100ab0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c1a1f20 con 0x7f958c100ab0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f9588ff9640 1 -- 172.21.15.28:0/144442550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f957c034440 con 0x7f958c100ab0 2024-03-22T02:21:17.632 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.627+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1a2330 con 0x7f958c100ab0 2024-03-22T02:21:17.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.629+0000 7f9588ff9640 1 -- 172.21.15.28:0/144442550 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f957c044070 con 0x7f958c100ab0 2024-03-22T02:21:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 Processor -- start 2024-03-22T02:21:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 -- start start 2024-03-22T02:21:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b519d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b544d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563449ab1040 con 0x563449b4ced0 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563449a599e0 con 0x563449b4c4e0 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9594da6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563449aacff0 con 0x563449b52040 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9588ff9640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45708/0 (socket says 172.21.15.28:45708) 2024-03-22T02:21:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43068/0 (socket says 172.21.15.28:43068) 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b519d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37418/0 (socket says 172.21.15.28:37418) 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 -- 172.21.15.28:0/1057145927 learned_addr learned my addr 172.21.15.28:0/1057145927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f9588ff9640 1 -- 172.21.15.28:0/144442550 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f957c0f1230 con 0x7f958c100ab0 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 -- 172.21.15.28:0/1057145927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 msgr2=0x563449b544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 -- 172.21.15.28:0/1057145927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 msgr2=0x563449b519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.630+0000 7f958bfff640 1 -- 172.21.15.28:0/1057145927 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634497c5210 con 0x563449b4c4e0 2024-03-22T02:21:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.631+0000 7f958bfff640 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9584011fd0 tx=0x7f958400fa50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2abbf81a8a7f81b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.631+0000 7f958bfff640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f958400ff50 tx=0x7f958401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.631+0000 7f9571ffb640 1 -- 172.21.15.28:0/1057145927 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958402b070 con 0x563449b4c4e0 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.631+0000 7f9571ffb640 1 -- 172.21.15.28:0/1057145927 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9584011630 con 0x563449b4c4e0 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.631+0000 7f9571ffb640 1 -- 172.21.15.28:0/1057145927 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584022070 con 0x563449b4c4e0 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- 172.21.15.28:0/1057145927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 msgr2=0x563449b4c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9584011fd0 tx=0x7f958400fa50 comp rx=0 tx=0).stop 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- 172.21.15.28:0/1057145927 shutdown_connections 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- 172.21.15.28:0/1057145927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4c900 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- 172.21.15.28:0/1057145927 >> 172.21.15.28:0/1057145927 conn(0x563449b47a50 msgr2=0x563449b47e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- 172.21.15.28:0/1057145927 shutdown_connections 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- 172.21.15.28:0/1057145927 wait complete. 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- start start 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 Processor -- start 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b56490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 0x563449b5da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563449b54f90 con 0x563449b52040 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563449a6cdb0 con 0x563449b4c4e0 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563449b55110 con 0x563449b4ced0 2024-03-22T02:21:17.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.632+0000 7f9594da6740 10 client.0 _ll_register_callbacks cb 0x5634497c4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 0x563449b5da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43082/0 (socket says 172.21.15.28:43082) 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b56490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 -- 172.21.15.28:0/2051132266 learned_addr learned my addr 172.21.15.28:0/2051132266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 -- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 msgr2=0x563449b56490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b56490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 -- 172.21.15.28:0/2051132266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 msgr2=0x563449b5da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 0x563449b5da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 -- 172.21.15.28:0/2051132266 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563449891620 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958bfff640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 0x563449b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f958affd640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9580012d00 tx=0x7f958000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d4c6105adbb948 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f956affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580012110 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9584011630 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9580004050 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9580025050 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f9594da6740 10 client.27520 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.633+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563449b60ea0 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9584011630 con 0x563449b4ced0 2024-03-22T02:21:17.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f9580010430 con 0x563449b4ced0 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 10 client.27520 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 10 client.27520 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 20 client.27520 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449b62590 0x563449b649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 4 client.27520 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563449abe580 con 0x563449b62590 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f958b7fe640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449b62590 0x563449b649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f958b7fe640 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449b62590 0x563449b649f0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6790d72c8dfbaad7 server_cookie=74246d8fb69a1b01 in_seq=0 out_seq=0 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.634+0000 7f956affd640 10 client.27520 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f956affd640 1 -- 172.21.15.28:0/2051132266 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563449abe580 con 0x563449b62590 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f956affd640 10 client.27520 handle_command_reply: tid=0 2024-03-22T02:21:17.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 client.27520 shutdown 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 msgr2=0x563449b569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9580012d00 tx=0x7f958000f790 comp rx=0 tx=0).stop 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 shutdown_connections 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449b62590 0x563449b649f0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b52040 0x563449b5da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f956affd640 0 client.27520 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4ced0 0x563449b569d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 --2- 172.21.15.28:0/2051132266 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b56490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 >> 172.21.15.28:0/2051132266 conn(0x563449b47a50 msgr2=0x563449b46c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 shutdown_connections 2024-03-22T02:21:17.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 1 -- 172.21.15.28:0/2051132266 wait complete. 2024-03-22T02:21:17.651 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.636+0000 7f9594da6740 20 client.27520 trim_cache size 0 max 16384 2024-03-22T02:21:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.652+0000 7f9594da6740 1 Processor -- start 2024-03-22T02:21:17.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.652+0000 7f9594da6740 1 -- start start 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b4b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1af440 con 0x563449b4ced0 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1acd10 con 0x563449b4c4e0 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f9594da6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1aab60 con 0x563449b52040 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37438/0 (socket says 172.21.15.28:37438) 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 -- 172.21.15.28:0/342535194 learned_addr learned my addr 172.21.15.28:0/342535194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958bfff640 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b4b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 -- 172.21.15.28:0/342535194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 msgr2=0x563449b4b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b4b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 -- 172.21.15.28:0/342535194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 msgr2=0x563449b4acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 -- 172.21.15.28:0/342535194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563449b4b730 con 0x563449b4ced0 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f958affd640 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9580009730 tx=0x7f9580012320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e78c7235cd17e24 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.653+0000 7f95717fa640 1 -- 172.21.15.28:0/342535194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958002d070 con 0x563449b4ced0 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f95717fa640 1 -- 172.21.15.28:0/342535194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f958000fe80 con 0x563449b4ced0 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f95717fa640 1 -- 172.21.15.28:0/342535194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580000e60 con 0x563449b4ced0 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 -- 172.21.15.28:0/342535194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 msgr2=0x563449b5d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9580009730 tx=0x7f9580012320 comp rx=0 tx=0).stop 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 -- 172.21.15.28:0/342535194 shutdown_connections 2024-03-22T02:21:17.660 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b52040 0x563449b4b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5d9d0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 --2- 172.21.15.28:0/342535194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b4c4e0 0x563449b4acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 -- 172.21.15.28:0/342535194 >> 172.21.15.28:0/342535194 conn(0x563449b47a50 msgr2=0x563449b56490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 -- 172.21.15.28:0/342535194 shutdown_connections 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.654+0000 7f9594da6740 1 -- 172.21.15.28:0/342535194 wait complete. 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 -- start start 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 Processor -- start 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 0x563449b5af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1ab1d0 con 0x563449b4ced0 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1ac640 con 0x563449b52040 2024-03-22T02:21:17.661 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1ab8a0 con 0x563449b4c4e0 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f9594da6740 10 client.0 _ll_register_callbacks cb 0x7f9580009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 0x563449b5af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45740/0 (socket says 172.21.15.28:45740) 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37446/0 (socket says 172.21.15.28:37446) 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 0x563449b5af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43100/0 (socket says 172.21.15.28:43100) 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 -- 172.21.15.28:0/3919287188 learned_addr learned my addr 172.21.15.28:0/3919287188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 -- 172.21.15.28:0/3919287188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 msgr2=0x563449b5af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 0x563449b5af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 -- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 msgr2=0x563449b5c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958affd640 1 -- 172.21.15.28:0/3919287188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563449b5cab0 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.655+0000 7f958bfff640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f958affd640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9580010c70 tx=0x7f95800127f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508aef28608c346c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580036060 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9580005070 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563449ab88d0 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f958003d050 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f9594da6740 10 client.17703 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x563449b5ddc0 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563449ab88d0 con 0x563449b4ced0 2024-03-22T02:21:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.656+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f9580030070 con 0x563449b4ced0 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 10 client.17703 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 10 client.17703 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 20 client.17703 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449bc9110 0x563449bcb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 4 client.17703 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563449a36770 con 0x563449bc9110 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f958b7fe640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449bc9110 0x563449bcb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f958b7fe640 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449bc9110 0x563449bcb530 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4610610ea298c31 server_cookie=c7faca4124011768 in_seq=0 out_seq=0 2024-03-22T02:21:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:17.657+0000 7f956affd640 10 client.17703 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f956affd640 1 -- 172.21.15.28:0/3919287188 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563449a36770 con 0x563449bc9110 2024-03-22T02:21:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f956affd640 10 client.17703 handle_command_reply: tid=0 2024-03-22T02:21:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 client.17703 shutdown 2024-03-22T02:21:18.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 msgr2=0x563449b5b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f9580010c70 tx=0x7f95800127f0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 shutdown_connections 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563449bc9110 0x563449bcb530 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563449b52040 0x563449b5c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563449b4ced0 0x563449b5b4b0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 --2- 172.21.15.28:0/3919287188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563449b4c4e0 0x563449b5af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f956affd640 0 client.17703 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 >> 172.21.15.28:0/3919287188 conn(0x563449b47a50 msgr2=0x563449b56120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 shutdown_connections 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 1 -- 172.21.15.28:0/3919287188 wait complete. 2024-03-22T02:21:18.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.005+0000 7f9594da6740 20 client.17703 trim_cache size 0 max 16384 2024-03-22T02:21:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:21:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:21:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:21:18.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 msgr2=0x7f95740a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f958400ff50 tx=0x7f958401e000 comp rx=0 tx=0).stop 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 msgr2=0x7f958c104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f957c000c00 tx=0x7f957c02c640 comp rx=0 tx=0).stop 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 shutdown_connections 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f95740a2fc0 0x7f95740a5480 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f958c102660 0x7f958c1035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f958c101ca0 0x7f958c103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 --2- 172.21.15.28:0/144442550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f958c100ab0 0x7f958c104a40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 >> 172.21.15.28:0/144442550 conn(0x7f958c0fc020 msgr2=0x7f958c0fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 shutdown_connections 2024-03-22T02:21:18.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:18.006+0000 7f95921bc640 1 -- 172.21.15.28:0/144442550 wait complete. 2024-03-22T02:21:18.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 718386 11.80 MB/sec execute 240 sec latency 292.324 ms 2024-03-22T02:21:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:18 smithi120 ceph-mon[36803]: pgmap v590: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 357 op/s 2024-03-22T02:21:18.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:18 smithi083 ceph-mon[36340]: pgmap v590: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 357 op/s 2024-03-22T02:21:18.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:18 smithi028 ceph-mon[31916]: pgmap v590: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 357 op/s 2024-03-22T02:21:19.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 721100 11.82 MB/sec execute 241 sec latency 15.382 ms 2024-03-22T02:21:20.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 723195 11.80 MB/sec execute 242 sec latency 27.215 ms 2024-03-22T02:21:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:20 smithi120 ceph-mon[36803]: pgmap v591: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 433 op/s 2024-03-22T02:21:20.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:20 smithi083 ceph-mon[36340]: pgmap v591: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 433 op/s 2024-03-22T02:21:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:20 smithi028 ceph-mon[31916]: pgmap v591: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 433 op/s 2024-03-22T02:21:21.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 724998 11.80 MB/sec execute 243 sec latency 21.482 ms 2024-03-22T02:21:22.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 727324 11.81 MB/sec execute 244 sec latency 25.159 ms 2024-03-22T02:21:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:22 smithi120 ceph-mon[36803]: pgmap v592: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 344 op/s 2024-03-22T02:21:22.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:22 smithi083 ceph-mon[36340]: pgmap v592: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 344 op/s 2024-03-22T02:21:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:22 smithi028 ceph-mon[31916]: pgmap v592: 129 pgs: 129 active+clean; 53 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.4 MiB/s wr, 344 op/s 2024-03-22T02:21:23.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 728645 11.79 MB/sec execute 245 sec latency 79.615 ms 2024-03-22T02:21:24.423 INFO:tasks.workunit.client.0.smithi028.stdout: 1 731468 11.81 MB/sec execute 246 sec latency 9.254 ms 2024-03-22T02:21:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:24 smithi120 ceph-mon[36803]: pgmap v593: 129 pgs: 129 active+clean; 56 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 393 op/s 2024-03-22T02:21:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:24 smithi083 ceph-mon[36340]: pgmap v593: 129 pgs: 129 active+clean; 56 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 393 op/s 2024-03-22T02:21:24.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:24 smithi028 ceph-mon[31916]: pgmap v593: 129 pgs: 129 active+clean; 56 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 393 op/s 2024-03-22T02:21:25.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 733025 11.79 MB/sec execute 247 sec latency 29.236 ms 2024-03-22T02:21:26.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 735433 11.80 MB/sec execute 248 sec latency 21.203 ms 2024-03-22T02:21:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:26 smithi120 ceph-mon[36803]: pgmap v594: 129 pgs: 129 active+clean; 51 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:21:26.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:26 smithi083 ceph-mon[36340]: pgmap v594: 129 pgs: 129 active+clean; 51 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:21:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:26 smithi028 ceph-mon[31916]: pgmap v594: 129 pgs: 129 active+clean; 51 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 428 op/s 2024-03-22T02:21:27.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 737788 11.79 MB/sec execute 249 sec latency 29.579 ms 2024-03-22T02:21:28.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 739159 11.78 MB/sec execute 250 sec latency 59.350 ms 2024-03-22T02:21:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:28 smithi120 ceph-mon[36803]: pgmap v595: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 403 op/s 2024-03-22T02:21:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:28 smithi083 ceph-mon[36340]: pgmap v595: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 403 op/s 2024-03-22T02:21:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:28 smithi028 ceph-mon[31916]: pgmap v595: 129 pgs: 129 active+clean; 55 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 403 op/s 2024-03-22T02:21:29.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 742103 11.80 MB/sec execute 251 sec latency 29.247 ms 2024-03-22T02:21:29.613 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:21:29.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:21:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 -- 172.21.15.28:0/774137105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28107870 msgr2=0x7f9e28109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 --2- 172.21.15.28:0/774137105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28107870 0x7f9e28109cc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f9e20009900 tx=0x7f9e2002bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 -- 172.21.15.28:0/774137105 shutdown_connections 2024-03-22T02:21:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 --2- 172.21.15.28:0/774137105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28107870 0x7f9e28109cc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 --2- 172.21.15.28:0/774137105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28104f40 0x7f9e28107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 --2- 172.21.15.28:0/774137105 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28102610 0x7f9e28104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.725+0000 7f9e304fb640 1 -- 172.21.15.28:0/774137105 >> 172.21.15.28:0/774137105 conn(0x7f9e280fc020 msgr2=0x7f9e280fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- 172.21.15.28:0/774137105 shutdown_connections 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- 172.21.15.28:0/774137105 wait complete. 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 Processor -- start 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- start start 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 0x7f9e2819eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28107870 0x7f9e281a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e28101200 con 0x7f9e28102610 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e28101080 con 0x7f9e28104f40 2024-03-22T02:21:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.726+0000 7f9e304fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e28101380 con 0x7f9e28107870 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50594/0 (socket says 172.21.15.28:50594) 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2da96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 0x7f9e2819eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 -- 172.21.15.28:0/4012849491 learned_addr learned my addr 172.21.15.28:0/4012849491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28107870 0x7f9e281a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 -- 172.21.15.28:0/4012849491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28107870 msgr2=0x7f9e281a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28107870 0x7f9e281a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 -- 172.21.15.28:0/4012849491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 msgr2=0x7f9e2819eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 0x7f9e2819eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 -- 172.21.15.28:0/4012849491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e281a60f0 con 0x7f9e28102610 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2da96640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 0x7f9e2819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e2e297640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400ebc0 tx=0x7f9e2400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/4012849491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24010630 con 0x7f9e28102610 2024-03-22T02:21:29.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/4012849491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e24010cb0 con 0x7f9e28102610 2024-03-22T02:21:29.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e281a63e0 con 0x7f9e28102610 2024-03-22T02:21:29.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/4012849491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24018bd0 con 0x7f9e28102610 2024-03-22T02:21:29.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.727+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e281a6930 con 0x7f9e28102610 2024-03-22T02:21:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/4012849491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9e240107d0 con 0x7f9e28102610 2024-03-22T02:21:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 Processor -- start 2024-03-22T02:21:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 -- start start 2024-03-22T02:21:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9de8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9deb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cd9d488c0 con 0x558cd9de3430 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558cd9cf0c20 con 0x558cd9de8f90 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e331c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558cd9d44870 con 0x558cd9de3e20 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.730+0000 7f9e1f7fe640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 0x7f9e140a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50608/0 (socket says 172.21.15.28:50608) 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 -- 172.21.15.28:0/4119890988 learned_addr learned my addr 172.21.15.28:0/4119890988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9deb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 0x7f9e140a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2e297640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9de8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e1f7fe640 1 -- 172.21.15.28:0/4012849491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e240cfda0 con 0x7f9e28102610 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 -- 172.21.15.28:0/4119890988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 msgr2=0x558cd9de8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9de8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 -- 172.21.15.28:0/4119890988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 msgr2=0x558cd9de3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 -- 172.21.15.28:0/4119890988 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd9d15150 con 0x558cd9de8f90 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2da96640 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9deb420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9e28069850 tx=0x7f9e18011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c3f3240a0fc38d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 0x7f9e140a5410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x558cd9de4680 tx=0x7f9e2002c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e1cbfe640 1 -- 172.21.15.28:0/4119890988 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18004030 con 0x558cd9de8f90 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e1cbfe640 1 -- 172.21.15.28:0/4119890988 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e18005500 con 0x558cd9de8f90 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.731+0000 7f9e1cbfe640 1 -- 172.21.15.28:0/4119890988 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18004e10 con 0x558cd9de8f90 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 -- 172.21.15.28:0/4119890988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 msgr2=0x558cd9deb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9deb420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9e28069850 tx=0x7f9e18011810 comp rx=0 tx=0).stop 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 -- 172.21.15.28:0/4119890988 shutdown_connections 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9deb420 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9de8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 --2- 172.21.15.28:0/4119890988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9de3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 -- 172.21.15.28:0/4119890988 >> 172.21.15.28:0/4119890988 conn(0x558cd9dde9a0 msgr2=0x558cd9ddeda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 -- 172.21.15.28:0/4119890988 shutdown_connections 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.732+0000 7f9e331c2740 1 -- 172.21.15.28:0/4119890988 wait complete. 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 -- start start 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 Processor -- start 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9dedeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9df4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558cd9debf40 con 0x558cd9de3430 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558cd9d04130 con 0x558cd9de8f90 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558cd9dec0c0 con 0x558cd9de3e20 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e2e297640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e331c2740 10 client.0 _ll_register_callbacks cb 0x558cd9d03b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e2e297640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50610/0 (socket says 172.21.15.28:50610) 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e2da96640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9dedeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e2e297640 1 -- 172.21.15.28:0/1833767465 learned_addr learned my addr 172.21.15.28:0/1833767465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.733+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9df4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 -- 172.21.15.28:0/1833767465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 msgr2=0x558cd9dedeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9dedeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 -- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 msgr2=0x558cd9df4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9df4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 -- 172.21.15.28:0/1833767465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd9b267d0 con 0x558cd9de3430 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9e240cd800 tx=0x7f9e2409c170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1180beebf0ea47d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e24010e60 con 0x558cd9de3430 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e240bc0c0 con 0x558cd9de3430 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e18005500 con 0x558cd9de3430 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9e240bc2c0 con 0x558cd9de3430 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e331c2740 10 client.17715 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:29.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558cd9a5c3f0 con 0x558cd9de3430 2024-03-22T02:21:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e18005500 con 0x558cd9de3430 2024-03-22T02:21:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.734+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f9e24096ce0 con 0x558cd9de3430 2024-03-22T02:21:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 10 client.17715 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 10 client.17715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 20 client.17715 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9df9770 0x558cd9dfbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 4 client.17715 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cd9d55e00 con 0x558cd9df9770 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e2da96640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9df9770 0x558cd9dfbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e2da96640 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9df9770 0x558cd9dfbc00 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3643bc1458d4deb server_cookie=b2a18b9f4d27bf07 in_seq=0 out_seq=0 2024-03-22T02:21:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.735+0000 7f9e11ffb640 10 client.17715 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1833767465 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cd9d55e00 con 0x558cd9df9770 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e11ffb640 10 client.17715 handle_command_reply: tid=0 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 client.17715 shutdown 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 msgr2=0x558cd9ded970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9e240cd800 tx=0x7f9e2409c170 comp rx=0 tx=0).stop 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 shutdown_connections 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9df9770 0x558cd9dfbc00 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de8f90 0x558cd9df4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3e20 0x558cd9dedeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e11ffb640 0 client.17715 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1833767465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de3430 0x558cd9ded970 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 >> 172.21.15.28:0/1833767465 conn(0x558cd9dde9a0 msgr2=0x558cd9dddc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 shutdown_connections 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 1 -- 172.21.15.28:0/1833767465 wait complete. 2024-03-22T02:21:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.738+0000 7f9e331c2740 20 client.17715 trim_cache size 0 max 16384 2024-03-22T02:21:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 Processor -- start 2024-03-22T02:21:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 -- start start 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9ddd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281b39c0 con 0x558cd9de8f90 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281b1290 con 0x558cd9de3e20 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e331c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281af0e0 con 0x558cd9de3430 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2da96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50626/0 (socket says 172.21.15.28:50626) 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1430312365 learned_addr learned my addr 172.21.15.28:0/1430312365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9ddd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1430312365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 msgr2=0x558cd9ddd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9ddd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1430312365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 msgr2=0x558cd9df4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1430312365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cd9a1c0f0 con 0x558cd9de8f90 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.757+0000 7f9e2da96640 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.758+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9e2002c7b0 tx=0x7f9e20005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283c8d839debdf15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.758+0000 7f9e13fff640 1 -- 172.21.15.28:0/1430312365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20055070 con 0x558cd9de8f90 2024-03-22T02:21:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.758+0000 7f9e13fff640 1 -- 172.21.15.28:0/1430312365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e200044b0 con 0x558cd9de8f90 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.758+0000 7f9e13fff640 1 -- 172.21.15.28:0/1430312365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20003850 con 0x558cd9de8f90 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- 172.21.15.28:0/1430312365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 msgr2=0x558cd9ddd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9e2002c7b0 tx=0x7f9e20005b50 comp rx=0 tx=0).stop 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- 172.21.15.28:0/1430312365 shutdown_connections 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9ddd8b0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1430312365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9ddd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- 172.21.15.28:0/1430312365 >> 172.21.15.28:0/1430312365 conn(0x558cd9dde9a0 msgr2=0x558cd9de2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- 172.21.15.28:0/1430312365 shutdown_connections 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- 172.21.15.28:0/1430312365 wait complete. 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- start start 2024-03-22T02:21:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 Processor -- start 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9df27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281af750 con 0x558cd9de8f90 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281b0bc0 con 0x558cd9de3e20 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9e281afe20 con 0x558cd9de3430 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e331c2740 10 client.0 _ll_register_callbacks cb 0x7f9e2409bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e2ea98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50628/0 (socket says 172.21.15.28:50628) 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e2da96640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1850557604 learned_addr learned my addr 172.21.15.28:0/1850557604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.759+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9df27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1850557604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 msgr2=0x558cd9df27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9df27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 msgr2=0x558cd9df2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 -- 172.21.15.28:0/1850557604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cd9df42e0 con 0x558cd9de8f90 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2da96640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e2ea98640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005ab0 tx=0x7f9e20005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bbcb12f534cd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:29.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2005d060 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f9e20004b50 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e200044b0 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f9e20055070 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e331c2740 10 client.17721 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558cd9e5fce0 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e200044b0 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.760+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f9e200506b0 con 0x558cd9de8f90 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 10 client.17721 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 10 client.17721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 20 client.17721 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9e606f0 0x558cd9e62b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 4 client.17721 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:21:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558cd9d50150 con 0x558cd9e606f0 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9e606f0 0x558cd9e62b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e2e297640 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9e606f0 0x558cd9e62b10 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f120a485e7a3ea35 server_cookie=5d9b347dfabb0d09 in_seq=0 out_seq=0 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.761+0000 7f9e11ffb640 10 client.17721 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e11ffb640 1 -- 172.21.15.28:0/1850557604 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558cd9d50150 con 0x558cd9e606f0 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e11ffb640 10 client.17721 handle_command_reply: tid=0 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 client.17721 shutdown 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 msgr2=0x558cd9df3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9e20005ab0 tx=0x7f9e20005780 comp rx=0 tx=0).stop 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 shutdown_connections 2024-03-22T02:21:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558cd9e606f0 0x558cd9e62b10 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9de8f90 0x558cd9df3cf0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558cd9de3e20 0x558cd9df2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 --2- 172.21.15.28:0/1850557604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558cd9de3430 0x558cd9df27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 >> 172.21.15.28:0/1850557604 conn(0x558cd9dde9a0 msgr2=0x558cd9ddd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e11ffb640 0 client.17721 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 shutdown_connections 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 1 -- 172.21.15.28:0/1850557604 wait complete. 2024-03-22T02:21:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.762+0000 7f9e331c2740 20 client.17721 trim_cache size 0 max 16384 2024-03-22T02:21:29.776 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:21:29.776 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:21:29.776 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:21:29.776 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:21:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 msgr2=0x7f9e140a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 0x7f9e140a5410 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x558cd9de4680 tx=0x7f9e2002c880 comp rx=0 tx=0).stop 2024-03-22T02:21:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 msgr2=0x7f9e2819e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9e2400ebc0 tx=0x7f9e2400c6a0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 shutdown_connections 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9e140a2f50 0x7f9e140a5410 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9e28107870 0x7f9e281a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9e28104f40 0x7f9e2819eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 --2- 172.21.15.28:0/4012849491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e28102610 0x7f9e2819e560 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 >> 172.21.15.28:0/4012849491 conn(0x7f9e280fc020 msgr2=0x7f9e28107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 shutdown_connections 2024-03-22T02:21:29.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.763+0000 7f9e304fb640 1 -- 172.21.15.28:0/4012849491 wait complete. 2024-03-22T02:21:29.778 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:21:29.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.887+0000 7f723dd25640 1 -- 172.21.15.28:0/4257555533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238107890 msgr2=0x7f7238109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.887+0000 7f723dd25640 1 --2- 172.21.15.28:0/4257555533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238107890 0x7f7238109ce0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7230009ca0 tx=0x7f723002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- 172.21.15.28:0/4257555533 shutdown_connections 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- 172.21.15.28:0/4257555533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238107890 0x7f7238109ce0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- 172.21.15.28:0/4257555533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238104f60 0x7f7238107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- 172.21.15.28:0/4257555533 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- 172.21.15.28:0/4257555533 >> 172.21.15.28:0/4257555533 conn(0x7f72380fc020 msgr2=0x7f72380fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- 172.21.15.28:0/4257555533 shutdown_connections 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- 172.21.15.28:0/4257555533 wait complete. 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 Processor -- start 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- start start 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 0x7f723819f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f723810d850 con 0x7f7238104f60 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f723810d6d0 con 0x7f7238102630 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.888+0000 7f723dd25640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f723810d9d0 con 0x7f7238107890 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f72377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50648/0 (socket says 172.21.15.28:50648) 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 -- 172.21.15.28:0/4265920794 learned_addr learned my addr 172.21.15.28:0/4265920794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:29.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f7237fff640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 0x7f723819f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 -- 172.21.15.28:0/4265920794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 msgr2=0x7f723819f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 0x7f723819f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 -- 172.21.15.28:0/4265920794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 msgr2=0x7f7238197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 -- 172.21.15.28:0/4265920794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f723819fa90 con 0x7f7238104f60 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f7237fff640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 0x7f723819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f72377fe640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f722ffff640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f722000e9f0 tx=0x7f722000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722000ccc0 con 0x7f7238104f60 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7220005500 con 0x7f7238104f60 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f723819fd80 con 0x7f7238104f60 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7220010690 con 0x7f7238104f60 2024-03-22T02:21:29.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.889+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72381a0270 con 0x7f7238104f60 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.890+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72380fdb10 con 0x7f7238104f60 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.891+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f72200046c0 con 0x7f7238104f60 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.892+0000 7f72357fa640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 0x7f72240a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.892+0000 7f72377fe640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 0x7f72240a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.892+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f72200cfa90 con 0x7f7238104f60 2024-03-22T02:21:29.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.892+0000 7f72377fe640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 0x7f72240a5460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7228009fc0 tx=0x7f7228005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:29.897+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f722009c4b0 con 0x7f7238104f60 2024-03-22T02:21:30.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.128+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f723802d3b0 con 0x7f7238104f60 2024-03-22T02:21:30.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.129+0000 7f72357fa640 1 -- 172.21.15.28:0/4265920794 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+11866 (secure 0 0 0) 0x7f722009bdb0 con 0x7f7238104f60 2024-03-22T02:21:30.130 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:21:30.130 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":33,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:21:10.369573+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_26924":{"gid":26924,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2468455656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2468455656},{"type":"v1","addr":"172.21.15.83:6837","nonce":2468455656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:21:30.131 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 33 2024-03-22T02:21:30.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 msgr2=0x7f72240a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 0x7f72240a5460 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f7228009fc0 tx=0x7f7228005f90 comp rx=0 tx=0).stop 2024-03-22T02:21:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 msgr2=0x7f72381983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f722000e9f0 tx=0x7f722000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 shutdown_connections 2024-03-22T02:21:30.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f72240a2fa0 0x7f72240a5460 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7238107890 0x7f723819f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7238104f60 0x7f72381983b0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 --2- 172.21.15.28:0/4265920794 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7238102630 0x7f7238197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 >> 172.21.15.28:0/4265920794 conn(0x7f72380fc020 msgr2=0x7f7238102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 shutdown_connections 2024-03-22T02:21:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.131+0000 7f723dd25640 1 -- 172.21.15.28:0/4265920794 wait complete. 2024-03-22T02:21:30.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:21:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/2343468644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad0108b90 msgr2=0x7f0ad010afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/2343468644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad0108b90 0x7f0ad010afc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009ca0 tx=0x7f0ac802c140 comp rx=0 tx=0).stop 2024-03-22T02:21:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/2343468644 shutdown_connections 2024-03-22T02:21:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/2343468644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 0x7f0ad010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/2343468644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad010b500 0x7f0ad010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/2343468644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad0108b90 0x7f0ad010afc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/2343468644 >> 172.21.15.28:0/2343468644 conn(0x7f0ad006e9f0 msgr2=0x7f0ad0070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/2343468644 shutdown_connections 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/2343468644 wait complete. 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.278+0000 7f0ad5a8d640 1 Processor -- start 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 -- start start 2024-03-22T02:21:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad010b500 0x7f0ad01a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 0x7f0ad01ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad011c600 con 0x7f0ad0108b90 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad011c480 con 0x7f0ad010b500 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0ad5a8d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad011c780 con 0x7f0ad010deb0 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50668/0 (socket says 172.21.15.28:50668) 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 -- 172.21.15.28:0/3518329636 learned_addr learned my addr 172.21.15.28:0/3518329636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0aceffd640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad010b500 0x7f0ad01a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acffff640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 0x7f0ad01ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 -- 172.21.15.28:0/3518329636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 msgr2=0x7f0ad01ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 0x7f0ad01ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 -- 172.21.15.28:0/3518329636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad010b500 msgr2=0x7f0ad01a72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad010b500 0x7f0ad01a72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 -- 172.21.15.28:0/3518329636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad01ae950 con 0x7f0ad0108b90 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.279+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8032520 tx=0x7f0ac8035cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.280+0000 7f0accff9640 1 -- 172.21.15.28:0/3518329636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8039040 con 0x7f0ad0108b90 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.280+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad01aebe0 con 0x7f0ad0108b90 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.280+0000 7f0accff9640 1 -- 172.21.15.28:0/3518329636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac8044070 con 0x7f0ad0108b90 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.280+0000 7f0accff9640 1 -- 172.21.15.28:0/3518329636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80043c0 con 0x7f0ad0108b90 2024-03-22T02:21:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.280+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad01af120 con 0x7f0ad0108b90 2024-03-22T02:21:30.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.282+0000 7f0accff9640 1 -- 172.21.15.28:0/3518329636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0ac8004600 con 0x7f0ad0108b90 2024-03-22T02:21:30.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.282+0000 7f0accff9640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 0x7f0ab80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0aceffd640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 0x7f0ab80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0accff9640 1 -- 172.21.15.28:0/3518329636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0ac80f1bf0 con 0x7f0ad0108b90 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 Processor -- start 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 -- start start 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0aceffd640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 0x7f0ab80a5460 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009bd0 tx=0x7f0ac4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b8063c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b8068d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b806b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608b7fc88d0 con 0x5608b8064210 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5608b7f70e00 con 0x5608b8063820 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.283+0000 7f0ad8670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608b7fc4880 con 0x5608b8069380 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0aceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b8068d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0aceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b8068d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50674/0 (socket says 172.21.15.28:50674) 2024-03-22T02:21:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0aceffd640 1 -- 172.21.15.28:0/3061021283 learned_addr learned my addr 172.21.15.28:0/3061021283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b8063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 -- 172.21.15.28:0/3061021283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 msgr2=0x5608b8063c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b8063c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 -- 172.21.15.28:0/3061021283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 msgr2=0x5608b8068d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b8068d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 -- 172.21.15.28:0/3061021283 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b7cdc3f0 con 0x5608b8069380 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0acffff640 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b806b810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000bb30 tx=0x7f0ac000f360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ce24b1b65f0e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/3061021283 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0015070 con 0x5608b8069380 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/3061021283 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac000fd90 con 0x5608b8069380 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.284+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/3061021283 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0011440 con 0x5608b8069380 2024-03-22T02:21:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- 172.21.15.28:0/3061021283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 msgr2=0x5608b806b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b806b810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ac000bb30 tx=0x7f0ac000f360 comp rx=0 tx=0).stop 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- 172.21.15.28:0/3061021283 shutdown_connections 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b806b810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b8068d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 --2- 172.21.15.28:0/3061021283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b8063c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- 172.21.15.28:0/3061021283 >> 172.21.15.28:0/3061021283 conn(0x5608b805ed90 msgr2=0x5608b805f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- 172.21.15.28:0/3061021283 shutdown_connections 2024-03-22T02:21:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- 172.21.15.28:0/3061021283 wait complete. 2024-03-22T02:21:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.285+0000 7f0ad8670740 1 -- start start 2024-03-22T02:21:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 Processor -- start 2024-03-22T02:21:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b8075280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608b806c330 con 0x5608b8064210 2024-03-22T02:21:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5608b7f84300 con 0x5608b8063820 2024-03-22T02:21:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5608b806c4b0 con 0x5608b8069380 2024-03-22T02:21:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50680/0 (socket says 172.21.15.28:50680) 2024-03-22T02:21:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 -- 172.21.15.28:0/1069085505 learned_addr learned my addr 172.21.15.28:0/1069085505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0acffff640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b8075280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0ad8670740 10 client.0 _ll_register_callbacks cb 0x5608b7fbfe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 -- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 msgr2=0x5608b8075280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b8075280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 -- 172.21.15.28:0/1069085505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 msgr2=0x5608b806dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b806dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0aceffd640 1 -- 172.21.15.28:0/1069085505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b7da67d0 con 0x5608b8064210 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.286+0000 7f0acffff640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b8075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40038a0 tx=0x7f0ac4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252a60576d8d4937 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac400e720 con 0x5608b8064210 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0ac400e8c0 con 0x5608b8064210 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac000fd90 con 0x5608b8064210 2024-03-22T02:21:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ac400eac0 con 0x5608b8064210 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0ad8670740 10 client.17739 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5608b8078640 con 0x5608b8064210 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac000fd90 con 0x5608b8064210 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.287+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f0ac401d410 con 0x5608b8064210 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 10 client.17739 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 10 client.17739 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 20 client.17739 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b8079dd0 0x5608b807c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 4 client.17739 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608b7fd5e10 con 0x5608b8079dd0 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.288+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b8079dd0 0x5608b807c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.289+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b8079dd0 0x5608b807c270 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b9cea771f56e88bc server_cookie=d1af5e47d9f38063 in_seq=0 out_seq=0 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.289+0000 7f0aaeffd640 10 client.17739 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.289+0000 7f0aaeffd640 1 -- 172.21.15.28:0/1069085505 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608b7fd5e10 con 0x5608b8079dd0 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.289+0000 7f0aaeffd640 10 client.17739 handle_command_reply: tid=0 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.289+0000 7f0ad8670740 1 client.17739 shutdown 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 msgr2=0x5608b806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40038a0 tx=0x7f0ac4004730 comp rx=0 tx=0).stop 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 shutdown_connections 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8069380 0x5608b8075280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8064210 0x5608b806e240 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b8079dd0 0x5608b807c270 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0aaeffd640 0 client.17739 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1069085505 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8063820 0x5608b806dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 >> 172.21.15.28:0/1069085505 conn(0x5608b805ed90 msgr2=0x5608b805dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 shutdown_connections 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 1 -- 172.21.15.28:0/1069085505 wait complete. 2024-03-22T02:21:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.290+0000 7f0ad8670740 20 client.17739 trim_cache size 0 max 16384 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.304+0000 7f0ad8670740 1 Processor -- start 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 -- start start 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 0x5608b8075240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 0x5608b805de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01bc0f0 con 0x5608b8063820 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01b99c0 con 0x5608b8069380 2024-03-22T02:21:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0ad8670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01b7810 con 0x5608b8064210 2024-03-22T02:21:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50692/0 (socket says 172.21.15.28:50692) 2024-03-22T02:21:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acf7fe640 1 -- 172.21.15.28:0/1988425714 learned_addr learned my addr 172.21.15.28:0/1988425714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 0x5608b8075240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acffff640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 0x5608b805de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 -- 172.21.15.28:0/1988425714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 msgr2=0x5608b805de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 0x5608b805de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 -- 172.21.15.28:0/1988425714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 msgr2=0x5608b806c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 -- 172.21.15.28:0/1988425714 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608b806c5e0 con 0x5608b8064210 2024-03-22T02:21:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0acffff640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 0x5608b805de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.305+0000 7f0aceffd640 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 0x5608b8075240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ac401d950 tx=0x7f0ac4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d75ce4b349e563c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ab57fa640 1 -- 172.21.15.28:0/1988425714 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4011920 con 0x5608b8064210 2024-03-22T02:21:30.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ab57fa640 1 -- 172.21.15.28:0/1988425714 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac400e720 con 0x5608b8064210 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ab57fa640 1 -- 172.21.15.28:0/1988425714 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac401d610 con 0x5608b8064210 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 -- 172.21.15.28:0/1988425714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 msgr2=0x5608b8075240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 0x5608b8075240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ac401d950 tx=0x7f0ac4000bc0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 -- 172.21.15.28:0/1988425714 shutdown_connections 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8069380 0x5608b805de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8064210 0x5608b8075240 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 --2- 172.21.15.28:0/1988425714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8063820 0x5608b806c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 -- 172.21.15.28:0/1988425714 >> 172.21.15.28:0/1988425714 conn(0x5608b805ed90 msgr2=0x5608b805d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.306+0000 7f0ad8670740 1 -- 172.21.15.28:0/1988425714 shutdown_connections 2024-03-22T02:21:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 -- 172.21.15.28:0/1988425714 wait complete. 2024-03-22T02:21:30.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 -- start start 2024-03-22T02:21:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 Processor -- start 2024-03-22T02:21:30.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8064210 0x5608b80732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8069380 0x5608b8074280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01b7e80 con 0x5608b8069380 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01b92f0 con 0x5608b8064210 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.307+0000 7f0ad8670740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad01b8550 con 0x5608b8063820 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0ad8670740 10 client.0 _ll_register_callbacks cb 0x7f0ac4004510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51616/0 (socket says 172.21.15.28:51616) 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 -- 172.21.15.28:0/4243858160 learned_addr learned my addr 172.21.15.28:0/4243858160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0aceffd640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8064210 0x5608b80732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acffff640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8069380 0x5608b8074280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 -- 172.21.15.28:0/4243858160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8064210 msgr2=0x5608b80732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8064210 0x5608b80732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 -- 172.21.15.28:0/4243858160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8069380 msgr2=0x5608b8074280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8069380 0x5608b8074280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 -- 172.21.15.28:0/4243858160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608b8074870 con 0x5608b8063820 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0acf7fe640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8003fd0 tx=0x7f0ac8032ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=690e595035db7521 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0aaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac80be610 con 0x5608b8063820 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ac400e720 con 0x5608b8063820 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f0ac80ba4f0 con 0x5608b8063820 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0ac80a1050 con 0x5608b8063820 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0ad8670740 10 client.27568 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.308+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5608b80e0590 con 0x5608b8063820 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ac400e720 con 0x5608b8063820 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f0ac803c030 con 0x5608b8063820 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 10 client.27568 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 10 client.27568 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 20 client.27568 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b80e0f60 0x5608b80e3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 4 client.27568 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608b7fd0160 con 0x5608b80e0f60 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0aceffd640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b80e0f60 0x5608b80e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0aceffd640 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b80e0f60 0x5608b80e3380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14d2a7a150ac7032 server_cookie=b87e15da59a3eb18 in_seq=0 out_seq=0 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.309+0000 7f0aaeffd640 10 client.27568 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0aaeffd640 1 -- 172.21.15.28:0/4243858160 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608b7fd0160 con 0x5608b80e0f60 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0aaeffd640 10 client.27568 handle_command_reply: tid=0 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 client.27568 shutdown 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 msgr2=0x5608b8072d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8003fd0 tx=0x7f0ac8032ad0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 shutdown_connections 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608b8069380 0x5608b8074280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5608b80e0f60 0x5608b80e3380 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5608b8064210 0x5608b80732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 --2- 172.21.15.28:0/4243858160 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5608b8063820 0x5608b8072d80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 >> 172.21.15.28:0/4243858160 conn(0x5608b805ed90 msgr2=0x5608b8062880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0aaeffd640 0 client.27568 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 shutdown_connections 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 1 -- 172.21.15.28:0/4243858160 wait complete. 2024-03-22T02:21:30.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad8670740 20 client.27568 trim_cache size 0 max 16384 2024-03-22T02:21:30.323 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:21:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 msgr2=0x7f0ab80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 0x7f0ab80a5460 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4009bd0 tx=0x7f0ac4005f90 comp rx=0 tx=0).stop 2024-03-22T02:21:30.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 msgr2=0x7f0ad01a6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.310+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8032520 tx=0x7f0ac8035cf0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 shutdown_connections 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0ab80a2fa0 0x7f0ab80a5460 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ad010deb0 0x7f0ad01ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ad010b500 0x7f0ad01a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 --2- 172.21.15.28:0/3518329636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ad0108b90 0x7f0ad01a6d90 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 >> 172.21.15.28:0/3518329636 conn(0x7f0ad006e9f0 msgr2=0x7f0ad010b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 shutdown_connections 2024-03-22T02:21:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.311+0000 7f0ad5a8d640 1 -- 172.21.15.28:0/3518329636 wait complete. 2024-03-22T02:21:30.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:21:30.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 743687 11.78 MB/sec execute 252 sec latency 27.574 ms 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 -- 172.21.15.28:0/3998816612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c075bd0 msgr2=0x7f294c074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 --2- 172.21.15.28:0/3998816612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c075bd0 0x7f294c074020 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f293c009900 tx=0x7f293c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 -- 172.21.15.28:0/3998816612 shutdown_connections 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 --2- 172.21.15.28:0/3998816612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 0x7f294c10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 --2- 172.21.15.28:0/3998816612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c0745f0 0x7f294c074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 --2- 172.21.15.28:0/3998816612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c075bd0 0x7f294c074020 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 -- 172.21.15.28:0/3998816612 >> 172.21.15.28:0/3998816612 conn(0x7f294c0fd130 msgr2=0x7f294c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 -- 172.21.15.28:0/3998816612 shutdown_connections 2024-03-22T02:21:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.442+0000 7f2951965640 1 -- 172.21.15.28:0/3998816612 wait complete. 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 Processor -- start 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 -- start start 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 0x7f294c19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 0x7f294c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10fa30 con 0x7f294c075bd0 2024-03-22T02:21:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10f8b0 con 0x7f294c0745f0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2951965640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f294c10fbb0 con 0x7f294c109a60 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50714/0 (socket says 172.21.15.28:50714) 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f294affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 0x7f294c19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 0x7f294c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.443+0000 7f2943fff640 1 -- 172.21.15.28:0/3639616414 learned_addr learned my addr 172.21.15.28:0/3639616414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 -- 172.21.15.28:0/3639616414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 msgr2=0x7f294c1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 0x7f294c1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 -- 172.21.15.28:0/3639616414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 msgr2=0x7f294c19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 0x7f294c19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 -- 172.21.15.28:0/3639616414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c1a1cf0 con 0x7f294c075bd0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f294affd640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 0x7f294c19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2943fff640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f293400e9f0 tx=0x7f293400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2950963640 1 -- 172.21.15.28:0/3639616414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293400cd80 con 0x7f294c075bd0 2024-03-22T02:21:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f294c1a1fe0 con 0x7f294c075bd0 2024-03-22T02:21:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2950963640 1 -- 172.21.15.28:0/3639616414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2934010910 con 0x7f294c075bd0 2024-03-22T02:21:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2950963640 1 -- 172.21.15.28:0/3639616414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340188f0 con 0x7f294c075bd0 2024-03-22T02:21:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.444+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f294c1a24f0 con 0x7f294c075bd0 2024-03-22T02:21:30.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.447+0000 7f2950963640 1 -- 172.21.15.28:0/3639616414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2934010ac0 con 0x7f294c075bd0 2024-03-22T02:21:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:30 smithi028 ceph-mon[31916]: pgmap v596: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f2950963640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 0x7f29380a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 Processor -- start 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f294affd640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 0x7f29380a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 -- start start 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 0x5590225c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 0x5590225c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590225228d0 con 0x5590225c3380 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5590224cae00 con 0x5590225bd820 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f295462e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55902251e880 con 0x5590225be210 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f2950963640 1 -- 172.21.15.28:0/3639616414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f29340cfab0 con 0x7f294c075bd0 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f294affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f2943fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 0x5590225c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f294affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54534/0 (socket says 172.21.15.28:54534) 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.448+0000 7f294affd640 1 -- 172.21.15.28:0/173802837 learned_addr learned my addr 172.21.15.28:0/173802837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f294b7fe640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 0x5590225c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f294affd640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 0x7f29380a5460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f293c02c240 tx=0x7f293c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 -- 172.21.15.28:0/173802837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 msgr2=0x5590225bdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 -- 172.21.15.28:0/173802837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 msgr2=0x5590225c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 0x5590225c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 -- 172.21.15.28:0/173802837 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590222363f0 con 0x5590225be210 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f294affd640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f294b7fe640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 0x5590225c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.449+0000 7f2943fff640 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 0x5590225c2d10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f29340cd1f0 tx=0x7f293409bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3107baf121208a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.450+0000 7f2941ffb640 1 -- 172.21.15.28:0/173802837 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293409a180 con 0x5590225be210 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.450+0000 7f2941ffb640 1 -- 172.21.15.28:0/173802837 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29340984f0 con 0x5590225be210 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.450+0000 7f2941ffb640 1 -- 172.21.15.28:0/173802837 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293409b530 con 0x5590225be210 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- 172.21.15.28:0/173802837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 msgr2=0x5590225c2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 0x5590225c2d10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f29340cd1f0 tx=0x7f293409bd80 comp rx=0 tx=0).stop 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- 172.21.15.28:0/173802837 shutdown_connections 2024-03-22T02:21:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225c3380 0x5590225c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225be210 0x5590225c2d10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 --2- 172.21.15.28:0/173802837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225bd820 0x5590225bdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- 172.21.15.28:0/173802837 >> 172.21.15.28:0/173802837 conn(0x5590225b8d90 msgr2=0x5590225b9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- 172.21.15.28:0/173802837 shutdown_connections 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- 172.21.15.28:0/173802837 wait complete. 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.451+0000 7f295462e740 1 -- start start 2024-03-22T02:21:30.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 Processor -- start 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 0x5590225c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 0x5590225cf350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5590225c6330 con 0x5590225bd820 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5590224de300 con 0x5590225be210 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5590225c64b0 con 0x5590225c3380 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f295462e740 10 client.0 _ll_register_callbacks cb 0x559022519e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f294affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 0x5590225c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 0x5590225cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54542/0 (socket says 172.21.15.28:54542) 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 -- 172.21.15.28:0/201310759 learned_addr learned my addr 172.21.15.28:0/201310759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 -- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 msgr2=0x5590225cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 0x5590225cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 -- 172.21.15.28:0/201310759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 msgr2=0x5590225c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 0x5590225c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f2943fff640 1 -- 172.21.15.28:0/201310759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590223007d0 con 0x5590225be210 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f294affd640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 0x5590225c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.452+0000 7f294b7fe640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 0x5590225cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f2943fff640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29340982e0 tx=0x7f2934009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c42b1dee44caca81 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340b4070 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29340984f0 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2934010430 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2934010630 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 10 client.27581 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5590225d26c0 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29340984f0 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f2934018bb0 con 0x5590225be210 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 10 client.27581 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 10 client.27581 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 20 client.27581 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5590225d3a10 0x5590225d5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 4 client.27581 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:30.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.453+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55902252fe10 con 0x5590225d3a10 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.454+0000 7f294affd640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5590225d3a10 0x5590225d5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.454+0000 7f294affd640 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5590225d3a10 0x5590225d5e00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e2474e8765cf8243 server_cookie=6f56f8e7b08423de in_seq=0 out_seq=0 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.454+0000 7f292affd640 10 client.27581 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f292affd640 1 -- 172.21.15.28:0/201310759 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55902252fe10 con 0x5590225d3a10 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f292affd640 10 client.27581 handle_command_reply: tid=0 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 client.27581 shutdown 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 msgr2=0x5590225c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29340982e0 tx=0x7f2934009700 comp rx=0 tx=0).stop 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 shutdown_connections 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5590225d3a10 0x5590225d5e00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225c3380 0x5590225cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f292affd640 0 client.27581 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225be210 0x5590225c8310 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 --2- 172.21.15.28:0/201310759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225bd820 0x5590225c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 >> 172.21.15.28:0/201310759 conn(0x5590225b8d90 msgr2=0x5590225b7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 shutdown_connections 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 1 -- 172.21.15.28:0/201310759 wait complete. 2024-03-22T02:21:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.455+0000 7f295462e740 20 client.27581 trim_cache size 0 max 16384 2024-03-22T02:21:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.471+0000 7f295462e740 1 Processor -- start 2024-03-22T02:21:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 -- start start 2024-03-22T02:21:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225b8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1af580 con 0x5590225be210 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1ace50 con 0x5590225c3380 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f295462e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1aaca0 con 0x5590225bd820 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54558/0 (socket says 172.21.15.28:54558) 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51668/0 (socket says 172.21.15.28:51668) 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50738/0 (socket says 172.21.15.28:50738) 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 -- 172.21.15.28:0/1248636878 learned_addr learned my addr 172.21.15.28:0/1248636878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 -- 172.21.15.28:0/1248636878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 msgr2=0x5590225b8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225b8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 -- 172.21.15.28:0/1248636878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 msgr2=0x5590225cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 -- 172.21.15.28:0/1248636878 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590225be5f0 con 0x5590225c3380 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.472+0000 7f294b7fe640 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2944013880 tx=0x7f2944013d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83de8338fb7c37e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f29417fa640 1 -- 172.21.15.28:0/1248636878 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2944004b20 con 0x5590225c3380 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f29417fa640 1 -- 172.21.15.28:0/1248636878 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2944004cc0 con 0x5590225c3380 2024-03-22T02:21:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f29417fa640 1 -- 172.21.15.28:0/1248636878 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29440042b0 con 0x5590225c3380 2024-03-22T02:21:30.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 -- 172.21.15.28:0/1248636878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 msgr2=0x5590225bc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2944013880 tx=0x7f2944013d40 comp rx=0 tx=0).stop 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 -- 172.21.15.28:0/1248636878 shutdown_connections 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225bc700 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 --2- 172.21.15.28:0/1248636878 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225b8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.473+0000 7f295462e740 1 -- 172.21.15.28:0/1248636878 >> 172.21.15.28:0/1248636878 conn(0x5590225b8d90 msgr2=0x5590225bbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- 172.21.15.28:0/1248636878 shutdown_connections 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- 172.21.15.28:0/1248636878 wait complete. 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- start start 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 Processor -- start 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1ab310 con 0x5590225be210 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1ac780 con 0x5590225c3380 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f294c1ab9e0 con 0x5590225bd820 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f295462e740 10 client.0 _ll_register_callbacks cb 0x7f2944009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f2943fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50746/0 (socket says 172.21.15.28:50746) 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f2943fff640 1 -- 172.21.15.28:0/2777475016 learned_addr learned my addr 172.21.15.28:0/2777475016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f294b7fe640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.474+0000 7f294affd640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 -- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 msgr2=0x5590225cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 -- 172.21.15.28:0/2777475016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 msgr2=0x5590225cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 -- 172.21.15.28:0/2777475016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590225ce030 con 0x5590225be210 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f294b7fe640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f2943fff640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f293409be10 tx=0x7f2934009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b31dffb8220b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29340b4070 con 0x5590225be210 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f2934097de0 con 0x5590225be210 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2944004cc0 con 0x5590225be210 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2934098120 con 0x5590225be210 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f295462e740 10 client.17763 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5590225cf330 con 0x5590225be210 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2944004cc0 con 0x5590225be210 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.475+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f2934005040 con 0x5590225be210 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 10 client.17763 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 10 client.17763 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 20 client.17763 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559022639f80 0x55902263c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 4 client.17763 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55902252a160 con 0x559022639f80 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f294affd640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559022639f80 0x55902263c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f292affd640 10 client.17763 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f294affd640 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559022639f80 0x55902263c370 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7496eb78ee20bbd0 server_cookie=135d35bacb708e92 in_seq=0 out_seq=0 2024-03-22T02:21:30.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f292affd640 1 -- 172.21.15.28:0/2777475016 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55902252a160 con 0x559022639f80 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f292affd640 10 client.17763 handle_command_reply: tid=0 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 client.17763 shutdown 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 msgr2=0x5590225cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f293409be10 tx=0x7f2934009770 comp rx=0 tx=0).stop 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 shutdown_connections 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x559022639f80 0x55902263c370 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5590225c3380 0x5590225cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5590225be210 0x5590225cca30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.476+0000 7f295462e740 1 --2- 172.21.15.28:0/2777475016 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5590225bd820 0x5590225cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 >> 172.21.15.28:0/2777475016 conn(0x5590225b8d90 msgr2=0x5590225bc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f292affd640 0 client.17763 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:21:30.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 shutdown_connections 2024-03-22T02:21:30.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f295462e740 1 -- 172.21.15.28:0/2777475016 wait complete. 2024-03-22T02:21:30.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f295462e740 20 client.17763 trim_cache size 0 max 16384 2024-03-22T02:21:30.485 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 msgr2=0x7f29380a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 0x7f29380a5460 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f293c02c240 tx=0x7f293c02df90 comp rx=0 tx=0).stop 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 msgr2=0x7f294c19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f293400e9f0 tx=0x7f293400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 shutdown_connections 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29380a2fa0 0x7f29380a5460 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f294c109a60 0x7f294c1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f294c075bd0 0x7f294c19a610 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 --2- 172.21.15.28:0/3639616414 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f294c0745f0 0x7f294c19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 >> 172.21.15.28:0/3639616414 conn(0x7f294c0fd130 msgr2=0x7f294c105960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 shutdown_connections 2024-03-22T02:21:30.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.477+0000 7f2951965640 1 -- 172.21.15.28:0/3639616414 wait complete. 2024-03-22T02:21:30.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:21:30.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- 172.21.15.28:0/2474658543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100dc0 msgr2=0x7f55e8101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- 172.21.15.28:0/2474658543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100dc0 0x7f55e8101220 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009900 tx=0x7f55d802bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- 172.21.15.28:0/2474658543 shutdown_connections 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- 172.21.15.28:0/2474658543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8101760 0x7f55e8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- 172.21.15.28:0/2474658543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100dc0 0x7f55e8101220 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- 172.21.15.28:0/2474658543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8100460 0x7f55e8100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- 172.21.15.28:0/2474658543 >> 172.21.15.28:0/2474658543 conn(0x7f55e80fc020 msgr2=0x7f55e80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- 172.21.15.28:0/2474658543 shutdown_connections 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- 172.21.15.28:0/2474658543 wait complete. 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 Processor -- start 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- start start 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 0x7f55e819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 0x7f55e81a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55e810f9d0 con 0x7f55e8100460 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55e810f850 con 0x7f55e8101760 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.600+0000 7f55f0302640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55e810fb50 con 0x7f55e8100dc0 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50762/0 (socket says 172.21.15.28:50762) 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 -- 172.21.15.28:0/847285260 learned_addr learned my addr 172.21.15.28:0/847285260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee89f640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 0x7f55e81a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ed89d640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 0x7f55e819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 -- 172.21.15.28:0/847285260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 msgr2=0x7f55e819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 0x7f55e819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 -- 172.21.15.28:0/847285260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 msgr2=0x7f55e81a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 0x7f55e81a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 -- 172.21.15.28:0/847285260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e81a1cf0 con 0x7f55e8100460 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee89f640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 0x7f55e81a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ed89d640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 0x7f55e819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55ee09e640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f55e400e9f0 tx=0x7f55e400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55df7fe640 1 -- 172.21.15.28:0/847285260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e400cd80 con 0x7f55e8100460 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55df7fe640 1 -- 172.21.15.28:0/847285260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e4010910 con 0x7f55e8100460 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e81a1fe0 con 0x7f55e8100460 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55df7fe640 1 -- 172.21.15.28:0/847285260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4018990 con 0x7f55e8100460 2024-03-22T02:21:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.601+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e81a2440 con 0x7f55e8100460 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.604+0000 7f55df7fe640 1 -- 172.21.15.28:0/847285260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f55e4010ac0 con 0x7f55e8100460 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.604+0000 7f55f2fca740 1 Processor -- start 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.604+0000 7f55f2fca740 1 -- start start 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.604+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.604+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568c210 0x556ce5690d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55f2fca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ce55f08d0 con 0x556ce568b820 2024-03-22T02:21:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55f2fca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ce5598e00 con 0x556ce568c210 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55f2fca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ce55ec880 con 0x556ce5691380 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55df7fe640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 0x7f55d40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568c210 0x556ce5690d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ed89d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ed89d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51710/0 (socket says 172.21.15.28:51710) 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ed89d640 1 -- 172.21.15.28:0/4111579714 learned_addr learned my addr 172.21.15.28:0/4111579714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50778/0 (socket says 172.21.15.28:50778) 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 0x7f55d40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55df7fe640 1 -- 172.21.15.28:0/847285260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f55e40cfc50 con 0x7f55e8100460 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 -- 172.21.15.28:0/4111579714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 msgr2=0x556ce5693810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 -- 172.21.15.28:0/4111579714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568c210 msgr2=0x556ce5690d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568c210 0x556ce5690d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 -- 172.21.15.28:0/4111579714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce53043f0 con 0x556ce568b820 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ed89d640 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f55e0014f10 tx=0x7f55e00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a772bf3a1f0ed55 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.605+0000 7f55ee89f640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 0x7f55d40a53e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009370 tx=0x7f55e0016000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55dcbfe640 1 -- 172.21.15.28:0/4111579714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e0024070 con 0x556ce568b820 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55dcbfe640 1 -- 172.21.15.28:0/4111579714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e000bbf0 con 0x556ce568b820 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55dcbfe640 1 -- 172.21.15.28:0/4111579714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e0005760 con 0x556ce568b820 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 -- 172.21.15.28:0/4111579714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 msgr2=0x556ce568bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f55e0014f10 tx=0x7f55e00117b0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 -- 172.21.15.28:0/4111579714 shutdown_connections 2024-03-22T02:21:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce5693810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568c210 0x556ce5690d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 --2- 172.21.15.28:0/4111579714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568bc40 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 -- 172.21.15.28:0/4111579714 >> 172.21.15.28:0/4111579714 conn(0x556ce5686d90 msgr2=0x556ce5687190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 -- 172.21.15.28:0/4111579714 shutdown_connections 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.606+0000 7f55f2fca740 1 -- 172.21.15.28:0/4111579714 wait complete. 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 -- start start 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 Processor -- start 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce5695dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce569d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556ce56942d0 con 0x556ce568c210 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556ce55ac300 con 0x556ce568b820 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556ce5694450 con 0x556ce5691380 2024-03-22T02:21:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55f2fca740 10 client.0 _ll_register_callbacks cb 0x556ce55e7e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55ed89d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce5695dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55ed89d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50792/0 (socket says 172.21.15.28:50792) 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55ed89d640 1 -- 172.21.15.28:0/415436588 learned_addr learned my addr 172.21.15.28:0/415436588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.607+0000 7f55ee89f640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce569d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 -- 172.21.15.28:0/415436588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 msgr2=0x556ce569d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce569d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 -- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 msgr2=0x556ce5695dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce5695dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 -- 172.21.15.28:0/415436588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce53ce7d0 con 0x556ce568c210 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ee09e640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce5695dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55ed89d640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f55d802cfd0 tx=0x7f55d8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de919dfada0e426 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d8005d50 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f55d8005230 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e000bbf0 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f55d804d050 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55f2fca740 10 client.17775 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556ce56a0700 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e000bbf0 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.608+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f55d8003a30 con 0x556ce568c210 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 10 client.17775 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 10 client.17775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 20 client.17775 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce56a1dd0 0x556ce56a4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 4 client.17775 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ce55fde10 con 0x556ce56a1dd0 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55ee09e640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce56a1dd0 0x556ce56a4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55d1ffb640 10 client.17775 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.609+0000 7f55ee09e640 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce56a1dd0 0x556ce56a4250 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1500a5896720055 server_cookie=f9b6f879d00e138c in_seq=0 out_seq=0 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55d1ffb640 1 -- 172.21.15.28:0/415436588 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ce55fde10 con 0x556ce56a1dd0 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55d1ffb640 10 client.17775 handle_command_reply: tid=0 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 client.17775 shutdown 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 msgr2=0x556ce5696300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f55d802cfd0 tx=0x7f55d8004270 comp rx=0 tx=0).stop 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 shutdown_connections 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce56a1dd0 0x556ce56a4250 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce5691380 0x556ce569d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568c210 0x556ce5696300 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55d1ffb640 0 client.17775 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 --2- 172.21.15.28:0/415436588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce5695dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 >> 172.21.15.28:0/415436588 conn(0x556ce5686d90 msgr2=0x556ce5685fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 shutdown_connections 2024-03-22T02:21:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 1 -- 172.21.15.28:0/415436588 wait complete. 2024-03-22T02:21:30.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.610+0000 7f55f2fca740 20 client.17775 trim_cache size 0 max 16384 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 Processor -- start 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 -- start start 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce5691380 0x556ce56a8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81af3e0 con 0x556ce568b820 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81accb0 con 0x556ce5691380 2024-03-22T02:21:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55f2fca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81aab00 con 0x556ce568c210 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50804/0 (socket says 172.21.15.28:50804) 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 -- 172.21.15.28:0/2669672340 learned_addr learned my addr 172.21.15.28:0/2669672340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ed89d640 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee89f640 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce5691380 0x556ce56a8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 -- 172.21.15.28:0/2669672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 msgr2=0x556ce569d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 -- 172.21.15.28:0/2669672340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce5691380 msgr2=0x556ce56a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce5691380 0x556ce56a8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.625+0000 7f55ee09e640 1 -- 172.21.15.28:0/2669672340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce5690ad0 con 0x556ce568b820 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f55e40cd6b0 tx=0x7f55e40bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66e5b88ecbd9f67b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55d3fff640 1 -- 172.21.15.28:0/2669672340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e40a00c0 con 0x556ce568b820 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55d3fff640 1 -- 172.21.15.28:0/2669672340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e40a4980 con 0x556ce568b820 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55d3fff640 1 -- 172.21.15.28:0/2669672340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e4018b30 con 0x556ce568b820 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55f2fca740 1 -- 172.21.15.28:0/2669672340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 msgr2=0x556ce568a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.626+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f55e40cd6b0 tx=0x7f55e40bc040 comp rx=0 tx=0).stop 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- 172.21.15.28:0/2669672340 shutdown_connections 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce5691380 0x556ce56a8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2669672340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce568b820 0x556ce568a700 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- 172.21.15.28:0/2669672340 >> 172.21.15.28:0/2669672340 conn(0x556ce5686d90 msgr2=0x556ce5689ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- 172.21.15.28:0/2669672340 shutdown_connections 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- 172.21.15.28:0/2669672340 wait complete. 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- start start 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 Processor -- start 2024-03-22T02:21:30.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce569aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81ab170 con 0x556ce5691380 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81ac5e0 con 0x556ce568b820 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f55e81ab840 con 0x556ce568c210 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55f2fca740 10 client.0 _ll_register_callbacks cb 0x7f55d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee09e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce569aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50810/0 (socket says 172.21.15.28:50810) 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 -- 172.21.15.28:0/2859655879 learned_addr learned my addr 172.21.15.28:0/2859655879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ed89d640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 -- 172.21.15.28:0/2859655879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 msgr2=0x556ce569b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 -- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 msgr2=0x556ce569aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.627+0000 7f55ee89f640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce569aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55ee89f640 1 -- 172.21.15.28:0/2859655879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce569c900 con 0x556ce5691380 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce569aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55ee89f640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f55e0005fd0 tx=0x7f55e0009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fce0798bf72f682 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55e0024040 con 0x556ce5691380 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f55e000bdf0 con 0x556ce5691380 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55e40a4980 con 0x556ce5691380 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f55e003d050 con 0x556ce5691380 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55f2fca740 10 client.17781 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556ce5708500 con 0x556ce5691380 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.628+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55e40a4980 con 0x556ce5691380 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f55e0020420 con 0x556ce5691380 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 10 client.17781 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 10 client.17781 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 20 client.17781 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce5708cf0 0x556ce570b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 4 client.17781 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ce55f8160 con 0x556ce5708cf0 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce5708cf0 0x556ce570b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55ee09e640 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce5708cf0 0x556ce570b110 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8916e4be57b1069 server_cookie=8ae25d3c7b2742fe in_seq=0 out_seq=0 2024-03-22T02:21:30.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.629+0000 7f55d1ffb640 10 client.17781 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:30.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55d1ffb640 1 -- 172.21.15.28:0/2859655879 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ce55f8160 con 0x556ce5708cf0 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55d1ffb640 10 client.17781 handle_command_reply: tid=0 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 client.17781 shutdown 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 msgr2=0x556ce569c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f55e0005fd0 tx=0x7f55e0009760 comp rx=0 tx=0).stop 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 shutdown_connections 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556ce5708cf0 0x556ce570b110 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556ce5691380 0x556ce569c310 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556ce568c210 0x556ce569b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55d1ffb640 0 client.17781 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 --2- 172.21.15.28:0/2859655879 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556ce568b820 0x556ce569aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.630+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 >> 172.21.15.28:0/2859655879 conn(0x556ce5686d90 msgr2=0x556ce568a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 shutdown_connections 2024-03-22T02:21:30.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f2fca740 1 -- 172.21.15.28:0/2859655879 wait complete. 2024-03-22T02:21:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f2fca740 20 client.17781 trim_cache size 0 max 16384 2024-03-22T02:21:30.638 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 msgr2=0x7f55d40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 0x7f55d40a53e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009370 tx=0x7f55e0016000 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 msgr2=0x7f55e819a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f55e400e9f0 tx=0x7f55e400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 shutdown_connections 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f55d40a2f20 0x7f55d40a53e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f55e8101760 0x7f55e81a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55e8100dc0 0x7f55e819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 --2- 172.21.15.28:0/847285260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55e8100460 0x7f55e819a0d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 >> 172.21.15.28:0/847285260 conn(0x7f55e80fc020 msgr2=0x7f55e80fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:30.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 shutdown_connections 2024-03-22T02:21:30.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:30.631+0000 7f55f0302640 1 -- 172.21.15.28:0/847285260 wait complete. 2024-03-22T02:21:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:30 smithi120 ceph-mon[36803]: pgmap v596: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-22T02:21:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:30 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4265920794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:21:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:30 smithi083 ceph-mon[36340]: pgmap v596: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.2 MiB/s wr, 419 op/s 2024-03-22T02:21:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:30 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4265920794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:21:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:30 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4265920794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:21:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- 172.21.15.28:0/3139053065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c107890 msgr2=0x7f7b1c109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3139053065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c107890 0x7f7b1c109ce0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009ca0 tx=0x7f7b1402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- 172.21.15.28:0/3139053065 shutdown_connections 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3139053065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c107890 0x7f7b1c109ce0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3139053065 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c104f60 0x7f7b1c107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3139053065 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c102630 0x7f7b1c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- 172.21.15.28:0/3139053065 >> 172.21.15.28:0/3139053065 conn(0x7f7b1c0fc020 msgr2=0x7f7b1c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- 172.21.15.28:0/3139053065 shutdown_connections 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- 172.21.15.28:0/3139053065 wait complete. 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 Processor -- start 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.100+0000 7f7b23af4640 1 -- start start 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c104f60 0x7f7b1c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c107890 0x7f7b1c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c10d850 con 0x7f7b1c102630 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c10d6d0 con 0x7f7b1c107890 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c10d9d0 con 0x7f7b1c104f60 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50840/0 (socket says 172.21.15.28:50840) 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 -- 172.21.15.28:0/3304427363 learned_addr learned my addr 172.21.15.28:0/3304427363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b2108f640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c104f60 0x7f7b1c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b22091640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c107890 0x7f7b1c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 -- 172.21.15.28:0/3304427363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c104f60 msgr2=0x7f7b1c198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c104f60 0x7f7b1c198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 -- 172.21.15.28:0/3304427363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c107890 msgr2=0x7f7b1c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c107890 0x7f7b1c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 -- 172.21.15.28:0/3304427363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1c19fa50 con 0x7f7b1c102630 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b21890640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800e9f0 tx=0x7f7b1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b12ffd640 1 -- 172.21.15.28:0/3304427363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1800cd80 con 0x7f7b1c102630 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b12ffd640 1 -- 172.21.15.28:0/3304427363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b18005500 con 0x7f7b1c102630 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1c19fd40 con 0x7f7b1c102630 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b12ffd640 1 -- 172.21.15.28:0/3304427363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18010690 con 0x7f7b1c102630 2024-03-22T02:21:31.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.101+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1c1a0180 con 0x7f7b1c102630 2024-03-22T02:21:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b12ffd640 1 -- 172.21.15.28:0/3304427363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7b180046c0 con 0x7f7b1c102630 2024-03-22T02:21:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 Processor -- start 2024-03-22T02:21:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 -- start start 2024-03-22T02:21:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 0x55c1b069dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1b05f9040 con 0x55c1b06988f0 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c1b05a19e0 con 0x55c1b06992e0 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.104+0000 7f7b267ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1b05f4ff0 con 0x55c1b069e450 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b21890640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 0x55c1b069dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50848/0 (socket says 172.21.15.28:50848) 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b21890640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 0x55c1b069dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54644/0 (socket says 172.21.15.28:54644) 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 -- 172.21.15.28:0/2930004946 learned_addr learned my addr 172.21.15.28:0/2930004946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51746/0 (socket says 172.21.15.28:51746) 2024-03-22T02:21:31.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b12ffd640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 0x7f7b080a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 -- 172.21.15.28:0/2930004946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 msgr2=0x55c1b069dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 0x55c1b069dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 -- 172.21.15.28:0/2930004946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 msgr2=0x55c1b0698d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 -- 172.21.15.28:0/2930004946 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b030d210 con 0x55c1b069e450 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b12ffd640 1 -- 172.21.15.28:0/3304427363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7b180cfbc0 con 0x7f7b1c102630 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 0x7f7b080a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b2108f640 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c005d00 tx=0x7f7b0c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfbc81ade2dc5829 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b07fff640 1 -- 172.21.15.28:0/2930004946 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c01c480 con 0x55c1b069e450 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b07fff640 1 -- 172.21.15.28:0/2930004946 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b0c01cb00 con 0x55c1b069e450 2024-03-22T02:21:31.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b22091640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 0x7f7b080a5480 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7b14006010 tx=0x7f7b1402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.105+0000 7f7b07fff640 1 -- 172.21.15.28:0/2930004946 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c011c10 con 0x55c1b069e450 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 -- 172.21.15.28:0/2930004946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 msgr2=0x55c1b06a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c005d00 tx=0x7f7b0c0107a0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 -- 172.21.15.28:0/2930004946 shutdown_connections 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a08e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06992e0 0x55c1b069dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 --2- 172.21.15.28:0/2930004946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06988f0 0x55c1b0698d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 -- 172.21.15.28:0/2930004946 >> 172.21.15.28:0/2930004946 conn(0x55c1b068fa50 msgr2=0x55c1b068fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 -- 172.21.15.28:0/2930004946 shutdown_connections 2024-03-22T02:21:31.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.106+0000 7f7b267ba740 1 -- 172.21.15.28:0/2930004946 wait complete. 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 -- start start 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 Processor -- start 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b06a2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c1b06a1400 con 0x55c1b06992e0 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c1b05b4db0 con 0x55c1b06988f0 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c1b06a1580 con 0x55c1b069e450 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b267ba740 10 client.0 _ll_register_callbacks cb 0x55c1b030ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b21890640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b06a2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b22091640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b22091640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51756/0 (socket says 172.21.15.28:51756) 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b22091640 1 -- 172.21.15.28:0/4270389281 learned_addr learned my addr 172.21.15.28:0/4270389281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50860/0 (socket says 172.21.15.28:50860) 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.107+0000 7f7b21890640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b06a2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54650/0 (socket says 172.21.15.28:54650) 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 -- 172.21.15.28:0/4270389281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 msgr2=0x55c1b06a9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 -- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 msgr2=0x55c1b06a2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b06a2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 -- 172.21.15.28:0/4270389281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b03d9620 con 0x55c1b06992e0 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b2108f640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000c00 tx=0x7f7b0c00bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdde67f831810748 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c01c890 con 0x55c1b06992e0 2024-03-22T02:21:31.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7b0c011910 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b0c01cb00 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b0c033050 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b267ba740 10 client.17793 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.108+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c1b06ad180 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b0c01cb00 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f7b0c025b00 con 0x55c1b06992e0 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 10 client.17793 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 10 client.17793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 20 client.17793 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b06ae580 0x55c1b06b0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 4 client.17793 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:31.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c1b0606580 con 0x55c1b06ae580 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b21890640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b06ae580 0x55c1b06b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b057fa640 10 client.17793 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.109+0000 7f7b21890640 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b06ae580 0x55c1b06b0970 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ddb95cabf21f706 server_cookie=62391c5a9e062ea8 in_seq=0 out_seq=0 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b057fa640 1 -- 172.21.15.28:0/4270389281 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c1b0606580 con 0x55c1b06ae580 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b057fa640 10 client.17793 handle_command_reply: tid=0 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 client.17793 shutdown 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 msgr2=0x55c1b06a2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000c00 tx=0x7f7b0c00bf50 comp rx=0 tx=0).stop 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 shutdown_connections 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b06ae580 0x55c1b06b0970 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06a9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b057fa640 0 client.17793 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a2dd0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 --2- 172.21.15.28:0/4270389281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b06a2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 >> 172.21.15.28:0/4270389281 conn(0x55c1b068fa50 msgr2=0x55c1b068ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 shutdown_connections 2024-03-22T02:21:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 1 -- 172.21.15.28:0/4270389281 wait complete. 2024-03-22T02:21:31.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.111+0000 7f7b267ba740 20 client.17793 trim_cache size 0 max 16384 2024-03-22T02:21:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 Processor -- start 2024-03-22T02:21:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 -- start start 2024-03-22T02:21:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b0692cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06931f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1ab020 con 0x55c1b06992e0 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a88f0 con 0x55c1b06988f0 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b267ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a6740 con 0x55c1b069e450 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b22091640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06931f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50870/0 (socket says 172.21.15.28:50870) 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b2108f640 1 -- 172.21.15.28:0/550964291 learned_addr learned my addr 172.21.15.28:0/550964291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.128+0000 7f7b21890640 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b0692cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 -- 172.21.15.28:0/550964291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 msgr2=0x55c1b0692cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b0692cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 -- 172.21.15.28:0/550964291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 msgr2=0x55c1b06a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 -- 172.21.15.28:0/550964291 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c1b06907c0 con 0x55c1b069e450 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b2108f640 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b22091640 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06931f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b1402c630 tx=0x7f7b14041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c412b02a3d966ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b077fe640 1 -- 172.21.15.28:0/550964291 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14039040 con 0x55c1b069e450 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b077fe640 1 -- 172.21.15.28:0/550964291 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b14053070 con 0x55c1b069e450 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.129+0000 7f7b077fe640 1 -- 172.21.15.28:0/550964291 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b14041470 con 0x55c1b069e450 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- 172.21.15.28:0/550964291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 msgr2=0x55c1b06931f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06931f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b1402c630 tx=0x7f7b14041000 comp rx=0 tx=0).stop 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- 172.21.15.28:0/550964291 shutdown_connections 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b069e450 0x55c1b06931f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 --2- 172.21.15.28:0/550964291 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b06988f0 0x55c1b0692cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- 172.21.15.28:0/550964291 >> 172.21.15.28:0/550964291 conn(0x55c1b068fa50 msgr2=0x55c1b06a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- 172.21.15.28:0/550964291 shutdown_connections 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- 172.21.15.28:0/550964291 wait complete. 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 -- start start 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.130+0000 7f7b267ba740 1 Processor -- start 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b06988f0 0x55c1b06a7130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b069e450 0x55c1b06a8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a6db0 con 0x55c1b06992e0 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a8220 con 0x55c1b069e450 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1c1a7480 con 0x55c1b06988f0 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b267ba740 10 client.0 _ll_register_callbacks cb 0x7f7b18005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50874/0 (socket says 172.21.15.28:50874) 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b21890640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b06988f0 0x55c1b06a7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 -- 172.21.15.28:0/3642255221 learned_addr learned my addr 172.21.15.28:0/3642255221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b22091640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b069e450 0x55c1b06a8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 -- 172.21.15.28:0/3642255221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b06988f0 msgr2=0x55c1b06a7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b06988f0 0x55c1b06a7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 -- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b069e450 msgr2=0x55c1b06a8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b069e450 0x55c1b06a8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 -- 172.21.15.28:0/3642255221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c1b06a8c00 con 0x55c1b06992e0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b2108f640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c025fd0 tx=0x7f7b0c00bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef43ac5ccada6d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.131+0000 7f7b057fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c004220 con 0x55c1b06992e0 2024-03-22T02:21:31.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b14053070 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f7b0c0043c0 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7b0c047050 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 10 client.17799 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55c1b06a9f40 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b14053070 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f7b0c02e8f0 con 0x55c1b06992e0 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 10 client.17799 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 10 client.17799 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 20 client.17799 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b0714df0 0x55c1b0717210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 4 client.17799 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:21:31.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c1b06008d0 con 0x55c1b0714df0 2024-03-22T02:21:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.132+0000 7f7b21890640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b0714df0 0x55c1b0717210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.133+0000 7f7b057fa640 10 client.17799 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.139 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.133+0000 7f7b21890640 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b0714df0 0x55c1b0717210 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fdb3b3504acf74d server_cookie=4a685ff76b48b6f7 in_seq=0 out_seq=0 2024-03-22T02:21:31.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b057fa640 1 -- 172.21.15.28:0/3642255221 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c1b06008d0 con 0x55c1b0714df0 2024-03-22T02:21:31.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b057fa640 10 client.17799 handle_command_reply: tid=0 2024-03-22T02:21:31.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 client.17799 shutdown 2024-03-22T02:21:31.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 msgr2=0x55c1b06a7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c025fd0 tx=0x7f7b0c00bae0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 shutdown_connections 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c1b0714df0 0x55c1b0717210 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c1b069e450 0x55c1b06a8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b057fa640 0 client.17799 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c1b06992e0 0x55c1b06a7670 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 --2- 172.21.15.28:0/3642255221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c1b06988f0 0x55c1b06a7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 >> 172.21.15.28:0/3642255221 conn(0x55c1b068fa50 msgr2=0x55c1b069d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 shutdown_connections 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 1 -- 172.21.15.28:0/3642255221 wait complete. 2024-03-22T02:21:31.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.408+0000 7f7b267ba740 20 client.17799 trim_cache size 0 max 16384 2024-03-22T02:21:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:21:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:21:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:21:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 msgr2=0x7f7b080a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 0x7f7b080a5480 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7b14006010 tx=0x7f7b1402e040 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 msgr2=0x7f7b1c197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f7b1800e9f0 tx=0x7f7b1800edf0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 shutdown_connections 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7b080a2fc0 0x7f7b080a5480 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7b1c107890 0x7f7b1c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7b1c104f60 0x7f7b1c198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 --2- 172.21.15.28:0/3304427363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b1c102630 0x7f7b1c197e30 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 >> 172.21.15.28:0/3304427363 conn(0x7f7b1c0fc020 msgr2=0x7f7b1c102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 shutdown_connections 2024-03-22T02:21:31.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:31.409+0000 7f7b23af4640 1 -- 172.21.15.28:0/3304427363 wait complete. 2024-03-22T02:21:31.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 745914 11.79 MB/sec execute 253 sec latency 141.722 ms 2024-03-22T02:21:31.644 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:21:31.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:21:31.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:21:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 -- 172.21.15.28:0/1223732770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658102610 msgr2=0x7fb658104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/1223732770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658102610 0x7fb658104a00 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fb650009900 tx=0x7fb65002bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 -- 172.21.15.28:0/1223732770 shutdown_connections 2024-03-22T02:21:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/1223732770 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658107870 0x7fb658109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/1223732770 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658104f40 0x7fb658107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/1223732770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658102610 0x7fb658104a00 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 -- 172.21.15.28:0/1223732770 >> 172.21.15.28:0/1223732770 conn(0x7fb6580fc040 msgr2=0x7fb6580fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 -- 172.21.15.28:0/1223732770 shutdown_connections 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.753+0000 7fb65fcaf640 1 -- 172.21.15.28:0/1223732770 wait complete. 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 Processor -- start 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 -- start start 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658104f40 0x7fb6581005c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 0x7fb658100b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb65810d840 con 0x7fb658107870 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb65810d6c0 con 0x7fb658104f40 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65fcaf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb65810d9c0 con 0x7fb658102610 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 0x7fb658100b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51264/0 (socket says 172.21.15.28:51264) 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65da4b640 1 -- 172.21.15.28:0/811740200 learned_addr learned my addr 172.21.15.28:0/811740200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65d24a640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658104f40 0x7fb6581005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 -- 172.21.15.28:0/811740200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 msgr2=0x7fb658101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 -- 172.21.15.28:0/811740200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658104f40 msgr2=0x7fb6581005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658104f40 0x7fb6581005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 -- 172.21.15.28:0/811740200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb658101390 con 0x7fb658107870 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65da4b640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.754+0000 7fb65e24c640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 0x7fb658100b00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb64800da10 tx=0x7fb64800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.755+0000 7fb656ffd640 1 -- 172.21.15.28:0/811740200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64800bd70 con 0x7fb658107870 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.755+0000 7fb656ffd640 1 -- 172.21.15.28:0/811740200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb648003e50 con 0x7fb658107870 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.755+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6581a1fd0 con 0x7fb658107870 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.755+0000 7fb656ffd640 1 -- 172.21.15.28:0/811740200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6480106a0 con 0x7fb658107870 2024-03-22T02:21:31.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.755+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6581a2440 con 0x7fb658107870 2024-03-22T02:21:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.757+0000 7fb656ffd640 1 -- 172.21.15.28:0/811740200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb648003970 con 0x7fb658107870 2024-03-22T02:21:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.757+0000 7fb662972740 1 Processor -- start 2024-03-22T02:21:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 -- start start 2024-03-22T02:21:31.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b43fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b49720 0x558364b4bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558364aa8570 con 0x558364b43bc0 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558364a50e10 con 0x558364b445b0 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb662972740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558364aa4520 con 0x558364b49720 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb656ffd640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 0x7fb6400a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b43fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b43fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59868/0 (socket says 172.21.15.28:59868) 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41128/0 (socket says 172.21.15.28:41128) 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65d24a640 1 -- 172.21.15.28:0/3124032439 learned_addr learned my addr 172.21.15.28:0/3124032439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65e24c640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 0x7fb6400a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65d24a640 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b49720 0x558364b4bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb656ffd640 1 -- 172.21.15.28:0/811740200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fb6480cfbc0 con 0x7fb658107870 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 -- 172.21.15.28:0/3124032439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b49720 msgr2=0x558364b4bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b49720 0x558364b4bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 -- 172.21.15.28:0/3124032439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 msgr2=0x558364b43fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.758+0000 7fb65da4b640 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b43fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb65da4b640 1 -- 172.21.15.28:0/3124032439 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583648847e0 con 0x558364b445b0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb65e24c640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 0x7fb6400a53e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb648008f90 tx=0x7fb64806ec10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb65da4b640 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb65002c7b0 tx=0x7fb65002ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32503b51998c90f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb63ffff640 1 -- 172.21.15.28:0/3124032439 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650042070 con 0x558364b445b0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb63ffff640 1 -- 172.21.15.28:0/3124032439 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb650004ba0 con 0x558364b445b0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.759+0000 7fb63ffff640 1 -- 172.21.15.28:0/3124032439 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650005310 con 0x558364b445b0 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- 172.21.15.28:0/3124032439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 msgr2=0x558364b490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb65002c7b0 tx=0x7fb65002ce00 comp rx=0 tx=0).stop 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- 172.21.15.28:0/3124032439 shutdown_connections 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b49720 0x558364b4bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b445b0 0x558364b490b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 --2- 172.21.15.28:0/3124032439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b43fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- 172.21.15.28:0/3124032439 >> 172.21.15.28:0/3124032439 conn(0x558364b3f130 msgr2=0x558364b3f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- 172.21.15.28:0/3124032439 shutdown_connections 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- 172.21.15.28:0/3124032439 wait complete. 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 -- start start 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.760+0000 7fb662972740 1 Processor -- start 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b4e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558364b4c670 con 0x558364b43bc0 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558364a64110 con 0x558364b49720 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558364b4c7f0 con 0x558364b445b0 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb662972740 10 client.0 _ll_register_callbacks cb 0x558364aa0be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59882/0 (socket says 172.21.15.28:59882) 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65d24a640 1 -- 172.21.15.28:0/754871694 learned_addr learned my addr 172.21.15.28:0/754871694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b4e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65d24a640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 -- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 msgr2=0x558364b552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 -- 172.21.15.28:0/754871694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 msgr2=0x558364b4dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 -- 172.21.15.28:0/754871694 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583648e4f70 con 0x558364b445b0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65d24a640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb65da4b640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b4e290 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558364b449b0 tx=0x7fb650004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1625d82f4491249a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.761+0000 7fb63d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650042040 con 0x558364b445b0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fb6500052b0 con 0x558364b445b0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb650004ba0 con 0x558364b445b0 2024-03-22T02:21:31.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb650057050 con 0x558364b445b0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 10 client.27634 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55836477c170 con 0x558364b445b0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb650004ba0 con 0x558364b445b0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fb650034440 con 0x558364b445b0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 10 client.27634 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 10 client.27634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 20 client.27634 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364b597c0 0x558364b5bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 4 client.27634 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.762+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558364ab5ab0 con 0x558364b597c0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.763+0000 7fb65d24a640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364b597c0 0x558364b5bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.763+0000 7fb63d7fa640 10 client.27634 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.763+0000 7fb65d24a640 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364b597c0 0x558364b5bbb0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28f5d003a08098d3 server_cookie=56888a29e22ccd80 in_seq=0 out_seq=0 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb63d7fa640 1 -- 172.21.15.28:0/754871694 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558364ab5ab0 con 0x558364b597c0 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb63d7fa640 10 client.27634 handle_command_reply: tid=0 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 client.27634 shutdown 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 msgr2=0x558364b4e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b4e290 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x558364b449b0 tx=0x7fb650004250 comp rx=0 tx=0).stop 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 shutdown_connections 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364b597c0 0x558364b5bbb0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b4e290 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 --2- 172.21.15.28:0/754871694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b4dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb63d7fa640 0 client.27634 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 >> 172.21.15.28:0/754871694 conn(0x558364b3f130 msgr2=0x558364b3e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 shutdown_connections 2024-03-22T02:21:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 1 -- 172.21.15.28:0/754871694 wait complete. 2024-03-22T02:21:31.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.764+0000 7fb662972740 20 client.27634 trim_cache size 0 max 16384 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 Processor -- start 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 -- start start 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b43bc0 0x558364b3e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b60310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581af370 con 0x558364b445b0 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581acc40 con 0x558364b49720 2024-03-22T02:21:31.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb662972740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581aaa90 con 0x558364b43bc0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59892/0 (socket says 172.21.15.28:59892) 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 -- 172.21.15.28:0/534985394 learned_addr learned my addr 172.21.15.28:0/534985394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65d24a640 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b60310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65d24a640 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b43bc0 0x558364b3e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 -- 172.21.15.28:0/534985394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b43bc0 msgr2=0x558364b3e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b43bc0 0x558364b3e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 -- 172.21.15.28:0/534985394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 msgr2=0x558364b60310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b60310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.782+0000 7fb65da4b640 1 -- 172.21.15.28:0/534985394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558364b48a30 con 0x558364b445b0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb65da4b640 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb650036040 tx=0x7fb650005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201f3e4ec5ead270 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb63f7fe640 1 -- 172.21.15.28:0/534985394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650042040 con 0x558364b445b0 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb63f7fe640 1 -- 172.21.15.28:0/534985394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb650009520 con 0x558364b445b0 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb63f7fe640 1 -- 172.21.15.28:0/534985394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650034bc0 con 0x558364b445b0 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 -- 172.21.15.28:0/534985394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 msgr2=0x558364b55290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb650036040 tx=0x7fb650005d50 comp rx=0 tx=0).stop 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 -- 172.21.15.28:0/534985394 shutdown_connections 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b60310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b445b0 0x558364b55290 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 --2- 172.21.15.28:0/534985394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b43bc0 0x558364b3e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.783+0000 7fb662972740 1 -- 172.21.15.28:0/534985394 >> 172.21.15.28:0/534985394 conn(0x558364b3f130 msgr2=0x558364b3db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- 172.21.15.28:0/534985394 shutdown_connections 2024-03-22T02:21:31.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- 172.21.15.28:0/534985394 wait complete. 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- start start 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 Processor -- start 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b53660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b53ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ab100 con 0x558364b43bc0 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ac570 con 0x558364b49720 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6581ab7d0 con 0x558364b445b0 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb662972740 10 client.0 _ll_register_callbacks cb 0x7fb64c009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb65d24a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59900/0 (socket says 172.21.15.28:59900) 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb65d24a640 1 -- 172.21.15.28:0/1056124207 learned_addr learned my addr 172.21.15.28:0/1056124207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb65da4b640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b53660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.784+0000 7fb65d24a640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b53ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 -- 172.21.15.28:0/1056124207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 msgr2=0x558364b53660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b53660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 -- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 msgr2=0x558364b53ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b53ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 -- 172.21.15.28:0/1056124207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558364b54290 con 0x558364b43bc0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb65d24a640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb650004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa7f0104467dad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64c005bb0 con 0x558364b43bc0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fb64c009040 con 0x558364b43bc0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb650009520 con 0x558364b43bc0 2024-03-22T02:21:31.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb64c005020 con 0x558364b43bc0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 10 client.17817 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x558364bc0920 con 0x558364b43bc0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb650009520 con 0x558364b43bc0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fb64c031070 con 0x558364b43bc0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 10 client.17817 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 10 client.17817 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 20 client.17817 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364bc1300 0x558364bc3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 4 client.17817 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.785+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558364aafe00 con 0x558364bc1300 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.786+0000 7fb65da4b640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364bc1300 0x558364bc3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.786+0000 7fb65da4b640 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364bc1300 0x558364bc3720 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225def1a808253b6 server_cookie=dd6942f65db4f25b in_seq=0 out_seq=0 2024-03-22T02:21:31.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.786+0000 7fb63d7fa640 10 client.17817 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb63d7fa640 1 -- 172.21.15.28:0/1056124207 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558364aafe00 con 0x558364bc1300 2024-03-22T02:21:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb63d7fa640 10 client.17817 handle_command_reply: tid=0 2024-03-22T02:21:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 client.17817 shutdown 2024-03-22T02:21:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 msgr2=0x558364b55050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb650004a60 comp rx=0 tx=0).stop 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 shutdown_connections 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558364bc1300 0x558364bc3720 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558364b49720 0x558364b53ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb63d7fa640 0 client.17817 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558364b445b0 0x558364b53660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 --2- 172.21.15.28:0/1056124207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558364b43bc0 0x558364b55050 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 >> 172.21.15.28:0/1056124207 conn(0x558364b3f130 msgr2=0x558364b3e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 shutdown_connections 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 1 -- 172.21.15.28:0/1056124207 wait complete. 2024-03-22T02:21:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.799+0000 7fb662972740 20 client.17817 trim_cache size 0 max 16384 2024-03-22T02:21:31.803 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:21:31.803 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:21:31.803 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "37f4e065-6e63-4b3f-87f4-f3a55c0ee491", 2024-03-22T02:21:31.803 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:21:31.803 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 msgr2=0x7fb6400a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 0x7fb6400a53e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb648008f90 tx=0x7fb64806ec10 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 msgr2=0x7fb658100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 0x7fb658100b00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb64800da10 tx=0x7fb64800ded0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 shutdown_connections 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb6400a2f20 0x7fb6400a53e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb658107870 0x7fb658100b00 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb658104f40 0x7fb6581005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 --2- 172.21.15.28:0/811740200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb658102610 0x7fb658101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 >> 172.21.15.28:0/811740200 conn(0x7fb6580fc040 msgr2=0x7fb658103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 shutdown_connections 2024-03-22T02:21:31.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.800+0000 7fb65fcaf640 1 -- 172.21.15.28:0/811740200 wait complete. 2024-03-22T02:21:31.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:21:31.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:21:31.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 -- 172.21.15.28:0/1039770897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 msgr2=0x7f5a28105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/1039770897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a28105bb0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009900 tx=0x7f5a2002bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 -- 172.21.15.28:0/1039770897 shutdown_connections 2024-03-22T02:21:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/1039770897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a28105bb0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/1039770897 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28102d80 0x7f5a281051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/1039770897 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28100410 0x7f5a28102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.932+0000 7f5a2fac2640 1 -- 172.21.15.28:0/1039770897 >> 172.21.15.28:0/1039770897 conn(0x7f5a280fc030 msgr2=0x7f5a280fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- 172.21.15.28:0/1039770897 shutdown_connections 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- 172.21.15.28:0/1039770897 wait complete. 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 Processor -- start 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- start start 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28100410 0x7f5a2819e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28102d80 0x7f5a2819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a28113e40 con 0x7f5a28105730 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a28113cc0 con 0x7f5a28102d80 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2fac2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5a28113fc0 con 0x7f5a28100410 2024-03-22T02:21:31.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59908/0 (socket says 172.21.15.28:59908) 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.933+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2887838502 learned_addr learned my addr 172.21.15.28:0/2887838502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28100410 0x7f5a2819e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28102d80 0x7f5a2819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2887838502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28100410 msgr2=0x7f5a2819e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28100410 0x7f5a2819e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2887838502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28102d80 msgr2=0x7f5a2819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28102d80 0x7f5a2819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2887838502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a281a6080 con 0x7f5a28105730 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5a20005ab0 tx=0x7f5a2002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a1effd640 1 -- 172.21.15.28:0/2887838502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20041070 con 0x7f5a28105730 2024-03-22T02:21:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a1effd640 1 -- 172.21.15.28:0/2887838502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a20004450 con 0x7f5a28105730 2024-03-22T02:21:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a281a6310 con 0x7f5a28105730 2024-03-22T02:21:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a1effd640 1 -- 172.21.15.28:0/2887838502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20003e60 con 0x7f5a28105730 2024-03-22T02:21:31.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.934+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a281a6800 con 0x7f5a28105730 2024-03-22T02:21:31.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.936+0000 7f5a1effd640 1 -- 172.21.15.28:0/2887838502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5a20044070 con 0x7f5a28105730 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 Processor -- start 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 -- start start 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a1effd640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 0x7f5a140a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43962920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 0x55fe43965420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe438c28c0 con 0x55fe4395de20 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe4386ac20 con 0x55fe4395d430 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a32781740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe438be870 con 0x55fe43962f90 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a1effd640 1 -- 172.21.15.28:0/2887838502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5a200f1430 con 0x7f5a28105730 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 0x7f5a140a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41192/0 (socket says 172.21.15.28:41192) 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.937+0000 7f5a2d85e640 1 -- 172.21.15.28:0/3789161216 learned_addr learned my addr 172.21.15.28:0/3789161216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 0x55fe43965420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43962920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 -- 172.21.15.28:0/3789161216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 msgr2=0x55fe43965420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 0x55fe43965420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 -- 172.21.15.28:0/3789161216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 msgr2=0x55fe43962920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43962920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 -- 172.21.15.28:0/3789161216 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe4388f150 con 0x55fe4395d430 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 0x55fe43965420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 0x7f5a140a53e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009fd0 tx=0x7f5a18009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a240114f0 tx=0x7f5a240119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e17ed6cb17094c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a13fff640 1 -- 172.21.15.28:0/3789161216 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2401e070 con 0x55fe4395d430 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a13fff640 1 -- 172.21.15.28:0/3789161216 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a240110f0 con 0x55fe4395d430 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.938+0000 7f5a13fff640 1 -- 172.21.15.28:0/3789161216 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24010830 con 0x55fe4395d430 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- 172.21.15.28:0/3789161216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 msgr2=0x55fe4395d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5a240114f0 tx=0x7f5a240119b0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- 172.21.15.28:0/3789161216 shutdown_connections 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe43962f90 0x55fe43965420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43962920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 --2- 172.21.15.28:0/3789161216 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395d850 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- 172.21.15.28:0/3789161216 >> 172.21.15.28:0/3789161216 conn(0x55fe439589a0 msgr2=0x55fe43958da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- 172.21.15.28:0/3789161216 shutdown_connections 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- 172.21.15.28:0/3789161216 wait complete. 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.939+0000 7f5a32781740 1 -- start start 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 Processor -- start 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fe43965f40 con 0x55fe4395de20 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55fe4387e130 con 0x55fe43962f90 2024-03-22T02:21:31.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55fe439660c0 con 0x55fe4395d430 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a32781740 10 client.0 _ll_register_callbacks cb 0x55fe4387db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d05d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d05d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59920/0 (socket says 172.21.15.28:59920) 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41196/0 (socket says 172.21.15.28:41196) 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51330/0 (socket says 172.21.15.28:51330) 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d05d640 1 -- 172.21.15.28:0/2427781945 learned_addr learned my addr 172.21.15.28:0/2427781945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2427781945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 msgr2=0x55fe43967940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2427781945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 msgr2=0x55fe43967e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2e05f640 1 -- 172.21.15.28:0/2427781945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe436a07d0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.940+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a200eee90 tx=0x7f5a200bd220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9295c1ff01c707e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2004b050 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5a200a80b0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a240110f0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5a200db050 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a32781740 10 client.27653 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe435d63f0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.941+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a240110f0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f5a200c5cd0 con 0x55fe43962f90 2024-03-22T02:21:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 10 client.27653 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 10 client.27653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 20 client.27653 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439736f0 0x55fe43975b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 4 client.27653 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe438cfe00 con 0x55fe439736f0 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439736f0 0x55fe43975b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439736f0 0x55fe43975b40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7636788c0e7a89 server_cookie=804feae3fc4e9f6b in_seq=0 out_seq=0 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.942+0000 7f5a117fa640 10 client.27653 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a117fa640 1 -- 172.21.15.28:0/2427781945 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe438cfe00 con 0x55fe439736f0 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a117fa640 10 client.27653 handle_command_reply: tid=0 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 client.27653 shutdown 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 msgr2=0x55fe4396eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5a200eee90 tx=0x7f5a200bd220 comp rx=0 tx=0).stop 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 shutdown_connections 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439736f0 0x55fe43975b40 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a117fa640 0 client.27653 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe43962f90 0x55fe4396eec0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe4395de20 0x55fe43967e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 --2- 172.21.15.28:0/2427781945 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395d430 0x55fe43967940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.943+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 >> 172.21.15.28:0/2427781945 conn(0x55fe439589a0 msgr2=0x55fe43957c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.944+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 shutdown_connections 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.944+0000 7f5a32781740 1 -- 172.21.15.28:0/2427781945 wait complete. 2024-03-22T02:21:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.944+0000 7f5a32781740 20 client.27653 trim_cache size 0 max 16384 2024-03-22T02:21:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 Processor -- start 2024-03-22T02:21:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 -- start start 2024-03-22T02:21:31.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281b38b0 con 0x55fe43962f90 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281b1180 con 0x55fe4395d430 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a32781740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281aefd0 con 0x55fe4395de20 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a2d05d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59922/0 (socket says 172.21.15.28:59922) 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.967+0000 7f5a2e05f640 1 -- 172.21.15.28:0/938299195 learned_addr learned my addr 172.21.15.28:0/938299195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 -- 172.21.15.28:0/938299195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 msgr2=0x55fe4395f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 -- 172.21.15.28:0/938299195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 msgr2=0x55fe4395f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 -- 172.21.15.28:0/938299195 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe435960f0 con 0x55fe4395de20 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ee80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009dc0 tx=0x7f5a18021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69348bf24ccefe1c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a137fe640 1 -- 172.21.15.28:0/938299195 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a1802e070 con 0x55fe4395de20 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a137fe640 1 -- 172.21.15.28:0/938299195 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a180044c0 con 0x55fe4395de20 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.968+0000 7f5a137fe640 1 -- 172.21.15.28:0/938299195 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18029900 con 0x55fe4395de20 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- 172.21.15.28:0/938299195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 msgr2=0x55fe4396ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ee80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009dc0 tx=0x7f5a18021000 comp rx=0 tx=0).stop 2024-03-22T02:21:31.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- 172.21.15.28:0/938299195 shutdown_connections 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4395f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ee80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 --2- 172.21.15.28:0/938299195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4395f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- 172.21.15.28:0/938299195 >> 172.21.15.28:0/938299195 conn(0x55fe439589a0 msgr2=0x55fe43967940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- 172.21.15.28:0/938299195 shutdown_connections 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- 172.21.15.28:0/938299195 wait complete. 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 -- start start 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.969+0000 7f5a32781740 1 Processor -- start 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4396da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281af640 con 0x55fe43962f90 2024-03-22T02:21:31.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281b0ab0 con 0x55fe4395d430 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5a281afd10 con 0x55fe4395de20 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a32781740 10 client.0 _ll_register_callbacks cb 0x7f5a200bd340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2e05f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4396da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41214/0 (socket says 172.21.15.28:41214) 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 -- 172.21.15.28:0/1132735230 learned_addr learned my addr 172.21.15.28:0/1132735230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 -- 172.21.15.28:0/1132735230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 msgr2=0x55fe4396ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 -- 172.21.15.28:0/1132735230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 msgr2=0x55fe4396da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4396da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 -- 172.21.15.28:0/1132735230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe4396e030 con 0x55fe4395d430 2024-03-22T02:21:31.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2e05f640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4396da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a2d85e640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5a24005c40 tx=0x7f5a24005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f97d0e1b1ea3fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.970+0000 7f5a117fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2401e040 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7f5a2400fdd0 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fe438ca150 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5a24003970 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 10 client.27659 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fe439d9a40 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55fe438ca150 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7f5a2402b380 con 0x55fe4395d430 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 10 client.27659 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:31.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 10 client.27659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 20 client.27659 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439da160 0x55fe439dc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 4 client.27659 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.971+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe43847540 con 0x55fe439da160 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.972+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439da160 0x55fe439dc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.972+0000 7f5a117fa640 10 client.27659 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.972+0000 7f5a2d05d640 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439da160 0x55fe439dc580 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccfface39763c707 server_cookie=21a66182ddfe9d09 in_seq=0 out_seq=0 2024-03-22T02:21:31.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a117fa640 1 -- 172.21.15.28:0/1132735230 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fe43847540 con 0x55fe439da160 2024-03-22T02:21:31.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a117fa640 10 client.27659 handle_command_reply: tid=0 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 client.27659 shutdown 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 msgr2=0x55fe4396c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5a24005c40 tx=0x7f5a24005bd0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 shutdown_connections 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55fe439da160 0x55fe439dc580 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fe43962f90 0x55fe4396da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a117fa640 0 client.27659 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55fe4395de20 0x55fe4396ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 --2- 172.21.15.28:0/1132735230 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55fe4395d430 0x55fe4396c4f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 >> 172.21.15.28:0/1132735230 conn(0x55fe439589a0 msgr2=0x55fe43967940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 shutdown_connections 2024-03-22T02:21:31.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 1 -- 172.21.15.28:0/1132735230 wait complete. 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.974+0000 7f5a32781740 20 client.27659 trim_cache size 0 max 16384 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "37f4e065-6e63-4b3f-87f4-f3a55c0ee491": { 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "37f4e065-6e63-4b3f-87f4-f3a55c0ee491", 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:21:31.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:21:31.983 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:21:31.983 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:21:31.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 msgr2=0x7f5a140a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 0x7f5a140a53e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5a18009fd0 tx=0x7f5a18009b80 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 msgr2=0x7f5a281a5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5a20005ab0 tx=0x7f5a2002c6c0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 shutdown_connections 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5a140a2f20 0x7f5a140a53e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a28105730 0x7f5a281a5a10 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5a28102d80 0x7f5a2819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 --2- 172.21.15.28:0/2887838502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5a28100410 0x7f5a2819e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 >> 172.21.15.28:0/2887838502 conn(0x7f5a280fc030 msgr2=0x7f5a28100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 shutdown_connections 2024-03-22T02:21:31.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:21:31.975+0000 7f5a2fac2640 1 -- 172.21.15.28:0/2887838502 wait complete. 2024-03-22T02:21:31.989 INFO:tasks.cephfs.filesystem:scrub status for tag:37f4e065-6e63-4b3f-87f4-f3a55c0ee491 - {'path': '/', 'tag': '37f4e065-6e63-4b3f-87f4-f3a55c0ee491', 'options': 'recursive,force'} 2024-03-22T02:21:32.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 747816 11.77 MB/sec execute 254 sec latency 42.832 ms 2024-03-22T02:21:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:32 smithi120 ceph-mon[36803]: pgmap v597: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-03-22T02:21:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:32 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:21:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:32 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:32 smithi083 ceph-mon[36340]: pgmap v597: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-03-22T02:21:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:32 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:21:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:32 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:32 smithi028 ceph-mon[31916]: pgmap v597: 129 pgs: 129 active+clean; 55 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 321 op/s 2024-03-22T02:21:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:32 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:21:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:32 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 749611 11.77 MB/sec execute 255 sec latency 102.092 ms 2024-03-22T02:21:33.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:33 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:21:33.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:33 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:33 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:21:33.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:33 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:33 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:21:33.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:33 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:33 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:21:33.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:33 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:33 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:21:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:33 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:33 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:21:33.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:33 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:21:34.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 752118 11.77 MB/sec execute 256 sec latency 36.571 ms 2024-03-22T02:21:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:34 smithi120 ceph-mon[36803]: pgmap v598: 129 pgs: 129 active+clean; 59 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-03-22T02:21:34.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:34 smithi083 ceph-mon[36340]: pgmap v598: 129 pgs: 129 active+clean; 59 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-03-22T02:21:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:34 smithi028 ceph-mon[31916]: pgmap v598: 129 pgs: 129 active+clean; 59 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-03-22T02:21:35.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 753892 11.76 MB/sec execute 257 sec latency 30.235 ms 2024-03-22T02:21:36.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 756636 11.78 MB/sec execute 258 sec latency 16.258 ms 2024-03-22T02:21:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:36 smithi120 ceph-mon[36803]: pgmap v599: 129 pgs: 129 active+clean; 58 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 395 op/s 2024-03-22T02:21:36.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:36 smithi083 ceph-mon[36340]: pgmap v599: 129 pgs: 129 active+clean; 58 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 395 op/s 2024-03-22T02:21:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:36 smithi028 ceph-mon[31916]: pgmap v599: 129 pgs: 129 active+clean; 58 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 395 op/s 2024-03-22T02:21:37.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 759298 11.79 MB/sec execute 259 sec latency 43.624 ms 2024-03-22T02:21:38.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 760618 11.77 MB/sec execute 260 sec latency 67.406 ms 2024-03-22T02:21:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:38 smithi120 ceph-mon[36803]: pgmap v600: 129 pgs: 129 active+clean; 50 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 402 op/s 2024-03-22T02:21:38.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:38 smithi083 ceph-mon[36340]: pgmap v600: 129 pgs: 129 active+clean; 50 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 402 op/s 2024-03-22T02:21:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:38 smithi028 ceph-mon[31916]: pgmap v600: 129 pgs: 129 active+clean; 50 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 402 op/s 2024-03-22T02:21:39.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 763422 11.79 MB/sec execute 261 sec latency 25.455 ms 2024-03-22T02:21:40.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 764939 11.77 MB/sec execute 262 sec latency 37.338 ms 2024-03-22T02:21:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:40 smithi120 ceph-mon[36803]: pgmap v601: 129 pgs: 129 active+clean; 55 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-03-22T02:21:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:40 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:21:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:40 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:21:40.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:40 smithi083 ceph-mon[36340]: pgmap v601: 129 pgs: 129 active+clean; 55 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-03-22T02:21:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:40 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:21:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:40 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:21:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:40 smithi028 ceph-mon[31916]: pgmap v601: 129 pgs: 129 active+clean; 55 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 436 op/s 2024-03-22T02:21:40.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:40 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:21:40.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:40 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:21:41.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 767349 11.78 MB/sec execute 263 sec latency 18.042 ms 2024-03-22T02:21:42.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 769857 11.78 MB/sec execute 264 sec latency 32.451 ms 2024-03-22T02:21:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:42 smithi120 ceph-mon[36803]: pgmap v602: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 416 op/s 2024-03-22T02:21:42.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:42 smithi083 ceph-mon[36340]: pgmap v602: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 416 op/s 2024-03-22T02:21:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:42 smithi028 ceph-mon[31916]: pgmap v602: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.9 MiB/s wr, 416 op/s 2024-03-22T02:21:43.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 771210 11.76 MB/sec execute 265 sec latency 68.564 ms 2024-03-22T02:21:44.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 773694 11.77 MB/sec execute 266 sec latency 19.944 ms 2024-03-22T02:21:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 -- 172.21.15.28:0/3659344012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 msgr2=0x7fa36c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 --2- 172.21.15.28:0/3659344012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c1073d0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fa368009900 tx=0x7fa36802bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:44.515 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 -- 172.21.15.28:0/3659344012 shutdown_connections 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 --2- 172.21.15.28:0/3659344012 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 0x7fa36c109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 --2- 172.21.15.28:0/3659344012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c1073d0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 --2- 172.21.15.28:0/3659344012 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 -- 172.21.15.28:0/3659344012 >> 172.21.15.28:0/3659344012 conn(0x7fa36c0fc040 msgr2=0x7fa36c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 -- 172.21.15.28:0/3659344012 shutdown_connections 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.512+0000 7fa374242640 1 -- 172.21.15.28:0/3659344012 wait complete. 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 Processor -- start 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 -- start start 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 0x7fa36c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.516 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d820 con 0x7fa36c104fe0 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d6a0 con 0x7fa36c1026b0 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa374242640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c10d9a0 con 0x7fa36c107910 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa3717dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa3717dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43884/0 (socket says 172.21.15.28:43884) 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa3717dd640 1 -- 172.21.15.28:0/1355601260 learned_addr learned my addr 172.21.15.28:0/1355601260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa371fde640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.513+0000 7fa3727df640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 0x7fa36c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 -- 172.21.15.28:0/1355601260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 msgr2=0x7fa36c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 0x7fa36c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 -- 172.21.15.28:0/1355601260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 msgr2=0x7fa36c197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 -- 172.21.15.28:0/1355601260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c19fa50 con 0x7fa36c104fe0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa371fde640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa368000c00 tx=0x7fa36802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa362ffd640 1 -- 172.21.15.28:0/1355601260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368041070 con 0x7fa36c104fe0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa362ffd640 1 -- 172.21.15.28:0/1355601260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa368004450 con 0x7fa36c104fe0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36c19fce0 con 0x7fa36c104fe0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa362ffd640 1 -- 172.21.15.28:0/1355601260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368034440 con 0x7fa36c104fe0 2024-03-22T02:21:44.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.514+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36c1a01d0 con 0x7fa36c104fe0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.516+0000 7fa362ffd640 1 -- 172.21.15.28:0/1355601260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa368044070 con 0x7fa36c104fe0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.516+0000 7fa376e1f740 1 Processor -- start 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 -- start start 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd08d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614cfc69040 con 0x5614cfd092e0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5614cfc119e0 con 0x5614cfd0e450 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa376e1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5614cfc64ff0 con 0x5614cfd088f0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa3727df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd08d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.525 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43898/0 (socket says 172.21.15.28:43898) 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa3727df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd08d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58666/0 (socket says 172.21.15.28:58666) 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 -- 172.21.15.28:0/1485457875 learned_addr learned my addr 172.21.15.28:0/1485457875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa362ffd640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 0x7fa3580a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 -- 172.21.15.28:0/1485457875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 msgr2=0x5614cfd08d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd08d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 -- 172.21.15.28:0/1485457875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 msgr2=0x5614cfd108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa371fde640 1 -- 172.21.15.28:0/1485457875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cf97d210 con 0x5614cfd092e0 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.517+0000 7fa3727df640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 0x7fa3580a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.518+0000 7fa362ffd640 1 -- 172.21.15.28:0/1355601260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa3680f12a0 con 0x7fa36c104fe0 2024-03-22T02:21:44.526 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.518+0000 7fa371fde640 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0115f0 tx=0x7fa35c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f76b7a8784d0431 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.518+0000 7fa357fff640 1 -- 172.21.15.28:0/1485457875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c01e070 con 0x5614cfd092e0 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.518+0000 7fa357fff640 1 -- 172.21.15.28:0/1485457875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa35c00fb20 con 0x5614cfd092e0 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.518+0000 7fa357fff640 1 -- 172.21.15.28:0/1485457875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c019bf0 con 0x5614cfd092e0 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- 172.21.15.28:0/1485457875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 msgr2=0x5614cfd0dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa3727df640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 0x7fa3580a5480 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa364009fd0 tx=0x7fa364004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0115f0 tx=0x7fa35c011ab0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- 172.21.15.28:0/1485457875 shutdown_connections 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd0dde0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1485457875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd08d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- 172.21.15.28:0/1485457875 >> 172.21.15.28:0/1485457875 conn(0x5614cfd03e60 msgr2=0x5614cfd04260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- 172.21.15.28:0/1485457875 shutdown_connections 2024-03-22T02:21:44.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- 172.21.15.28:0/1485457875 wait complete. 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- start start 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 Processor -- start 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd19e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.519+0000 7fa376e1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614cfd11400 con 0x5614cfd092e0 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5614cfc24db0 con 0x5614cfd0e450 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5614cfd11580 con 0x5614cfd088f0 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 10 client.0 _ll_register_callbacks cb 0x5614cf97ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43906/0 (socket says 172.21.15.28:43906) 2024-03-22T02:21:44.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 -- 172.21.15.28:0/1579074253 learned_addr learned my addr 172.21.15.28:0/1579074253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa371fde640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3727df640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd19e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 -- 172.21.15.28:0/1579074253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 msgr2=0x5614cfd128f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd128f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 -- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 msgr2=0x5614cfd19e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd19e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 -- 172.21.15.28:0/1579074253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614cfa49620 con 0x5614cfd092e0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3717dd640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa3680ee990 tx=0x7fa3680b93a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73e10f1da9fd72b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36804b070 con 0x5614cfd092e0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa3680bba00 con 0x5614cfd092e0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa35c00fb20 con 0x5614cfd092e0 2024-03-22T02:21:44.529 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa3680bd4d0 con 0x5614cfd092e0 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 10 client.17847 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.520+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5614cfd1d1e0 con 0x5614cfd092e0 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa35c00fb20 con 0x5614cfd092e0 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fa3680035f0 con 0x5614cfd092e0 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 10 client.17847 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 10 client.17847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 20 client.17847 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd1e910 0x5614cfd20d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 4 client.17847 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614cfc76580 con 0x5614cfd1e910 2024-03-22T02:21:44.530 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.521+0000 7fa371fde640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd1e910 0x5614cfd20d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.522+0000 7fa3557fa640 10 client.17847 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.522+0000 7fa371fde640 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd1e910 0x5614cfd20d20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8107146aeb80b62d server_cookie=adfa94e504a2f8d8 in_seq=0 out_seq=0 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa3557fa640 1 -- 172.21.15.28:0/1579074253 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5614cfc76580 con 0x5614cfd1e910 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa3557fa640 10 client.17847 handle_command_reply: tid=0 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 client.17847 shutdown 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 msgr2=0x5614cfd12e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fa3680ee990 tx=0x7fa3680b93a0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 shutdown_connections 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd1e910 0x5614cfd20d20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd0e450 0x5614cfd19e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd092e0 0x5614cfd12e30 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.531 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa3557fa640 0 client.17847 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 --2- 172.21.15.28:0/1579074253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd128f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 >> 172.21.15.28:0/1579074253 conn(0x5614cfd03e60 msgr2=0x5614cfcfec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 shutdown_connections 2024-03-22T02:21:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 1 -- 172.21.15.28:0/1579074253 wait complete. 2024-03-22T02:21:44.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.523+0000 7fa376e1f740 20 client.17847 trim_cache size 0 max 16384 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 Processor -- start 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 -- start start 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd19e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd0e450 0x5614cfcfe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1ab010 con 0x5614cfd088f0 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1a88e0 con 0x5614cfd092e0 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa376e1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1a6730 con 0x5614cfd0e450 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa371fde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa371fde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43910/0 (socket says 172.21.15.28:43910) 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa371fde640 1 -- 172.21.15.28:0/803073618 learned_addr learned my addr 172.21.15.28:0/803073618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa3727df640 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd0e450 0x5614cfcfe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.539+0000 7fa3717dd640 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd19e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 -- 172.21.15.28:0/803073618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd0e450 msgr2=0x5614cfcfe3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd0e450 0x5614cfcfe3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 -- 172.21.15.28:0/803073618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 msgr2=0x5614cfd19e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd19e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.545 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 -- 172.21.15.28:0/803073618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614cfd09b70 con 0x5614cfd088f0 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa371fde640 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0115f0 tx=0x7fa35c019840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bbc59aa865c8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa3577fe640 1 -- 172.21.15.28:0/803073618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c02f070 con 0x5614cfd088f0 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa3577fe640 1 -- 172.21.15.28:0/803073618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa35c029070 con 0x5614cfd088f0 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.540+0000 7fa3577fe640 1 -- 172.21.15.28:0/803073618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c01e040 con 0x5614cfd088f0 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- 172.21.15.28:0/803073618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 msgr2=0x5614cfd0db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0115f0 tx=0x7fa35c019840 comp rx=0 tx=0).stop 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- 172.21.15.28:0/803073618 shutdown_connections 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd0e450 0x5614cfcfe3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd19e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- 172.21.15.28:0/803073618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd088f0 0x5614cfd0db20 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- 172.21.15.28:0/803073618 >> 172.21.15.28:0/803073618 conn(0x5614cfd03e60 msgr2=0x5614cfd07950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.546 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- 172.21.15.28:0/803073618 shutdown_connections 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- 172.21.15.28:0/803073618 wait complete. 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- start start 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 Processor -- start 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd17c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd18190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1a6da0 con 0x5614cfd0e450 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1a8210 con 0x5614cfd092e0 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c1a7470 con 0x5614cfd088f0 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa376e1f740 10 client.0 _ll_register_callbacks cb 0x7fa368005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa3727df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.547 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa3717dd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd18190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa371fde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd17c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa3727df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43920/0 (socket says 172.21.15.28:43920) 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.541+0000 7fa3727df640 1 -- 172.21.15.28:0/3608508428 learned_addr learned my addr 172.21.15.28:0/3608508428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 -- 172.21.15.28:0/3608508428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 msgr2=0x5614cfd17c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd17c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 -- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 msgr2=0x5614cfd18190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd18190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 -- 172.21.15.28:0/3608508428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614cfd19790 con 0x5614cfd0e450 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3717dd640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd18190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3727df640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa36400fdd0 tx=0x7fa364004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda7a64a5c91a4e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:44.548 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36402a450 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa35c029070 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa36402a5f0 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa364038050 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa376e1f740 10 client.17853 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5614cfd85400 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.542+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa35c029070 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fa36401c7f0 con 0x5614cfd0e450 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 10 client.17853 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 10 client.17853 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 20 client.17853 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd85eb0 0x5614cfd882d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:44.549 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 4 client.17853 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:21:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614cfc708d0 con 0x5614cfd85eb0 2024-03-22T02:21:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa371fde640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd85eb0 0x5614cfd882d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa371fde640 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd85eb0 0x5614cfd882d0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1981757e9ec2eeb server_cookie=eb795c5a2b5a83b0 in_seq=0 out_seq=0 2024-03-22T02:21:44.550 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.543+0000 7fa3557fa640 10 client.17853 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:44 smithi120 ceph-mon[36803]: pgmap v603: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 407 op/s 2024-03-22T02:21:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:44 smithi083 ceph-mon[36340]: pgmap v603: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 407 op/s 2024-03-22T02:21:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa3557fa640 1 -- 172.21.15.28:0/3608508428 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614cfc708d0 con 0x5614cfd85eb0 2024-03-22T02:21:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa3557fa640 10 client.17853 handle_command_reply: tid=0 2024-03-22T02:21:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 client.17853 shutdown 2024-03-22T02:21:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 msgr2=0x5614cfd191a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fa36400fdd0 tx=0x7fa364004ab0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 shutdown_connections 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5614cfd85eb0 0x5614cfd882d0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa3557fa640 0 client.17853 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614cfd0e450 0x5614cfd191a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5614cfd092e0 0x5614cfd18190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 --2- 172.21.15.28:0/3608508428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5614cfd088f0 0x5614cfd17c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 >> 172.21.15.28:0/3608508428 conn(0x5614cfd03e60 msgr2=0x5614cfd09d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 shutdown_connections 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 1 -- 172.21.15.28:0/3608508428 wait complete. 2024-03-22T02:21:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.805+0000 7fa376e1f740 20 client.17853 trim_cache size 0 max 16384 2024-03-22T02:21:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:21:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:21:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:21:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 msgr2=0x7fa3580a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 0x7fa3580a5480 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa364009fd0 tx=0x7fa364004a40 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 msgr2=0x7fa36c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa368000c00 tx=0x7fa36802c640 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 shutdown_connections 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa3580a2fc0 0x7fa3580a5480 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa36c107910 0x7fa36c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c104fe0 0x7fa36c198370 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 --2- 172.21.15.28:0/1355601260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa36c1026b0 0x7fa36c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 >> 172.21.15.28:0/1355601260 conn(0x7fa36c0fc040 msgr2=0x7fa36c103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 shutdown_connections 2024-03-22T02:21:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:44.806+0000 7fa374242640 1 -- 172.21.15.28:0/1355601260 wait complete. 2024-03-22T02:21:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:44 smithi028 ceph-mon[31916]: pgmap v603: 129 pgs: 129 active+clean; 57 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 407 op/s 2024-03-22T02:21:45.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 774737 11.75 MB/sec execute 267 sec latency 88.111 ms 2024-03-22T02:21:46.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 777536 11.77 MB/sec execute 268 sec latency 10.708 ms 2024-03-22T02:21:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:46 smithi120 ceph-mon[36803]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.0 MiB/s wr, 415 op/s 2024-03-22T02:21:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:46 smithi083 ceph-mon[36340]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.0 MiB/s wr, 415 op/s 2024-03-22T02:21:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:46 smithi028 ceph-mon[31916]: pgmap v604: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.0 MiB/s wr, 415 op/s 2024-03-22T02:21:47.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 779043 11.75 MB/sec execute 269 sec latency 43.217 ms 2024-03-22T02:21:48.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 781391 11.76 MB/sec execute 270 sec latency 46.875 ms 2024-03-22T02:21:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:48 smithi120 ceph-mon[36803]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 331 op/s 2024-03-22T02:21:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:48 smithi083 ceph-mon[36340]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 331 op/s 2024-03-22T02:21:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:48 smithi028 ceph-mon[31916]: pgmap v605: 129 pgs: 129 active+clean; 60 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 331 op/s 2024-03-22T02:21:49.424 INFO:tasks.workunit.client.0.smithi028.stdout: 1 782748 11.73 MB/sec execute 271 sec latency 47.586 ms 2024-03-22T02:21:50.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 785068 11.74 MB/sec execute 272 sec latency 17.261 ms 2024-03-22T02:21:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:50 smithi120 ceph-mon[36803]: pgmap v606: 129 pgs: 129 active+clean; 53 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 411 op/s 2024-03-22T02:21:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:50 smithi083 ceph-mon[36340]: pgmap v606: 129 pgs: 129 active+clean; 53 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 411 op/s 2024-03-22T02:21:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:50 smithi028 ceph-mon[31916]: pgmap v606: 129 pgs: 129 active+clean; 53 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 3.3 MiB/s wr, 411 op/s 2024-03-22T02:21:51.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 787244 11.73 MB/sec execute 273 sec latency 41.426 ms 2024-03-22T02:21:52.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 788971 11.73 MB/sec execute 274 sec latency 44.115 ms 2024-03-22T02:21:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:52 smithi120 ceph-mon[36803]: pgmap v607: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:21:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:52 smithi083 ceph-mon[36340]: pgmap v607: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:21:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:52 smithi028 ceph-mon[31916]: pgmap v607: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:21:53.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 791912 11.75 MB/sec execute 275 sec latency 12.874 ms 2024-03-22T02:21:54.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 793353 11.73 MB/sec execute 276 sec latency 40.423 ms 2024-03-22T02:21:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:54 smithi120 ceph-mon[36803]: pgmap v608: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 350 op/s 2024-03-22T02:21:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:54 smithi083 ceph-mon[36340]: pgmap v608: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 350 op/s 2024-03-22T02:21:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:54 smithi028 ceph-mon[31916]: pgmap v608: 129 pgs: 129 active+clean; 54 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.3 MiB/s wr, 350 op/s 2024-03-22T02:21:55.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 795727 11.74 MB/sec execute 277 sec latency 14.209 ms 2024-03-22T02:21:56.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 798515 11.74 MB/sec execute 278 sec latency 25.043 ms 2024-03-22T02:21:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:56 smithi120 ceph-mon[36803]: pgmap v609: 129 pgs: 129 active+clean; 56 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:21:56.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:56 smithi083 ceph-mon[36340]: pgmap v609: 129 pgs: 129 active+clean; 56 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:21:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:56 smithi028 ceph-mon[31916]: pgmap v609: 129 pgs: 129 active+clean; 56 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 409 op/s 2024-03-22T02:21:57.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 800017 11.73 MB/sec execute 279 sec latency 43.503 ms 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 -- 172.21.15.28:0/1867395597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138101ca0 msgr2=0x7fa138102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 --2- 172.21.15.28:0/1867395597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138101ca0 0x7fa138102120 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fa128009900 tx=0x7fa12802bc30 comp rx=0 tx=0).stop 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 -- 172.21.15.28:0/1867395597 shutdown_connections 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 --2- 172.21.15.28:0/1867395597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138102660 0x7fa138108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 --2- 172.21.15.28:0/1867395597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138101ca0 0x7fa138102120 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 --2- 172.21.15.28:0/1867395597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138100ab0 0x7fa138100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 -- 172.21.15.28:0/1867395597 >> 172.21.15.28:0/1867395597 conn(0x7fa1380fc020 msgr2=0x7fa1380fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 -- 172.21.15.28:0/1867395597 shutdown_connections 2024-03-22T02:21:57.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.897+0000 7fa13f80e640 1 -- 172.21.15.28:0/1867395597 wait complete. 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 Processor -- start 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 -- start start 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 0x7fa138069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138101ca0 0x7fa13819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa13810c730 con 0x7fa138102660 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa13810c5b0 con 0x7fa138100ab0 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13f80e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa13810c8b0 con 0x7fa138101ca0 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 0x7fa138069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138101ca0 0x7fa13819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 0x7fa138069d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:36128/0 (socket says 172.21.15.28:36128) 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33620/0 (socket says 172.21.15.28:33620) 2024-03-22T02:21:57.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13d5aa640 1 -- 172.21.15.28:0/3156705321 learned_addr learned my addr 172.21.15.28:0/3156705321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 -- 172.21.15.28:0/3156705321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138101ca0 msgr2=0x7fa13819cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138101ca0 0x7fa13819cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 -- 172.21.15.28:0/3156705321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 msgr2=0x7fa138069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 0x7fa138069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.898+0000 7fa13ddab640 1 -- 172.21.15.28:0/3156705321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13819fac0 con 0x7fa138102660 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa13ddab640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa130009fd0 tx=0x7fa13000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa12e7fc640 1 -- 172.21.15.28:0/3156705321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13000ce20 con 0x7fa138102660 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa12e7fc640 1 -- 172.21.15.28:0/3156705321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa130005020 con 0x7fa138102660 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa12e7fc640 1 -- 172.21.15.28:0/3156705321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130010720 con 0x7fa138102660 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13819fd50 con 0x7fa138102660 2024-03-22T02:21:57.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.899+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1381a0210 con 0x7fa138102660 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.901+0000 7fa12e7fc640 1 -- 172.21.15.28:0/3156705321 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa130005540 con 0x7fa138102660 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 Processor -- start 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 -- start start 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa12e7fc640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 0x7fa1240a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047eed0 0x562fc04839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fc03e3040 con 0x562fc047e4e0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562fc038b9e0 con 0x562fc047eed0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa1423f7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fc03deff0 con 0x562fc0484040 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 0x7fa1240a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33624/0 (socket says 172.21.15.28:33624) 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13cda9640 1 -- 172.21.15.28:0/117941979 learned_addr learned my addr 172.21.15.28:0/117941979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33652/0 (socket says 172.21.15.28:33652) 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa13cda9640 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047eed0 0x562fc04839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.902+0000 7fa12e7fc640 1 -- 172.21.15.28:0/3156705321 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa1300cfe50 con 0x7fa138102660 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 -- 172.21.15.28:0/117941979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047eed0 msgr2=0x562fc04839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047eed0 0x562fc04839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 -- 172.21.15.28:0/117941979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 msgr2=0x562fc047e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 -- 172.21.15.28:0/117941979 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc00f7210 con 0x562fc0484040 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13cda9640 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 0x7fa1240a5500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa1340058c0 tx=0x7fa134005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa13ddab640 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa1300cd8b0 tx=0x7fa130005940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a8da2b0673217e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa1237fe640 1 -- 172.21.15.28:0/117941979 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300a9780 con 0x562fc0484040 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa1237fe640 1 -- 172.21.15.28:0/117941979 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1300a5a40 con 0x562fc0484040 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.903+0000 7fa1237fe640 1 -- 172.21.15.28:0/117941979 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b0400 con 0x562fc0484040 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- 172.21.15.28:0/117941979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 msgr2=0x562fc04864d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa1300cd8b0 tx=0x7fa130005940 comp rx=0 tx=0).stop 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- 172.21.15.28:0/117941979 shutdown_connections 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc04864d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047eed0 0x562fc04839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 --2- 172.21.15.28:0/117941979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047e4e0 0x562fc047e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- 172.21.15.28:0/117941979 >> 172.21.15.28:0/117941979 conn(0x562fc0479a50 msgr2=0x562fc0479e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- 172.21.15.28:0/117941979 shutdown_connections 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- 172.21.15.28:0/117941979 wait complete. 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 -- start start 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.904+0000 7fa1423f7740 1 Processor -- start 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc0488940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc048f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562fc0486f30 con 0x562fc047eed0 2024-03-22T02:21:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562fc039edb0 con 0x562fc047e4e0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562fc04870b0 con 0x562fc0484040 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 10 client.0 _ll_register_callbacks cb 0x562fc00f6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc0488940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:36144/0 (socket says 172.21.15.28:36144) 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13d5aa640 1 -- 172.21.15.28:0/2715682563 learned_addr learned my addr 172.21.15.28:0/2715682563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13ddab640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc048f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 -- 172.21.15.28:0/2715682563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 msgr2=0x562fc048f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc048f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 -- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 msgr2=0x562fc0488400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 -- 172.21.15.28:0/2715682563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc01c3620 con 0x562fc047eed0 2024-03-22T02:21:57.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa13cda9640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc0488940 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x562fc047f2d0 tx=0x7fa128000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e7274d91ef5541 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa120ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa128003a00 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa128003ba0 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.905+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1300a5a40 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa128052050 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 10 client.17865 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562fc0492d40 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1300a5a40 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fa128040420 con 0x562fc047eed0 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 10 client.17865 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:57.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 20 client.17865 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc0494410 0x562fc0496840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 4 client.17865 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fc03f0580 con 0x562fc0494410 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc0494410 0x562fc0496840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa120ff9640 10 client.17865 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.906+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc0494410 0x562fc0496840 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=876b9655b3098baa server_cookie=de649c17c4f8db8c in_seq=0 out_seq=0 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa120ff9640 1 -- 172.21.15.28:0/2715682563 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562fc03f0580 con 0x562fc0494410 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa120ff9640 10 client.17865 handle_command_reply: tid=0 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 client.17865 shutdown 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 msgr2=0x562fc0488940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc0488940 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x562fc047f2d0 tx=0x7fa128000bc0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 shutdown_connections 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc0494410 0x562fc0496840 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc048f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc0488940 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa120ff9640 0 client.17865 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2715682563 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc0488400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 >> 172.21.15.28:0/2715682563 conn(0x562fc0479a50 msgr2=0x562fc0478c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 shutdown_connections 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.907+0000 7fa1423f7740 1 -- 172.21.15.28:0/2715682563 wait complete. 2024-03-22T02:21:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.908+0000 7fa1423f7740 20 client.17865 trim_cache size 0 max 16384 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 Processor -- start 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 -- start start 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc04783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381ad0b0 con 0x562fc047eed0 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381aa980 con 0x562fc047e4e0 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa1423f7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381a87d0 con 0x562fc0484040 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc04783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13ddab640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33668/0 (socket says 172.21.15.28:33668) 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33654/0 (socket says 172.21.15.28:33654) 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13ddab640 1 -- 172.21.15.28:0/2063635117 learned_addr learned my addr 172.21.15.28:0/2063635117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.924+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc04783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:36156/0 (socket says 172.21.15.28:36156) 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 -- 172.21.15.28:0/2063635117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 msgr2=0x562fc04783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc04783c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 -- 172.21.15.28:0/2063635117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 msgr2=0x562fc048f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 -- 172.21.15.28:0/2063635117 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fc0496ec0 con 0x562fc0484040 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13cda9640 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa13ddab640 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1300985d0 tx=0x7fa13009bbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3312c0df5e891e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa122ffd640 1 -- 172.21.15.28:0/2063635117 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b4070 con 0x562fc0484040 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa122ffd640 1 -- 172.21.15.28:0/2063635117 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1300b5270 con 0x562fc0484040 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.925+0000 7fa122ffd640 1 -- 172.21.15.28:0/2063635117 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1300a46f0 con 0x562fc0484040 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- 172.21.15.28:0/2063635117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 msgr2=0x562fc0478900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa1300985d0 tx=0x7fa13009bbd0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- 172.21.15.28:0/2063635117 shutdown_connections 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc0484040 0x562fc0478900 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- 172.21.15.28:0/2063635117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc047e4e0 0x562fc04783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- 172.21.15.28:0/2063635117 >> 172.21.15.28:0/2063635117 conn(0x562fc0479a50 msgr2=0x562fc047ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- 172.21.15.28:0/2063635117 shutdown_connections 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- 172.21.15.28:0/2063635117 wait complete. 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- start start 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 Processor -- start 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 0x562fc048e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381a8e40 con 0x562fc047eed0 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381aa2b0 con 0x562fc0484040 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.926+0000 7fa1423f7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa1381a9510 con 0x562fc047e4e0 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa1423f7740 10 client.0 _ll_register_callbacks cb 0x7fa128004440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13cda9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33680/0 (socket says 172.21.15.28:33680) 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 -- 172.21.15.28:0/444430603 learned_addr learned my addr 172.21.15.28:0/444430603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13ddab640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 0x562fc048e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 -- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 msgr2=0x562fc048e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 0x562fc048e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 -- 172.21.15.28:0/444430603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 msgr2=0x562fc048d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 -- 172.21.15.28:0/444430603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fc047a7c0 con 0x562fc047e4e0 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13cda9640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13ddab640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 0x562fc048e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:21:57.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa13d5aa640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa134009840 tx=0x7fa134013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa3b418a07ad4bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa120ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134005230 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1300b5270 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9274+0+0 (secure 0 0 0) 0x7fa1340053d0 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.927+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fa13403a050 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 10 client.27694 fetch_fsmap learned FSMap version 33 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x562fc04fa7f0 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1300b5270 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fa134026040 con 0x562fc047e4e0 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 10 client.27694 fetch_fsmap finished waiting for FSMap version 33 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 10 client.27694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:21:57.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 20 client.27694 populate_metadata read hostname 'smithi028' 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc04fb1d0 0x562fc04fd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 4 client.27694 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562fc03ea8d0 con 0x562fc04fb1d0 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.928+0000 7fa13cda9640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc04fb1d0 0x562fc04fd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.929+0000 7fa13cda9640 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc04fb1d0 0x562fc04fd5f0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38054c17e77903b3 server_cookie=7cf165cfa4e2a688 in_seq=0 out_seq=0 2024-03-22T02:21:57.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:57.929+0000 7fa120ff9640 10 client.27694 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.227+0000 7fa120ff9640 1 -- 172.21.15.28:0/444430603 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562fc03ea8d0 con 0x562fc04fb1d0 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.227+0000 7fa120ff9640 10 client.27694 handle_command_reply: tid=0 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.227+0000 7fa1423f7740 1 client.27694 shutdown 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.227+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 msgr2=0x562fc048d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.227+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa134009840 tx=0x7fa134013040 comp rx=0 tx=0).stop 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 shutdown_connections 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562fc04fb1d0 0x562fc04fd5f0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562fc0484040 0x562fc048e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562fc047eed0 0x562fc048d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 --2- 172.21.15.28:0/444430603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562fc047e4e0 0x562fc048d1d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 >> 172.21.15.28:0/444430603 conn(0x562fc0479a50 msgr2=0x562fc047f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa120ff9640 0 client.27694 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 shutdown_connections 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 1 -- 172.21.15.28:0/444430603 wait complete. 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.228+0000 7fa1423f7740 20 client.27694 trim_cache size 0 max 16384 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:21:58.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:21:58.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 msgr2=0x7fa1240a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:58.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 0x7fa1240a5500 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa1340058c0 tx=0x7fa134005f90 comp rx=0 tx=0).stop 2024-03-22T02:21:58.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 msgr2=0x7fa13819f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:21:58.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fa130009fd0 tx=0x7fa13000ef10 comp rx=0 tx=0).stop 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 shutdown_connections 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa1240a3040 0x7fa1240a5500 secure :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa1340058c0 tx=0x7fa134005f90 comp rx=0 tx=0).stop 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa138102660 0x7fa13819f1b0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa138101ca0 0x7fa13819cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 --2- 172.21.15.28:0/3156705321 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa138100ab0 0x7fa138069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 >> 172.21.15.28:0/3156705321 conn(0x7fa1380fc020 msgr2=0x7fa138104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 shutdown_connections 2024-03-22T02:21:58.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:21:58.229+0000 7fa13f80e640 1 -- 172.21.15.28:0/3156705321 wait complete. 2024-03-22T02:21:58.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 802233 11.73 MB/sec execute 280 sec latency 236.510 ms 2024-03-22T02:21:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:58 smithi120 ceph-mon[36803]: pgmap v610: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 329 op/s 2024-03-22T02:21:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:58 smithi120 ceph-mon[36803]: daemon mds.h restarted 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-mon[36340]: pgmap v610: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 329 op/s 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-mon[36340]: daemon mds.h restarted 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setuser ceph since I am not root 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setgroup ceph since I am not root 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-mds[66500]: -- 172.21.15.83:0/2172858370 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed4b511a40 con 0x55ed4b4e4800 2024-03-22T02:21:58.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:21:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: starting mds.h at 2024-03-22T02:21:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:58 smithi028 ceph-mon[31916]: pgmap v610: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.4 MiB/s wr, 329 op/s 2024-03-22T02:21:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:58 smithi028 ceph-mon[31916]: daemon mds.h restarted 2024-03-22T02:21:59.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 803470 11.71 MB/sec execute 281 sec latency 54.415 ms 2024-03-22T02:21:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:59 smithi120 ceph-mon[36803]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T02:21:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:59 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/3084572880,v1:172.21.15.83:6837/3084572880] up:boot 2024-03-22T02:21:59.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:59 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:21:59.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:59 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:21:59.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:21:59 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:21:59.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:59 smithi083 ceph-mon[36340]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T02:21:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:59 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/3084572880,v1:172.21.15.83:6837/3084572880] up:boot 2024-03-22T02:21:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:59 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:21:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:59 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:21:59.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:21:59 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:21:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:59 smithi028 ceph-mon[31916]: osdmap e96: 12 total, 12 up, 12 in 2024-03-22T02:21:59.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:59 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/3084572880,v1:172.21.15.83:6837/3084572880] up:boot 2024-03-22T02:21:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:59 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:21:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:59 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:21:59.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:21:59 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:22:00.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 806095 11.73 MB/sec execute 282 sec latency 24.289 ms 2024-03-22T02:22:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:00 smithi120 ceph-mon[36803]: pgmap v612: 129 pgs: 129 active+clean; 59 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 351 op/s 2024-03-22T02:22:00.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:00 smithi083 ceph-mon[36340]: pgmap v612: 129 pgs: 129 active+clean; 59 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 351 op/s 2024-03-22T02:22:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:00 smithi028 ceph-mon[31916]: pgmap v612: 129 pgs: 129 active+clean; 59 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 351 op/s 2024-03-22T02:22:01.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 808323 11.71 MB/sec execute 283 sec latency 31.641 ms 2024-03-22T02:22:01.990 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:22:01.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 -- 172.21.15.28:0/2725233139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 msgr2=0x7f6ee8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 --2- 172.21.15.28:0/2725233139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee8102100 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009900 tx=0x7f6ed802bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 -- 172.21.15.28:0/2725233139 shutdown_connections 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 --2- 172.21.15.28:0/2725233139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8102640 0x7f6ee8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 --2- 172.21.15.28:0/2725233139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee8102100 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 --2- 172.21.15.28:0/2725233139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8100a90 0x7f6ee8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 -- 172.21.15.28:0/2725233139 >> 172.21.15.28:0/2725233139 conn(0x7f6ee80fc020 msgr2=0x7f6ee80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 -- 172.21.15.28:0/2725233139 shutdown_connections 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.099+0000 7f6eef284640 1 -- 172.21.15.28:0/2725233139 wait complete. 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 Processor -- start 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 -- start start 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8100a90 0x7f6ee8069880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8102640 0x7f6ee819f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee810c710 con 0x7f6ee8101c80 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee810c590 con 0x7f6ee8102640 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eef284640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee810c890 con 0x7f6ee8100a90 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eed020640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8100a90 0x7f6ee8069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39970/0 (socket says 172.21.15.28:39970) 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eed821640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8102640 0x7f6ee819f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 -- 172.21.15.28:0/3305978336 learned_addr learned my addr 172.21.15.28:0/3305978336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 -- 172.21.15.28:0/3305978336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8100a90 msgr2=0x7f6ee8069880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8100a90 0x7f6ee8069880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 -- 172.21.15.28:0/3305978336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8102640 msgr2=0x7f6ee819f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8102640 0x7f6ee819f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 -- 172.21.15.28:0/3305978336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee819fae0 con 0x7f6ee8101c80 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.100+0000 7f6eec81f640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009230 tx=0x7f6ed802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.101+0000 7f6ede7fc640 1 -- 172.21.15.28:0/3305978336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8041070 con 0x7f6ee8101c80 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.101+0000 7f6ede7fc640 1 -- 172.21.15.28:0/3305978336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed8004450 con 0x7f6ee8101c80 2024-03-22T02:22:02.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.101+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee819fd70 con 0x7f6ee8101c80 2024-03-22T02:22:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.101+0000 7f6ede7fc640 1 -- 172.21.15.28:0/3305978336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed8003e20 con 0x7f6ee8101c80 2024-03-22T02:22:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.101+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ee81a0200 con 0x7f6ee8101c80 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.103+0000 7f6ede7fc640 1 -- 172.21.15.28:0/3305978336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6ed8044070 con 0x7f6ee8101c80 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 Processor -- start 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ede7fc640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 0x7f6ed40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 -- start start 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eed821640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 0x7f6ed40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740dc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ede7fc640 1 -- 172.21.15.28:0/3305978336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6ed80f1370 con 0x7f6ee8101c80 2024-03-22T02:22:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d740398c0 con 0x556d740d4e20 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556d73fe1c20 con 0x556d740d4430 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d74035870 con 0x556d740d9f90 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39978/0 (socket says 172.21.15.28:39978) 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 -- 172.21.15.28:0/478649423 learned_addr learned my addr 172.21.15.28:0/478649423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eed020640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eed821640 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 -- 172.21.15.28:0/478649423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 msgr2=0x556d740dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 -- 172.21.15.28:0/478649423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 msgr2=0x556d740d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eec81f640 1 -- 172.21.15.28:0/478649423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d74006150 con 0x556d740d4e20 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.104+0000 7f6eed821640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 0x7f6ed40a53e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0005fd0 tx=0x7f6ee0011bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6eec81f640 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80bd440 tx=0x7f6ed8032fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e94dbe2e3ab4bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6ed37fe640 1 -- 172.21.15.28:0/478649423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed804b040 con 0x556d740d4e20 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6ed37fe640 1 -- 172.21.15.28:0/478649423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed80bd740 con 0x556d740d4e20 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6ed37fe640 1 -- 172.21.15.28:0/478649423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80035f0 con 0x556d740d4e20 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/478649423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 msgr2=0x556d740d9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.105+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80bd440 tx=0x7f6ed8032fc0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/478649423 shutdown_connections 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740d9920 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/478649423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/478649423 >> 172.21.15.28:0/478649423 conn(0x556d740cf9a0 msgr2=0x556d740cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/478649423 shutdown_connections 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/478649423 wait complete. 2024-03-22T02:22:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- start start 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 Processor -- start 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d4430 0x556d740e29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d9f90 0x556d740e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d740398c0 con 0x556d740d4e20 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556d73ff5130 con 0x556d740d9f90 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.106+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556d740dcd00 con 0x556d740d4430 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ef1e6c740 10 client.0 _ll_register_callbacks cb 0x556d73ff4b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39984/0 (socket says 172.21.15.28:39984) 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 -- 172.21.15.28:0/1420587417 learned_addr learned my addr 172.21.15.28:0/1420587417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eed821640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d9f90 0x556d740e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eed020640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d4430 0x556d740e29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 -- 172.21.15.28:0/1420587417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d4430 msgr2=0x556d740e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d4430 0x556d740e29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 -- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d9f90 msgr2=0x556d740e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d9f90 0x556d740e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 -- 172.21.15.28:0/1420587417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d73e177d0 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ba110 tx=0x7f6ed8005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a648d9215c5ee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed804b050 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6ed80f1170 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed80bd740 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6ed80bb7e0 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ef1e6c740 10 client.17889 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556d73d4d3f0 con 0x556d740d4e20 2024-03-22T02:22:02.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.107+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed80bd740 con 0x556d740d4e20 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f6ed80d1ce0 con 0x556d740d4e20 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 10 client.17889 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 10 client.17889 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 20 client.17889 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d740ee630 0x556d740f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 4 client.17889 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d74046e00 con 0x556d740ee630 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6eed020640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d740ee630 0x556d740f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6ed0ff9640 10 client.17889 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.108+0000 7f6eed020640 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d740ee630 0x556d740f0a20 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5cd9fd4c63e00d7 server_cookie=f0ea939dad5c0d48 in_seq=0 out_seq=0 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1420587417 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d74046e00 con 0x556d740ee630 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ed0ff9640 10 client.17889 handle_command_reply: tid=0 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 client.17889 shutdown 2024-03-22T02:22:02.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 msgr2=0x556d740e2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80ba110 tx=0x7f6ed8005770 comp rx=0 tx=0).stop 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 shutdown_connections 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d740ee630 0x556d740f0a20 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d9f90 0x556d740e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e2f30 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1420587417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d4430 0x556d740e29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ed0ff9640 0 client.17889 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 >> 172.21.15.28:0/1420587417 conn(0x556d740cf9a0 msgr2=0x556d740cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 shutdown_connections 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1420587417 wait complete. 2024-03-22T02:22:02.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.109+0000 7f6ef1e6c740 20 client.17889 trim_cache size 0 max 16384 2024-03-22T02:22:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 Processor -- start 2024-03-22T02:22:02.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 -- start start 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 0x556d740e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81ab070 con 0x556d740d4430 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81a8940 con 0x556d740d4e20 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81a6790 con 0x556d740d9f90 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eed020640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 0x556d740e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eed020640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39992/0 (socket says 172.21.15.28:39992) 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eed821640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eed020640 1 -- 172.21.15.28:0/751663331 learned_addr learned my addr 172.21.15.28:0/751663331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eec81f640 1 -- 172.21.15.28:0/751663331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 msgr2=0x556d740e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eec81f640 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.127+0000 7f6eec81f640 1 -- 172.21.15.28:0/751663331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 msgr2=0x556d740e2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6eec81f640 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6eec81f640 1 -- 172.21.15.28:0/751663331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d73d0d0f0 con 0x556d740d4e20 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6eed020640 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6eec81f640 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 0x556d740e9f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80eedd0 tx=0x7f6ed80057e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0f1777a15f2a98b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6ed2ffd640 1 -- 172.21.15.28:0/751663331 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed804b070 con 0x556d740d4e20 2024-03-22T02:22:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6ed2ffd640 1 -- 172.21.15.28:0/751663331 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed80bd740 con 0x556d740d4e20 2024-03-22T02:22:02.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.128+0000 7f6ed2ffd640 1 -- 172.21.15.28:0/751663331 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed80bba40 con 0x556d740d4e20 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/751663331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 msgr2=0x556d740e9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 0x556d740e9f30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6ed80eedd0 tx=0x7f6ed80057e0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/751663331 shutdown_connections 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4e20 0x556d740e9f30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/751663331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4430 0x556d740e2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/751663331 >> 172.21.15.28:0/751663331 conn(0x556d740cf9a0 msgr2=0x556d740e1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/751663331 shutdown_connections 2024-03-22T02:22:02.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/751663331 wait complete. 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- start start 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 Processor -- start 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81a6e00 con 0x556d740d4e20 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81a8270 con 0x556d740d4430 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee81a74d0 con 0x556d740d9f90 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6ef1e6c740 10 client.0 _ll_register_callbacks cb 0x7f6ed80bd510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.129+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eec81f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39998/0 (socket says 172.21.15.28:39998) 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eec81f640 1 -- 172.21.15.28:0/1865765964 learned_addr learned my addr 172.21.15.28:0/1865765964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed821640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 -- 172.21.15.28:0/1865765964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 msgr2=0x556d740e8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 -- 172.21.15.28:0/1865765964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 msgr2=0x556d740e7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 -- 172.21.15.28:0/1865765964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d740e90e0 con 0x556d740d4430 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6eed020640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740e75f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4000c00 tx=0x7f6ee4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79524be0ae9fd6bb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ed0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee4005020 con 0x556d740d4430 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ed80bd740 con 0x556d740d4430 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6ee40051c0 con 0x556d740d4430 2024-03-22T02:22:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6ee4030050 con 0x556d740d4430 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ef1e6c740 10 client.27719 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.130+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556d741548e0 con 0x556d740d4430 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ed80bd740 con 0x556d740d4430 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f6ee400f620 con 0x556d740d4430 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 10 client.27719 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 10 client.27719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 20 client.27719 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d741552c0 0x556d741576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 4 client.27719 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556d740e27e0 con 0x556d741552c0 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d741552c0 0x556d741576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ed0ff9640 10 client.27719 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6eec81f640 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d741552c0 0x556d741576b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149816fb47b9adfa server_cookie=f1974c91c1eea227 in_seq=0 out_seq=0 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ed0ff9640 1 -- 172.21.15.28:0/1865765964 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556d740e27e0 con 0x556d741552c0 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.131+0000 7f6ed0ff9640 10 client.27719 handle_command_reply: tid=0 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 client.27719 shutdown 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 msgr2=0x556d740e75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740e75f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4000c00 tx=0x7f6ee4005850 comp rx=0 tx=0).stop 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 shutdown_connections 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556d741552c0 0x556d741576b0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556d740d9f90 0x556d740e8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ed0ff9640 0 client.27719 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d740d4e20 0x556d740e7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 --2- 172.21.15.28:0/1865765964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556d740d4430 0x556d740e75f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 >> 172.21.15.28:0/1865765964 conn(0x556d740cf9a0 msgr2=0x556d740d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 shutdown_connections 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 1 -- 172.21.15.28:0/1865765964 wait complete. 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6ef1e6c740 20 client.27719 trim_cache size 0 max 16384 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:22:02.141 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:02.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 msgr2=0x7f6ed40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 0x7f6ed40a53e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ee0005fd0 tx=0x7f6ee0011bc0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 msgr2=0x7f6ee819cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f6ed8009230 tx=0x7f6ed802c650 comp rx=0 tx=0).stop 2024-03-22T02:22:02.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 shutdown_connections 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6ed40a2f20 0x7f6ed40a53e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6ee8102640 0x7f6ee819f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ee8101c80 0x7f6ee819cbe0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 --2- 172.21.15.28:0/3305978336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6ee8100a90 0x7f6ee8069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.132+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 >> 172.21.15.28:0/3305978336 conn(0x7f6ee80fc020 msgr2=0x7f6ee8104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.133+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 shutdown_connections 2024-03-22T02:22:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.133+0000 7f6eef284640 1 -- 172.21.15.28:0/3305978336 wait complete. 2024-03-22T02:22:02.144 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:22:02.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:22:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- 172.21.15.28:0/715447254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784107890 msgr2=0x7fe784109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- 172.21.15.28:0/715447254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784107890 0x7fe784109ce0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009ca0 tx=0x7fe77c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- 172.21.15.28:0/715447254 shutdown_connections 2024-03-22T02:22:02.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- 172.21.15.28:0/715447254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784107890 0x7fe784109ce0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- 172.21.15.28:0/715447254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784104f60 0x7fe784107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- 172.21.15.28:0/715447254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784102630 0x7fe784104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- 172.21.15.28:0/715447254 >> 172.21.15.28:0/715447254 conn(0x7fe7840fc020 msgr2=0x7fe7840fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- 172.21.15.28:0/715447254 shutdown_connections 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- 172.21.15.28:0/715447254 wait complete. 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 Processor -- start 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- start start 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784102630 0x7fe784197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784107890 0x7fe78419f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410d850 con 0x7fe784104f60 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410d6d0 con 0x7fe784102630 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe78ad81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe78410d9d0 con 0x7fe784107890 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe783fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.249+0000 7fe788b1d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784102630 0x7fe784197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40014/0 (socket says 172.21.15.28:40014) 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 -- 172.21.15.28:0/630817459 learned_addr learned my addr 172.21.15.28:0/630817459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe78931e640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784107890 0x7fe78419f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 -- 172.21.15.28:0/630817459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784107890 msgr2=0x7fe78419f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784107890 0x7fe78419f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 -- 172.21.15.28:0/630817459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784102630 msgr2=0x7fe784197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784102630 0x7fe784197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 -- 172.21.15.28:0/630817459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78419fac0 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe783fff640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e9c0 tx=0x7fe77000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77000cd80 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe770005500 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe770010690 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78419fdb0 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.250+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7841a0270 con 0x7fe784104f60 2024-03-22T02:22:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.251+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7840fdb10 con 0x7fe784104f60 2024-03-22T02:22:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.252+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe7700046c0 con 0x7fe784104f60 2024-03-22T02:22:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.253+0000 7fe781ffb640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 0x7fe7740a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.253+0000 7fe788b1d640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 0x7fe7740a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.253+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe7700cfb40 con 0x7fe784104f60 2024-03-22T02:22:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.253+0000 7fe788b1d640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 0x7fe7740a53e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe778009bd0 tx=0x7fe778005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.258+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe77009c560 con 0x7fe784104f60 2024-03-22T02:22:02.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 810050 11.72 MB/sec execute 284 sec latency 16.097 ms 2024-03-22T02:22:02.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.496+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe784062890 con 0x7fe784104f60 2024-03-22T02:22:02.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.499+0000 7fe781ffb640 1 -- 172.21.15.28:0/630817459 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11866 (secure 0 0 0) 0x7fe77009be60 con 0x7fe784104f60 2024-03-22T02:22:02.499 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:22:02.499 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:21:58.483135+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:22:02.500 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-03-22T02:22:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 msgr2=0x7fe7740a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 0x7fe7740a53e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe778009bd0 tx=0x7fe778005f90 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 msgr2=0x7fe7841983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe77000e9c0 tx=0x7fe77000ee80 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 shutdown_connections 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe7740a2f20 0x7fe7740a53e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe784107890 0x7fe78419f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe784104f60 0x7fe7841983e0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 --2- 172.21.15.28:0/630817459 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe784102630 0x7fe784197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 >> 172.21.15.28:0/630817459 conn(0x7fe7840fc020 msgr2=0x7fe784102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 shutdown_connections 2024-03-22T02:22:02.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.501+0000 7fe78ad81640 1 -- 172.21.15.28:0/630817459 wait complete. 2024-03-22T02:22:02.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:22:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 -- 172.21.15.28:0/2347763315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190100a90 msgr2=0x7f5190100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 --2- 172.21.15.28:0/2347763315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190100a90 0x7f5190100eb0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5180009900 tx=0x7f518002bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 -- 172.21.15.28:0/2347763315 shutdown_connections 2024-03-22T02:22:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 --2- 172.21.15.28:0/2347763315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190102640 0x7f5190108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 --2- 172.21.15.28:0/2347763315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 0x7f5190102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 --2- 172.21.15.28:0/2347763315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190100a90 0x7f5190100eb0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 -- 172.21.15.28:0/2347763315 >> 172.21.15.28:0/2347763315 conn(0x7f51900fc020 msgr2=0x7f51900fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 -- 172.21.15.28:0/2347763315 shutdown_connections 2024-03-22T02:22:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.624+0000 7f519509a640 1 -- 172.21.15.28:0/2347763315 wait complete. 2024-03-22T02:22:02.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 Processor -- start 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 -- start start 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 0x7f5190198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 0x7f519019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c700 con 0x7f5190102640 2024-03-22T02:22:02.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c580 con 0x7f5190100a90 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.625+0000 7f519509a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f519010c880 con 0x7f5190101c80 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518f59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 0x7f519019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33090/0 (socket says 172.21.15.28:33090) 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 -- 172.21.15.28:0/48394546 learned_addr learned my addr 172.21.15.28:0/48394546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518e59c640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 0x7f5190198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 -- 172.21.15.28:0/48394546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 msgr2=0x7f5190198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 0x7f5190198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 -- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 msgr2=0x7f519019f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 0x7f519019f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 -- 172.21.15.28:0/48394546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519019fb00 con 0x7f5190100a90 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518f59e640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 0x7f519019f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f518ed9d640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5180000c00 tx=0x7f518002c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f5177fff640 1 -- 172.21.15.28:0/48394546 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180041070 con 0x7f5190100a90 2024-03-22T02:22:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519019fd90 con 0x7f5190100a90 2024-03-22T02:22:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f5177fff640 1 -- 172.21.15.28:0/48394546 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5180004450 con 0x7f5190100a90 2024-03-22T02:22:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f5177fff640 1 -- 172.21.15.28:0/48394546 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180034440 con 0x7f5190100a90 2024-03-22T02:22:02.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.626+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51901a0210 con 0x7f5190100a90 2024-03-22T02:22:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.628+0000 7f5177fff640 1 -- 172.21.15.28:0/48394546 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5180044070 con 0x7f5190100a90 2024-03-22T02:22:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5177fff640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 0x7f51780a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 Processor -- start 2024-03-22T02:22:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 -- start start 2024-03-22T02:22:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5177fff640 1 -- 172.21.15.28:0/48394546 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f51800f12a0 con 0x7f5190100a90 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f518e59c640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 0x7f51780a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a8b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a8dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7c09e68d0 con 0x55f7c0a85c30 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f7c098ee00 con 0x55f7c0a8b790 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f5197922740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f7c09e2880 con 0x55f7c0a86620 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.629+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40038/0 (socket says 172.21.15.28:40038) 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 -- 172.21.15.28:0/3926661130 learned_addr learned my addr 172.21.15.28:0/3926661130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518f59e640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a8dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518e59c640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a8b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 -- 172.21.15.28:0/3926661130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 msgr2=0x55f7c0a8b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518e59c640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 0x7f51780a53e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51901993f0 tx=0x7f5184011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a8b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 -- 172.21.15.28:0/3926661130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 msgr2=0x55f7c0a8dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a8dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 -- 172.21.15.28:0/3926661130 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7c06fa3f0 con 0x55f7c0a85c30 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518f59e640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a8dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f518ed9d640 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f51800bd400 tx=0x7f5180005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cda125a0e6f13ea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f51757fa640 1 -- 172.21.15.28:0/3926661130 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518004b070 con 0x55f7c0a85c30 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f51757fa640 1 -- 172.21.15.28:0/3926661130 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51800eee50 con 0x55f7c0a85c30 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.630+0000 7f51757fa640 1 -- 172.21.15.28:0/3926661130 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51800bba00 con 0x55f7c0a85c30 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- 172.21.15.28:0/3926661130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 msgr2=0x55f7c0a86050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f51800bd400 tx=0x7f5180005e50 comp rx=0 tx=0).stop 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- 172.21.15.28:0/3926661130 shutdown_connections 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a8dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a8b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 --2- 172.21.15.28:0/3926661130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a86050 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- 172.21.15.28:0/3926661130 >> 172.21.15.28:0/3926661130 conn(0x55f7c0a7cd90 msgr2=0x55f7c0a7d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- 172.21.15.28:0/3926661130 shutdown_connections 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- 172.21.15.28:0/3926661130 wait complete. 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 -- start start 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.631+0000 7f5197922740 1 Processor -- start 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a85c30 0x55f7c0a8feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a97430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f7c0a8e500 con 0x55f7c0a86620 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f7c09a2300 con 0x55f7c0a8b790 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f7c0a8e680 con 0x55f7c0a85c30 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 10 client.0 _ll_register_callbacks cb 0x55f7c09dde30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a85c30 0x55f7c0a8feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40050/0 (socket says 172.21.15.28:40050) 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a97430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 -- 172.21.15.28:0/2613701030 learned_addr learned my addr 172.21.15.28:0/2613701030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 -- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a85c30 msgr2=0x55f7c0a8feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a85c30 0x55f7c0a8feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 -- 172.21.15.28:0/2613701030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 msgr2=0x55f7c0a97430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a97430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 -- 172.21.15.28:0/2613701030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7c07c47d0 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f518e59c640 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f518400fad0 tx=0x7f5184004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc846f4a433e1999 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5172ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5184017040 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f518401e6d0 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.632+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51800eee50 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f518401e8d0 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 10 client.17913 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f7c0a9a810 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51800eee50 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f5184025070 con 0x55f7c0a86620 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 10 client.17913 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 10 client.17913 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 20 client.17913 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a9bfa0 0x55f7c0a9e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 4 client.17913 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7c09f3e10 con 0x55f7c0a9bfa0 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.633+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a9bfa0 0x55f7c0a9e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5172ff5640 10 client.17913 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a9bfa0 0x55f7c0a9e3c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4a7433c4fc42b7c server_cookie=3706d566672ddedf in_seq=0 out_seq=0 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5172ff5640 1 -- 172.21.15.28:0/2613701030 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f7c09f3e10 con 0x55f7c0a9bfa0 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5172ff5640 10 client.17913 handle_command_reply: tid=0 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 client.17913 shutdown 2024-03-22T02:22:02.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 msgr2=0x55f7c0a903f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f518400fad0 tx=0x7f5184004730 comp rx=0 tx=0).stop 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 shutdown_connections 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a9bfa0 0x55f7c0a9e3c0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a8b790 0x55f7c0a97430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a86620 0x55f7c0a903f0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 --2- 172.21.15.28:0/2613701030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a85c30 0x55f7c0a8feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.634+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 >> 172.21.15.28:0/2613701030 conn(0x55f7c0a7cd90 msgr2=0x55f7c0a7bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.635+0000 7f5172ff5640 0 client.17913 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.635+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 shutdown_connections 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.635+0000 7f5197922740 1 -- 172.21.15.28:0/2613701030 wait complete. 2024-03-22T02:22:02.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.635+0000 7f5197922740 20 client.17913 trim_cache size 0 max 16384 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 Processor -- start 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 -- start start 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a7c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 0x55f7c0aa2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51901ab130 con 0x55f7c0a85c30 2024-03-22T02:22:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51901a8a00 con 0x55f7c0a86620 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.648+0000 7f5197922740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51901a6870 con 0x55f7c0a7cd90 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40058/0 (socket says 172.21.15.28:40058) 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 0x55f7c0aa2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518e59c640 1 -- 172.21.15.28:0/2970892377 learned_addr learned my addr 172.21.15.28:0/2970892377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a7c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 -- 172.21.15.28:0/2970892377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a7cd90 msgr2=0x55f7c0a7c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a7c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 -- 172.21.15.28:0/2970892377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 msgr2=0x55f7c0a973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 -- 172.21.15.28:0/2970892377 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7c0a8aec0 con 0x55f7c0a86620 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518e59c640 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f518f59e640 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 0x55f7c0aa2cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5188009fd0 tx=0x7f5188009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a06ae3f71448fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f5174ff9640 1 -- 172.21.15.28:0/2970892377 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188012a40 con 0x55f7c0a86620 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f5174ff9640 1 -- 172.21.15.28:0/2970892377 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5188003e50 con 0x55f7c0a86620 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.649+0000 7f5174ff9640 1 -- 172.21.15.28:0/2970892377 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5188004700 con 0x55f7c0a86620 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- 172.21.15.28:0/2970892377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 msgr2=0x55f7c0aa2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 0x55f7c0aa2cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5188009fd0 tx=0x7f5188009500 comp rx=0 tx=0).stop 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- 172.21.15.28:0/2970892377 shutdown_connections 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a86620 0x55f7c0aa2cc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 --2- 172.21.15.28:0/2970892377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a7c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- 172.21.15.28:0/2970892377 >> 172.21.15.28:0/2970892377 conn(0x55f7c0a8fa70 msgr2=0x55f7c0a80880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- 172.21.15.28:0/2970892377 shutdown_connections 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- 172.21.15.28:0/2970892377 wait complete. 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 -- start start 2024-03-22T02:22:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.650+0000 7f5197922740 1 Processor -- start 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a95700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51901a6ec0 con 0x55f7c0a85c30 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51901a8330 con 0x55f7c0a7cd90 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51901a7590 con 0x55f7c0a86620 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 10 client.0 _ll_register_callbacks cb 0x7f518400f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40064/0 (socket says 172.21.15.28:40064) 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33122/0 (socket says 172.21.15.28:33122) 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 -- 172.21.15.28:0/2997400328 learned_addr learned my addr 172.21.15.28:0/2997400328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518f59e640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a95700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 -- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 msgr2=0x55f7c0a95700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a95700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 -- 172.21.15.28:0/2997400328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 msgr2=0x55f7c0a96f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 -- 172.21.15.28:0/2997400328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7c0a95f30 con 0x55f7c0a85c30 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f518e59c640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5184011940 tx=0x7f518400fa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3043acb794f28859 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5172ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518403b060 con 0x55f7c0a85c30 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f518401e250 con 0x55f7c0a85c30 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.651+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5188003e50 con 0x55f7c0a85c30 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5184042050 con 0x55f7c0a85c30 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 10 client.17919 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f7c0b02f20 con 0x55f7c0a85c30 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5188003e50 con 0x55f7c0a85c30 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f5184035070 con 0x55f7c0a85c30 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 10 client.17919 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 10 client.17919 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 20 client.17919 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a8b790 0x55f7c0b05910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 4 client.17919 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7c09ee160 con 0x55f7c0a8b790 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.652+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a8b790 0x55f7c0b05910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5172ff5640 10 client.17919 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f518ed9d640 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a8b790 0x55f7c0b05910 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f9108ac7f5d0476d server_cookie=825dbef3a7d2a18 in_seq=0 out_seq=0 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5172ff5640 1 -- 172.21.15.28:0/2997400328 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7c09ee160 con 0x55f7c0a8b790 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5172ff5640 10 client.17919 handle_command_reply: tid=0 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 client.17919 shutdown 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 msgr2=0x55f7c0a97460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5184011940 tx=0x7f518400fa90 comp rx=0 tx=0).stop 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 shutdown_connections 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f7c0a86620 0x55f7c0a95700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f7c0a85c30 0x55f7c0a97460 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55f7c0a8b790 0x55f7c0b05910 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 --2- 172.21.15.28:0/2997400328 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f7c0a7cd90 0x55f7c0a96f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5172ff5640 0 client.17919 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 >> 172.21.15.28:0/2997400328 conn(0x55f7c0a8fa70 msgr2=0x55f7c0a7b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 shutdown_connections 2024-03-22T02:22:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 1 -- 172.21.15.28:0/2997400328 wait complete. 2024-03-22T02:22:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.653+0000 7f5197922740 20 client.17919 trim_cache size 0 max 16384 2024-03-22T02:22:02.661 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 msgr2=0x7f51780a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 0x7f51780a53e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f51901993f0 tx=0x7f5184011440 comp rx=0 tx=0).stop 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 msgr2=0x7f5190197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5180000c00 tx=0x7f518002c640 comp rx=0 tx=0).stop 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 shutdown_connections 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51780a2f20 0x7f51780a53e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5190102640 0x7f519019f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5190101c80 0x7f5190198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 --2- 172.21.15.28:0/48394546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5190100a90 0x7f5190197ee0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 >> 172.21.15.28:0/48394546 conn(0x7f51900fc020 msgr2=0x7f5190068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 shutdown_connections 2024-03-22T02:22:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.654+0000 7f519509a640 1 -- 172.21.15.28:0/48394546 wait complete. 2024-03-22T02:22:02.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:22:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:02 smithi120 ceph-mon[36803]: pgmap v613: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-22T02:22:02.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:02 smithi083 ceph-mon[36340]: pgmap v613: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 -- 172.21.15.28:0/2654331648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 msgr2=0x7f3920102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 --2- 172.21.15.28:0/2654331648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f3920102100 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3914009900 tx=0x7f391402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 -- 172.21.15.28:0/2654331648 shutdown_connections 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 --2- 172.21.15.28:0/2654331648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f3920108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 --2- 172.21.15.28:0/2654331648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f3920102100 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 --2- 172.21.15.28:0/2654331648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 0x7f3920100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 -- 172.21.15.28:0/2654331648 >> 172.21.15.28:0/2654331648 conn(0x7f39200fc020 msgr2=0x7f39200fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 -- 172.21.15.28:0/2654331648 shutdown_connections 2024-03-22T02:22:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 -- 172.21.15.28:0/2654331648 wait complete. 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.778+0000 7f3926866640 1 Processor -- start 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 -- start start 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 0x7f39200698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f392019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f392010c6f0 con 0x7f3920101c80 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f392010c570 con 0x7f3920102640 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3926866640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f392010c870 con 0x7f3920100a90 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40088/0 (socket says 172.21.15.28:40088) 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 -- 172.21.15.28:0/3362705118 learned_addr learned my addr 172.21.15.28:0/3362705118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3924e03640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f392019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391ffff640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 0x7f39200698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 -- 172.21.15.28:0/3362705118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 msgr2=0x7f39200698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 0x7f39200698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 -- 172.21.15.28:0/3362705118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 msgr2=0x7f392019f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f392019f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 -- 172.21.15.28:0/3362705118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392019fb00 con 0x7f3920101c80 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f3924e03640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f392019f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.779+0000 7f391f7fe640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3914009230 tx=0x7f3914005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.780+0000 7f391d7fa640 1 -- 172.21.15.28:0/3362705118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140044d0 con 0x7f3920101c80 2024-03-22T02:22:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.780+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39201969a0 con 0x7f3920101c80 2024-03-22T02:22:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.780+0000 7f391d7fa640 1 -- 172.21.15.28:0/3362705118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39140054e0 con 0x7f3920101c80 2024-03-22T02:22:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.780+0000 7f391d7fa640 1 -- 172.21.15.28:0/3362705118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914003850 con 0x7f3920101c80 2024-03-22T02:22:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.780+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3920196e90 con 0x7f3920101c80 2024-03-22T02:22:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.782+0000 7f391d7fa640 1 -- 172.21.15.28:0/3362705118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f391403d640 con 0x7f3920101c80 2024-03-22T02:22:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.782+0000 7f3929449740 1 Processor -- start 2024-03-22T02:22:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.782+0000 7f3929449740 1 -- start start 2024-03-22T02:22:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.782+0000 7f391d7fa640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 0x7f39080a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb71210 0x55c3cfb75d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391d7fa640 1 -- 172.21.15.28:0/3362705118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f39140f53d0 con 0x7f3920101c80 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb78810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfad58d0 con 0x55c3cfb70820 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfa7de00 con 0x55c3cfb76380 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3929449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfad1880 con 0x55c3cfb71210 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3924e03640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 0x7f39080a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40090/0 (socket says 172.21.15.28:40090) 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 -- 172.21.15.28:0/3594849309 learned_addr learned my addr 172.21.15.28:0/3594849309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3924e03640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb78810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391f7fe640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb71210 0x55c3cfb75d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 -- 172.21.15.28:0/3594849309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb71210 msgr2=0x55c3cfb75d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb71210 0x55c3cfb75d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 -- 172.21.15.28:0/3594849309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 msgr2=0x55c3cfb78810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb78810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 -- 172.21.15.28:0/3594849309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3cf7e93f0 con 0x55c3cfb70820 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3924e03640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb78810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f3924e03640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 0x7f39080a5580 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f39180098b0 tx=0x7f391800eee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f391ffff640 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3910009ca0 tx=0x7f3910011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483420820bb53c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.783+0000 7f39067fc640 1 -- 172.21.15.28:0/3594849309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910009e50 con 0x55c3cfb70820 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f39067fc640 1 -- 172.21.15.28:0/3594849309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39100035f0 con 0x55c3cfb70820 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f39067fc640 1 -- 172.21.15.28:0/3594849309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910019650 con 0x55c3cfb70820 2024-03-22T02:22:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 -- 172.21.15.28:0/3594849309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 msgr2=0x55c3cfb70c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f3910009ca0 tx=0x7f3910011810 comp rx=0 tx=0).stop 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 -- 172.21.15.28:0/3594849309 shutdown_connections 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb78810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb71210 0x55c3cfb75d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 --2- 172.21.15.28:0/3594849309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb70820 0x55c3cfb70c40 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 -- 172.21.15.28:0/3594849309 >> 172.21.15.28:0/3594849309 conn(0x55c3cfb6bd90 msgr2=0x55c3cfb6c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 -- 172.21.15.28:0/3594849309 shutdown_connections 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.784+0000 7f3929449740 1 -- 172.21.15.28:0/3594849309 wait complete. 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 -- start start 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 Processor -- start 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb82270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfb79330 con 0x55c3cfb71210 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfa91300 con 0x55c3cfb76380 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c3cfb794b0 con 0x55c3cfb70820 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3929449740 10 client.0 _ll_register_callbacks cb 0x55c3cfacce30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f391ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40096/0 (socket says 172.21.15.28:40096) 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f391f7fe640 1 -- 172.21.15.28:0/2796611463 learned_addr learned my addr 172.21.15.28:0/2796611463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.785+0000 7f3924e03640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb82270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 -- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 msgr2=0x55c3cfb7acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 -- 172.21.15.28:0/2796611463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 msgr2=0x55c3cfb82270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb82270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 -- 172.21.15.28:0/2796611463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3cf8b37d0 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391ffff640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f391f7fe640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3914009430 tx=0x7f391403dba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0984eda3c8515d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140d9070 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f39140bfa90 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39100035f0 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f39140c1e90 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 10 client.17931 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c3cfb79330 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39100035f0 con 0x55c3cfb71210 2024-03-22T02:22:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f3914044070 con 0x55c3cfb71210 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 10 client.17931 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 10 client.17931 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 20 client.17931 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfb86d10 0x55c3cfb89160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 4 client.17931 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.786+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3cfae2e10 con 0x55c3cfb86d10 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.787+0000 7f391ffff640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfb86d10 0x55c3cfb89160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.787+0000 7f391ffff640 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfb86d10 0x55c3cfb89160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f5114136ab826b0 server_cookie=57f2dbb132a0b69f in_seq=0 out_seq=0 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.787+0000 7f38ff7fe640 10 client.17931 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f38ff7fe640 1 -- 172.21.15.28:0/2796611463 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c3cfae2e10 con 0x55c3cfb86d10 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f38ff7fe640 10 client.17931 handle_command_reply: tid=0 2024-03-22T02:22:02.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 client.17931 shutdown 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 msgr2=0x55c3cfb7b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f3914009430 tx=0x7f391403dba0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 shutdown_connections 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfb86d10 0x55c3cfb89160 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb76380 0x55c3cfb82270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f38ff7fe640 0 client.17931 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb7b230 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 --2- 172.21.15.28:0/2796611463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 >> 172.21.15.28:0/2796611463 conn(0x55c3cfb6bd90 msgr2=0x55c3cfb6afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 shutdown_connections 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 1 -- 172.21.15.28:0/2796611463 wait complete. 2024-03-22T02:22:02.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.788+0000 7f3929449740 20 client.17931 trim_cache size 0 max 16384 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 Processor -- start 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 -- start start 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 0x55c3cfb6f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb76380 0x55c3cfb8da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39201ab0e0 con 0x55c3cfb71210 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39201a89b0 con 0x55c3cfb70820 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3929449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39201a6800 con 0x55c3cfb76380 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 0x55c3cfb6f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40106/0 (socket says 172.21.15.28:40106) 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 -- 172.21.15.28:0/92830567 learned_addr learned my addr 172.21.15.28:0/92830567 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f3924e03640 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb76380 0x55c3cfb8da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 -- 172.21.15.28:0/92830567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb76380 msgr2=0x55c3cfb8da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb76380 0x55c3cfb8da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 -- 172.21.15.28:0/92830567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 msgr2=0x55c3cfb6f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 0x55c3cfb6f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391f7fe640 1 -- 172.21.15.28:0/92830567 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3cfb715f0 con 0x55c3cfb71210 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.803+0000 7f391ffff640 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 0x55c3cfb6f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f391f7fe640 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f391402c7b0 tx=0x7f39140c12d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32cb8d88d87f33e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3905ffb640 1 -- 172.21.15.28:0/92830567 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140d9070 con 0x55c3cfb71210 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3905ffb640 1 -- 172.21.15.28:0/92830567 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3914004c80 con 0x55c3cfb71210 2024-03-22T02:22:02.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3905ffb640 1 -- 172.21.15.28:0/92830567 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39140bfa90 con 0x55c3cfb71210 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 -- 172.21.15.28:0/92830567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 msgr2=0x55c3cfb82230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f391402c7b0 tx=0x7f39140c12d0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 -- 172.21.15.28:0/92830567 shutdown_connections 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb76380 0x55c3cfb8da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb71210 0x55c3cfb82230 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 --2- 172.21.15.28:0/92830567 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb70820 0x55c3cfb6f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 -- 172.21.15.28:0/92830567 >> 172.21.15.28:0/92830567 conn(0x55c3cfb6bd90 msgr2=0x55c3cfb6eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 -- 172.21.15.28:0/92830567 shutdown_connections 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.804+0000 7f3929449740 1 -- 172.21.15.28:0/92830567 wait complete. 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 -- start start 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 Processor -- start 2024-03-22T02:22:02.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb71210 0x55c3cfb80390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 0x55c3cfb81350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39201a6e70 con 0x55c3cfb76380 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39201a82e0 con 0x55c3cfb71210 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39201a7540 con 0x55c3cfb70820 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3929449740 10 client.0 _ll_register_callbacks cb 0x7f39140bda80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3924e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 0x55c3cfb81350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f391ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb71210 0x55c3cfb80390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3924e03640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 0x55c3cfb81350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40116/0 (socket says 172.21.15.28:40116) 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.805+0000 7f3924e03640 1 -- 172.21.15.28:0/4244226507 learned_addr learned my addr 172.21.15.28:0/4244226507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 -- 172.21.15.28:0/4244226507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb71210 msgr2=0x55c3cfb80390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb71210 0x55c3cfb80390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 -- 172.21.15.28:0/4244226507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 msgr2=0x55c3cfb81350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 0x55c3cfb81350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 -- 172.21.15.28:0/4244226507 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3cfb81940 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f391ffff640 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7fe50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3910009c70 tx=0x7f3910011e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d32f4e4040bd582 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f38ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100035f0 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3914004c80 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f3910016040 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f39100052c0 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f3929449740 10 client.27760 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c3cfbed5c0 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.806+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3914004c80 con 0x55c3cfb70820 2024-03-22T02:22:02.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f3910025a40 con 0x55c3cfb70820 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 10 client.27760 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 10 client.27760 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 20 client.27760 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfbedda0 0x55c3cfbf01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 4 client.27760 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c3cfadd160 con 0x55c3cfbedda0 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f391f7fe640 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfbedda0 0x55c3cfbf01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f391f7fe640 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfbedda0 0x55c3cfbf01c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df4f6c3db15c7979 server_cookie=ae6ba45d803c1ba1 in_seq=0 out_seq=0 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f38ff7fe640 10 client.27760 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f38ff7fe640 1 -- 172.21.15.28:0/4244226507 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c3cfadd160 con 0x55c3cfbedda0 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f38ff7fe640 10 client.27760 handle_command_reply: tid=0 2024-03-22T02:22:02.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.807+0000 7f3929449740 1 client.27760 shutdown 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 msgr2=0x55c3cfb7fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7fe50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3910009c70 tx=0x7f3910011e50 comp rx=0 tx=0).stop 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 shutdown_connections 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55c3cfbedda0 0x55c3cfbf01c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3cfb76380 0x55c3cfb81350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c3cfb71210 0x55c3cfb80390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f38ff7fe640 0 client.27760 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 --2- 172.21.15.28:0/4244226507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c3cfb70820 0x55c3cfb7fe50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 >> 172.21.15.28:0/4244226507 conn(0x55c3cfb6bd90 msgr2=0x55c3cfb6f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 shutdown_connections 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 1 -- 172.21.15.28:0/4244226507 wait complete. 2024-03-22T02:22:02.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3929449740 20 client.27760 trim_cache size 0 max 16384 2024-03-22T02:22:02.816 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:02.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 msgr2=0x7f39080a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 0x7f39080a5580 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f39180098b0 tx=0x7f391800eee0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 msgr2=0x7f392019cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f3914009230 tx=0x7f3914005d50 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 shutdown_connections 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39080a30c0 0x7f39080a5580 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3920102640 0x7f392019f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920101c80 0x7f392019cbd0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.808+0000 7f3926866640 1 --2- 172.21.15.28:0/3362705118 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3920100a90 0x7f39200698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.809+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 >> 172.21.15.28:0/3362705118 conn(0x7f39200fc020 msgr2=0x7f3920105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.809+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 shutdown_connections 2024-03-22T02:22:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.809+0000 7f3926866640 1 -- 172.21.15.28:0/3362705118 wait complete. 2024-03-22T02:22:02.818 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:02 smithi028 ceph-mon[31916]: pgmap v613: 129 pgs: 129 active+clean; 56 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-03-22T02:22:02.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:22:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- 172.21.15.28:0/309803265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 msgr2=0x7efdb81073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- 172.21.15.28:0/309803265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb81073a0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7efda4009900 tx=0x7efda402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- 172.21.15.28:0/309803265 shutdown_connections 2024-03-22T02:22:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- 172.21.15.28:0/309803265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 0x7efdb8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- 172.21.15.28:0/309803265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb81073a0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- 172.21.15.28:0/309803265 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb8104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- 172.21.15.28:0/309803265 >> 172.21.15.28:0/309803265 conn(0x7efdb80fc030 msgr2=0x7efdb80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- 172.21.15.28:0/309803265 shutdown_connections 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- 172.21.15.28:0/309803265 wait complete. 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 Processor -- start 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- start start 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 0x7efdb81a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdb810d820 con 0x7efdb8104fb0 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdb810d6a0 con 0x7efdb8102680 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.929+0000 7efdbecd4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdb810d9a0 con 0x7efdb81078e0 2024-03-22T02:22:02.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40122/0 (socket says 172.21.15.28:40122) 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdbca70640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 -- 172.21.15.28:0/2808152271 learned_addr learned my addr 172.21.15.28:0/2808152271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdbd271640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 0x7efdb81a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 -- 172.21.15.28:0/2808152271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 msgr2=0x7efdb81a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 0x7efdb81a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 -- 172.21.15.28:0/2808152271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 msgr2=0x7efdb819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 -- 172.21.15.28:0/2808152271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb81a1d40 con 0x7efdb8104fb0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdbca70640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efda4000c00 tx=0x7efda402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb5ffb640 1 -- 172.21.15.28:0/2808152271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4041070 con 0x7efdb8104fb0 2024-03-22T02:22:02.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb5ffb640 1 -- 172.21.15.28:0/2808152271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda4004450 con 0x7efdb8104fb0 2024-03-22T02:22:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdb5ffb640 1 -- 172.21.15.28:0/2808152271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4034440 con 0x7efdb8104fb0 2024-03-22T02:22:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.930+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb81a1fd0 con 0x7efdb8104fb0 2024-03-22T02:22:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.931+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb81a23e0 con 0x7efdb8104fb0 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdb5ffb640 1 -- 172.21.15.28:0/2808152271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7efda4044070 con 0x7efdb8104fb0 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 Processor -- start 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 -- start start 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47eef5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47eefd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d47ee5a8d0 con 0x55d47eef6210 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d47ee02e00 con 0x55d47eefb380 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.933+0000 7efdc1997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d47ee56880 con 0x55d47eef5820 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdb5ffb640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 0x7efda80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbd271640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47eef5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40132/0 (socket says 172.21.15.28:40132) 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47eefd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 -- 172.21.15.28:0/3365391757 learned_addr learned my addr 172.21.15.28:0/3365391757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbd271640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 0x7efda80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 -- 172.21.15.28:0/3365391757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 msgr2=0x55d47eef5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47eef5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 -- 172.21.15.28:0/3365391757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 msgr2=0x55d47eefd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdb5ffb640 1 -- 172.21.15.28:0/2808152271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efda40f12e0 con 0x7efdb8104fb0 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47eefd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 -- 172.21.15.28:0/3365391757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d47eb6e3f0 con 0x55d47eef6210 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdb7fff640 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47eefd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbca70640 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdac0115f0 tx=0x7efdac011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f01185c85e5fda server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efdbd271640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 0x7efda80a53e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55d47eef6a70 tx=0x7efdb0004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efda2ffd640 1 -- 172.21.15.28:0/3365391757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01e070 con 0x55d47eef6210 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efda2ffd640 1 -- 172.21.15.28:0/3365391757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac00fb20 con 0x55d47eef6210 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.934+0000 7efda2ffd640 1 -- 172.21.15.28:0/3365391757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac019bf0 con 0x55d47eef6210 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 -- 172.21.15.28:0/3365391757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 msgr2=0x55d47eefad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdac0115f0 tx=0x7efdac011ab0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 -- 172.21.15.28:0/3365391757 shutdown_connections 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47eefd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef6210 0x55d47eefad10 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 --2- 172.21.15.28:0/3365391757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47eef5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 -- 172.21.15.28:0/3365391757 >> 172.21.15.28:0/3365391757 conn(0x55d47eef0d90 msgr2=0x55d47eef1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 -- 172.21.15.28:0/3365391757 shutdown_connections 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.935+0000 7efdc1997740 1 -- 172.21.15.28:0/3365391757 wait complete. 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 -- start start 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 Processor -- start 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef00290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d47eefe330 con 0x55d47eef5820 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d47ee16300 con 0x55d47eefb380 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d47eefe4b0 con 0x55d47eef6210 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdc1997740 10 client.0 _ll_register_callbacks cb 0x55d47ee51e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40140/0 (socket says 172.21.15.28:40140) 2024-03-22T02:22:02.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 -- 172.21.15.28:0/3326098758 learned_addr learned my addr 172.21.15.28:0/3326098758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef00290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbd271640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 -- 172.21.15.28:0/3326098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 msgr2=0x55d47ef00290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef00290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 -- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 msgr2=0x55d47ef072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdbca70640 1 -- 172.21.15.28:0/3326098758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d47ec387d0 con 0x55d47eef5820 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.936+0000 7efdb7fff640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef00290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdbca70640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdac0115c0 tx=0x7efdac00f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a18d76ae18352b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01e040 con 0x55d47eef5820 2024-03-22T02:22:02.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efdac004040 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac00fb20 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdac030050 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 10 client.17949 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d47ef0a640 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac00fb20 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7efdac021070 con 0x55d47eef5820 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 10 client.17949 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 10 client.17949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 20 client.17949 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef0bae0 0x55d47ef0df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 4 client.17949 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.937+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d47ee67e10 con 0x55d47ef0bae0 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.938+0000 7efdb7fff640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef0bae0 0x55d47ef0df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.938+0000 7efdb7fff640 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef0bae0 0x55d47ef0df10 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5d1c896e84d0eef server_cookie=778e5c6768716f4c in_seq=0 out_seq=0 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.938+0000 7efd97fff640 10 client.17949 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efd97fff640 1 -- 172.21.15.28:0/3326098758 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d47ee67e10 con 0x55d47ef0bae0 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efd97fff640 10 client.17949 handle_command_reply: tid=0 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 client.17949 shutdown 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 msgr2=0x55d47eeffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdac0115c0 tx=0x7efdac00f870 comp rx=0 tx=0).stop 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 shutdown_connections 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef0bae0 0x55d47ef0df10 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef00290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efd97fff640 0 client.17949 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 --2- 172.21.15.28:0/3326098758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47eeffd50 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 >> 172.21.15.28:0/3326098758 conn(0x55d47eef0d90 msgr2=0x55d47eeeffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 shutdown_connections 2024-03-22T02:22:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 1 -- 172.21.15.28:0/3326098758 wait complete. 2024-03-22T02:22:02.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.939+0000 7efdc1997740 20 client.17949 trim_cache size 0 max 16384 2024-03-22T02:22:02.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 Processor -- start 2024-03-22T02:22:02.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 -- start start 2024-03-22T02:22:02.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef07290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef12d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81af550 con 0x55d47eef5820 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81ace20 con 0x55d47eefb380 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdc1997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81aac70 con 0x55d47eef6210 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40144/0 (socket says 172.21.15.28:40144) 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 -- 172.21.15.28:0/814822536 learned_addr learned my addr 172.21.15.28:0/814822536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbd271640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef12d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdb7fff640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef07290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 -- 172.21.15.28:0/814822536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 msgr2=0x55d47ef07290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef07290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 -- 172.21.15.28:0/814822536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 msgr2=0x55d47ef12d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef12d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbca70640 1 -- 172.21.15.28:0/814822536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d47eefe600 con 0x55d47eef5820 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.955+0000 7efdbd271640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef12d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.956+0000 7efdbca70640 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efdac006010 tx=0x7efdac005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c075a6cfd661bc62 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.956+0000 7efda27fc640 1 -- 172.21.15.28:0/814822536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01e040 con 0x55d47eef5820 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.956+0000 7efda27fc640 1 -- 172.21.15.28:0/814822536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac027070 con 0x55d47eef5820 2024-03-22T02:22:02.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.956+0000 7efda27fc640 1 -- 172.21.15.28:0/814822536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac003e10 con 0x55d47eef5820 2024-03-22T02:22:02.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- 172.21.15.28:0/814822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 msgr2=0x55d47ef127d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7efdac006010 tx=0x7efdac005d30 comp rx=0 tx=0).stop 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- 172.21.15.28:0/814822536 shutdown_connections 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eefb380 0x55d47ef12d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef6210 0x55d47ef07290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 --2- 172.21.15.28:0/814822536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eef5820 0x55d47ef127d0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- 172.21.15.28:0/814822536 >> 172.21.15.28:0/814822536 conn(0x55d47eef0d90 msgr2=0x55d47eef4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- 172.21.15.28:0/814822536 shutdown_connections 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- 172.21.15.28:0/814822536 wait complete. 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 -- start start 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 Processor -- start 2024-03-22T02:22:02.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.957+0000 7efdc1997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81ab2e0 con 0x55d47eefb380 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81ac750 con 0x55d47eef6210 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efdb81ab9b0 con 0x55d47eef5820 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdc1997740 10 client.0 _ll_register_callbacks cb 0x7efdac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbd271640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:57546/0 (socket says 172.21.15.28:57546) 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbd271640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40156/0 (socket says 172.21.15.28:40156) 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 -- 172.21.15.28:0/2487270176 learned_addr learned my addr 172.21.15.28:0/2487270176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:02.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdb7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33246/0 (socket says 172.21.15.28:33246) 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 -- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 msgr2=0x55d47ef04740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 -- 172.21.15.28:0/2487270176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 msgr2=0x55d47ef6fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 -- 172.21.15.28:0/2487270176 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d47ef70340 con 0x55d47eef5820 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbd271640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efdbca70640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efdac011510 tx=0x7efdac009e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d1a55f6e39cab17 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efd97fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.958+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac01e040 con 0x55d47eef5820 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdac027070 con 0x55d47eef5820 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efdac0052d0 con 0x55d47eef5820 2024-03-22T02:22:02.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7efdac042050 con 0x55d47eef5820 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 10 client.27778 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55d47ef715c0 con 0x55d47eef5820 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdac027070 con 0x55d47eef5820 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7efdac021070 con 0x55d47eef5820 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 10 client.27778 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 10 client.27778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 20 client.27778 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef71ea0 0x55d47ef742c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 4 client.27778 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d47ee62160 con 0x55d47ef71ea0 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.959+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef71ea0 0x55d47ef742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.960+0000 7efdb7fff640 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef71ea0 0x55d47ef742c0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93e4bccf4cbf25ba server_cookie=9256c4ceb3ccf0d7 in_seq=0 out_seq=0 2024-03-22T02:22:02.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.960+0000 7efd97fff640 10 client.27778 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efd97fff640 1 -- 172.21.15.28:0/2487270176 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d47ee62160 con 0x55d47ef71ea0 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efd97fff640 10 client.27778 handle_command_reply: tid=0 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 client.27778 shutdown 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 msgr2=0x55d47ef04200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efdac011510 tx=0x7efdac009e00 comp rx=0 tx=0).stop 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 shutdown_connections 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d47ef71ea0 0x55d47ef742c0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d47eefb380 0x55d47ef6fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efd97fff640 0 client.27778 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d47eef6210 0x55d47ef04740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 --2- 172.21.15.28:0/2487270176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d47eef5820 0x55d47ef04200 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.962+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 >> 172.21.15.28:0/2487270176 conn(0x55d47eef0d90 msgr2=0x55d47eeef740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 shutdown_connections 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdc1997740 1 -- 172.21.15.28:0/2487270176 wait complete. 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdc1997740 20 client.27778 trim_cache size 0 max 16384 2024-03-22T02:22:02.968 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 msgr2=0x7efda80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 0x7efda80a53e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55d47eef6a70 tx=0x7efdb0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 msgr2=0x7efdb819a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efda4000c00 tx=0x7efda402c640 comp rx=0 tx=0).stop 2024-03-22T02:22:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 shutdown_connections 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efda80a2f20 0x7efda80a53e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdb81078e0 0x7efdb81a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efdb8104fb0 0x7efdb819a660 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 --2- 172.21.15.28:0/2808152271 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efdb8102680 0x7efdb819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 >> 172.21.15.28:0/2808152271 conn(0x7efdb80fc030 msgr2=0x7efdb8101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 shutdown_connections 2024-03-22T02:22:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:02.963+0000 7efdbecd4640 1 -- 172.21.15.28:0/2808152271 wait complete. 2024-03-22T02:22:03.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 812697 11.72 MB/sec execute 285 sec latency 31.180 ms 2024-03-22T02:22:03.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:03 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/630817459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:03.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:03 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/630817459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:03.975 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:22:03.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:22:03.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:22:03.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:03 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/630817459' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- 172.21.15.28:0/802451025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 msgr2=0x7f51d0100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 --2- 172.21.15.28:0/802451025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0100eb0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009900 tx=0x7f51c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- 172.21.15.28:0/802451025 shutdown_connections 2024-03-22T02:22:04.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 --2- 172.21.15.28:0/802451025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 0x7f51d0108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 --2- 172.21.15.28:0/802451025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 0x7f51d0102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 --2- 172.21.15.28:0/802451025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0100eb0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- 172.21.15.28:0/802451025 >> 172.21.15.28:0/802451025 conn(0x7f51d00fc020 msgr2=0x7f51d00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- 172.21.15.28:0/802451025 shutdown_connections 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- 172.21.15.28:0/802451025 wait complete. 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 Processor -- start 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.089+0000 7f51d64c9640 1 -- start start 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 0x7f51d0198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 0x7f51d019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010c700 con 0x7f51d0100a90 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010c580 con 0x7f51d0102640 2024-03-22T02:22:04.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51d010c880 con 0x7f51d0101c80 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 0x7f51d0198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40174/0 (socket says 172.21.15.28:40174) 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 -- 172.21.15.28:0/3484107 learned_addr learned my addr 172.21.15.28:0/3484107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d4a66640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 0x7f51d019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 -- 172.21.15.28:0/3484107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 msgr2=0x7f51d0198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 0x7f51d0198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 -- 172.21.15.28:0/3484107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 msgr2=0x7f51d019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 0x7f51d019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 -- 172.21.15.28:0/3484107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d019fb20 con 0x7f51d0100a90 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cffff640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f51c0000c00 tx=0x7f51c002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cd7fa640 1 -- 172.21.15.28:0/3484107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0041070 con 0x7f51d0100a90 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d0103040 con 0x7f51d0100a90 2024-03-22T02:22:04.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cd7fa640 1 -- 172.21.15.28:0/3484107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c0004450 con 0x7f51d0100a90 2024-03-22T02:22:04.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51cd7fa640 1 -- 172.21.15.28:0/3484107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0034440 con 0x7f51d0100a90 2024-03-22T02:22:04.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.090+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d0103530 con 0x7f51d0100a90 2024-03-22T02:22:04.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51cd7fa640 1 -- 172.21.15.28:0/3484107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f51c0044070 con 0x7f51d0100a90 2024-03-22T02:22:04.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 Processor -- start 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 -- start start 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55572b62c570 con 0x55572b6c7bc0 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55572b5d4e10 con 0x55572b6cd720 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d9193740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55572b628520 con 0x55572b6c85b0 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51cd7fa640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 0x7f51b80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.093+0000 7f51d4a66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33278/0 (socket says 172.21.15.28:33278) 2024-03-22T02:22:04.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 -- 172.21.15.28:0/733721599 learned_addr learned my addr 172.21.15.28:0/733721599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51d4a66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c7fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40182/0 (socket says 172.21.15.28:40182) 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51d4a66640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 0x7f51b80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cd7fa640 1 -- 172.21.15.28:0/3484107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f51c00f12d0 con 0x7f51d0100a90 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 -- 172.21.15.28:0/733721599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 msgr2=0x55572b6cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 -- 172.21.15.28:0/733721599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 msgr2=0x55572b6c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 -- 172.21.15.28:0/733721599 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572b4087e0 con 0x55572b6cd720 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51d4a66640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 0x7f51b80a53e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005c40 tx=0x7f51c8009bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51c4010dd0 tx=0x7f51c4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440dd88441cd6e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51b67fc640 1 -- 172.21.15.28:0/733721599 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c401e070 con 0x55572b6cd720 2024-03-22T02:22:04.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51b67fc640 1 -- 172.21.15.28:0/733721599 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c401aae0 con 0x55572b6cd720 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.094+0000 7f51b67fc640 1 -- 172.21.15.28:0/733721599 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c4019af0 con 0x55572b6cd720 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- 172.21.15.28:0/733721599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 msgr2=0x55572b6cfbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51c4010dd0 tx=0x7f51c4005f90 comp rx=0 tx=0).stop 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- 172.21.15.28:0/733721599 shutdown_connections 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6cfbb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 --2- 172.21.15.28:0/733721599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- 172.21.15.28:0/733721599 >> 172.21.15.28:0/733721599 conn(0x55572b6c3130 msgr2=0x55572b6c3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- 172.21.15.28:0/733721599 shutdown_connections 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- 172.21.15.28:0/733721599 wait complete. 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.095+0000 7f51d9193740 1 -- start start 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 Processor -- start 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55572b6d06d0 con 0x55572b6c85b0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55572b5e8110 con 0x55572b6c7bc0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55572b6d0850 con 0x55572b6cd720 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d9193740 10 client.0 _ll_register_callbacks cb 0x55572b624be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40186/0 (socket says 172.21.15.28:40186) 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 -- 172.21.15.28:0/3560091413 learned_addr learned my addr 172.21.15.28:0/3560091413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51d4a66640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 -- 172.21.15.28:0/3560091413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 msgr2=0x55572b6d9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 -- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 msgr2=0x55572b6d1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cf7fe640 1 -- 172.21.15.28:0/3560091413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572b468f70 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.096+0000 7f51cffff640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009d90 tx=0x7f51c4019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3df0fffa3d7b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c401e030 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f51c4004020 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c401aae0 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f51c401a910 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 10 client.17967 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55572b300170 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c401aae0 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f51c4019b00 con 0x55572b6c85b0 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 10 client.17967 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:04.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 10 client.17967 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 20 client.17967 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b6dd8c0 0x55572b6dfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 4 client.17967 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55572b639ab0 con 0x55572b6dd8c0 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.097+0000 7f51cffff640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b6dd8c0 0x55572b6dfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.098+0000 7f51af7fe640 10 client.17967 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.098+0000 7f51cffff640 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b6dd8c0 0x55572b6dfcb0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d40814556ce4b06 server_cookie=f87aff42c430e362 in_seq=0 out_seq=0 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.098+0000 7f51af7fe640 1 -- 172.21.15.28:0/3560091413 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55572b639ab0 con 0x55572b6dd8c0 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.098+0000 7f51af7fe640 10 client.17967 handle_command_reply: tid=0 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.098+0000 7f51d9193740 1 client.17967 shutdown 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 msgr2=0x55572b6d22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009d90 tx=0x7f51c4019590 comp rx=0 tx=0).stop 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 shutdown_connections 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b6dd8c0 0x55572b6dfcb0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d22e0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51af7fe640 0 client.17967 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 --2- 172.21.15.28:0/3560091413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 >> 172.21.15.28:0/3560091413 conn(0x55572b6c3130 msgr2=0x55572b6c23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 shutdown_connections 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 1 -- 172.21.15.28:0/3560091413 wait complete. 2024-03-22T02:22:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.099+0000 7f51d9193740 20 client.17967 trim_cache size 0 max 16384 2024-03-22T02:22:04.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.117+0000 7f51d9193740 1 Processor -- start 2024-03-22T02:22:04.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 -- start start 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6c9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01ab0a0 con 0x55572b6c7bc0 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01a8970 con 0x55572b6cd720 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d9193740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01a67c0 con 0x55572b6c85b0 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51d4a66640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:57594/0 (socket says 172.21.15.28:57594) 2024-03-22T02:22:04.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40190/0 (socket says 172.21.15.28:40190) 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cf7fe640 1 -- 172.21.15.28:0/255107167 learned_addr learned my addr 172.21.15.28:0/255107167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 -- 172.21.15.28:0/255107167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 msgr2=0x55572b6d92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 -- 172.21.15.28:0/255107167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 msgr2=0x55572b6c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 -- 172.21.15.28:0/255107167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55572b6e0020 con 0x55572b6c7bc0 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51cffff640 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f51c00ee990 tx=0x7f51c00e82b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b20d7c3c5e8188c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.118+0000 7f51b5ffb640 1 -- 172.21.15.28:0/255107167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c004b040 con 0x55572b6c7bc0 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51b5ffb640 1 -- 172.21.15.28:0/255107167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c00c6000 con 0x55572b6c7bc0 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51b5ffb640 1 -- 172.21.15.28:0/255107167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c00bd730 con 0x55572b6c7bc0 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 -- 172.21.15.28:0/255107167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 msgr2=0x55572b6c8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f51c00ee990 tx=0x7f51c00e82b0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 -- 172.21.15.28:0/255107167 shutdown_connections 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6cd720 0x55572b6c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6c85b0 0x55572b6d92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 --2- 172.21.15.28:0/255107167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c7bc0 0x55572b6c8e40 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 -- 172.21.15.28:0/255107167 >> 172.21.15.28:0/255107167 conn(0x55572b6c3130 msgr2=0x55572b6c23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 -- 172.21.15.28:0/255107167 shutdown_connections 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.119+0000 7f51d9193740 1 -- 172.21.15.28:0/255107167 wait complete. 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 -- start start 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 Processor -- start 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01a6e30 con 0x55572b6c85b0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01a82a0 con 0x55572b6c7bc0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01a7500 con 0x55572b6cd720 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d9193740 10 client.0 _ll_register_callbacks cb 0x7f51c00bd500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40206/0 (socket says 172.21.15.28:40206) 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51cf7fe640 1 -- 172.21.15.28:0/2775467922 learned_addr learned my addr 172.21.15.28:0/2775467922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51d4a66640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.120+0000 7f51cffff640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 -- 172.21.15.28:0/2775467922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 msgr2=0x55572b6d80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 -- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 msgr2=0x55572b6d6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 -- 172.21.15.28:0/2775467922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55572b6d86d0 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51cf7fe640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009e10 tx=0x7f51c4010b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d790f33f7c633c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c401e070 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f51c401acc0 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c00c6000 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f51c4004040 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51d9193740 10 client.17973 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55572b743ce0 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51c00c6000 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.121+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f51c40035f0 con 0x55572b6c85b0 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 10 client.17973 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:04.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 20 client.17973 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b7446e0 0x55572b746b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 4 client.17973 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55572b633e00 con 0x55572b7446e0 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51cffff640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b7446e0 0x55572b746b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51cffff640 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b7446e0 0x55572b746b00 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d539ba882d27302a server_cookie=6c5b694c99504d01 in_seq=0 out_seq=0 2024-03-22T02:22:04.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.122+0000 7f51af7fe640 10 client.17973 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51af7fe640 1 -- 172.21.15.28:0/2775467922 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55572b633e00 con 0x55572b7446e0 2024-03-22T02:22:04.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51af7fe640 10 client.17973 handle_command_reply: tid=0 2024-03-22T02:22:04.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 client.17973 shutdown 2024-03-22T02:22:04.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 msgr2=0x55572b6d7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f51c4009e10 tx=0x7f51c4010b60 comp rx=0 tx=0).stop 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 shutdown_connections 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55572b7446e0 0x55572b746b00 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55572b6cd720 0x55572b6d80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55572b6c85b0 0x55572b6d7140 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 --2- 172.21.15.28:0/2775467922 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55572b6c7bc0 0x55572b6d6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 >> 172.21.15.28:0/2775467922 conn(0x55572b6c3130 msgr2=0x55572b6c6390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51af7fe640 0 client.17973 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 shutdown_connections 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 1 -- 172.21.15.28:0/2775467922 wait complete. 2024-03-22T02:22:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.123+0000 7f51d9193740 20 client.17973 trim_cache size 0 max 16384 2024-03-22T02:22:04.131 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:04.131 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:22:04.131 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "96319dfc-c605-41ce-bfa9-06652ee35caa", 2024-03-22T02:22:04.131 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:22:04.131 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 msgr2=0x7f51b80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 0x7f51b80a53e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f51c8005c40 tx=0x7f51c8009bb0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 msgr2=0x7f51d0197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f51c0000c00 tx=0x7f51c002c640 comp rx=0 tx=0).stop 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 shutdown_connections 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f51b80a2f20 0x7f51b80a53e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f51d0102640 0x7f51d019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51d0101c80 0x7f51d0198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 --2- 172.21.15.28:0/3484107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51d0100a90 0x7f51d0197f00 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 >> 172.21.15.28:0/3484107 conn(0x7f51d00fc020 msgr2=0x7f51d0068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 shutdown_connections 2024-03-22T02:22:04.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.124+0000 7f51d64c9640 1 -- 172.21.15.28:0/3484107 wait complete. 2024-03-22T02:22:04.135 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:22:04.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 -- 172.21.15.28:0/3076847351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c102640 msgr2=0x7fe08c108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 --2- 172.21.15.28:0/3076847351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c102640 0x7fe08c108ed0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fe084009ca0 tx=0x7fe08402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 -- 172.21.15.28:0/3076847351 shutdown_connections 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 --2- 172.21.15.28:0/3076847351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c102640 0x7fe08c108ed0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 --2- 172.21.15.28:0/3076847351 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c101c80 0x7fe08c102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 --2- 172.21.15.28:0/3076847351 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c100a90 0x7fe08c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 -- 172.21.15.28:0/3076847351 >> 172.21.15.28:0/3076847351 conn(0x7fe08c0fc020 msgr2=0x7fe08c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 -- 172.21.15.28:0/3076847351 shutdown_connections 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.244+0000 7fe093332640 1 -- 172.21.15.28:0/3076847351 wait complete. 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 Processor -- start 2024-03-22T02:22:04.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 -- start start 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c100a90 0x7fe08c197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 0x7fe08c19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c10c6f0 con 0x7fe08c101c80 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c10c570 con 0x7fe08c102640 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe093332640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c10c870 con 0x7fe08c100a90 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c100a90 0x7fe08c197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40226/0 (socket says 172.21.15.28:40226) 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 -- 172.21.15.28:0/3873091112 learned_addr learned my addr 172.21.15.28:0/3873091112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 0x7fe08c19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 -- 172.21.15.28:0/3873091112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c100a90 msgr2=0x7fe08c197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c100a90 0x7fe08c197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 -- 172.21.15.28:0/3873091112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 msgr2=0x7fe08c19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 0x7fe08c19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.245+0000 7fe0908cd640 1 -- 172.21.15.28:0/3873091112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe08c19fad0 con 0x7fe08c101c80 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 0x7fe08c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00ea60 tx=0x7fe07c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe0827fc640 1 -- 172.21.15.28:0/3873091112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c00cde0 con 0x7fe08c101c80 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08c102f00 con 0x7fe08c101c80 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe0827fc640 1 -- 172.21.15.28:0/3873091112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07c0035f0 con 0x7fe08c101c80 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe0827fc640 1 -- 172.21.15.28:0/3873091112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c010670 con 0x7fe08c101c80 2024-03-22T02:22:04.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.246+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe08c1034a0 con 0x7fe08c101c80 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.248+0000 7fe0827fc640 1 -- 172.21.15.28:0/3873091112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe07c0037a0 con 0x7fe08c101c80 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 Processor -- start 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 -- start start 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3534d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c35355420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c352b28c0 con 0x558c35352f90 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558c3525ac20 con 0x558c3534de20 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe095ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c352ae870 con 0x558c3534d430 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0827fc640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 0x7fe0780a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c35355420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0918cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3534d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33334/0 (socket says 172.21.15.28:33334) 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 -- 172.21.15.28:0/752680905 learned_addr learned my addr 172.21.15.28:0/752680905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 0x7fe0780a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0827fc640 1 -- 172.21.15.28:0/3873091112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe07c0cfa10 con 0x7fe08c101c80 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 -- 172.21.15.28:0/752680905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 msgr2=0x558c3534d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3534d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 -- 172.21.15.28:0/752680905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 msgr2=0x558c35355420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c35355420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0910ce640 1 -- 172.21.15.28:0/752680905 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c3527f150 con 0x558c3534de20 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0918cf640 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3534d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.249+0000 7fe0908cd640 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c35355420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.250+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 0x7fe0780a5460 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe084005e30 tx=0x7fe084032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.250+0000 7fe0910ce640 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe088010dd0 tx=0x7fe088005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05dbd0692d64d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.250+0000 7fe0777fe640 1 -- 172.21.15.28:0/752680905 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08801e070 con 0x558c3534de20 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.250+0000 7fe0777fe640 1 -- 172.21.15.28:0/752680905 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe088004220 con 0x558c3534de20 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.250+0000 7fe0777fe640 1 -- 172.21.15.28:0/752680905 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08801a830 con 0x558c3534de20 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- 172.21.15.28:0/752680905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 msgr2=0x558c35352920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe088010dd0 tx=0x7fe088005f90 comp rx=0 tx=0).stop 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- 172.21.15.28:0/752680905 shutdown_connections 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c35355420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35352920 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- 172.21.15.28:0/752680905 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3534d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- 172.21.15.28:0/752680905 >> 172.21.15.28:0/752680905 conn(0x558c353489a0 msgr2=0x558c35348da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- 172.21.15.28:0/752680905 shutdown_connections 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- 172.21.15.28:0/752680905 wait complete. 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 -- start start 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 Processor -- start 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c35357a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.251+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35357f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558c35355f40 con 0x558c35352f90 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558c3526e130 con 0x558c3534de20 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558c353560c0 con 0x558c3534d430 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 10 client.0 _ll_register_callbacks cb 0x558c3526db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c35357a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35357f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 -- 172.21.15.28:0/4101060286 learned_addr learned my addr 172.21.15.28:0/4101060286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 -- 172.21.15.28:0/4101060286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 msgr2=0x558c35357a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c35357a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 -- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 msgr2=0x558c35357f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35357f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 -- 172.21.15.28:0/4101060286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c350907d0 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe0918cf640 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe084036000 tx=0x7fe0840095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919512325d697af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe074ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084053060 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe088004220 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe084004040 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe084039040 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 10 client.17985 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.252+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558c34fc63f0 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe088004220 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fe08404e470 con 0x558c35352f90 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 10 client.17985 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:04.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 10 client.17985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 20 client.17985 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353637d0 0x558c35365c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 4 client.17985 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c352bfe00 con 0x558c353637d0 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe0910ce640 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353637d0 0x558c35365c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe0910ce640 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353637d0 0x558c35365c70 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=769b4a84400719b5 server_cookie=bc8c9961f3244aec in_seq=0 out_seq=0 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.253+0000 7fe074ff9640 10 client.17985 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe074ff9640 1 -- 172.21.15.28:0/4101060286 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c352bfe00 con 0x558c353637d0 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe074ff9640 10 client.17985 handle_command_reply: tid=0 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 client.17985 shutdown 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 msgr2=0x558c3535eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fe084036000 tx=0x7fe0840095c0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 shutdown_connections 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353637d0 0x558c35365c70 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c35352f90 0x558c3535eea0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c3534de20 0x558c35357f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe074ff9640 0 client.17985 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 --2- 172.21.15.28:0/4101060286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c35357a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 >> 172.21.15.28:0/4101060286 conn(0x558c353489a0 msgr2=0x558c35347c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 shutdown_connections 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 1 -- 172.21.15.28:0/4101060286 wait complete. 2024-03-22T02:22:04.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.255+0000 7fe095ff0740 20 client.17985 trim_cache size 0 max 16384 2024-03-22T02:22:04.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.276+0000 7fe095ff0740 1 Processor -- start 2024-03-22T02:22:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.277+0000 7fe095ff0740 1 -- start start 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534de20 0x558c3535ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3536a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1ab190 con 0x558c3534d430 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1a8a60 con 0x558c35352f90 2024-03-22T02:22:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.278+0000 7fe095ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1a68b0 con 0x558c3534de20 2024-03-22T02:22:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.279+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.279+0000 7fe0910ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40258/0 (socket says 172.21.15.28:40258) 2024-03-22T02:22:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.279+0000 7fe0910ce640 1 -- 172.21.15.28:0/3521387325 learned_addr learned my addr 172.21.15.28:0/3521387325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.279+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534de20 0x558c3535ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.279+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3536a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 -- 172.21.15.28:0/3521387325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534de20 msgr2=0x558c3535ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534de20 0x558c3535ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 -- 172.21.15.28:0/3521387325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 msgr2=0x558c3536a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3536a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 -- 172.21.15.28:0/3521387325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c34f860f0 con 0x558c3534d430 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0918cf640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3536a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe088010d70 tx=0x7fe088003cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83eb81c2a57c5c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.280+0000 7fe076ffd640 1 -- 172.21.15.28:0/3521387325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08801e040 con 0x558c3534d430 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.281+0000 7fe076ffd640 1 -- 172.21.15.28:0/3521387325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0880041c0 con 0x558c3534d430 2024-03-22T02:22:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.281+0000 7fe076ffd640 1 -- 172.21.15.28:0/3521387325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880053c0 con 0x558c3534d430 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.282+0000 7fe095ff0740 1 -- 172.21.15.28:0/3521387325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 msgr2=0x558c3534c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.282+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fe088010d70 tx=0x7fe088003cc0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 -- 172.21.15.28:0/3521387325 shutdown_connections 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3536a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534de20 0x558c3535ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3521387325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534d430 0x558c3534c310 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 -- 172.21.15.28:0/3521387325 >> 172.21.15.28:0/3521387325 conn(0x558c353489a0 msgr2=0x558c3534bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 -- 172.21.15.28:0/3521387325 shutdown_connections 2024-03-22T02:22:04.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.283+0000 7fe095ff0740 1 -- 172.21.15.28:0/3521387325 wait complete. 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.284+0000 7fe095ff0740 1 -- start start 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.284+0000 7fe095ff0740 1 Processor -- start 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3535c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3535d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1a6f20 con 0x558c3534de20 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1a8390 con 0x558c35352f90 2024-03-22T02:22:04.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe08c1a75f0 con 0x558c3534d430 2024-03-22T02:22:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe095ff0740 10 client.0 _ll_register_callbacks cb 0x7fe084032f80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0918cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3535d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0908cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40264/0 (socket says 172.21.15.28:40264) 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0918cf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3535d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33358/0 (socket says 172.21.15.28:33358) 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0908cd640 1 -- 172.21.15.28:0/3798727939 learned_addr learned my addr 172.21.15.28:0/3798727939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.285+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3535c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 -- 172.21.15.28:0/3798727939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 msgr2=0x558c3535c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3535c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 -- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 msgr2=0x558c3535d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3535d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 -- 172.21.15.28:0/3798727939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c3535dee0 con 0x558c3534de20 2024-03-22T02:22:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe0908cd640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe07c003ee0 tx=0x7fe07c06a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe50ac28cba6bee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.286+0000 7fe074ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:04.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c09fe80 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe07c0983a0 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0880041c0 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe07c0a4740 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe095ff0740 10 client.17991 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.287+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558c353dab80 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0880041c0 con 0x558c3534de20 2024-03-22T02:22:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fe07c09b310 con 0x558c3534de20 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 10 client.17991 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 10 client.17991 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 20 client.17991 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353db540 0x558c353dd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 4 client.17991 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.288+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c352ba150 con 0x558c353db540 2024-03-22T02:22:04.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.290+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353db540 0x558c353dd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:04.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.291+0000 7fe0910ce640 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353db540 0x558c353dd960 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ee70143246e568d server_cookie=40a092215932696d in_seq=0 out_seq=0 2024-03-22T02:22:04.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.291+0000 7fe074ff9640 10 client.17991 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.292+0000 7fe074ff9640 1 -- 172.21.15.28:0/3798727939 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558c352ba150 con 0x558c353db540 2024-03-22T02:22:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.292+0000 7fe074ff9640 10 client.17991 handle_command_reply: tid=0 2024-03-22T02:22:04.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.292+0000 7fe095ff0740 1 client.17991 shutdown 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 msgr2=0x558c3535c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fe07c003ee0 tx=0x7fe07c06a920 comp rx=0 tx=0).stop 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 shutdown_connections 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558c353db540 0x558c353dd960 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558c35352f90 0x558c3535d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c3534de20 0x558c3535c8e0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 --2- 172.21.15.28:0/3798727939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558c3534d430 0x558c3535c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 >> 172.21.15.28:0/3798727939 conn(0x558c353489a0 msgr2=0x558c3534c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.293+0000 7fe074ff9640 0 client.17991 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.294+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 shutdown_connections 2024-03-22T02:22:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.294+0000 7fe095ff0740 1 -- 172.21.15.28:0/3798727939 wait complete. 2024-03-22T02:22:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.294+0000 7fe095ff0740 20 client.17991 trim_cache size 0 max 16384 2024-03-22T02:22:04.295 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "96319dfc-c605-41ce-bfa9-06652ee35caa": { 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "96319dfc-c605-41ce-bfa9-06652ee35caa", 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:22:04.296 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 msgr2=0x7fe0780a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 0x7fe0780a5460 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe084005e30 tx=0x7fe084032590 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 msgr2=0x7fe08c1983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00ea60 tx=0x7fe07c00ef20 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 shutdown_connections 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe0780a2fa0 0x7fe0780a5460 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe08c102640 0x7fe08c19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe08c101c80 0x7fe08c1983f0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 --2- 172.21.15.28:0/3873091112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe08c100a90 0x7fe08c197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.295+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 >> 172.21.15.28:0/3873091112 conn(0x7fe08c0fc020 msgr2=0x7fe08c0fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.296+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 shutdown_connections 2024-03-22T02:22:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:04.296+0000 7fe093332640 1 -- 172.21.15.28:0/3873091112 wait complete. 2024-03-22T02:22:04.309 INFO:tasks.cephfs.filesystem:scrub status for tag:96319dfc-c605-41ce-bfa9-06652ee35caa - {'path': '/', 'tag': '96319dfc-c605-41ce-bfa9-06652ee35caa', 'options': 'recursive,force'} 2024-03-22T02:22:04.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 813835 11.70 MB/sec execute 286 sec latency 39.734 ms 2024-03-22T02:22:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:04 smithi028 ceph-mon[31916]: pgmap v614: 129 pgs: 129 active+clean; 52 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:22:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:04 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:22:04.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:04 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:04 smithi083 ceph-mon[36340]: pgmap v614: 129 pgs: 129 active+clean; 52 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:22:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:04 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:22:04.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:04 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:04 smithi120 ceph-mon[36803]: pgmap v614: 129 pgs: 129 active+clean; 52 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:22:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:04 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:22:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:04 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 815822 11.70 MB/sec execute 287 sec latency 259.505 ms 2024-03-22T02:22:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:05 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:22:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:05 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:05 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:22:05.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:05 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:05 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:22:05.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:05 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:05 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:22:05.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:05 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:05 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:22:05.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:05 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:05.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:05 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:22:05.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:05 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:06.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 817586 11.69 MB/sec execute 288 sec latency 29.642 ms 2024-03-22T02:22:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:06 smithi083 ceph-mon[36340]: pgmap v615: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 445 op/s 2024-03-22T02:22:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:06 smithi028 ceph-mon[31916]: pgmap v615: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 445 op/s 2024-03-22T02:22:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:06 smithi120 ceph-mon[36803]: pgmap v615: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.1 MiB/s wr, 445 op/s 2024-03-22T02:22:07.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 820391 11.71 MB/sec execute 289 sec latency 20.834 ms 2024-03-22T02:22:08.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 822626 11.70 MB/sec execute 290 sec latency 46.276 ms 2024-03-22T02:22:08.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:08 smithi083 ceph-mon[36340]: pgmap v616: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 447 op/s 2024-03-22T02:22:08.800 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:08 smithi120 ceph-mon[36803]: pgmap v616: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 447 op/s 2024-03-22T02:22:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:08 smithi028 ceph-mon[31916]: pgmap v616: 129 pgs: 129 active+clean; 51 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.1 MiB/s wr, 447 op/s 2024-03-22T02:22:09.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 824160 11.70 MB/sec execute 291 sec latency 90.107 ms 2024-03-22T02:22:10.425 INFO:tasks.workunit.client.0.smithi028.stdout: 1 826603 11.70 MB/sec execute 292 sec latency 37.766 ms 2024-03-22T02:22:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:10 smithi083 ceph-mon[36340]: pgmap v617: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-22T02:22:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:10 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:22:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:10 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:22:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:10 smithi028 ceph-mon[31916]: pgmap v617: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-22T02:22:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:10 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:22:10.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:10 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:22:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:10 smithi120 ceph-mon[36803]: pgmap v617: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.0 MiB/s wr, 392 op/s 2024-03-22T02:22:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:10 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:22:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:10 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:22:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.324+0000 7f237eb5c640 1 -- 172.21.15.28:0/3123635601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378101ca0 msgr2=0x7f2378102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.324+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3123635601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378101ca0 0x7f2378102120 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f236c009860 tx=0x7f236c02bb70 comp rx=0 tx=0).stop 2024-03-22T02:22:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- 172.21.15.28:0/3123635601 shutdown_connections 2024-03-22T02:22:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3123635601 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378102660 0x7f2378108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3123635601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378101ca0 0x7f2378102120 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3123635601 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 0x7f2378100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- 172.21.15.28:0/3123635601 >> 172.21.15.28:0/3123635601 conn(0x7f23780fc020 msgr2=0x7f23780fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- 172.21.15.28:0/3123635601 shutdown_connections 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- 172.21.15.28:0/3123635601 wait complete. 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 Processor -- start 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- start start 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 0x7f2378069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378101ca0 0x7f237819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f237810c710 con 0x7f2378102660 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f237810c590 con 0x7f2378100ab0 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237eb5c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f237810c890 con 0x7f2378101ca0 2024-03-22T02:22:11.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.325+0000 7f237c8f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 0x7f2378069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40284/0 (socket says 172.21.15.28:40284) 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 -- 172.21.15.28:0/3010391353 learned_addr learned my addr 172.21.15.28:0/3010391353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f2377fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378101ca0 0x7f237819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 -- 172.21.15.28:0/3010391353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378101ca0 msgr2=0x7f237819cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378101ca0 0x7f237819cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 -- 172.21.15.28:0/3010391353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 msgr2=0x7f2378069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 0x7f2378069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 -- 172.21.15.28:0/3010391353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237819fae0 con 0x7f2378102660 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f236800e940 tx=0x7f236800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f2375ffb640 1 -- 172.21.15.28:0/3010391353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236800cd80 con 0x7f2378102660 2024-03-22T02:22:11.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f2375ffb640 1 -- 172.21.15.28:0/3010391353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23680035f0 con 0x7f2378102660 2024-03-22T02:22:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237819fd70 con 0x7f2378102660 2024-03-22T02:22:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f2375ffb640 1 -- 172.21.15.28:0/3010391353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368010690 con 0x7f2378102660 2024-03-22T02:22:11.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.326+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23781a01e0 con 0x7f2378102660 2024-03-22T02:22:11.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.328+0000 7f2375ffb640 1 -- 172.21.15.28:0/3010391353 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f23680037a0 con 0x7f2378102660 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 Processor -- start 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2375ffb640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 0x7f23600a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 -- start start 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d69de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2377fff640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 0x7f23600a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d6a450 0x556c85d6c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c85cc5040 con 0x556c85d648f0 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556c85c6d9e0 con 0x556c85d652e0 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2381815740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556c85cc0ff0 con 0x556c85d6a450 2024-03-22T02:22:11.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2375ffb640 1 -- 172.21.15.28:0/3010391353 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f23680cfb90 con 0x7f2378102660 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40288/0 (socket says 172.21.15.28:40288) 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 -- 172.21.15.28:0/2136476534 learned_addr learned my addr 172.21.15.28:0/2136476534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f2377fff640 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d69de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237d0f9640 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d6a450 0x556c85d6c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 -- 172.21.15.28:0/2136476534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d6a450 msgr2=0x556c85d6c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d6a450 0x556c85d6c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 -- 172.21.15.28:0/2136476534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 msgr2=0x556c85d69de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.329+0000 7f237c8f8640 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d69de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f237c8f8640 1 -- 172.21.15.28:0/2136476534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c859d9210 con 0x556c85d648f0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f2377fff640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 0x7f23600a5480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f236c02c180 tx=0x7f236c0323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f237c8f8640 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2370011470 tx=0x7f2370011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea335468770ba213 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f235effd640 1 -- 172.21.15.28:0/2136476534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2370003970 con 0x556c85d648f0 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f235effd640 1 -- 172.21.15.28:0/2136476534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2370004020 con 0x556c85d648f0 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.330+0000 7f235effd640 1 -- 172.21.15.28:0/2136476534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237000f930 con 0x556c85d648f0 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- 172.21.15.28:0/2136476534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 msgr2=0x556c85d64d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f2370011470 tx=0x7f2370011930 comp rx=0 tx=0).stop 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- 172.21.15.28:0/2136476534 shutdown_connections 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d6a450 0x556c85d6c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d69de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- 172.21.15.28:0/2136476534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d648f0 0x556c85d64d10 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- 172.21.15.28:0/2136476534 >> 172.21.15.28:0/2136476534 conn(0x556c85d5ba50 msgr2=0x556c85d5be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- 172.21.15.28:0/2136476534 shutdown_connections 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- 172.21.15.28:0/2136476534 wait complete. 2024-03-22T02:22:11.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- start start 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 Processor -- start 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d6e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d75e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556c85d6d400 con 0x556c85d6a450 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556c85c80db0 con 0x556c85d652e0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556c85d6d580 con 0x556c85d648f0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.331+0000 7f2381815740 10 client.0 _ll_register_callbacks cb 0x556c859d8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f237d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d75e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f237c8f8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d6e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33408/0 (socket says 172.21.15.28:33408) 2024-03-22T02:22:11.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 -- 172.21.15.28:0/765288242 learned_addr learned my addr 172.21.15.28:0/765288242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 -- 172.21.15.28:0/765288242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 msgr2=0x556c85d6e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d6e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 -- 172.21.15.28:0/765288242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 msgr2=0x556c85d75e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d75e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 -- 172.21.15.28:0/765288242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c85aa5620 con 0x556c85d652e0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2377fff640 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f236c005b60 tx=0x7f236c02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ac58647f0ebc86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2357fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c039040 con 0x556c85d652e0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2370004020 con 0x556c85d652e0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f236c034740 con 0x556c85d652e0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f236c059050 con 0x556c85d652e0 2024-03-22T02:22:11.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2381815740 10 client.27827 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.332+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556c85d79240 con 0x556c85d652e0 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2370004020 con 0x556c85d652e0 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f236c03f070 con 0x556c85d652e0 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 10 client.27827 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 10 client.27827 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 20 client.27827 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85d7a940 0x556c85d7cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 4 client.27827 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c85cd2580 con 0x556c85d7a940 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f237c8f8640 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85d7a940 0x556c85d7cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f237c8f8640 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85d7a940 0x556c85d7cd90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=464931fa45bcaef5 server_cookie=b2258e16b69359ed in_seq=0 out_seq=0 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.333+0000 7f2357fff640 10 client.27827 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2357fff640 1 -- 172.21.15.28:0/765288242 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c85cd2580 con 0x556c85d7a940 2024-03-22T02:22:11.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2357fff640 10 client.27827 handle_command_reply: tid=0 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 client.27827 shutdown 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 msgr2=0x556c85d6ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f236c005b60 tx=0x7f236c02bb70 comp rx=0 tx=0).stop 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 shutdown_connections 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85d7a940 0x556c85d7cd90 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d75e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d652e0 0x556c85d6ee40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 --2- 172.21.15.28:0/765288242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d6e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2357fff640 0 client.27827 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 >> 172.21.15.28:0/765288242 conn(0x556c85d5ba50 msgr2=0x556c85d5ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 shutdown_connections 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 1 -- 172.21.15.28:0/765288242 wait complete. 2024-03-22T02:22:11.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.335+0000 7f2381815740 20 client.27827 trim_cache size 0 max 16384 2024-03-22T02:22:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 Processor -- start 2024-03-22T02:22:11.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 -- start start 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 0x556c85d69930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d652e0 0x556c85d75dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23781aae90 con 0x556c85d6a450 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a8760 con 0x556c85d648f0 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f2381815740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a65b0 con 0x556c85d652e0 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f237d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f237d0f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40312/0 (socket says 172.21.15.28:40312) 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f237c8f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 0x556c85d69930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.351+0000 7f237d0f9640 1 -- 172.21.15.28:0/3411023355 learned_addr learned my addr 172.21.15.28:0/3411023355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f2377fff640 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d652e0 0x556c85d75dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 -- 172.21.15.28:0/3411023355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d652e0 msgr2=0x556c85d75dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d652e0 0x556c85d75dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 -- 172.21.15.28:0/3411023355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 msgr2=0x556c85d69930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 0x556c85d69930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 -- 172.21.15.28:0/3411023355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c85d656c0 con 0x556c85d6a450 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237c8f8640 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 0x556c85d69930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f237d0f9640 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f23680982b0 tx=0x7f23680092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a83fd53d507f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f235e7fc640 1 -- 172.21.15.28:0/3411023355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236809be50 con 0x556c85d6a450 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f235e7fc640 1 -- 172.21.15.28:0/3411023355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23680b6070 con 0x556c85d6a450 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.352+0000 7f235e7fc640 1 -- 172.21.15.28:0/3411023355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23680babc0 con 0x556c85d6a450 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- 172.21.15.28:0/3411023355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 msgr2=0x556c85d65b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f23680982b0 tx=0x7f23680092d0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- 172.21.15.28:0/3411023355 shutdown_connections 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d6a450 0x556c85d65b70 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d652e0 0x556c85d75dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 --2- 172.21.15.28:0/3411023355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d648f0 0x556c85d69930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- 172.21.15.28:0/3411023355 >> 172.21.15.28:0/3411023355 conn(0x556c85d5ba50 msgr2=0x556c85d5f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- 172.21.15.28:0/3411023355 shutdown_connections 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- 172.21.15.28:0/3411023355 wait complete. 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 -- start start 2024-03-22T02:22:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.353+0000 7f2381815740 1 Processor -- start 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d73960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d6a450 0x556c85d74eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a6c20 con 0x556c85d652e0 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a8090 con 0x556c85d6a450 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a72f0 con 0x556c85d648f0 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 10 client.0 _ll_register_callbacks cb 0x7f2368005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40322/0 (socket says 172.21.15.28:40322) 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 -- 172.21.15.28:0/1938624197 learned_addr learned my addr 172.21.15.28:0/1938624197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f237c8f8640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d73960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f237d0f9640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d6a450 0x556c85d74eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 -- 172.21.15.28:0/1938624197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 msgr2=0x556c85d73960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d73960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 -- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d6a450 msgr2=0x556c85d74eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d6a450 0x556c85d74eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 -- 172.21.15.28:0/1938624197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c85d754a0 con 0x556c85d652e0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2377fff640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f236c003740 tx=0x7f236c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482e92163238ccff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2357fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c061070 con 0x556c85d652e0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f236c02cae0 con 0x556c85d652e0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23680b6070 con 0x556c85d652e0 2024-03-22T02:22:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f236c06d050 con 0x556c85d652e0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 10 client.18009 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.354+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556c85de11f0 con 0x556c85d652e0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23680b6070 con 0x556c85d652e0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f236c03f050 con 0x556c85d652e0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 10 client.18009 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 10 client.18009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 20 client.18009 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85de1bd0 0x556c85de3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 4 client.18009 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556c85ccc8d0 con 0x556c85de1bd0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f237c8f8640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85de1bd0 0x556c85de3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f237c8f8640 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85de1bd0 0x556c85de3ff0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f480f6d6abbed14 server_cookie=8dc8f4c6d15dc5b in_seq=0 out_seq=0 2024-03-22T02:22:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.355+0000 7f2357fff640 10 client.18009 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:11.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 828196 11.68 MB/sec execute 293 sec latency 51.471 ms 2024-03-22T02:22:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2357fff640 1 -- 172.21.15.28:0/1938624197 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556c85ccc8d0 con 0x556c85de1bd0 2024-03-22T02:22:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2357fff640 10 client.18009 handle_command_reply: tid=0 2024-03-22T02:22:11.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 client.18009 shutdown 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 msgr2=0x556c85d73ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f236c003740 tx=0x7f236c005f10 comp rx=0 tx=0).stop 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 shutdown_connections 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556c85de1bd0 0x556c85de3ff0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2357fff640 0 client.18009 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556c85d6a450 0x556c85d74eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556c85d652e0 0x556c85d73ea0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 --2- 172.21.15.28:0/1938624197 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556c85d648f0 0x556c85d73960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 >> 172.21.15.28:0/1938624197 conn(0x556c85d5ba50 msgr2=0x556c85d5f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 shutdown_connections 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 1 -- 172.21.15.28:0/1938624197 wait complete. 2024-03-22T02:22:11.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.653+0000 7f2381815740 20 client.18009 trim_cache size 0 max 16384 2024-03-22T02:22:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:22:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:22:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:22:11.657 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 msgr2=0x7f23600a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 0x7f23600a5480 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f236c02c180 tx=0x7f236c0323c0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 msgr2=0x7f237819f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f236800e940 tx=0x7f236800ee00 comp rx=0 tx=0).stop 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 shutdown_connections 2024-03-22T02:22:11.658 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f23600a2fc0 0x7f23600a5480 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2378102660 0x7f237819f1d0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2378101ca0 0x7f237819cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 --2- 172.21.15.28:0/3010391353 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2378100ab0 0x7f2378069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 >> 172.21.15.28:0/3010391353 conn(0x7f23780fc020 msgr2=0x7f2378104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 shutdown_connections 2024-03-22T02:22:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:11.654+0000 7f237eb5c640 1 -- 172.21.15.28:0/3010391353 wait complete. 2024-03-22T02:22:12.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 830577 11.69 MB/sec execute 294 sec latency 187.305 ms 2024-03-22T02:22:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:12 smithi028 ceph-mon[31916]: pgmap v618: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-03-22T02:22:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:12 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:12 smithi120 ceph-mon[36803]: pgmap v618: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-03-22T02:22:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:12 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:12 smithi083 ceph-mon[36340]: pgmap v618: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-03-22T02:22:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:12 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:22:13.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 831848 11.67 MB/sec execute 295 sec latency 42.042 ms 2024-03-22T02:22:13.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:13.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:13.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:22:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:22:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:22:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:22:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:22:14.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 834392 11.69 MB/sec execute 296 sec latency 62.266 ms 2024-03-22T02:22:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:14 smithi028 ceph-mon[31916]: pgmap v619: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-22T02:22:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:14 smithi120 ceph-mon[36803]: pgmap v619: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-22T02:22:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:14 smithi083 ceph-mon[36340]: pgmap v619: 129 pgs: 129 active+clean; 54 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.5 MiB/s wr, 309 op/s 2024-03-22T02:22:15.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 836010 11.67 MB/sec execute 297 sec latency 35.443 ms 2024-03-22T02:22:16.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 838386 11.68 MB/sec execute 298 sec latency 22.151 ms 2024-03-22T02:22:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:16 smithi028 ceph-mon[31916]: pgmap v620: 129 pgs: 129 active+clean; 49 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2024-03-22T02:22:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:16 smithi120 ceph-mon[36803]: pgmap v620: 129 pgs: 129 active+clean; 49 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2024-03-22T02:22:17.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:16 smithi083 ceph-mon[36340]: pgmap v620: 129 pgs: 129 active+clean; 49 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.3 MiB/s wr, 464 op/s 2024-03-22T02:22:17.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 840919 11.68 MB/sec execute 299 sec latency 36.309 ms 2024-03-22T02:22:18.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 842699 11.67 MB/sec execute 300 sec latency 28.991 ms 2024-03-22T02:22:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:18 smithi028 ceph-mon[31916]: pgmap v621: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 408 op/s 2024-03-22T02:22:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:18 smithi120 ceph-mon[36803]: pgmap v621: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 408 op/s 2024-03-22T02:22:19.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:18 smithi083 ceph-mon[36340]: pgmap v621: 129 pgs: 129 active+clean; 50 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.7 MiB/s wr, 408 op/s 2024-03-22T02:22:19.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 845262 11.69 MB/sec execute 301 sec latency 53.528 ms 2024-03-22T02:22:20.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 847197 11.67 MB/sec execute 302 sec latency 29.559 ms 2024-03-22T02:22:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:20 smithi028 ceph-mon[31916]: pgmap v622: 129 pgs: 129 active+clean; 51 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 459 op/s 2024-03-22T02:22:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:20 smithi120 ceph-mon[36803]: pgmap v622: 129 pgs: 129 active+clean; 51 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 459 op/s 2024-03-22T02:22:21.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:20 smithi083 ceph-mon[36340]: pgmap v622: 129 pgs: 129 active+clean; 51 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.3 MiB/s wr, 459 op/s 2024-03-22T02:22:21.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 849166 11.68 MB/sec execute 303 sec latency 21.056 ms 2024-03-22T02:22:22.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 851736 11.68 MB/sec execute 304 sec latency 46.934 ms 2024-03-22T02:22:22.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:22 smithi028 ceph-mon[31916]: pgmap v623: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 402 op/s 2024-03-22T02:22:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:22 smithi120 ceph-mon[36803]: pgmap v623: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 402 op/s 2024-03-22T02:22:23.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:22 smithi083 ceph-mon[36340]: pgmap v623: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 402 op/s 2024-03-22T02:22:23.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 853180 11.67 MB/sec execute 305 sec latency 37.369 ms 2024-03-22T02:22:24.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 855750 11.68 MB/sec execute 306 sec latency 51.884 ms 2024-03-22T02:22:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.744+0000 7f8169dc9640 1 -- 172.21.15.28:0/94477589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81641026d0 msgr2=0x7f8164104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.744+0000 7f8169dc9640 1 --2- 172.21.15.28:0/94477589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81641026d0 0x7f8164104ac0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f8154009900 tx=0x7f815402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 -- 172.21.15.28:0/94477589 shutdown_connections 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 --2- 172.21.15.28:0/94477589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 0x7f8164109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 --2- 172.21.15.28:0/94477589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8164105000 0x7f81641073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 --2- 172.21.15.28:0/94477589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81641026d0 0x7f8164104ac0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 -- 172.21.15.28:0/94477589 >> 172.21.15.28:0/94477589 conn(0x7f81640fc020 msgr2=0x7f81640fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 -- 172.21.15.28:0/94477589 shutdown_connections 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 -- 172.21.15.28:0/94477589 wait complete. 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 Processor -- start 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 -- start start 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81641026d0 0x7f8164102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.745+0000 7f8169dc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 0x7f8164100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f816410d840 con 0x7f8164105000 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f816410d6c0 con 0x7f81641026d0 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f816410d9c0 con 0x7f8164107930 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53932/0 (socket says 172.21.15.28:53932) 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 -- 172.21.15.28:0/1165918261 learned_addr learned my addr 172.21.15.28:0/1165918261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f81637fe640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81641026d0 0x7f8164102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8163fff640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 0x7f8164100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 -- 172.21.15.28:0/1165918261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 msgr2=0x7f8164100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 0x7f8164100bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 -- 172.21.15.28:0/1165918261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81641026d0 msgr2=0x7f8164102040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81641026d0 0x7f8164102040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 -- 172.21.15.28:0/1165918261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8164101450 con 0x7f8164105000 2024-03-22T02:22:24.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9c0 tx=0x7f815800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8160ff9640 1 -- 172.21.15.28:0/1165918261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800cd80 con 0x7f8164105000 2024-03-22T02:22:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8160ff9640 1 -- 172.21.15.28:0/1165918261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8158005500 con 0x7f8164105000 2024-03-22T02:22:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81641a1f70 con 0x7f8164105000 2024-03-22T02:22:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8160ff9640 1 -- 172.21.15.28:0/1165918261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8158010690 con 0x7f8164105000 2024-03-22T02:22:24.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.746+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81641a23d0 con 0x7f8164105000 2024-03-22T02:22:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f8160ff9640 1 -- 172.21.15.28:0/1165918261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f81580046c0 con 0x7f8164105000 2024-03-22T02:22:24.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 Processor -- start 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 -- start start 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 0x5635e7f174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635e7e74040 con 0x5635e7f15040 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5635e7e1c9e0 con 0x5635e7f0f4e0 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.749+0000 7f816c9b3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5635e7e6fff0 con 0x5635e7f0fed0 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8163fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 0x5635e7f174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:57016/0 (socket says 172.21.15.28:57016) 2024-03-22T02:22:24.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8163fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53068/0 (socket says 172.21.15.28:53068) 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 -- 172.21.15.28:0/3207130592 learned_addr learned my addr 172.21.15.28:0/3207130592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8160ff9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 0x7f814c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8163fff640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 0x7f814c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 -- 172.21.15.28:0/3207130592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 msgr2=0x5635e7f0f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 -- 172.21.15.28:0/3207130592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 msgr2=0x5635e7f174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8160ff9640 1 -- 172.21.15.28:0/1165918261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f81580cfb50 con 0x7f8164105000 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 0x5635e7f174d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 -- 172.21.15.28:0/3207130592 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e7b88210 con 0x5635e7f0fed0 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8162ffd640 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 0x5635e7f174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:24.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f81637fe640 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8154009900 tx=0x7f8154009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0a98f68356964a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8163fff640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 0x7f814c0a5480 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5635e7f10730 tx=0x7f815c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8149ffb640 1 -- 172.21.15.28:0/3207130592 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815402cb80 con 0x5635e7f0fed0 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.750+0000 7f8149ffb640 1 -- 172.21.15.28:0/3207130592 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8154005040 con 0x5635e7f0fed0 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f8149ffb640 1 -- 172.21.15.28:0/3207130592 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81540048c0 con 0x5635e7f0fed0 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 -- 172.21.15.28:0/3207130592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 msgr2=0x5635e7f149d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8154009900 tx=0x7f8154009560 comp rx=0 tx=0).stop 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 -- 172.21.15.28:0/3207130592 shutdown_connections 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f15040 0x5635e7f174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f149d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3207130592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 -- 172.21.15.28:0/3207130592 >> 172.21.15.28:0/3207130592 conn(0x5635e7f0aa50 msgr2=0x5635e7f0ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 -- 172.21.15.28:0/3207130592 shutdown_connections 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.751+0000 7f816c9b3740 1 -- 172.21.15.28:0/3207130592 wait complete. 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 -- start start 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 Processor -- start 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f199c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f20a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5635e7f17f90 con 0x5635e7f0f4e0 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5635e7e2fdb0 con 0x5635e7f15040 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5635e7f18110 con 0x5635e7f0fed0 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f816c9b3740 10 client.0 _ll_register_callbacks cb 0x5635e7b87a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f81637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f199c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f8163fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f20a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f81637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53942/0 (socket says 172.21.15.28:53942) 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.752+0000 7f81637fe640 1 -- 172.21.15.28:0/1416169283 learned_addr learned my addr 172.21.15.28:0/1416169283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 -- 172.21.15.28:0/1416169283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 msgr2=0x5635e7f199c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f199c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 -- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 msgr2=0x5635e7f20a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f20a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 -- 172.21.15.28:0/1416169283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e7c54620 con 0x5635e7f0f4e0 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8163fff640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f20a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f81637fe640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f815402c170 tx=0x7f81540054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705d6bdb49343b74 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154004ad0 con 0x5635e7f0f4e0 2024-03-22T02:22:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8154004c70 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8154005040 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8154056050 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f816c9b3740 10 client.18021 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5635e7f23d70 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8154005040 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.753+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f8154045420 con 0x5635e7f0f4e0 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 10 client.18021 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 10 client.18021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 20 client.18021 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f250c0 0x5635e7f274b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 4 client.18021 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635e7e81580 con 0x5635e7f250c0 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f250c0 0x5635e7f274b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f250c0 0x5635e7f274b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90ba721d3e72ec5 server_cookie=c9fa57047ce7d478 in_seq=0 out_seq=0 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.754+0000 7f8142ffd640 10 client.18021 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f8142ffd640 1 -- 172.21.15.28:0/1416169283 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635e7e81580 con 0x5635e7f250c0 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f8142ffd640 10 client.18021 handle_command_reply: tid=0 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 client.18021 shutdown 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 msgr2=0x5635e7f19480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f815402c170 tx=0x7f81540054a0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 shutdown_connections 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f250c0 0x5635e7f274b0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f20a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f199c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f8142ffd640 0 client.18021 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1416169283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f19480 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 >> 172.21.15.28:0/1416169283 conn(0x5635e7f0aa50 msgr2=0x5635e7f09c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 shutdown_connections 2024-03-22T02:22:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 1 -- 172.21.15.28:0/1416169283 wait complete. 2024-03-22T02:22:24.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.755+0000 7f816c9b3740 20 client.18021 trim_cache size 0 max 16384 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.770+0000 7f816c9b3740 1 Processor -- start 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 -- start start 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f18750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81641af410 con 0x5635e7f0fed0 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81641acce0 con 0x5635e7f15040 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f816c9b3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81641aab30 con 0x5635e7f0f4e0 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53954/0 (socket says 172.21.15.28:53954) 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 -- 172.21.15.28:0/1783039054 learned_addr learned my addr 172.21.15.28:0/1783039054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f81637fe640 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8163fff640 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f18750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 -- 172.21.15.28:0/1783039054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0f4e0 msgr2=0x5635e7f0e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 -- 172.21.15.28:0/1783039054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 msgr2=0x5635e7f18750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f18750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 -- 172.21.15.28:0/1783039054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635e7f0a3a0 con 0x5635e7f0fed0 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f8162ffd640 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f815800c6a0 tx=0x7f81580bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bdf90ce15c5c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f81497fa640 1 -- 172.21.15.28:0/1783039054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81580b4070 con 0x5635e7f0fed0 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f81497fa640 1 -- 172.21.15.28:0/1783039054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f815809c510 con 0x5635e7f0fed0 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.771+0000 7f81497fa640 1 -- 172.21.15.28:0/1783039054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81580cd2f0 con 0x5635e7f0fed0 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- 172.21.15.28:0/1783039054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 msgr2=0x5635e7f209c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f815800c6a0 tx=0x7f81580bc040 comp rx=0 tx=0).stop 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- 172.21.15.28:0/1783039054 shutdown_connections 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f18750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0fed0 0x5635e7f209c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 --2- 172.21.15.28:0/1783039054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0f4e0 0x5635e7f0e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- 172.21.15.28:0/1783039054 >> 172.21.15.28:0/1783039054 conn(0x5635e7f0aa50 msgr2=0x5635e7f0dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:24.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- 172.21.15.28:0/1783039054 shutdown_connections 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- 172.21.15.28:0/1783039054 wait complete. 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.772+0000 7f816c9b3740 1 -- start start 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 Processor -- start 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f1de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f89320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81641ab1a0 con 0x5635e7f0f4e0 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81641ac610 con 0x5635e7f15040 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81641ab870 con 0x5635e7f0fed0 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f816c9b3740 10 client.0 _ll_register_callbacks cb 0x7f815809bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53964/0 (socket says 172.21.15.28:53964) 2024-03-22T02:22:24.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 -- 172.21.15.28:0/3424721023 learned_addr learned my addr 172.21.15.28:0/3424721023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f8162ffd640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f1de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f8163fff640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f89320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 -- 172.21.15.28:0/3424721023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 msgr2=0x5635e7f1de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f1de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 -- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 msgr2=0x5635e7f89320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f89320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 -- 172.21.15.28:0/3424721023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635e7f89890 con 0x5635e7f0f4e0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f81637fe640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8154001eb0 tx=0x7f815402cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934f3c6720d28d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f8142ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.773+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154045cb0 con 0x5635e7f0f4e0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8154005510 con 0x5635e7f0f4e0 2024-03-22T02:22:24.780 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f815809c510 con 0x5635e7f0f4e0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8154045630 con 0x5635e7f0f4e0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 10 client.18027 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5635e7f8abf0 con 0x5635e7f0f4e0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f815809c510 con 0x5635e7f0f4e0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f8154004d90 con 0x5635e7f0f4e0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 10 client.18027 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 10 client.18027 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 20 client.18027 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f8b5c0 0x5635e7f8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 4 client.18027 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635e7e7b8d0 con 0x5635e7f8b5c0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.774+0000 7f8162ffd640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f8b5c0 0x5635e7f8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:24.781 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.775+0000 7f8142ffd640 10 client.18027 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:24.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:24.775+0000 7f8162ffd640 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f8b5c0 0x5635e7f8d9e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53f0c50af86ef2e1 server_cookie=48163efab8a9b473 in_seq=0 out_seq=0 2024-03-22T02:22:24.830 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:24 smithi028 ceph-mon[31916]: pgmap v624: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 389 op/s 2024-03-22T02:22:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:24 smithi120 ceph-mon[36803]: pgmap v624: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 389 op/s 2024-03-22T02:22:25.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:24 smithi083 ceph-mon[36340]: pgmap v624: 129 pgs: 129 active+clean; 53 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 389 op/s 2024-03-22T02:22:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f8142ffd640 1 -- 172.21.15.28:0/3424721023 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635e7e7b8d0 con 0x5635e7f8b5c0 2024-03-22T02:22:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f8142ffd640 10 client.18027 handle_command_reply: tid=0 2024-03-22T02:22:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 client.18027 shutdown 2024-03-22T02:22:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 msgr2=0x5635e7f1d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f8154001eb0 tx=0x7f815402cec0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 shutdown_connections 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5635e7f8b5c0 0x5635e7f8d9e0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5635e7f15040 0x5635e7f89320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5635e7f0fed0 0x5635e7f1de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f8142ffd640 0 client.18027 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 --2- 172.21.15.28:0/3424721023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5635e7f0f4e0 0x5635e7f1d900 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 >> 172.21.15.28:0/3424721023 conn(0x5635e7f0aa50 msgr2=0x5635e7f0e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 shutdown_connections 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 1 -- 172.21.15.28:0/3424721023 wait complete. 2024-03-22T02:22:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.057+0000 7f816c9b3740 20 client.18027 trim_cache size 0 max 16384 2024-03-22T02:22:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:22:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:22:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:22:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 msgr2=0x7f814c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 0x7f814c0a5480 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x5635e7f10730 tx=0x7f815c004ab0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 msgr2=0x7f8164100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f815800e9c0 tx=0x7f815800ee80 comp rx=0 tx=0).stop 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 shutdown_connections 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f814c0a2fc0 0x7f814c0a5480 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8164107930 0x7f8164100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8164105000 0x7f8164100680 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 --2- 172.21.15.28:0/1165918261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81641026d0 0x7f8164102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 >> 172.21.15.28:0/1165918261 conn(0x7f81640fc020 msgr2=0x7f8164103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 shutdown_connections 2024-03-22T02:22:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:25.058+0000 7f8169dc9640 1 -- 172.21.15.28:0/1165918261 wait complete. 2024-03-22T02:22:25.426 INFO:tasks.workunit.client.0.smithi028.stdout: 1 856461 11.65 MB/sec execute 307 sec latency 153.414 ms 2024-03-22T02:22:26.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 859231 11.67 MB/sec execute 308 sec latency 12.037 ms 2024-03-22T02:22:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:26 smithi028 ceph-mon[31916]: pgmap v625: 129 pgs: 129 active+clean; 57 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 466 op/s 2024-03-22T02:22:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:26 smithi120 ceph-mon[36803]: pgmap v625: 129 pgs: 129 active+clean; 57 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 466 op/s 2024-03-22T02:22:27.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:26 smithi083 ceph-mon[36340]: pgmap v625: 129 pgs: 129 active+clean; 57 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.5 MiB/s wr, 466 op/s 2024-03-22T02:22:27.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 860792 11.65 MB/sec execute 309 sec latency 45.993 ms 2024-03-22T02:22:28.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 863147 11.66 MB/sec execute 310 sec latency 13.000 ms 2024-03-22T02:22:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:28 smithi028 ceph-mon[31916]: pgmap v626: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-03-22T02:22:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:28 smithi120 ceph-mon[36803]: pgmap v626: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-03-22T02:22:29.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:28 smithi083 ceph-mon[36340]: pgmap v626: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-03-22T02:22:29.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 865351 11.65 MB/sec execute 311 sec latency 30.419 ms 2024-03-22T02:22:30.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 866981 11.65 MB/sec execute 312 sec latency 58.263 ms 2024-03-22T02:22:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:30 smithi028 ceph-mon[31916]: pgmap v627: 129 pgs: 129 active+clean; 53 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:22:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:30 smithi120 ceph-mon[36803]: pgmap v627: 129 pgs: 129 active+clean; 53 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:22:31.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:30 smithi083 ceph-mon[36340]: pgmap v627: 129 pgs: 129 active+clean; 53 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:22:31.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 869661 11.65 MB/sec execute 313 sec latency 23.591 ms 2024-03-22T02:22:32.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 871726 11.65 MB/sec execute 314 sec latency 30.183 ms 2024-03-22T02:22:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:32 smithi028 ceph-mon[31916]: pgmap v628: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:22:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:32 smithi120 ceph-mon[36803]: pgmap v628: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:22:33.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:32 smithi083 ceph-mon[36340]: pgmap v628: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:22:33.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 873888 11.66 MB/sec execute 315 sec latency 24.866 ms 2024-03-22T02:22:34.310 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:22:34.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:22:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/2549137205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a0102130 msgr2=0x7fb6a0102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/2549137205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a0102130 0x7fb6a0102550 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb69002bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/2549137205 shutdown_connections 2024-03-22T02:22:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/2549137205 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a01005e0 0x7fb6a0100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/2549137205 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a00ffc20 0x7fb6a01000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/2549137205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a0102130 0x7fb6a0102550 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/2549137205 >> 172.21.15.28:0/2549137205 conn(0x7fb6a00fb7a0 msgr2=0x7fb6a00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/2549137205 shutdown_connections 2024-03-22T02:22:34.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.422+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/2549137205 wait complete. 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 Processor -- start 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 -- start start 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a0102130 0x7fb6a01a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010fa50 con 0x7fb6a01005e0 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.423+0000 7fb6a4d5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010f8d0 con 0x7fb6a00ffc20 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb6a4d5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a010fbd0 con 0x7fb6a0102130 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a0102130 0x7fb6a01a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44566/0 (socket says 172.21.15.28:44566) 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51152/0 (socket says 172.21.15.28:51152) 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb695d9b640 1 -- 172.21.15.28:0/3847674207 learned_addr learned my addr 172.21.15.28:0/3847674207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 -- 172.21.15.28:0/3847674207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a0102130 msgr2=0x7fb6a01a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a0102130 0x7fb6a01a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 -- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 msgr2=0x7fb6a019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 -- 172.21.15.28:0/3847674207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a01a1cc0 con 0x7fb6a00ffc20 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb695d9b640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.424+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb690009230 tx=0x7fb69002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.425+0000 7fb697fff640 1 -- 172.21.15.28:0/3847674207 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690041070 con 0x7fb6a00ffc20 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.425+0000 7fb697fff640 1 -- 172.21.15.28:0/3847674207 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb690004450 con 0x7fb6a00ffc20 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.425+0000 7fb697fff640 1 -- 172.21.15.28:0/3847674207 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb690003df0 con 0x7fb6a00ffc20 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.425+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a01a1f50 con 0x7fb6a00ffc20 2024-03-22T02:22:34.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.425+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a01a2360 con 0x7fb6a00ffc20 2024-03-22T02:22:34.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 876364 11.66 MB/sec execute 316 sec latency 45.457 ms 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.427+0000 7fb697fff640 1 -- 172.21.15.28:0/3847674207 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb690044070 con 0x7fb6a00ffc20 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 Processor -- start 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb697fff640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 0x7fb68c0a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb69ed9d640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 0x7fb68c0a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb697fff640 1 -- 172.21.15.28:0/3847674207 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb6900f10f0 con 0x7fb6a00ffc20 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 -- start start 2024-03-22T02:22:34.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d66850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d6e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f86ccb8c0 con 0x557f86d66e20 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f86c73c20 con 0x557f86d66430 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f86cc7870 con 0x557f86d6bf90 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d66850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44582/0 (socket says 172.21.15.28:44582) 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb695d9b640 1 -- 172.21.15.28:0/2435074923 learned_addr learned my addr 172.21.15.28:0/2435074923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb69ed9d640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 0x7fb68c0a5530 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb698005fd0 tx=0x7fb698009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.428+0000 7fb69ed9d640 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d6e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 -- 172.21.15.28:0/2435074923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 msgr2=0x557f86d6e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d6e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 -- 172.21.15.28:0/2435074923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 msgr2=0x557f86d6b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 -- 172.21.15.28:0/2435074923 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f86c98150 con 0x557f86d66430 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb695d9b640 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d66850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb6900eeb50 tx=0x7fb690000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2556e340419df45 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb694d99640 1 -- 172.21.15.28:0/2435074923 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69004b040 con 0x557f86d66430 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb694d99640 1 -- 172.21.15.28:0/2435074923 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6900dd050 con 0x557f86d66430 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.429+0000 7fb694d99640 1 -- 172.21.15.28:0/2435074923 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900035f0 con 0x557f86d66430 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2435074923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 msgr2=0x557f86d66850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d66850 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb6900eeb50 tx=0x7fb690000bc0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2435074923 shutdown_connections 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d6e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d6b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2435074923 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d66850 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2435074923 >> 172.21.15.28:0/2435074923 conn(0x557f86d619a0 msgr2=0x557f86d61da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2435074923 shutdown_connections 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2435074923 wait complete. 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- start start 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 Processor -- start 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d70670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d77bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f86d6ed00 con 0x557f86d66e20 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f86c87130 con 0x557f86d66430 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f86d6ee80 con 0x557f86d6bf90 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb6a7a21740 10 client.0 _ll_register_callbacks cb 0x557f86c86b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.430+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d70670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44586/0 (socket says 172.21.15.28:44586) 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb695d9b640 1 -- 172.21.15.28:0/2542591048 learned_addr learned my addr 172.21.15.28:0/2542591048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69ed9d640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d77bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 -- 172.21.15.28:0/2542591048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 msgr2=0x557f86d77bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d77bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 -- 172.21.15.28:0/2542591048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 msgr2=0x557f86d70bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 -- 172.21.15.28:0/2542591048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f86aa97d0 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb695d9b640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb69e59c640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d70670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb6900092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51fe9a257c7b1a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900db060 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb6900c1120 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6900dd050 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb69004b050 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb6a7a21740 10 client.27863 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.431+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557f869df3f0 con 0x557f86d66430 2024-03-22T02:22:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6900dd050 con 0x557f86d66430 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fb6900d62f0 con 0x557f86d66430 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 10 client.27863 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 10 client.27863 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 20 client.27863 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86d7c270 0x557f86d7e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 4 client.27863 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f86cd8e00 con 0x557f86d7c270 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb695d9b640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86d7c270 0x557f86d7e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.432+0000 7fb685ffb640 10 client.27863 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb695d9b640 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86d7c270 0x557f86d7e680 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48c76663130992de server_cookie=ba24aa6711cec2e6 in_seq=0 out_seq=0 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb685ffb640 1 -- 172.21.15.28:0/2542591048 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f86cd8e00 con 0x557f86d7c270 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb685ffb640 10 client.27863 handle_command_reply: tid=0 2024-03-22T02:22:34.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 client.27863 shutdown 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 msgr2=0x557f86d70670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d70670 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb690009900 tx=0x7fb6900092d0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 shutdown_connections 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86d7c270 0x557f86d7e680 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d6bf90 0x557f86d77bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d66e20 0x557f86d70bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/2542591048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d70670 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 >> 172.21.15.28:0/2542591048 conn(0x557f86d619a0 msgr2=0x557f86d60c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb685ffb640 0 client.27863 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.433+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 shutdown_connections 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.434+0000 7fb6a7a21740 1 -- 172.21.15.28:0/2542591048 wait complete. 2024-03-22T02:22:34.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.434+0000 7fb6a7a21740 20 client.27863 trim_cache size 0 max 16384 2024-03-22T02:22:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 Processor -- start 2024-03-22T02:22:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 -- start start 2024-03-22T02:22:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66e20 0x557f86d77b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86d6f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01af420 con 0x557f86d6bf90 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01accf0 con 0x557f86d66e20 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01aab40 con 0x557f86d66430 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86d6f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56396/0 (socket says 172.21.15.28:56396) 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66e20 0x557f86d77b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 -- 172.21.15.28:0/1551747317 learned_addr learned my addr 172.21.15.28:0/1551747317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 -- 172.21.15.28:0/1551747317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66e20 msgr2=0x557f86d77b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66e20 0x557f86d77b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 -- 172.21.15.28:0/1551747317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 msgr2=0x557f86d6f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86d6f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 -- 172.21.15.28:0/1551747317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8699f0f0 con 0x557f86d66430 2024-03-22T02:22:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.454+0000 7fb69e59c640 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb690006010 tx=0x7fb690000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6ec028d858d74f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.455+0000 7fb687fff640 1 -- 172.21.15.28:0/1551747317 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900db070 con 0x557f86d66430 2024-03-22T02:22:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.455+0000 7fb687fff640 1 -- 172.21.15.28:0/1551747317 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6900dd040 con 0x557f86d66430 2024-03-22T02:22:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.455+0000 7fb687fff640 1 -- 172.21.15.28:0/1551747317 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900c5e20 con 0x557f86d66430 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- 172.21.15.28:0/1551747317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 msgr2=0x557f86d6f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb690006010 tx=0x7fb690000bc0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- 172.21.15.28:0/1551747317 shutdown_connections 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86d6f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66e20 0x557f86d77b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/1551747317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66430 0x557f86d6f100 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- 172.21.15.28:0/1551747317 >> 172.21.15.28:0/1551747317 conn(0x557f86d619a0 msgr2=0x557f86d68380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- 172.21.15.28:0/1551747317 shutdown_connections 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- 172.21.15.28:0/1551747317 wait complete. 2024-03-22T02:22:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- start start 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 Processor -- start 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d74850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 0x557f86d74d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86de05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01ab1b0 con 0x557f86d6bf90 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01ac620 con 0x557f86d66430 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a01ab880 con 0x557f86d66e20 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb6a7a21740 10 client.0 _ll_register_callbacks cb 0x7fb690005dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.456+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86de05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 0x557f86d74d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb69ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86de05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44604/0 (socket says 172.21.15.28:44604) 2024-03-22T02:22:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb69ed9d640 1 -- 172.21.15.28:0/3493130107 learned_addr learned my addr 172.21.15.28:0/3493130107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d74850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 -- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 msgr2=0x557f86d74850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d74850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 -- 172.21.15.28:0/3493130107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 msgr2=0x557f86de05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86de05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 -- 172.21.15.28:0/3493130107 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f86d77df0 con 0x557f86d66e20 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb695d9b640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 0x557f86d74d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb688004700 tx=0x7fb688020890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f94b959f3d6db075 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688004250 con 0x557f86d66e20 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb6880043f0 con 0x557f86d66e20 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6900dd040 con 0x557f86d66e20 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fb6880220b0 con 0x557f86d66e20 2024-03-22T02:22:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb6a7a21740 10 client.27868 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.457+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557f86de1c00 con 0x557f86d66e20 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6900dd040 con 0x557f86d66e20 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fb688003bc0 con 0x557f86d66e20 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 10 client.27868 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 10 client.27868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 20 client.27868 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86de25b0 0x557f86de49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 4 client.27868 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f86cd3150 con 0x557f86de25b0 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86de25b0 0x557f86de49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb69e59c640 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86de25b0 0x557f86de49d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644b774e1bdf15ab server_cookie=2ab7f44b7b2fef71 in_seq=0 out_seq=0 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.458+0000 7fb685ffb640 10 client.27868 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:34.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.459+0000 7fb685ffb640 1 -- 172.21.15.28:0/3493130107 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f86cd3150 con 0x557f86de25b0 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.459+0000 7fb685ffb640 10 client.27868 handle_command_reply: tid=0 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.459+0000 7fb6a7a21740 1 client.27868 shutdown 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.459+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 msgr2=0x557f86d74d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.459+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 0x557f86d74d90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb688004700 tx=0x7fb688020890 comp rx=0 tx=0).stop 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 shutdown_connections 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557f86de25b0 0x557f86de49d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f86d6bf90 0x557f86de05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f86d66e20 0x557f86d74d90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb685ffb640 0 client.27868 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 --2- 172.21.15.28:0/3493130107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f86d66430 0x557f86d74850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 >> 172.21.15.28:0/3493130107 conn(0x557f86d619a0 msgr2=0x557f86d6f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 shutdown_connections 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 1 -- 172.21.15.28:0/3493130107 wait complete. 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a7a21740 20 client.27868 trim_cache size 0 max 16384 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:22:34.466 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 msgr2=0x7fb68c0a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 0x7fb68c0a5530 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb698005fd0 tx=0x7fb698009500 comp rx=0 tx=0).stop 2024-03-22T02:22:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 msgr2=0x7fb6a019a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb690009230 tx=0x7fb69002c650 comp rx=0 tx=0).stop 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 shutdown_connections 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb68c0a3070 0x7fb68c0a5530 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6a0102130 0x7fb6a01a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6a01005e0 0x7fb6a019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 --2- 172.21.15.28:0/3847674207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb6a00ffc20 0x7fb6a019a0a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 >> 172.21.15.28:0/3847674207 conn(0x7fb6a00fb7a0 msgr2=0x7fb6a00fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 shutdown_connections 2024-03-22T02:22:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.460+0000 7fb6a4d5e640 1 -- 172.21.15.28:0/3847674207 wait complete. 2024-03-22T02:22:34.472 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:22:34.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:22:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.635+0000 7f32605d7640 1 -- 172.21.15.28:0/863539520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258102dc0 msgr2=0x7f3258103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.635+0000 7f32605d7640 1 --2- 172.21.15.28:0/863539520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258102dc0 0x7f3258103240 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3248009900 tx=0x7f324802bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 -- 172.21.15.28:0/863539520 shutdown_connections 2024-03-22T02:22:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 --2- 172.21.15.28:0/863539520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f325810a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 --2- 172.21.15.28:0/863539520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258102dc0 0x7f3258103240 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 --2- 172.21.15.28:0/863539520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258101bd0 0x7f3258101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 -- 172.21.15.28:0/863539520 >> 172.21.15.28:0/863539520 conn(0x7f32580fd140 msgr2=0x7f32580ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 -- 172.21.15.28:0/863539520 shutdown_connections 2024-03-22T02:22:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 -- 172.21.15.28:0/863539520 wait complete. 2024-03-22T02:22:34.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.636+0000 7f32605d7640 1 Processor -- start 2024-03-22T02:22:34.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 -- start start 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 0x7f325819e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258102dc0 0x7f325819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f325810fb10 con 0x7f3258101bd0 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f325810f990 con 0x7f3258103780 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f32605d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f325810fc90 con 0x7f3258102dc0 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f325eb74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f325eb74640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51202/0 (socket says 172.21.15.28:51202) 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f325eb74640 1 -- 172.21.15.28:0/4088088413 learned_addr learned my addr 172.21.15.28:0/4088088413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f325e373640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 0x7f325819e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.637+0000 7f325db72640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258102dc0 0x7f325819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 -- 172.21.15.28:0/4088088413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258102dc0 msgr2=0x7f325819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258102dc0 0x7f325819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 -- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 msgr2=0x7f325819e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 0x7f325819e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 -- 172.21.15.28:0/4088088413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32581a5f70 con 0x7f3258103780 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325e373640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 0x7f325819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f325eb74640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f325000eb10 tx=0x7f325000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250004b20 con 0x7f3258103780 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32581a6260 con 0x7f3258103780 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.638+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3250004cc0 con 0x7f3258103780 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.639+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32581a6780 con 0x7f3258103780 2024-03-22T02:22:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.639+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250010790 con 0x7f3258103780 2024-03-22T02:22:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.640+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f323c0052c0 con 0x7f3258103780 2024-03-22T02:22:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.641+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3250044b40 con 0x7f3258103780 2024-03-22T02:22:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.642+0000 7f324f7fe640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 0x7f32440a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.642+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f32500cfd90 con 0x7f3258103780 2024-03-22T02:22:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.642+0000 7f325e373640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 0x7f32440a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.643+0000 7f325e373640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 0x7f32440a5460 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3258102c20 tx=0x7f3254008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:34.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.649+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f325009c750 con 0x7f3258103780 2024-03-22T02:22:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:34 smithi028 ceph-mon[31916]: pgmap v629: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.892+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f323c0055b0 con 0x7f3258103780 2024-03-22T02:22:34.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.893+0000 7f324f7fe640 1 -- 172.21.15.28:0/4088088413 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11866 (secure 0 0 0) 0x7f325009c050 con 0x7f3258103780 2024-03-22T02:22:34.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:22:34.894 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":35,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:21:58.483135+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:22:34.894 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 msgr2=0x7f32440a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 0x7f32440a5460 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3258102c20 tx=0x7f3254008040 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 msgr2=0x7f32581a5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f325000eb10 tx=0x7f325000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 shutdown_connections 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32440a2fa0 0x7f32440a5460 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3258103780 0x7f32581a5900 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3258102dc0 0x7f325819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 --2- 172.21.15.28:0/4088088413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3258101bd0 0x7f325819e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 >> 172.21.15.28:0/4088088413 conn(0x7f32580fd140 msgr2=0x7f325810cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:34.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 shutdown_connections 2024-03-22T02:22:34.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:34.895+0000 7f32605d7640 1 -- 172.21.15.28:0/4088088413 wait complete. 2024-03-22T02:22:34.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:22:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:34 smithi120 ceph-mon[36803]: pgmap v629: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:35.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:34 smithi083 ceph-mon[36340]: pgmap v629: 129 pgs: 129 active+clean; 52 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- 172.21.15.28:0/3540909823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 msgr2=0x7fbf8410a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- 172.21.15.28:0/3540909823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8410a000 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c009ca0 tx=0x7fbf7c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- 172.21.15.28:0/3540909823 shutdown_connections 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- 172.21.15.28:0/3540909823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8410a000 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- 172.21.15.28:0/3540909823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84102db0 0x7fbf84103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- 172.21.15.28:0/3540909823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84101bc0 0x7fbf84101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- 172.21.15.28:0/3540909823 >> 172.21.15.28:0/3540909823 conn(0x7fbf840fd130 msgr2=0x7fbf840ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- 172.21.15.28:0/3540909823 shutdown_connections 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- 172.21.15.28:0/3540909823 wait complete. 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 Processor -- start 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 -- start start 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84101bc0 0x7fbf84197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.018+0000 7fbf8b288640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 0x7fbf841983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf8b288640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf8b288640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410d840 con 0x7fbf84103770 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf8b288640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410d6c0 con 0x7fbf84102db0 2024-03-22T02:22:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf8b288640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8410d9c0 con 0x7fbf84101bc0 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf88823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 0x7fbf841983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44624/0 (socket says 172.21.15.28:44624) 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 -- 172.21.15.28:0/4124081592 learned_addr learned my addr 172.21.15.28:0/4124081592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89024640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84101bc0 0x7fbf84197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 -- 172.21.15.28:0/4124081592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84101bc0 msgr2=0x7fbf84197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84101bc0 0x7fbf84197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 -- 172.21.15.28:0/4124081592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 msgr2=0x7fbf841983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 0x7fbf841983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 -- 172.21.15.28:0/4124081592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8419faa0 con 0x7fbf84103770 2024-03-22T02:22:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf88823640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 0x7fbf841983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf89825640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c037040 tx=0x7fbf7c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf7a7fc640 1 -- 172.21.15.28:0/4124081592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c041020 con 0x7fbf84103770 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf7a7fc640 1 -- 172.21.15.28:0/4124081592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7c034cd0 con 0x7fbf84103770 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf7a7fc640 1 -- 172.21.15.28:0/4124081592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c039040 con 0x7fbf84103770 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.019+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8419fd30 con 0x7fbf84103770 2024-03-22T02:22:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.020+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf841a0210 con 0x7fbf84103770 2024-03-22T02:22:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.022+0000 7fbf7a7fc640 1 -- 172.21.15.28:0/4124081592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fbf7c044020 con 0x7fbf84103770 2024-03-22T02:22:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 Processor -- start 2024-03-22T02:22:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 -- start start 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ec7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3ec9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ff3e228d0 con 0x561ff3ec2620 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ff3dcae00 con 0x561ff3ec1c30 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561ff3e1e880 con 0x561ff3ec7790 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf7a7fc640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 0x7fbf700a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ec7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51226/0 (socket says 172.21.15.28:51226) 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 -- 172.21.15.28:0/2494662436 learned_addr learned my addr 172.21.15.28:0/2494662436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3ec9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf89825640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 0x7fbf700a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf7a7fc640 1 -- 172.21.15.28:0/4124081592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbf7c0f1600 con 0x7fbf84103770 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 -- 172.21.15.28:0/2494662436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 msgr2=0x561ff3ec9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3ec9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 -- 172.21.15.28:0/2494662436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 msgr2=0x561ff3ec7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ec7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.023+0000 7fbf88823640 1 -- 172.21.15.28:0/2494662436 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff3b363f0 con 0x561ff3ec1c30 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.024+0000 7fbf88823640 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbf74004650 tx=0x7fbf74009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5caad46a3571cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.024+0000 7fbf89825640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 0x7fbf700a5460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0ef060 tx=0x7fbf7c034a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.024+0000 7fbf6f7fe640 1 -- 172.21.15.28:0/2494662436 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400fc50 con 0x561ff3ec1c30 2024-03-22T02:22:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.024+0000 7fbf6f7fe640 1 -- 172.21.15.28:0/2494662436 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7400fdf0 con 0x561ff3ec1c30 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.024+0000 7fbf6f7fe640 1 -- 172.21.15.28:0/2494662436 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7400e5d0 con 0x561ff3ec1c30 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2494662436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 msgr2=0x561ff3ec2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbf74004650 tx=0x7fbf74009b80 comp rx=0 tx=0).stop 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2494662436 shutdown_connections 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3ec9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ec7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2494662436 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3ec2050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2494662436 >> 172.21.15.28:0/2494662436 conn(0x561ff3eb8d90 msgr2=0x561ff3eb9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2494662436 shutdown_connections 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2494662436 wait complete. 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 -- start start 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 Processor -- start 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ecc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.025+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ff3eca740 con 0x561ff3ec2620 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561ff3dde300 con 0x561ff3ec7790 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561ff3eca8c0 con 0x561ff3ec1c30 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 10 client.0 _ll_register_callbacks cb 0x561ff3e19e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44638/0 (socket says 172.21.15.28:44638) 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 -- 172.21.15.28:0/403715169 learned_addr learned my addr 172.21.15.28:0/403715169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf88823640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf88823640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ecc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 -- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 msgr2=0x561ff3ecc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ecc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 -- 172.21.15.28:0/403715169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 msgr2=0x561ff3ed3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 -- 172.21.15.28:0/403715169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff3c007d0 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf89024640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbf80012700 tx=0x7fbf80012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7546d53d25999b2c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80004050 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbf800041f0 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf7400fdf0 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbf80025050 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.026+0000 7fbf8de7c740 10 client.18063 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561ff3ed6b90 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7400fdf0 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fbf80003a40 con 0x561ff3ec2620 2024-03-22T02:22:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 10 client.18063 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 10 client.18063 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 20 client.18063 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3ed8350 0x561ff3eda790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 4 client.18063 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ff3e2fe10 con 0x561ff3ed8350 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.027+0000 7fbf88823640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3ed8350 0x561ff3eda790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf88823640 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3ed8350 0x561ff3eda790 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4bd21c866f7d1fb server_cookie=6638ee152e4b422b in_seq=0 out_seq=0 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf6cff9640 10 client.18063 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf6cff9640 1 -- 172.21.15.28:0/403715169 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ff3e2fe10 con 0x561ff3ed8350 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf6cff9640 10 client.18063 handle_command_reply: tid=0 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 client.18063 shutdown 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 msgr2=0x561ff3ecc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbf80012700 tx=0x7fbf80012bc0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 shutdown_connections 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3ed8350 0x561ff3eda790 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ecc710 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf6cff9640 0 client.18063 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/403715169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ecc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.028+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 >> 172.21.15.28:0/403715169 conn(0x561ff3eb8d90 msgr2=0x561ff3eb7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.029+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 shutdown_connections 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.029+0000 7fbf8de7c740 1 -- 172.21.15.28:0/403715169 wait complete. 2024-03-22T02:22:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.029+0000 7fbf8de7c740 20 client.18063 trim_cache size 0 max 16384 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 Processor -- start 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 -- start start 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3eb7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3eb7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841ab030 con 0x561ff3ec2620 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841a8900 con 0x561ff3ec1c30 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841a6750 con 0x561ff3ec7790 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44654/0 (socket says 172.21.15.28:44654) 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 -- 172.21.15.28:0/2628482510 learned_addr learned my addr 172.21.15.28:0/2628482510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf88823640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3eb7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf88823640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3eb7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 -- 172.21.15.28:0/2628482510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 msgr2=0x561ff3eb7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3eb7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 -- 172.21.15.28:0/2628482510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 msgr2=0x561ff3eb7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3eb7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 -- 172.21.15.28:0/2628482510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ff3ecb260 con 0x561ff3ec2620 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf88823640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3eb7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.044+0000 7fbf89024640 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbf80001eb0 tx=0x7fbf80010710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c031c1a3047bb7c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf6effd640 1 -- 172.21.15.28:0/2628482510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002d020 con 0x561ff3ec2620 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf6effd640 1 -- 172.21.15.28:0/2628482510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf80005550 con 0x561ff3ec2620 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf6effd640 1 -- 172.21.15.28:0/2628482510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf800035f0 con 0x561ff3ec2620 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2628482510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 msgr2=0x561ff3ed3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbf80001eb0 tx=0x7fbf80010710 comp rx=0 tx=0).stop 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2628482510 shutdown_connections 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec7790 0x561ff3eb7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed3710 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/2628482510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec1c30 0x561ff3eb7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2628482510 >> 172.21.15.28:0/2628482510 conn(0x561ff3eb8d90 msgr2=0x561ff3ec3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2628482510 shutdown_connections 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.045+0000 7fbf8de7c740 1 -- 172.21.15.28:0/2628482510 wait complete. 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 -- start start 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 Processor -- start 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ed1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841a6dc0 con 0x561ff3ec2620 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841a8230 con 0x561ff3ec7790 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf841a7490 con 0x561ff3ec1c30 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf8de7c740 10 client.0 _ll_register_callbacks cb 0x7fbf80009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf89024640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44666/0 (socket says 172.21.15.28:44666) 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf89024640 1 -- 172.21.15.28:0/1161576286 learned_addr learned my addr 172.21.15.28:0/1161576286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf88823640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ed1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.046+0000 7fbf88823640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 -- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 msgr2=0x561ff3ed1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ed1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 -- 172.21.15.28:0/1161576286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 msgr2=0x561ff3ed27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 -- 172.21.15.28:0/1161576286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ff3ed2da0 con 0x561ff3ec2620 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf89024640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e00 tx=0x7fbf80009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19cd8872223c64b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002d040 con 0x561ff3ec2620 2024-03-22T02:22:35.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf80005550 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbf80022040 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fbf8003b050 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf8de7c740 10 client.18069 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x561ff3f3ea70 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf80005550 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.047+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fbf8000f790 con 0x561ff3ec2620 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 10 client.18069 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 10 client.18069 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 20 client.18069 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3f3f390 0x561ff3f417b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 4 client.18069 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561ff3ecbfc0 con 0x561ff3f3f390 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf88823640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3f3f390 0x561ff3f417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf6cff9640 10 client.18069 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf88823640 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3f3f390 0x561ff3f417b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c35e36a1af33c812 server_cookie=f6437de438f2204b in_seq=0 out_seq=0 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf6cff9640 1 -- 172.21.15.28:0/1161576286 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561ff3ecbfc0 con 0x561ff3f3f390 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf6cff9640 10 client.18069 handle_command_reply: tid=0 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 client.18069 shutdown 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 msgr2=0x561ff3ed17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009e00 tx=0x7fbf80009660 comp rx=0 tx=0).stop 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 shutdown_connections 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x561ff3f3f390 0x561ff3f417b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf6cff9640 0 client.18069 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561ff3ec7790 0x561ff3ed27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ff3ec2620 0x561ff3ed17a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 --2- 172.21.15.28:0/1161576286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561ff3ec1c30 0x561ff3ed1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.048+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 >> 172.21.15.28:0/1161576286 conn(0x561ff3eb8d90 msgr2=0x561ff3ebbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 shutdown_connections 2024-03-22T02:22:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8de7c740 1 -- 172.21.15.28:0/1161576286 wait complete. 2024-03-22T02:22:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8de7c740 20 client.18069 trim_cache size 0 max 16384 2024-03-22T02:22:35.057 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 msgr2=0x7fbf700a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 0x7fbf700a5460 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c0ef060 tx=0x7fbf7c034a60 comp rx=0 tx=0).stop 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 msgr2=0x7fbf8419f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c037040 tx=0x7fbf7c032f40 comp rx=0 tx=0).stop 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 shutdown_connections 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf700a2fa0 0x7fbf700a5460 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf84103770 0x7fbf8419f430 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf84102db0 0x7fbf841983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 --2- 172.21.15.28:0/4124081592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf84101bc0 0x7fbf84197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.049+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 >> 172.21.15.28:0/4124081592 conn(0x7fbf840fd130 msgr2=0x7fbf841060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.050+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 shutdown_connections 2024-03-22T02:22:35.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.050+0000 7fbf8b288640 1 -- 172.21.15.28:0/4124081592 wait complete. 2024-03-22T02:22:35.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:22:35.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 -- 172.21.15.28:0/1044440932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 msgr2=0x7f651010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 --2- 172.21.15.28:0/1044440932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651010a000 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6508009ca0 tx=0x7f650802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 -- 172.21.15.28:0/1044440932 shutdown_connections 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 --2- 172.21.15.28:0/1044440932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651010a000 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 --2- 172.21.15.28:0/1044440932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 0x7f6510103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 --2- 172.21.15.28:0/1044440932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 -- 172.21.15.28:0/1044440932 >> 172.21.15.28:0/1044440932 conn(0x7f65100fd130 msgr2=0x7f65100ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 -- 172.21.15.28:0/1044440932 shutdown_connections 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 -- 172.21.15.28:0/1044440932 wait complete. 2024-03-22T02:22:35.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.171+0000 7f6514cd4640 1 Processor -- start 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 -- start start 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 0x7f65101983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f651010d840 con 0x7f6510103770 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f651010d6c0 con 0x7f6510101bc0 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f6514cd4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f651010d9c0 con 0x7f6510102db0 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 0x7f65101983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51262/0 (socket says 172.21.15.28:51262) 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 -- 172.21.15.28:0/3063020678 learned_addr learned my addr 172.21.15.28:0/3063020678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650ed9d640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 -- 172.21.15.28:0/3063020678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 msgr2=0x7f65101983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 0x7f65101983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 -- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 msgr2=0x7f651019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.172+0000 7f650e59c640 1 -- 172.21.15.28:0/3063020678 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651019faa0 con 0x7f6510101bc0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f650e59c640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f650000e9c0 tx=0x7f650000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f64ff7fe640 1 -- 172.21.15.28:0/3063020678 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650000cd80 con 0x7f6510101bc0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651019fd90 con 0x7f6510101bc0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f64ff7fe640 1 -- 172.21.15.28:0/3063020678 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65000035f0 con 0x7f6510101bc0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f64ff7fe640 1 -- 172.21.15.28:0/3063020678 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500010690 con 0x7f6510101bc0 2024-03-22T02:22:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.173+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65101a0220 con 0x7f6510101bc0 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 Processor -- start 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 -- start start 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f64ff7fe640 1 -- 172.21.15.28:0/3063020678 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f65000037a0 con 0x7f6510101bc0 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a6210 0x55919c4aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4ad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55919c40a8d0 con 0x55919c4a5820 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55919c3b2e00 con 0x55919c4a6210 2024-03-22T02:22:35.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f6517997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55919c406880 con 0x55919c4ab380 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44692/0 (socket says 172.21.15.28:44692) 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f650ed9d640 1 -- 172.21.15.28:0/1816612179 learned_addr learned my addr 172.21.15.28:0/1816612179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f650dd9b640 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.176+0000 7f650e59c640 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a6210 0x55919c4aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 -- 172.21.15.28:0/1816612179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 msgr2=0x55919c4ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 -- 172.21.15.28:0/1816612179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a6210 msgr2=0x55919c4aad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a6210 0x55919c4aad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 -- 172.21.15.28:0/1816612179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919c11e3f0 con 0x55919c4a5820 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f64ff7fe640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 0x7f64f80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650e59c640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 0x7f64f80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6508009ca0 tx=0x7f65080323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fc30fdc9f9e19d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f64ff7fe640 1 -- 172.21.15.28:0/3063020678 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f65000cfc80 con 0x7f6510101bc0 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f64fcff9640 1 -- 172.21.15.28:0/1816612179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508043070 con 0x55919c4a5820 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f64fcff9640 1 -- 172.21.15.28:0/1816612179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6508032c90 con 0x55919c4a5820 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.177+0000 7f64fcff9640 1 -- 172.21.15.28:0/1816612179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508039040 con 0x55919c4a5820 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 -- 172.21.15.28:0/1816612179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 msgr2=0x55919c4a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f650e59c640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 0x7f64f80a53e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f65000cd6e0 tx=0x7f65000d3760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6508009ca0 tx=0x7f65080323c0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 -- 172.21.15.28:0/1816612179 shutdown_connections 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a6210 0x55919c4aad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 --2- 172.21.15.28:0/1816612179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4a5c40 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 -- 172.21.15.28:0/1816612179 >> 172.21.15.28:0/1816612179 conn(0x55919c4a0d90 msgr2=0x55919c4a1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 -- 172.21.15.28:0/1816612179 shutdown_connections 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.178+0000 7f6517997740 1 -- 172.21.15.28:0/1816612179 wait complete. 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 -- start start 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 Processor -- start 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4ab380 0x55919c4b7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55919c4ae2d0 con 0x55919c4a5820 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55919c3c6300 con 0x55919c4ab380 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55919c4ae450 con 0x55919c4a6210 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f6517997740 10 client.0 _ll_register_callbacks cb 0x55919c401e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650dd9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44708/0 (socket says 172.21.15.28:44708) 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 -- 172.21.15.28:0/2659104166 learned_addr learned my addr 172.21.15.28:0/2659104166 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 -- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 msgr2=0x55919c4b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 -- 172.21.15.28:0/2659104166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4ab380 msgr2=0x55919c4b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4ab380 0x55919c4b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 -- 172.21.15.28:0/2659104166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919c1e87d0 con 0x55919c4a5820 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.179+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6508005c70 tx=0x7f65080095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f10870bb01bdf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650804c070 con 0x55919c4a5820 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6508048650 con 0x55919c4a5820 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6508032c90 con 0x55919c4a5820 2024-03-22T02:22:35.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6508058050 con 0x55919c4a5820 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 10 client.18081 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55919c4ba590 con 0x55919c4a5820 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6508032c90 con 0x55919c4a5820 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f6508034920 con 0x55919c4a5820 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 10 client.18081 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 10 client.18081 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 20 client.18081 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c4bb8e0 0x55919c4bdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 4 client.18081 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.180+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55919c417e10 con 0x55919c4bb8e0 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.181+0000 7f650dd9b640 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c4bb8e0 0x55919c4bdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.181+0000 7f650dd9b640 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c4bb8e0 0x55919c4bdcd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c37bedeb6c955fef server_cookie=c12d5d568de39525 in_seq=0 out_seq=0 2024-03-22T02:22:35.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.181+0000 7f64f5ffb640 10 client.18081 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2659104166 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55919c417e10 con 0x55919c4bb8e0 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f64f5ffb640 10 client.18081 handle_command_reply: tid=0 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 client.18081 shutdown 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 msgr2=0x55919c4afca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f6508005c70 tx=0x7f65080095d0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 shutdown_connections 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c4bb8e0 0x55919c4bdcd0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4ab380 0x55919c4b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 --2- 172.21.15.28:0/2659104166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a5820 0x55919c4afca0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f64f5ffb640 0 client.18081 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 >> 172.21.15.28:0/2659104166 conn(0x55919c4a0d90 msgr2=0x55919c49ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.182+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 shutdown_connections 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.183+0000 7f6517997740 1 -- 172.21.15.28:0/2659104166 wait complete. 2024-03-22T02:22:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.183+0000 7f6517997740 20 client.18081 trim_cache size 0 max 16384 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.196+0000 7f6517997740 1 Processor -- start 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.196+0000 7f6517997740 1 -- start start 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b0140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4a4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65101ad260 con 0x55919c4a6210 2024-03-22T02:22:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65101aab30 con 0x55919c4a5820 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f6517997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f65101a8980 con 0x55919c4ab380 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44724/0 (socket says 172.21.15.28:44724) 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 -- 172.21.15.28:0/1121223995 learned_addr learned my addr 172.21.15.28:0/1121223995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 -- 172.21.15.28:0/1121223995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 msgr2=0x55919c4a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 -- 172.21.15.28:0/1121223995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 msgr2=0x55919c4b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b0140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 -- 172.21.15.28:0/1121223995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55919c4be040 con 0x55919c4a6210 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650ed9d640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f650dd9b640 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x55919c4a1d00 tx=0x7f6504005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fddd778fb64ff61 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f64f7fff640 1 -- 172.21.15.28:0/1121223995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504004250 con 0x55919c4a6210 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f64f7fff640 1 -- 172.21.15.28:0/1121223995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6504012e50 con 0x55919c4a6210 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.197+0000 7f64f7fff640 1 -- 172.21.15.28:0/1121223995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504003c50 con 0x55919c4a6210 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- 172.21.15.28:0/1121223995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 msgr2=0x55919c4b71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x55919c4a1d00 tx=0x7f6504005f90 comp rx=0 tx=0).stop 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- 172.21.15.28:0/1121223995 shutdown_connections 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4ab380 0x55919c4a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4a6210 0x55919c4b71b0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 --2- 172.21.15.28:0/1121223995 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- 172.21.15.28:0/1121223995 >> 172.21.15.28:0/1121223995 conn(0x55919c4a0d90 msgr2=0x55919c4a3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- 172.21.15.28:0/1121223995 shutdown_connections 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- 172.21.15.28:0/1121223995 wait complete. 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 -- start start 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.198+0000 7f6517997740 1 Processor -- start 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f65101a8ff0 con 0x55919c4ab380 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f65101aa460 con 0x55919c4a5820 2024-03-22T02:22:35.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f65101a96c0 con 0x55919c4a6210 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 10 client.0 _ll_register_callbacks cb 0x7f65000d3630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44730/0 (socket says 172.21.15.28:44730) 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 -- 172.21.15.28:0/2729132516 learned_addr learned my addr 172.21.15.28:0/2729132516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650dd9b640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 -- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 msgr2=0x55919c4b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 -- 172.21.15.28:0/2729132516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 msgr2=0x55919c4b48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 -- 172.21.15.28:0/2729132516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55919c4b6420 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f650ed9d640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6508005e30 tx=0x7f6508032b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb65f5d738840ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f64f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650804c040 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6508034650 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.199+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6504012e50 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f650806c050 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 10 client.18087 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55919c521ac0 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6504012e50 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f650803f540 con 0x55919c4ab380 2024-03-22T02:22:35.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 10 client.18087 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 10 client.18087 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 20 client.18087 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c522370 0x55919c524790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 4 client.18087 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55919c412160 con 0x55919c522370 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f650dd9b640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c522370 0x55919c524790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f64f5ffb640 10 client.18087 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.200+0000 7f650dd9b640 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c522370 0x55919c524790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79b3e38f3b73fb10 server_cookie=da3472da69f272f9 in_seq=0 out_seq=0 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f64f5ffb640 1 -- 172.21.15.28:0/2729132516 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55919c412160 con 0x55919c522370 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f64f5ffb640 10 client.18087 handle_command_reply: tid=0 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 client.18087 shutdown 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 msgr2=0x55919c4b5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f6508005e30 tx=0x7f6508032b00 comp rx=0 tx=0).stop 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 shutdown_connections 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55919c522370 0x55919c524790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55919c4ab380 0x55919c4b5e30 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55919c4a6210 0x55919c4b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f64f5ffb640 0 client.18087 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 --2- 172.21.15.28:0/2729132516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55919c4a5820 0x55919c4b48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 >> 172.21.15.28:0/2729132516 conn(0x55919c4a0d90 msgr2=0x55919c4a4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 shutdown_connections 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 1 -- 172.21.15.28:0/2729132516 wait complete. 2024-03-22T02:22:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6517997740 20 client.18087 trim_cache size 0 max 16384 2024-03-22T02:22:35.211 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 msgr2=0x7f64f80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 0x7f64f80a53e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f65000cd6e0 tx=0x7f65000d3760 comp rx=0 tx=0).stop 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 msgr2=0x7f6510197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f650000e9c0 tx=0x7f650000ee80 comp rx=0 tx=0).stop 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 shutdown_connections 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f64f80a2f20 0x7f64f80a53e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6510103770 0x7f651019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6510102db0 0x7f65101983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 --2- 172.21.15.28:0/3063020678 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6510101bc0 0x7f6510197e80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.201+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 >> 172.21.15.28:0/3063020678 conn(0x7f65100fd130 msgr2=0x7f65101060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.202+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 shutdown_connections 2024-03-22T02:22:35.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.202+0000 7f6514cd4640 1 -- 172.21.15.28:0/3063020678 wait complete. 2024-03-22T02:22:35.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:22:35.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 -- 172.21.15.28:0/2933046604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf98102e50 msgr2=0x7faf981032d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/2933046604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf98102e50 0x7faf981032d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009930 tx=0x7faf8c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 -- 172.21.15.28:0/2933046604 shutdown_connections 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/2933046604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 0x7faf9810a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/2933046604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf98102e50 0x7faf981032d0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/2933046604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf980ffe80 0x7faf981002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 -- 172.21.15.28:0/2933046604 >> 172.21.15.28:0/2933046604 conn(0x7faf980fb790 msgr2=0x7faf980fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 -- 172.21.15.28:0/2933046604 shutdown_connections 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 -- 172.21.15.28:0/2933046604 wait complete. 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.330+0000 7faf9dfc9640 1 Processor -- start 2024-03-22T02:22:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 -- start start 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf98102e50 0x7faf981983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 0x7faf9819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf9810d840 con 0x7faf980ffe80 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf9810d6c0 con 0x7faf98103810 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf9dfc9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf9810d9c0 con 0x7faf98102e50 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44742/0 (socket says 172.21.15.28:44742) 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 -- 172.21.15.28:0/3618196457 learned_addr learned my addr 172.21.15.28:0/3618196457 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf97fff640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 0x7faf9819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf96ffd640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf98102e50 0x7faf981983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 -- 172.21.15.28:0/3618196457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf98102e50 msgr2=0x7faf981983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf98102e50 0x7faf981983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 -- 172.21.15.28:0/3618196457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 msgr2=0x7faf9819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 0x7faf9819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 -- 172.21.15.28:0/3618196457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9819fad0 con 0x7faf980ffe80 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.331+0000 7faf977fe640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e9c0 tx=0x7faf8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.332+0000 7faf94ff9640 1 -- 172.21.15.28:0/3618196457 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8800cd80 con 0x7faf980ffe80 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.332+0000 7faf94ff9640 1 -- 172.21.15.28:0/3618196457 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf880035f0 con 0x7faf980ffe80 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.332+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9819fdc0 con 0x7faf980ffe80 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.332+0000 7faf94ff9640 1 -- 172.21.15.28:0/3618196457 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88010690 con 0x7faf980ffe80 2024-03-22T02:22:35.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.332+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf981a0220 con 0x7faf980ffe80 2024-03-22T02:22:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7faf94ff9640 1 -- 172.21.15.28:0/3618196457 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faf880037a0 con 0x7faf980ffe80 2024-03-22T02:22:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 Processor -- start 2024-03-22T02:22:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 -- start start 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b645824c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b645829d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6457898d0 con 0x55b64582a380 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b645731e00 con 0x55b645825210 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b645785880 con 0x55b645824820 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7faf96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7faf96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44746/0 (socket says 172.21.15.28:44746) 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.334+0000 7faf977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b645829d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 -- 172.21.15.28:0/1479564735 learned_addr learned my addr 172.21.15.28:0/1479564735 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf97fff640 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b645824c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf94ff9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 0x7faf800a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf97fff640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 0x7faf800a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 -- 172.21.15.28:0/1479564735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 msgr2=0x55b645824c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b645824c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 -- 172.21.15.28:0/1479564735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 msgr2=0x55b645829d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b645829d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 -- 172.21.15.28:0/1479564735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b64549d3f0 con 0x55b64582a380 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf94ff9640 1 -- 172.21.15.28:0/3618196457 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf880cfbd0 con 0x7faf980ffe80 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009230 tx=0x7faf8c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a8428f04b2c64e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf97fff640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 0x7faf800a53e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x55b645825a70 tx=0x7faf90004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf7dffb640 1 -- 172.21.15.28:0/1479564735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c02cb80 con 0x55b64582a380 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf7dffb640 1 -- 172.21.15.28:0/1479564735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf8c005040 con 0x55b64582a380 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.335+0000 7faf7dffb640 1 -- 172.21.15.28:0/1479564735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c004920 con 0x55b64582a380 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1479564735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 msgr2=0x55b64582c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009230 tx=0x7faf8c02c650 comp rx=0 tx=0).stop 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1479564735 shutdown_connections 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b64582c810 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b645829d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1479564735 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b645824c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1479564735 >> 172.21.15.28:0/1479564735 conn(0x55b64581fd90 msgr2=0x55b645820190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1479564735 shutdown_connections 2024-03-22T02:22:35.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1479564735 wait complete. 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 -- start start 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.336+0000 7fafa0bb5740 1 Processor -- start 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645824820 0x55b6458330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b64583a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b64582d2d0 con 0x55b645825210 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b645745300 con 0x55b645824820 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b64582d450 con 0x55b64582a380 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 10 client.0 _ll_register_callbacks cb 0x55b645780e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf977fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645824820 0x55b6458330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44760/0 (socket says 172.21.15.28:44760) 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 -- 172.21.15.28:0/3740982523 learned_addr learned my addr 172.21.15.28:0/3740982523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf97fff640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b64583a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 -- 172.21.15.28:0/3740982523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 msgr2=0x55b64583a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b64583a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 -- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645824820 msgr2=0x55b6458330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645824820 0x55b6458330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 -- 172.21.15.28:0/3740982523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6455677d0 con 0x55b645825210 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf96ffd640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7faf8c02c7b0 tx=0x7faf8c005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0118b9bdf85f0e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c052060 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf8c004ad0 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.337+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf8c005040 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faf8c059050 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 10 client.18099 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b64583d9d0 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf8c005040 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7faf8c034c80 con 0x55b645825210 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 10 client.18099 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 10 client.18099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:35.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 20 client.18099 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b64583ed20 0x55b645841110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 4 client.18099 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b645796e10 con 0x55b64583ed20 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.338+0000 7faf977fe640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b64583ed20 0x55b645841110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.339+0000 7faf76ffd640 10 client.18099 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.339+0000 7faf977fe640 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b64583ed20 0x55b645841110 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec677005443cb512 server_cookie=c1083eba847ee7e7 in_seq=0 out_seq=0 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7faf76ffd640 1 -- 172.21.15.28:0/3740982523 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b645796e10 con 0x55b64583ed20 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7faf76ffd640 10 client.18099 handle_command_reply: tid=0 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 client.18099 shutdown 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 msgr2=0x55b645833620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7faf8c02c7b0 tx=0x7faf8c005480 comp rx=0 tx=0).stop 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 shutdown_connections 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b64583ed20 0x55b645841110 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b64583a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645825210 0x55b645833620 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7faf76ffd640 0 client.18099 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/3740982523 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645824820 0x55b6458330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 >> 172.21.15.28:0/3740982523 conn(0x55b64581fd90 msgr2=0x55b64581efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 shutdown_connections 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 1 -- 172.21.15.28:0/3740982523 wait complete. 2024-03-22T02:22:35.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.340+0000 7fafa0bb5740 20 client.18099 trim_cache size 0 max 16384 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 Processor -- start 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 -- start start 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b64581ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf981ab080 con 0x55b64582a380 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf981a8950 con 0x55b645825210 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.353+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf981a67a0 con 0x55b645824820 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44772/0 (socket says 172.21.15.28:44772) 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 -- 172.21.15.28:0/656581875 learned_addr learned my addr 172.21.15.28:0/656581875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf96ffd640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf977fe640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b64581ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 -- 172.21.15.28:0/656581875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 msgr2=0x55b64581ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b64581ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 -- 172.21.15.28:0/656581875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 msgr2=0x55b64583a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 -- 172.21.15.28:0/656581875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6458255f0 con 0x55b64582a380 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf96ffd640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf97fff640 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7faf90013470 tx=0x7faf90005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7075b488bd69c48e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf7d7fa640 1 -- 172.21.15.28:0/656581875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90004050 con 0x55b64582a380 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf7d7fa640 1 -- 172.21.15.28:0/656581875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90026030 con 0x55b64582a380 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.354+0000 7faf7d7fa640 1 -- 172.21.15.28:0/656581875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90022cb0 con 0x55b64582a380 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- 172.21.15.28:0/656581875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 msgr2=0x55b645823700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7faf90013470 tx=0x7faf90005bd0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- 172.21.15.28:0/656581875 shutdown_connections 2024-03-22T02:22:35.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b64582a380 0x55b645823700 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/656581875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b645824820 0x55b64581ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- 172.21.15.28:0/656581875 >> 172.21.15.28:0/656581875 conn(0x55b64581fd90 msgr2=0x55b645822ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- 172.21.15.28:0/656581875 shutdown_connections 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- 172.21.15.28:0/656581875 wait complete. 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 -- start start 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.355+0000 7fafa0bb5740 1 Processor -- start 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b6458384e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf981a6e10 con 0x55b645824820 2024-03-22T02:22:35.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf981a8280 con 0x55b645825210 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf981a74e0 con 0x55b64582a380 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 10 client.0 _ll_register_callbacks cb 0x7faf8809bd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44786/0 (socket says 172.21.15.28:44786) 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 -- 172.21.15.28:0/1922288824 learned_addr learned my addr 172.21.15.28:0/1922288824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf97fff640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b6458384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 -- 172.21.15.28:0/1922288824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 msgr2=0x55b6458384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b6458384e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 -- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 msgr2=0x55b64583a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 -- 172.21.15.28:0/1922288824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b645838bf0 con 0x55b645824820 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf977fe640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c6a0 tx=0x7faf8809bb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfb410e9ceb81798 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8809c5f0 con 0x55b645824820 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf880cd3d0 con 0x55b645824820 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.356+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf90026030 con 0x55b645824820 2024-03-22T02:22:35.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7faf880a4900 con 0x55b645824820 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 10 client.18105 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55b6458a5ab0 con 0x55b645824820 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90026030 con 0x55b645824820 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7faf8809b340 con 0x55b645824820 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 10 client.18105 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 10 client.18105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 20 client.18105 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6458a6390 0x55b6458a87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 4 client.18105 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:22:35.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b645791160 con 0x55b6458a6390 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6458a6390 0x55b6458a87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf76ffd640 10 client.18105 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.357+0000 7faf96ffd640 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6458a6390 0x55b6458a87b0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdb436fbae3dfab0 server_cookie=a6e587d332dcf460 in_seq=0 out_seq=0 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf76ffd640 1 -- 172.21.15.28:0/1922288824 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b645791160 con 0x55b6458a6390 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf76ffd640 10 client.18105 handle_command_reply: tid=0 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 client.18105 shutdown 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 msgr2=0x55b645839ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c6a0 tx=0x7faf8809bb50 comp rx=0 tx=0).stop 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 shutdown_connections 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6458a6390 0x55b6458a87b0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b64582a380 0x55b6458384e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b645825210 0x55b64583a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf76ffd640 0 client.18105 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 --2- 172.21.15.28:0/1922288824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b645824820 0x55b645839ed0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 >> 172.21.15.28:0/1922288824 conn(0x55b64581fd90 msgr2=0x55b645823880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 shutdown_connections 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 1 -- 172.21.15.28:0/1922288824 wait complete. 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7fafa0bb5740 20 client.18105 trim_cache size 0 max 16384 2024-03-22T02:22:35.368 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:22:35.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 msgr2=0x7faf800a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 0x7faf800a53e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x55b645825a70 tx=0x7faf90004ab0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 msgr2=0x7faf98197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7faf8800e9c0 tx=0x7faf8800ee80 comp rx=0 tx=0).stop 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.358+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 shutdown_connections 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf800a2f20 0x7faf800a53e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf98103810 0x7faf9819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf98102e50 0x7faf981983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 --2- 172.21.15.28:0/3618196457 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf980ffe80 0x7faf98197eb0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 >> 172.21.15.28:0/3618196457 conn(0x7faf980fb790 msgr2=0x7faf981061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 shutdown_connections 2024-03-22T02:22:35.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:35.359+0000 7faf9dfc9640 1 -- 172.21.15.28:0/3618196457 wait complete. 2024-03-22T02:22:35.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 877667 11.64 MB/sec execute 317 sec latency 88.918 ms 2024-03-22T02:22:35.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:35 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4088088413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:35 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4088088413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:36.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:35 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4088088413' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:22:36.372 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:22:36.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:22:36.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:22:36.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 880195 11.65 MB/sec execute 318 sec latency 22.800 ms 2024-03-22T02:22:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.491+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1567774617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 msgr2=0x7f8c2c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.491+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1567774617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c102100 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009900 tx=0x7f8c1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1567774617 shutdown_connections 2024-03-22T02:22:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1567774617 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1567774617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c102100 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1567774617 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 0x7f8c2c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1567774617 >> 172.21.15.28:0/1567774617 conn(0x7f8c2c0fc020 msgr2=0x7f8c2c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1567774617 shutdown_connections 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1567774617 wait complete. 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 Processor -- start 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- start start 2024-03-22T02:22:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 0x7f8c2c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c19f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c10c710 con 0x7f8c2c101c80 2024-03-22T02:22:36.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c10c590 con 0x7f8c2c102640 2024-03-22T02:22:36.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.492+0000 7f8c30aaa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c10c890 con 0x7f8c2c100a90 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44818/0 (socket says 172.21.15.28:44818) 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c19f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 0x7f8c2c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1413124963 learned_addr learned my addr 172.21.15.28:0/1413124963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1413124963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 msgr2=0x7f8c2c197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 0x7f8c2c197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1413124963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 msgr2=0x7f8c2c19f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c19f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1413124963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c19fb00 con 0x7f8c2c101c80 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c19f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009230 tx=0x7f8c1402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c237fe640 1 -- 172.21.15.28:0/1413124963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14041070 con 0x7f8c2c101c80 2024-03-22T02:22:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c19fd90 con 0x7f8c2c101c80 2024-03-22T02:22:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c237fe640 1 -- 172.21.15.28:0/1413124963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c14004450 con 0x7f8c2c101c80 2024-03-22T02:22:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c1a0270 con 0x7f8c2c101c80 2024-03-22T02:22:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.493+0000 7f8c237fe640 1 -- 172.21.15.28:0/1413124963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14003e20 con 0x7f8c2c101c80 2024-03-22T02:22:36.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.496+0000 7f8c237fe640 1 -- 172.21.15.28:0/1413124963 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8c14044070 con 0x7f8c2c101c80 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.496+0000 7f8c33768740 1 Processor -- start 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.496+0000 7f8c33768740 1 -- start start 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dca20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 0x55ef9dca4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dc01570 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dba9e10 con 0x55ef9dca2720 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c33768740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dbfd520 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c237fe640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 0x7f8c180a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 0x55ef9dca4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dca20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dca20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56596/0 (socket says 172.21.15.28:56596) 2024-03-22T02:22:36.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2a59c640 1 -- 172.21.15.28:0/1594053009 learned_addr learned my addr 172.21.15.28:0/1594053009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 0x7f8c180a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c237fe640 1 -- 172.21.15.28:0/1413124963 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8c140f13b0 con 0x7f8c2c101c80 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1594053009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 msgr2=0x55ef9dca20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dca20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1594053009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 msgr2=0x55ef9dc9cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 -- 172.21.15.28:0/1594053009 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef9d9dd7e0 con 0x55ef9dca2720 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 0x55ef9dca4bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8c140eee10 tx=0x7f8c1408f400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cf65bb790261af6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.497+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 0x7f8c180a53e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55ef9dc9de10 tx=0x7f8c24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c20ff9640 1 -- 172.21.15.28:0/1594053009 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1404b070 con 0x55ef9dca2720 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c20ff9640 1 -- 172.21.15.28:0/1594053009 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c14004c60 con 0x55ef9dca2720 2024-03-22T02:22:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c20ff9640 1 -- 172.21.15.28:0/1594053009 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c140c60e0 con 0x55ef9dca2720 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 -- 172.21.15.28:0/1594053009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 msgr2=0x55ef9dca4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 0x55ef9dca4bb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8c140eee10 tx=0x7f8c1408f400 comp rx=0 tx=0).stop 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 -- 172.21.15.28:0/1594053009 shutdown_connections 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dca2720 0x55ef9dca4bb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dca20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 --2- 172.21.15.28:0/1594053009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 -- 172.21.15.28:0/1594053009 >> 172.21.15.28:0/1594053009 conn(0x55ef9dc98130 msgr2=0x55ef9dc98530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 -- 172.21.15.28:0/1594053009 shutdown_connections 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.498+0000 7f8c33768740 1 -- 172.21.15.28:0/1594053009 wait complete. 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 -- start start 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 Processor -- start 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcaf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dca5490 con 0x55ef9dca2720 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dbbd110 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ef9dca5610 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c33768740 10 client.0 _ll_register_callbacks cb 0x55ef9dbf9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44828/0 (socket says 172.21.15.28:44828) 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 -- 172.21.15.28:0/2420917098 learned_addr learned my addr 172.21.15.28:0/2420917098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56610/0 (socket says 172.21.15.28:56610) 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcaf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 -- 172.21.15.28:0/2420917098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 msgr2=0x55ef9dcaf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 -- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 msgr2=0x55ef9dcaf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcaf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.499+0000 7f8c2ad9d640 1 -- 172.21.15.28:0/2420917098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef9da3df70 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f8c24011f30 tx=0x7f8c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=686703286466568b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2401f460 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c14004c60 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8c2401f600 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c2401f800 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 10 client.18117 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ef9d8d5170 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c14004c60 con 0x55ef9dca2720 2024-03-22T02:22:36.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f8c24028070 con 0x55ef9dca2720 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 10 client.18117 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 10 client.18117 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 20 client.18117 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dcbd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 4 client.18117 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.500+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef9dc0eab0 con 0x55ef9dcbb230 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.501+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dcbd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.501+0000 7f8c11ffb640 10 client.18117 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.501+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dcbd6d0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8071f689f21bef8a server_cookie=e8a13ca1c0a5ebf0 in_seq=0 out_seq=0 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c11ffb640 1 -- 172.21.15.28:0/2420917098 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef9dc0eab0 con 0x55ef9dcbb230 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c11ffb640 10 client.18117 handle_command_reply: tid=0 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 client.18117 shutdown 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 msgr2=0x55ef9dcb68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f8c24011f30 tx=0x7f8c240046c0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 shutdown_connections 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dcbd6d0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c11ffb640 0 client.18117 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dcb68b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcaf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 --2- 172.21.15.28:0/2420917098 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcaf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 >> 172.21.15.28:0/2420917098 conn(0x55ef9dc98130 msgr2=0x55ef9dc973f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 shutdown_connections 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 1 -- 172.21.15.28:0/2420917098 wait complete. 2024-03-22T02:22:36.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.502+0000 7f8c33768740 20 client.18117 trim_cache size 0 max 16384 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 Processor -- start 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 -- start start 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dc9e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1ab0b0 con 0x55ef9dca2720 2024-03-22T02:22:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1a8980 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.523+0000 7f8c33768740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1a67d0 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dc9e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51422/0 (socket says 172.21.15.28:51422) 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 -- 172.21.15.28:0/526147937 learned_addr learned my addr 172.21.15.28:0/526147937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c2ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dc9e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44840/0 (socket says 172.21.15.28:44840) 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 -- 172.21.15.28:0/526147937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 msgr2=0x55ef9dc9baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 -- 172.21.15.28:0/526147937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 msgr2=0x55ef9dc9e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dc9e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 -- 172.21.15.28:0/526147937 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef9dc98ea0 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8c140ba150 tx=0x7f8c1402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8fd52ae7a8cf311 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c13fff640 1 -- 172.21.15.28:0/526147937 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1404b040 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c13fff640 1 -- 172.21.15.28:0/526147937 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c14004c60 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.524+0000 7f8c13fff640 1 -- 172.21.15.28:0/526147937 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c140bba80 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- 172.21.15.28:0/526147937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 msgr2=0x55ef9dcb6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8c140ba150 tx=0x7f8c1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- 172.21.15.28:0/526147937 shutdown_connections 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dca2720 0x55ef9dc9e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb6870 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 --2- 172.21.15.28:0/526147937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dc9baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- 172.21.15.28:0/526147937 >> 172.21.15.28:0/526147937 conn(0x55ef9dc98130 msgr2=0x55ef9dc9b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- 172.21.15.28:0/526147937 shutdown_connections 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- 172.21.15.28:0/526147937 wait complete. 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.525+0000 7f8c33768740 1 -- start start 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 Processor -- start 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dca2720 0x55ef9dcb5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1a6e40 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1a82b0 con 0x55ef9dc9d5b0 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c1a7510 con 0x55ef9dca2720 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c33768740 10 client.0 _ll_register_callbacks cb 0x7f8c24011df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:36.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44842/0 (socket says 172.21.15.28:44842) 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 -- 172.21.15.28:0/1891208403 learned_addr learned my addr 172.21.15.28:0/1891208403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2ad9d640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dca2720 0x55ef9dcb5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 -- 172.21.15.28:0/1891208403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dca2720 msgr2=0x55ef9dcb5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dca2720 0x55ef9dcb5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 -- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 msgr2=0x55ef9dcb4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.526+0000 7f8c2a59c640 1 -- 172.21.15.28:0/1891208403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef9dcb5740 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c2a59c640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c004a80 tx=0x7f8c1c011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9d84ae3e701536 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c01fe40 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8c1c004b20 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c14004c60 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f8c1c034050 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c33768740 10 client.18123 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ef9dcb6b50 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c14004c60 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f8c1c02b790 con 0x55ef9dc9cbc0 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.527+0000 7f8c33768740 10 client.18123 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c33768740 10 client.18123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c33768740 20 client.18123 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dd23ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c33768740 4 client.18123 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef9dc08e00 con 0x55ef9dcbb230 2024-03-22T02:22:36.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.528+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dd23ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.529+0000 7f8c11ffb640 10 client.18123 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.529+0000 7f8c29d9b640 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dd23ce0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a92bf6ca6612df4 server_cookie=30945bb3f5d05329 in_seq=0 out_seq=0 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c11ffb640 1 -- 172.21.15.28:0/1891208403 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef9dc08e00 con 0x55ef9dcbb230 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c11ffb640 10 client.18123 handle_command_reply: tid=0 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 client.18123 shutdown 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 msgr2=0x55ef9dcb3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c004a80 tx=0x7f8c1c011ee0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 shutdown_connections 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ef9dcbb230 0x55ef9dd23ce0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ef9dca2720 0x55ef9dcb5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ef9dc9d5b0 0x55ef9dcb4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 --2- 172.21.15.28:0/1891208403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef9dc9cbc0 0x55ef9dcb3c00 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 >> 172.21.15.28:0/1891208403 conn(0x55ef9dc98130 msgr2=0x55ef9dc9bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c11ffb640 0 client.18123 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 shutdown_connections 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.531+0000 7f8c33768740 1 -- 172.21.15.28:0/1891208403 wait complete. 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c33768740 20 client.18123 trim_cache size 0 max 16384 2024-03-22T02:22:36.536 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:36.537 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:22:36.537 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c45c983d-0e3f-424f-b7be-e2993644556f", 2024-03-22T02:22:36.537 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:22:36.537 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:36.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 msgr2=0x7f8c180a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 0x7f8c180a53e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x55ef9dc9de10 tx=0x7f8c24005f90 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 msgr2=0x7f8c2c198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009230 tx=0x7f8c1402c650 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 shutdown_connections 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8c180a2f20 0x7f8c180a53e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8c2c102640 0x7f8c2c19f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c2c101c80 0x7f8c2c198420 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 --2- 172.21.15.28:0/1413124963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c2c100a90 0x7f8c2c197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 >> 172.21.15.28:0/1413124963 conn(0x7f8c2c0fc020 msgr2=0x7f8c2c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 shutdown_connections 2024-03-22T02:22:36.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.532+0000 7f8c30aaa640 1 -- 172.21.15.28:0/1413124963 wait complete. 2024-03-22T02:22:36.547 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:22:36.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 -- 172.21.15.28:0/2430088837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24102690 msgr2=0x7ffa24104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 --2- 172.21.15.28:0/2430088837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24102690 0x7ffa24104a80 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009ca0 tx=0x7ffa2002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 -- 172.21.15.28:0/2430088837 shutdown_connections 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 --2- 172.21.15.28:0/2430088837 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa241078f0 0x7ffa24109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 --2- 172.21.15.28:0/2430088837 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24104fc0 0x7ffa241073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 --2- 172.21.15.28:0/2430088837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa24102690 0x7ffa24104a80 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 -- 172.21.15.28:0/2430088837 >> 172.21.15.28:0/2430088837 conn(0x7ffa240fc040 msgr2=0x7ffa240fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 -- 172.21.15.28:0/2430088837 shutdown_connections 2024-03-22T02:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.659+0000 7ffa2c088640 1 -- 172.21.15.28:0/2430088837 wait complete. 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 Processor -- start 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 -- start start 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24102690 0x7ffa24102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 0x7ffa24100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2410d820 con 0x7ffa241078f0 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2410d6a0 con 0x7ffa24102690 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2c088640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2410d9a0 con 0x7ffa24104fc0 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44850/0 (socket says 172.21.15.28:44850) 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 -- 172.21.15.28:0/809864652 learned_addr learned my addr 172.21.15.28:0/809864652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa29623640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 0x7ffa24100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa29e24640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24102690 0x7ffa24102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 -- 172.21.15.28:0/809864652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 msgr2=0x7ffa24100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 0x7ffa24100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 -- 172.21.15.28:0/809864652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24102690 msgr2=0x7ffa24102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24102690 0x7ffa24102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa2a625640 1 -- 172.21.15.28:0/809864652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa24101410 con 0x7ffa241078f0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.660+0000 7ffa29623640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 0x7ffa24100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa2a625640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e970 tx=0x7ffa1c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa1affd640 1 -- 172.21.15.28:0/809864652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c00ccc0 con 0x7ffa241078f0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa241a1fe0 con 0x7ffa241078f0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa1affd640 1 -- 172.21.15.28:0/809864652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa1c00ce60 con 0x7ffa241078f0 2024-03-22T02:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa1affd640 1 -- 172.21.15.28:0/809864652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c010490 con 0x7ffa241078f0 2024-03-22T02:22:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.661+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa241a2450 con 0x7ffa241078f0 2024-03-22T02:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa1affd640 1 -- 172.21.15.28:0/809864652 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ffa1c005020 con 0x7ffa241078f0 2024-03-22T02:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 Processor -- start 2024-03-22T02:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 -- start start 2024-03-22T02:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 0x563c01961850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c01966920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01966f90 0x563c01969420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c018c68c0 con 0x563c01961430 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c0186ec20 con 0x563c01966f90 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563c018c2870 con 0x563c01961e20 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa29623640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 0x563c01961850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa29e24640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c01966920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa29623640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 0x563c01961850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44860/0 (socket says 172.21.15.28:44860) 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.664+0000 7ffa29623640 1 -- 172.21.15.28:0/3913582862 learned_addr learned my addr 172.21.15.28:0/3913582862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29623640 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01966f90 0x563c01969420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa1affd640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 0x7ffa100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 -- 172.21.15.28:0/3913582862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01966f90 msgr2=0x563c01969420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01966f90 0x563c01969420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 -- 172.21.15.28:0/3913582862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 msgr2=0x563c01961850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 0x563c01961850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 -- 172.21.15.28:0/3913582862 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c01893150 con 0x563c01961e20 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa2a625640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 0x7ffa100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa1affd640 1 -- 172.21.15.28:0/809864652 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ffa1c0cfcc0 con 0x7ffa241078f0 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa29e24640 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c01966920 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffa200324c0 tx=0x7ffa200323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f0b4515dd9df482 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa0ffff640 1 -- 172.21.15.28:0/3913582862 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20043070 con 0x563c01961e20 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa0ffff640 1 -- 172.21.15.28:0/3913582862 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20032c90 con 0x563c01961e20 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa0ffff640 1 -- 172.21.15.28:0/3913582862 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20039040 con 0x563c01961e20 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.665+0000 7ffa2a625640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 0x7ffa100a53e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c09bce0 tx=0x7ffa1c0122b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3913582862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 msgr2=0x563c01966920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c01966920 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffa200324c0 tx=0x7ffa200323d0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3913582862 shutdown_connections 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01966f90 0x563c01969420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c01966920 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3913582862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961430 0x563c01961850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.666+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3913582862 >> 172.21.15.28:0/3913582862 conn(0x563c0195c9a0 msgr2=0x563c0195cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3913582862 shutdown_connections 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3913582862 wait complete. 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- start start 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 Processor -- start 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c0196be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01972e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563c01969ee0 con 0x563c01966f90 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563c01882130 con 0x563c01961430 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563c0196a060 con 0x563c01961e20 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.667+0000 7ffa2ec7a740 10 client.0 _ll_register_callbacks cb 0x563c01881b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c0196be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51448/0 (socket says 172.21.15.28:51448) 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29623640 1 -- 172.21.15.28:0/1717977791 learned_addr learned my addr 172.21.15.28:0/1717977791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29623640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01972e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 -- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 msgr2=0x563c0196b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 -- 172.21.15.28:0/1717977791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 msgr2=0x563c01972e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01972e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 -- 172.21.15.28:0/1717977791 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c016a47d0 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29623640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa29e24640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c0196be10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x563c0195d910 tx=0x7ffa20009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe88418dccf27445 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2004c030 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa20032c90 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.668+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ffa20048450 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffa20059050 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 10 client.27955 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c015da3f0 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20032c90 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7ffa2003f4b0 con 0x563c01961e20 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 10 client.27955 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 10 client.27955 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 20 client.27955 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c01977770 0x563c01979b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 4 client.27955 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c018d3e00 con 0x563c01977770 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa29623640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c01977770 0x563c01979b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa0d7fa640 10 client.27955 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.669+0000 7ffa29623640 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c01977770 0x563c01979b90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=930485e0cfd33863 server_cookie=6e7752e67deba3af in_seq=0 out_seq=0 2024-03-22T02:22:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/1717977791 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563c018d3e00 con 0x563c01977770 2024-03-22T02:22:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa0d7fa640 10 client.27955 handle_command_reply: tid=0 2024-03-22T02:22:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 client.27955 shutdown 2024-03-22T02:22:36.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 msgr2=0x563c0196be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c0196be10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x563c0195d910 tx=0x7ffa20009560 comp rx=0 tx=0).stop 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 shutdown_connections 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c01977770 0x563c01979b90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01972e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961e20 0x563c0196be10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa0d7fa640 0 client.27955 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/1717977791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 >> 172.21.15.28:0/1717977791 conn(0x563c0195c9a0 msgr2=0x563c0195bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 shutdown_connections 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/1717977791 wait complete. 2024-03-22T02:22:36.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.671+0000 7ffa2ec7a740 20 client.27955 trim_cache size 0 max 16384 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 Processor -- start 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 -- start start 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01966f90 0x563c0196be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241af4c0 con 0x563c01961e20 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241acd90 con 0x563c01961430 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241aabe0 con 0x563c01966f90 2024-03-22T02:22:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29e24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29e24640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44866/0 (socket says 172.21.15.28:44866) 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29623640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51450/0 (socket says 172.21.15.28:51450) 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29e24640 1 -- 172.21.15.28:0/55084013 learned_addr learned my addr 172.21.15.28:0/55084013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.689+0000 7ffa29623640 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01966f90 0x563c0196be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 -- 172.21.15.28:0/55084013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01966f90 msgr2=0x563c0196be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01966f90 0x563c0196be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 -- 172.21.15.28:0/55084013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 msgr2=0x563c0196b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 -- 172.21.15.28:0/55084013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c0159a0f0 con 0x563c01961e20 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29623640 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa29e24640 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ffa20002230 tx=0x7ffa2002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57b45ff2a7713c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa0f7fe640 1 -- 172.21.15.28:0/55084013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2004c020 con 0x563c01961e20 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa0f7fe640 1 -- 172.21.15.28:0/55084013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20003e50 con 0x563c01961e20 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.690+0000 7ffa0f7fe640 1 -- 172.21.15.28:0/55084013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20039070 con 0x563c01961e20 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/55084013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 msgr2=0x563c01972e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ffa20002230 tx=0x7ffa2002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/55084013 shutdown_connections 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01966f90 0x563c0196be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01961e20 0x563c01972e10 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/55084013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961430 0x563c0196b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/55084013 >> 172.21.15.28:0/55084013 conn(0x563c0195c9a0 msgr2=0x563c019633b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/55084013 shutdown_connections 2024-03-22T02:22:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/55084013 wait complete. 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 -- start start 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.691+0000 7ffa2ec7a740 1 Processor -- start 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961e20 0x563c01970540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01971500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241ab250 con 0x563c01966f90 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241ac6c0 con 0x563c01961e20 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa241ab920 con 0x563c01961430 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 10 client.0 _ll_register_callbacks cb 0x7ffa20009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29e24640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961e20 0x563c01970540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56680/0 (socket says 172.21.15.28:56680) 2024-03-22T02:22:36.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 -- 172.21.15.28:0/3414955352 learned_addr learned my addr 172.21.15.28:0/3414955352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01971500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 -- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961e20 msgr2=0x563c01970540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961e20 0x563c01970540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 -- 172.21.15.28:0/3414955352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 msgr2=0x563c01971500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01971500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 -- 172.21.15.28:0/3414955352 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c01971af0 con 0x563c01961430 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa29623640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x563c0197a3e0 tx=0x7ffa14015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f77a2b4b0d8bdc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1402b020 con 0x563c01961430 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa20003e50 con 0x563c01961430 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ffa140107c0 con 0x563c01961430 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.692+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ffa14039050 con 0x563c01961430 2024-03-22T02:22:36.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 10 client.27961 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x563c01972ea0 con 0x563c01961430 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa20003e50 con 0x563c01961430 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7ffa14027ce0 con 0x563c01961430 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 10 client.27961 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 10 client.27961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 20 client.27961 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c019de050 0x563c019e0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 4 client.27961 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563c018ce150 con 0x563c019de050 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa29e24640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c019de050 0x563c019e0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.693+0000 7ffa0d7fa640 10 client.27961 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa29e24640 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c019de050 0x563c019e0470 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=569ccf93478f22e server_cookie=624ff567d5b29381 in_seq=0 out_seq=0 2024-03-22T02:22:36.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa0d7fa640 1 -- 172.21.15.28:0/3414955352 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563c018ce150 con 0x563c019de050 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa0d7fa640 10 client.27961 handle_command_reply: tid=0 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 client.27961 shutdown 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 msgr2=0x563c01970000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x563c0197a3e0 tx=0x7ffa14015040 comp rx=0 tx=0).stop 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 shutdown_connections 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563c019de050 0x563c019e0470 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa0d7fa640 0 client.27961 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c01966f90 0x563c01971500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563c01961e20 0x563c01970540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 --2- 172.21.15.28:0/3414955352 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563c01961430 0x563c01970000 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 >> 172.21.15.28:0/3414955352 conn(0x563c0195c9a0 msgr2=0x563c019633b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 shutdown_connections 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 1 -- 172.21.15.28:0/3414955352 wait complete. 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.694+0000 7ffa2ec7a740 20 client.27961 trim_cache size 0 max 16384 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: "c45c983d-0e3f-424f-b7be-e2993644556f": { 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:22:36.702 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c45c983d-0e3f-424f-b7be-e2993644556f", 2024-03-22T02:22:36.703 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:22:36.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:22:36.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:22:36.703 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 msgr2=0x7ffa100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 0x7ffa100a53e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c09bce0 tx=0x7ffa1c0122b0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 msgr2=0x7ffa24100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00e970 tx=0x7ffa1c00ed70 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 shutdown_connections 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa100a2f20 0x7ffa100a53e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa241078f0 0x7ffa24100b80 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa24104fc0 0x7ffa24100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 --2- 172.21.15.28:0/809864652 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa24102690 0x7ffa24102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 >> 172.21.15.28:0/809864652 conn(0x7ffa240fc040 msgr2=0x7ffa24102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 shutdown_connections 2024-03-22T02:22:36.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:22:36.695+0000 7ffa2c088640 1 -- 172.21.15.28:0/809864652 wait complete. 2024-03-22T02:22:36.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:36 smithi028 ceph-mon[31916]: pgmap v630: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:22:36.706 INFO:tasks.cephfs.filesystem:scrub status for tag:c45c983d-0e3f-424f-b7be-e2993644556f - {'path': '/', 'tag': 'c45c983d-0e3f-424f-b7be-e2993644556f', 'options': 'recursive,force'} 2024-03-22T02:22:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:36 smithi120 ceph-mon[36803]: pgmap v630: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:22:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:36 smithi083 ceph-mon[36340]: pgmap v630: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 423 op/s 2024-03-22T02:22:37.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 881727 11.64 MB/sec execute 319 sec latency 23.744 ms 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:37 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:22:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:37.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:22:37.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:22:37.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:37.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:37 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:38.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:22:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:22:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:22:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:37 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:22:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 -- 172.21.15.28:0/3424229932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c41004c0 msgr2=0x7f17c41008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 --2- 172.21.15.28:0/3424229932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c41004c0 0x7f17c41008e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009900 tx=0x7f17b402bc30 comp rx=0 tx=0).stop 2024-03-22T02:22:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 -- 172.21.15.28:0/3424229932 shutdown_connections 2024-03-22T02:22:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 --2- 172.21.15.28:0/3424229932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41017c0 0x7f17c4101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 --2- 172.21.15.28:0/3424229932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c4100e20 0x7f17c4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 --2- 172.21.15.28:0/3424229932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c41004c0 0x7f17c41008e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 -- 172.21.15.28:0/3424229932 >> 172.21.15.28:0/3424229932 conn(0x7f17c40fc060 msgr2=0x7f17c40fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 -- 172.21.15.28:0/3424229932 shutdown_connections 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.154+0000 7f17c8e88640 1 -- 172.21.15.28:0/3424229932 wait complete. 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 Processor -- start 2024-03-22T02:22:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 -- start start 2024-03-22T02:22:38.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41004c0 0x7f17c419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 0x7f17c41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410fa50 con 0x7f17c4100e20 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410f8d0 con 0x7f17c41017c0 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c8e88640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410fbd0 con 0x7f17c41004c0 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c2d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 0x7f17c41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44886/0 (socket says 172.21.15.28:44886) 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1561388503 learned_addr learned my addr 172.21.15.28:0/1561388503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.155+0000 7f17c259c640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41004c0 0x7f17c419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1561388503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41004c0 msgr2=0x7f17c419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41004c0 0x7f17c419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1561388503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 msgr2=0x7f17c41a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 0x7f17c41a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1561388503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c41a1c10 con 0x7f17c4100e20 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 0x7f17c41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f17b800e9f0 tx=0x7f17b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17b37fe640 1 -- 172.21.15.28:0/1561388503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b800cd80 con 0x7f17c4100e20 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c41a1f00 con 0x7f17c4100e20 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17b37fe640 1 -- 172.21.15.28:0/1561388503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b8010910 con 0x7f17c4100e20 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17b37fe640 1 -- 172.21.15.28:0/1561388503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b80188f0 con 0x7f17c4100e20 2024-03-22T02:22:38.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.156+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c41a23c0 con 0x7f17c4100e20 2024-03-22T02:22:38.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.158+0000 7f17b37fe640 1 -- 172.21.15.28:0/1561388503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f17b8010ac0 con 0x7f17c4100e20 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 Processor -- start 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 -- start start 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee714ed0 0x558aee7199d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee71c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aee679040 con 0x558aee7144e0 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aee6219e0 con 0x558aee714ed0 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17cba74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aee674ff0 con 0x558aee71a040 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17b37fe640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 0x7f17ac0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17c259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee714ed0 0x558aee7199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.159+0000 7f17c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44896/0 (socket says 172.21.15.28:44896) 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 -- 172.21.15.28:0/1224094790 learned_addr learned my addr 172.21.15.28:0/1224094790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee71c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 0x7f17ac0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 -- 172.21.15.28:0/1224094790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 msgr2=0x558aee71c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee71c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17b37fe640 1 -- 172.21.15.28:0/1561388503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f17b80cfbf0 con 0x7f17c4100e20 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 -- 172.21.15.28:0/1224094790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee714ed0 msgr2=0x558aee7199d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee714ed0 0x558aee7199d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 -- 172.21.15.28:0/1224094790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aee38d210 con 0x558aee7144e0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009fd0 tx=0x7f17bc013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1088f8f2defca1db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17b0ff9640 1 -- 172.21.15.28:0/1224094790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc022020 con 0x558aee7144e0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17b0ff9640 1 -- 172.21.15.28:0/1224094790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17bc00f8b0 con 0x558aee7144e0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17b0ff9640 1 -- 172.21.15.28:0/1224094790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc010e10 con 0x558aee7144e0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.160+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 0x7f17ac0a5480 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f17bc023000 tx=0x7f17bc021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 -- 172.21.15.28:0/1224094790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 msgr2=0x558aee714900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009fd0 tx=0x7f17bc013830 comp rx=0 tx=0).stop 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 -- 172.21.15.28:0/1224094790 shutdown_connections 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee71c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee714ed0 0x558aee7199d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 --2- 172.21.15.28:0/1224094790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee714900 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 -- 172.21.15.28:0/1224094790 >> 172.21.15.28:0/1224094790 conn(0x558aee70fa50 msgr2=0x558aee70fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 -- 172.21.15.28:0/1224094790 shutdown_connections 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.161+0000 7f17cba74740 1 -- 172.21.15.28:0/1224094790 wait complete. 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 -- start start 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 Processor -- start 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee71e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee7259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558aee71cff0 con 0x558aee714ed0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558aee634db0 con 0x558aee7144e0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558aee71d170 con 0x558aee71a040 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17cba74740 10 client.0 _ll_register_callbacks cb 0x558aee38ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44898/0 (socket says 172.21.15.28:44898) 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1372582036 learned_addr learned my addr 172.21.15.28:0/1372582036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c259c640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee71e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee7259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1372582036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 msgr2=0x558aee7259f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee7259f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 msgr2=0x558aee71e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee71e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.162+0000 7f17c1d9b640 1 -- 172.21.15.28:0/1372582036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aee459620 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f17b8098930 tx=0x7f17b806ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe57c2ebee4c083 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8026040 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f17b806bdb0 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558aee686580 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f17b80a0000 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 10 client.18153 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558aee728e20 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558aee686580 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f17b8097d40 con 0x558aee714ed0 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 10 client.18153 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 10 client.18153 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 20 client.18153 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee72a4b0 0x558aee72c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 4 client.18153 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558aee6808d0 con 0x558aee72a4b0 2024-03-22T02:22:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.163+0000 7f17c259c640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee72a4b0 0x558aee72c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.164+0000 7f17a9ffb640 10 client.18153 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:38.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.164+0000 7f17c259c640 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee72a4b0 0x558aee72c900 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92f3f631d18322bc server_cookie=6ab40fbf7c1d418d in_seq=0 out_seq=0 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17a9ffb640 1 -- 172.21.15.28:0/1372582036 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558aee6808d0 con 0x558aee72a4b0 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17a9ffb640 10 client.18153 handle_command_reply: tid=0 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 client.18153 shutdown 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 msgr2=0x558aee71ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f17b8098930 tx=0x7f17b806ac10 comp rx=0 tx=0).stop 2024-03-22T02:22:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 shutdown_connections 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee72a4b0 0x558aee72c900 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee71a040 0x558aee7259f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee714ed0 0x558aee71ea20 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 --2- 172.21.15.28:0/1372582036 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee71e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17a9ffb640 0 client.18153 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 >> 172.21.15.28:0/1372582036 conn(0x558aee70fa50 msgr2=0x558aee70ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 shutdown_connections 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 1 -- 172.21.15.28:0/1372582036 wait complete. 2024-03-22T02:22:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.165+0000 7f17cba74740 20 client.18153 trim_cache size 0 max 16384 2024-03-22T02:22:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 Processor -- start 2024-03-22T02:22:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 -- start start 2024-03-22T02:22:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee7259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 0x558aee715760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41af2f0 con 0x558aee7144e0 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41acbc0 con 0x558aee71a040 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17cba74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41aaa10 con 0x558aee714ed0 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44906/0 (socket says 172.21.15.28:44906) 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 -- 172.21.15.28:0/37071510 learned_addr learned my addr 172.21.15.28:0/37071510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 0x558aee715760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee7259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 -- 172.21.15.28:0/37071510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 msgr2=0x558aee7259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee7259b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 -- 172.21.15.28:0/37071510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 msgr2=0x558aee715760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 0x558aee715760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 -- 172.21.15.28:0/37071510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558aee71c890 con 0x558aee7144e0 2024-03-22T02:22:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 0x558aee715760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.181+0000 7f17c259c640 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f17b402cfd0 tx=0x7f17b4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb0253e17fce165 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17abfff640 1 -- 172.21.15.28:0/37071510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b402cd20 con 0x558aee7144e0 2024-03-22T02:22:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17abfff640 1 -- 172.21.15.28:0/37071510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b4046440 con 0x558aee7144e0 2024-03-22T02:22:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17abfff640 1 -- 172.21.15.28:0/37071510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4005040 con 0x558aee7144e0 2024-03-22T02:22:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 -- 172.21.15.28:0/37071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 msgr2=0x558aee70ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f17b402cfd0 tx=0x7f17b4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 -- 172.21.15.28:0/37071510 shutdown_connections 2024-03-22T02:22:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee71a040 0x558aee715760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee7259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 --2- 172.21.15.28:0/37071510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee7144e0 0x558aee70ead0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 -- 172.21.15.28:0/37071510 >> 172.21.15.28:0/37071510 conn(0x558aee70fa50 msgr2=0x558aee70e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 -- 172.21.15.28:0/37071510 shutdown_connections 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.182+0000 7f17cba74740 1 -- 172.21.15.28:0/37071510 wait complete. 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 -- start start 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 Processor -- start 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee723760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41ab080 con 0x558aee71a040 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41ac4f0 con 0x558aee7144e0 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17c41ab750 con 0x558aee714ed0 2024-03-22T02:22:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c1d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee723760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44922/0 (socket says 172.21.15.28:44922) 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51524/0 (socket says 172.21.15.28:51524) 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17c2d9d640 1 -- 172.21.15.28:0/2522203384 learned_addr learned my addr 172.21.15.28:0/2522203384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.183+0000 7f17cba74740 10 client.0 _ll_register_callbacks cb 0x7f17b8096980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 -- 172.21.15.28:0/2522203384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 msgr2=0x558aee723760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee723760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 -- 172.21.15.28:0/2522203384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 msgr2=0x558aee724770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 -- 172.21.15.28:0/2522203384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558aee724d60 con 0x558aee7144e0 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c2d9d640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:38.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee723760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17c259c640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f17b402c170 tx=0x7f17b4004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68a5b9b20def770a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4053020 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f17b402cd20 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b4046440 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f17b405e050 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17cba74740 10 client.27980 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x558aee7909d0 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.184+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b4046440 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f17b4004880 con 0x558aee7144e0 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 10 client.27980 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 10 client.27980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 20 client.27980 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee791370 0x558aee793790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 4 client.27980 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558aee5fe770 con 0x558aee791370 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee791370 0x558aee793790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17a9ffb640 10 client.27980 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:38.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.185+0000 7f17c1d9b640 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee791370 0x558aee793790 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce52cafe01fc9d3 server_cookie=4185dcc770acf054 in_seq=0 out_seq=0 2024-03-22T02:22:38.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 883850 11.64 MB/sec execute 320 sec latency 210.788 ms 2024-03-22T02:22:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.495+0000 7f17a9ffb640 1 -- 172.21.15.28:0/2522203384 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558aee5fe770 con 0x558aee791370 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.495+0000 7f17a9ffb640 10 client.27980 handle_command_reply: tid=0 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.495+0000 7f17cba74740 1 client.27980 shutdown 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.495+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 msgr2=0x558aee723220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.495+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f17b402c170 tx=0x7f17b4004530 comp rx=0 tx=0).stop 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 shutdown_connections 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558aee791370 0x558aee793790 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558aee71a040 0x558aee724770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17a9ffb640 0 client.27980 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558aee714ed0 0x558aee723760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 --2- 172.21.15.28:0/2522203384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558aee7144e0 0x558aee723220 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 >> 172.21.15.28:0/2522203384 conn(0x558aee70fa50 msgr2=0x558aee70ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 shutdown_connections 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 1 -- 172.21.15.28:0/2522203384 wait complete. 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17cba74740 20 client.27980 trim_cache size 0 max 16384 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:22:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 msgr2=0x7f17ac0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 0x7f17ac0a5480 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f17bc023000 tx=0x7f17bc021000 comp rx=0 tx=0).stop 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 msgr2=0x7f17c419a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f17b800e9f0 tx=0x7f17b800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 shutdown_connections 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17ac0a2fc0 0x7f17ac0a5480 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17c41017c0 0x7f17c41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4100e20 0x7f17c419a580 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.496+0000 7f17c8e88640 1 --2- 172.21.15.28:0/1561388503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17c41004c0 0x7f17c419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.497+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 >> 172.21.15.28:0/1561388503 conn(0x7f17c40fc060 msgr2=0x7f17c40fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.497+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 shutdown_connections 2024-03-22T02:22:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:38.497+0000 7f17c8e88640 1 -- 172.21.15.28:0/1561388503 wait complete. 2024-03-22T02:22:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:38 smithi028 ceph-mon[31916]: pgmap v631: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:22:38.899 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:38 smithi120 ceph-mon[36803]: pgmap v631: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:22:38.915 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:38 smithi083 ceph-mon[36340]: pgmap v631: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:22:39.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 885086 11.62 MB/sec execute 321 sec latency 248.592 ms 2024-03-22T02:22:40.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 887647 11.63 MB/sec execute 322 sec latency 114.505 ms 2024-03-22T02:22:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:40 smithi028 ceph-mon[31916]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 416 op/s 2024-03-22T02:22:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:40 smithi120 ceph-mon[36803]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 416 op/s 2024-03-22T02:22:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:40 smithi083 ceph-mon[36340]: pgmap v632: 129 pgs: 129 active+clean; 60 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 416 op/s 2024-03-22T02:22:41.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 889225 11.62 MB/sec execute 323 sec latency 28.153 ms 2024-03-22T02:22:42.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 891609 11.63 MB/sec execute 324 sec latency 21.250 ms 2024-03-22T02:22:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:42 smithi028 ceph-mon[31916]: pgmap v633: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-03-22T02:22:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:42 smithi120 ceph-mon[36803]: pgmap v633: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-03-22T02:22:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:42 smithi083 ceph-mon[36340]: pgmap v633: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-03-22T02:22:43.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 894318 11.63 MB/sec execute 325 sec latency 32.409 ms 2024-03-22T02:22:44.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 896110 11.63 MB/sec execute 326 sec latency 30.716 ms 2024-03-22T02:22:44.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:44 smithi028 ceph-mon[31916]: pgmap v634: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:44 smithi120 ceph-mon[36803]: pgmap v634: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:44 smithi083 ceph-mon[36340]: pgmap v634: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-03-22T02:22:45.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 898488 11.64 MB/sec execute 327 sec latency 80.211 ms 2024-03-22T02:22:45.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:45 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:22:45.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:45 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:22:46.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:45 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:22:46.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:45 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:22:46.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:45 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:22:46.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:45 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:22:46.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 900341 11.62 MB/sec execute 328 sec latency 25.843 ms 2024-03-22T02:22:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:46 smithi120 ceph-mon[36803]: pgmap v635: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 464 op/s 2024-03-22T02:22:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:46 smithi083 ceph-mon[36340]: pgmap v635: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 464 op/s 2024-03-22T02:22:47.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:46 smithi028 ceph-mon[31916]: pgmap v635: 129 pgs: 129 active+clean; 55 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 464 op/s 2024-03-22T02:22:47.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 902437 11.62 MB/sec execute 329 sec latency 28.247 ms 2024-03-22T02:22:48.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 905122 11.63 MB/sec execute 330 sec latency 20.256 ms 2024-03-22T02:22:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:48 smithi120 ceph-mon[36803]: pgmap v636: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 387 op/s 2024-03-22T02:22:49.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:48 smithi083 ceph-mon[36340]: pgmap v636: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 387 op/s 2024-03-22T02:22:49.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:48 smithi028 ceph-mon[31916]: pgmap v636: 129 pgs: 129 active+clean; 56 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.8 MiB/s wr, 387 op/s 2024-03-22T02:22:49.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 906574 11.62 MB/sec execute 331 sec latency 29.385 ms 2024-03-22T02:22:50.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 908995 11.63 MB/sec execute 332 sec latency 70.855 ms 2024-03-22T02:22:50.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:50 smithi028 ceph-mon[31916]: pgmap v637: 129 pgs: 129 active+clean; 59 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 453 op/s 2024-03-22T02:22:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:50 smithi120 ceph-mon[36803]: pgmap v637: 129 pgs: 129 active+clean; 59 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 453 op/s 2024-03-22T02:22:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:50 smithi083 ceph-mon[36340]: pgmap v637: 129 pgs: 129 active+clean; 59 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.5 MiB/s wr, 453 op/s 2024-03-22T02:22:51.427 INFO:tasks.workunit.client.0.smithi028.stdout: 1 910615 11.62 MB/sec execute 333 sec latency 31.327 ms 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 -- 172.21.15.28:0/2077655155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 msgr2=0x7fc040104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 --2- 172.21.15.28:0/2077655155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040104a40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009ca0 tx=0x7fc03c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 -- 172.21.15.28:0/2077655155 shutdown_connections 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 --2- 172.21.15.28:0/2077655155 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0401078b0 0x7fc040109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 --2- 172.21.15.28:0/2077655155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc040104f80 0x7fc040107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 --2- 172.21.15.28:0/2077655155 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040104a40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.593+0000 7fc048066640 1 -- 172.21.15.28:0/2077655155 >> 172.21.15.28:0/2077655155 conn(0x7fc0400fc020 msgr2=0x7fc0400fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- 172.21.15.28:0/2077655155 shutdown_connections 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- 172.21.15.28:0/2077655155 wait complete. 2024-03-22T02:22:51.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 Processor -- start 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- start start 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc040104f80 0x7fc0401983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc04010d850 con 0x7fc0401078b0 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc04010d6d0 con 0x7fc040102650 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc048066640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc04010d9d0 con 0x7fc040104f80 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc046603640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc046603640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41328/0 (socket says 172.21.15.28:41328) 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc046603640 1 -- 172.21.15.28:0/532347200 learned_addr learned my addr 172.21.15.28:0/532347200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:51.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc045e02640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.594+0000 7fc045601640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc040104f80 0x7fc0401983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 -- 172.21.15.28:0/532347200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc040104f80 msgr2=0x7fc0401983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc040104f80 0x7fc0401983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 -- 172.21.15.28:0/532347200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 msgr2=0x7fc040197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 -- 172.21.15.28:0/532347200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc04019fa90 con 0x7fc0401078b0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc045e02640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc046603640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc03800d950 tx=0x7fc03800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc036ffd640 1 -- 172.21.15.28:0/532347200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038017030 con 0x7fc0401078b0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc04019fd80 con 0x7fc0401078b0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc036ffd640 1 -- 172.21.15.28:0/532347200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc038004020 con 0x7fc0401078b0 2024-03-22T02:22:51.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc036ffd640 1 -- 172.21.15.28:0/532347200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc038003c00 con 0x7fc0401078b0 2024-03-22T02:22:51.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.595+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0401a0220 con 0x7fc0401078b0 2024-03-22T02:22:51.607 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.598+0000 7fc036ffd640 1 -- 172.21.15.28:0/532347200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc03801a070 con 0x7fc0401078b0 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.598+0000 7fc04ac48740 1 Processor -- start 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 -- start start 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242792c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x56242793ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562427891040 con 0x56242792ffb0 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5624278399e0 con 0x56242792c460 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc04ac48740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56242788cff0 con 0x56242792ce50 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045601640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242792c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53408/0 (socket says 172.21.15.28:53408) 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 -- 172.21.15.28:0/1468480740 learned_addr learned my addr 172.21.15.28:0/1468480740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045601640 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x56242793ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc036ffd640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 0x7fc02c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc046603640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 0x7fc02c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 -- 172.21.15.28:0/1468480740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 msgr2=0x56242792c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242792c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 -- 172.21.15.28:0/1468480740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 msgr2=0x56242793ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.599+0000 7fc045e02640 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x56242793ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc036ffd640 1 -- 172.21.15.28:0/532347200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc0380cf040 con 0x7fc0401078b0 2024-03-22T02:22:51.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc045e02640 1 -- 172.21.15.28:0/1468480740 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624275a5210 con 0x56242792ce50 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc045601640 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x56242793ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc045e02640 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc03c037040 tx=0x7fc03c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d178a650c9feedc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc02bfff640 1 -- 172.21.15.28:0/1468480740 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c043070 con 0x56242792ce50 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc02bfff640 1 -- 172.21.15.28:0/1468480740 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc03c032c90 con 0x56242792ce50 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc02bfff640 1 -- 172.21.15.28:0/1468480740 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c039040 con 0x56242792ce50 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.600+0000 7fc046603640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 0x7fc02c0a5480 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc0380977c0 tx=0x7fc038095d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 -- 172.21.15.28:0/1468480740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 msgr2=0x56242792f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc03c037040 tx=0x7fc03c005ef0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 -- 172.21.15.28:0/1468480740 shutdown_connections 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x56242793ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242792f940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 --2- 172.21.15.28:0/1468480740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242792c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 -- 172.21.15.28:0/1468480740 >> 172.21.15.28:0/1468480740 conn(0x562427927a50 msgr2=0x562427927e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.601+0000 7fc04ac48740 1 -- 172.21.15.28:0/1468480740 shutdown_connections 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 -- 172.21.15.28:0/1468480740 wait complete. 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 -- start start 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 Processor -- start 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242793ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x562427946280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56242793d7a0 con 0x56242792ffb0 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56242784cdb0 con 0x56242792c460 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56242793d920 con 0x56242792ce50 2024-03-22T02:22:51.611 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc04ac48740 10 client.0 _ll_register_callbacks cb 0x5624275a4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045601640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242793ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53410/0 (socket says 172.21.15.28:53410) 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 -- 172.21.15.28:0/2376032405 learned_addr learned my addr 172.21.15.28:0/2376032405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045601640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x562427946280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 -- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 msgr2=0x56242793ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242793ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 -- 172.21.15.28:0/2376032405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 msgr2=0x562427946280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x562427946280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 -- 172.21.15.28:0/2376032405 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562427671620 con 0x56242792ce50 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045601640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x562427946280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:22:51.612 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc045e02640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0324c0 tx=0x7fc03c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c5f41ee52629e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.603+0000 7fc0297fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c04c070 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc03c032c90 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc03c0349f0 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc03c058050 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 10 client.27991 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56242793d7a0 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc03c032c90 con 0x56242792ce50 2024-03-22T02:22:51.613 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fc03c0489c0 con 0x56242792ce50 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 10 client.27991 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 10 client.27991 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 20 client.27991 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242794a8f0 0x56242794cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 4 client.27991 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56242789e580 con 0x56242794a8f0 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.604+0000 7fc045601640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242794a8f0 0x56242794cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.605+0000 7fc045601640 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242794a8f0 0x56242794cce0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d1b0b0120af022 server_cookie=e0009dcd90f17dcf in_seq=0 out_seq=0 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.605+0000 7fc0297fa640 10 client.27991 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc0297fa640 1 -- 172.21.15.28:0/2376032405 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56242789e580 con 0x56242794a8f0 2024-03-22T02:22:51.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc0297fa640 10 client.27991 handle_command_reply: tid=0 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 client.27991 shutdown 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 msgr2=0x56242793f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc03c0324c0 tx=0x7fc03c0095c0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 shutdown_connections 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242794a8f0 0x56242794cce0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792ffb0 0x562427946280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792ce50 0x56242793f240 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2376032405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792c460 0x56242793ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc0297fa640 0 client.27991 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 >> 172.21.15.28:0/2376032405 conn(0x562427927a50 msgr2=0x562427926c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 shutdown_connections 2024-03-22T02:22:51.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 1 -- 172.21.15.28:0/2376032405 wait complete. 2024-03-22T02:22:51.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.606+0000 7fc04ac48740 20 client.27991 trim_cache size 0 max 16384 2024-03-22T02:22:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 Processor -- start 2024-03-22T02:22:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 -- start start 2024-03-22T02:22:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562427927a50 0x5624279394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.639 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427951570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401ab040 con 0x56242792c460 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401a8910 con 0x56242792ce50 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc04ac48740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401a6760 con 0x562427927a50 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc045e02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc045e02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41354/0 (socket says 172.21.15.28:41354) 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.637+0000 7fc045e02640 1 -- 172.21.15.28:0/4087613566 learned_addr learned my addr 172.21.15.28:0/4087613566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045601640 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562427927a50 0x5624279394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.640 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045601640 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427951570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 -- 172.21.15.28:0/4087613566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562427927a50 msgr2=0x5624279394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562427927a50 0x5624279394f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 -- 172.21.15.28:0/4087613566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 msgr2=0x562427951570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427951570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.641 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 -- 172.21.15.28:0/4087613566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56242792d230 con 0x56242792c460 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.638+0000 7fc045e02640 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc03c048d90 tx=0x7fc03c005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983b1b82402d299 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.639+0000 7fc02b7fe640 1 -- 172.21.15.28:0/4087613566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c04c070 con 0x56242792c460 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.639+0000 7fc02b7fe640 1 -- 172.21.15.28:0/4087613566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc03c034cd0 con 0x56242792c460 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.639+0000 7fc02b7fe640 1 -- 172.21.15.28:0/4087613566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc03c039070 con 0x56242792c460 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 -- 172.21.15.28:0/4087613566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 msgr2=0x562427946240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc03c048d90 tx=0x7fc03c005d80 comp rx=0 tx=0).stop 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 -- 172.21.15.28:0/4087613566 shutdown_connections 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427951570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56242792c460 0x562427946240 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 --2- 172.21.15.28:0/4087613566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562427927a50 0x5624279394f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 -- 172.21.15.28:0/4087613566 >> 172.21.15.28:0/4087613566 conn(0x56242793e8c0 msgr2=0x562427938de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.640+0000 7fc04ac48740 1 -- 172.21.15.28:0/4087613566 shutdown_connections 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 -- 172.21.15.28:0/4087613566 wait complete. 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 -- start start 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 Processor -- start 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792c460 0x562427943e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427944e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401a6dd0 con 0x562427927a50 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401a8240 con 0x56242792ce50 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0401a74a0 con 0x56242792c460 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.641+0000 7fc04ac48740 10 client.0 _ll_register_callbacks cb 0x7fc030009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045e02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792c460 0x562427943e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41358/0 (socket says 172.21.15.28:41358) 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 -- 172.21.15.28:0/2603048447 learned_addr learned my addr 172.21.15.28:0/2603048447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427944e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 -- 172.21.15.28:0/2603048447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792c460 msgr2=0x562427943e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792c460 0x562427943e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 -- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 msgr2=0x562427944e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427944e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 -- 172.21.15.28:0/2603048447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562427945470 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc045601640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc030000c00 tx=0x7fc03c0329a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95d714945fdd504 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc0297fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.642+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030019040 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc030009040 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc03c034cd0 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc0300261f0 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc04ac48740 10 client.18177 fetch_fsmap learned FSMap version 35 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5624279c1c60 con 0x562427927a50 2024-03-22T02:22:51.647 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.643+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc03c034cd0 con 0x562427927a50 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fc030032070 con 0x562427927a50 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 10 client.18177 fetch_fsmap finished waiting for FSMap version 35 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 10 client.18177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 20 client.18177 populate_metadata read hostname 'smithi028' 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242792ffb0 0x5624279c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 4 client.18177 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624278988d0 con 0x56242792ffb0 2024-03-22T02:22:51.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc045e02640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242792ffb0 0x5624279c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:22:51.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc0297fa640 10 client.18177 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:51.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.644+0000 7fc045e02640 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242792ffb0 0x5624279c46d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea2467284f5a7bb server_cookie=25a671978e3e0636 in_seq=0 out_seq=0 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.911+0000 7fc0297fa640 1 -- 172.21.15.28:0/2603048447 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5624278988d0 con 0x56242792ffb0 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.911+0000 7fc0297fa640 10 client.18177 handle_command_reply: tid=0 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.911+0000 7fc04ac48740 1 client.18177 shutdown 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 msgr2=0x562427943930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc030000c00 tx=0x7fc03c0329a0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 shutdown_connections 2024-03-22T02:22:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56242792ffb0 0x5624279c46d0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56242792ce50 0x562427944e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc0297fa640 0 client.18177 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56242792c460 0x562427943e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 --2- 172.21.15.28:0/2603048447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562427927a50 0x562427943930 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 >> 172.21.15.28:0/2603048447 conn(0x56242793e8c0 msgr2=0x562427938de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 shutdown_connections 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.912+0000 7fc04ac48740 1 -- 172.21.15.28:0/2603048447 wait complete. 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc04ac48740 20 client.18177 trim_cache size 0 max 16384 2024-03-22T02:22:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:22:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:22:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:22:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:22:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 msgr2=0x7fc02c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 0x7fc02c0a5480 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc0380977c0 tx=0x7fc038095d80 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 msgr2=0x7fc04019f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc03800d950 tx=0x7fc03800de10 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 shutdown_connections 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc02c0a2fc0 0x7fc02c0a5480 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0401078b0 0x7fc04019f420 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc040104f80 0x7fc0401983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 --2- 172.21.15.28:0/532347200 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc040102650 0x7fc040197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.913+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 >> 172.21.15.28:0/532347200 conn(0x7fc0400fc020 msgr2=0x7fc040102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.914+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 shutdown_connections 2024-03-22T02:22:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:22:51.914+0000 7fc048066640 1 -- 172.21.15.28:0/532347200 wait complete. 2024-03-22T02:22:52.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 912624 11.62 MB/sec execute 334 sec latency 177.813 ms 2024-03-22T02:22:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:52 smithi120 ceph-mon[36803]: pgmap v638: 129 pgs: 129 active+clean; 59 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 350 op/s 2024-03-22T02:22:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:52 smithi083 ceph-mon[36340]: pgmap v638: 129 pgs: 129 active+clean; 59 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 350 op/s 2024-03-22T02:22:53.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:52 smithi028 ceph-mon[31916]: pgmap v638: 129 pgs: 129 active+clean; 59 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 350 op/s 2024-03-22T02:22:53.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 914751 11.61 MB/sec execute 335 sec latency 30.110 ms 2024-03-22T02:22:54.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 916596 11.61 MB/sec execute 336 sec latency 33.552 ms 2024-03-22T02:22:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:54 smithi120 ceph-mon[36803]: pgmap v639: 129 pgs: 129 active+clean; 62 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-03-22T02:22:55.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:54 smithi083 ceph-mon[36340]: pgmap v639: 129 pgs: 129 active+clean; 62 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-03-22T02:22:55.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:54 smithi028 ceph-mon[31916]: pgmap v639: 129 pgs: 129 active+clean; 62 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-03-22T02:22:55.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 917787 11.64 MB/sec execute 337 sec latency 111.716 ms 2024-03-22T02:22:56.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 919963 11.65 MB/sec execute 338 sec latency 227.092 ms 2024-03-22T02:22:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:56 smithi120 ceph-mon[36803]: pgmap v640: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 454 op/s 2024-03-22T02:22:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:56 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:22:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:56 smithi083 ceph-mon[36340]: pgmap v640: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 454 op/s 2024-03-22T02:22:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:56 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:22:57.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:56 smithi028 ceph-mon[31916]: pgmap v640: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 454 op/s 2024-03-22T02:22:57.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:56 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:22:57.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 919963 11.61 MB/sec execute 339 sec latency 1227.181 ms 2024-03-22T02:22:58.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 919963 11.58 MB/sec execute 340 sec latency 2227.237 ms 2024-03-22T02:22:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:22:58 smithi120 ceph-mon[36803]: pgmap v641: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 361 op/s 2024-03-22T02:22:59.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:22:58 smithi083 ceph-mon[36340]: pgmap v641: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 361 op/s 2024-03-22T02:22:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:22:58 smithi028 ceph-mon[31916]: pgmap v641: 129 pgs: 129 active+clean; 55 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 361 op/s 2024-03-22T02:22:59.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 919963 11.54 MB/sec execute 341 sec latency 3227.330 ms 2024-03-22T02:23:00.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 920040 11.51 MB/sec execute 342 sec latency 4175.335 ms 2024-03-22T02:23:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:00 smithi120 ceph-mon[36803]: pgmap v642: 129 pgs: 129 active+clean; 45 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-03-22T02:23:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:00 smithi083 ceph-mon[36340]: pgmap v642: 129 pgs: 129 active+clean; 45 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-03-22T02:23:01.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:00 smithi028 ceph-mon[31916]: pgmap v642: 129 pgs: 129 active+clean; 45 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-03-22T02:23:01.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 921427 11.50 MB/sec execute 343 sec latency 33.477 ms 2024-03-22T02:23:02.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 924060 11.51 MB/sec execute 344 sec latency 86.563 ms 2024-03-22T02:23:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:02 smithi120 ceph-mon[36803]: pgmap v643: 129 pgs: 129 active+clean; 41 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-03-22T02:23:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:02 smithi083 ceph-mon[36340]: pgmap v643: 129 pgs: 129 active+clean; 41 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-03-22T02:23:03.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:02 smithi028 ceph-mon[31916]: pgmap v643: 129 pgs: 129 active+clean; 41 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-03-22T02:23:03.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 926666 11.51 MB/sec execute 345 sec latency 28.999 ms 2024-03-22T02:23:04.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 928515 11.51 MB/sec execute 346 sec latency 43.761 ms 2024-03-22T02:23:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:04 smithi120 ceph-mon[36803]: pgmap v644: 129 pgs: 129 active+clean; 40 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 324 op/s 2024-03-22T02:23:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.005+0000 7f73963bc640 1 -- 172.21.15.28:0/3291506726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390101bc0 msgr2=0x7f7390101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.005+0000 7f73963bc640 1 --2- 172.21.15.28:0/3291506726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390101bc0 0x7f7390101fe0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f7378009900 tx=0x7f737802bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- 172.21.15.28:0/3291506726 shutdown_connections 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- 172.21.15.28:0/3291506726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 0x7f739010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- 172.21.15.28:0/3291506726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390102db0 0x7f7390103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- 172.21.15.28:0/3291506726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390101bc0 0x7f7390101fe0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- 172.21.15.28:0/3291506726 >> 172.21.15.28:0/3291506726 conn(0x7f73900fd150 msgr2=0x7f73900ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- 172.21.15.28:0/3291506726 shutdown_connections 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- 172.21.15.28:0/3291506726 wait complete. 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 Processor -- start 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- start start 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 0x7f7390197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 0x7f739019f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f739010d810 con 0x7f7390102db0 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f739010d690 con 0x7f7390103770 2024-03-22T02:23:05.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.006+0000 7f73963bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f739010d990 con 0x7f7390101bc0 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 0x7f7390197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f7394959640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 0x7f739019f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52094/0 (socket says 172.21.15.28:52094) 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 0x7f7390197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33812/0 (socket says 172.21.15.28:33812) 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 -- 172.21.15.28:0/2017631727 learned_addr learned my addr 172.21.15.28:0/2017631727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 -- 172.21.15.28:0/2017631727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 msgr2=0x7f7390197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 0x7f7390197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 -- 172.21.15.28:0/2017631727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 msgr2=0x7f739019f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 0x7f739019f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 -- 172.21.15.28:0/2017631727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739019f9f0 con 0x7f7390102db0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738f7fe640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f738000e9f0 tx=0x7f738000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738d7fa640 1 -- 172.21.15.28:0/2017631727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738000cd80 con 0x7f7390102db0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738d7fa640 1 -- 172.21.15.28:0/2017631727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7380010910 con 0x7f7390102db0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f738d7fa640 1 -- 172.21.15.28:0/2017631727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7380018990 con 0x7f7390102db0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739019fce0 con 0x7f7390102db0 2024-03-22T02:23:05.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.007+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73901a01d0 con 0x7f7390102db0 2024-03-22T02:23:05.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:04 smithi083 ceph-mon[36340]: pgmap v644: 129 pgs: 129 active+clean; 40 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 324 op/s 2024-03-22T02:23:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738d7fa640 1 -- 172.21.15.28:0/2017631727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7380010ac0 con 0x7f7390102db0 2024-03-22T02:23:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 Processor -- start 2024-03-22T02:23:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 -- start start 2024-03-22T02:23:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72cb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738d7fa640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 0x7f737c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 0x560cd72d34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cd7230040 con 0x560cd72cb4e0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560cd71d89e0 con 0x560cd72cbed0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7398fa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560cd722bff0 con 0x560cd72d1040 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738f7fe640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 0x7f737c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41854/0 (socket says 172.21.15.28:41854) 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738ffff640 1 -- 172.21.15.28:0/27398760 learned_addr learned my addr 172.21.15.28:0/27398760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f738d7fa640 1 -- 172.21.15.28:0/2017631727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f73800cfaf0 con 0x7f7390102db0 2024-03-22T02:23:05.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.010+0000 7f7394959640 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 0x560cd72d34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 -- 172.21.15.28:0/27398760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 msgr2=0x560cd72d34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 0x560cd72d34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 -- 172.21.15.28:0/27398760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 msgr2=0x560cd72cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 -- 172.21.15.28:0/27398760 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd6f44210 con 0x560cd72cbed0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f7394959640 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 0x560cd72d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738f7fe640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 0x7f737c0a5480 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7380098890 tx=0x7f7380018420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f738ffff640 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7378009900 tx=0x7f7378009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99cbbd7c36dde4ca server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f73867fc640 1 -- 172.21.15.28:0/27398760 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f737802cb80 con 0x560cd72cbed0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f73867fc640 1 -- 172.21.15.28:0/27398760 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7378005040 con 0x560cd72cbed0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.011+0000 7f73867fc640 1 -- 172.21.15.28:0/27398760 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378004920 con 0x560cd72cbed0 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- 172.21.15.28:0/27398760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 msgr2=0x560cd72d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7378009900 tx=0x7f7378009560 comp rx=0 tx=0).stop 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- 172.21.15.28:0/27398760 shutdown_connections 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72d1040 0x560cd72d34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cbed0 0x560cd72d09d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 --2- 172.21.15.28:0/27398760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- 172.21.15.28:0/27398760 >> 172.21.15.28:0/27398760 conn(0x560cd72c6a50 msgr2=0x560cd72c6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- 172.21.15.28:0/27398760 shutdown_connections 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- 172.21.15.28:0/27398760 wait complete. 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 -- start start 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.012+0000 7f7398fa3740 1 Processor -- start 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72d5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560cd72d3f90 con 0x560cd72d1040 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560cd71ebdb0 con 0x560cd72cb4e0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560cd72d4110 con 0x560cd72cbed0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7398fa3740 10 client.0 _ll_register_callbacks cb 0x560cd6f43a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7394959640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72d5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33822/0 (socket says 172.21.15.28:33822) 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52108/0 (socket says 172.21.15.28:52108) 2024-03-22T02:23:05.022 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 -- 172.21.15.28:0/3938027874 learned_addr learned my addr 172.21.15.28:0/3938027874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 -- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 msgr2=0x560cd72d5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72d5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 -- 172.21.15.28:0/3938027874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 msgr2=0x560cd72dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 -- 172.21.15.28:0/3938027874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd7010620 con 0x560cd72cbed0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f7394959640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f738ffff640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f737802cfd0 tx=0x7f73780054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7937d9dc9ea2d578 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f736f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.013+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7378004ad0 con 0x560cd72cbed0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7378005040 con 0x560cd72cbed0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f7378004c70 con 0x560cd72cbed0 2024-03-22T02:23:05.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7378004e70 con 0x560cd72cbed0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 10 client.28009 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560cd72d3fe0 con 0x560cd72cbed0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7378005040 con 0x560cd72cbed0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f7378045420 con 0x560cd72cbed0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 10 client.28009 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 10 client.28009 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 20 client.28009 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd72e10e0 0x560cd72e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 4 client.28009 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cd723d580 con 0x560cd72e10e0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.014+0000 7f7394959640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd72e10e0 0x560cd72e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.015+0000 7f7394959640 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd72e10e0 0x560cd72e34d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=816ca48a819cbfd6 server_cookie=fedf1c3d57f579d9 in_seq=0 out_seq=0 2024-03-22T02:23:05.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.015+0000 7f736f7fe640 10 client.28009 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f736f7fe640 1 -- 172.21.15.28:0/3938027874 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560cd723d580 con 0x560cd72e10e0 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f736f7fe640 10 client.28009 handle_command_reply: tid=0 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 client.28009 shutdown 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 msgr2=0x560cd72d59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f737802cfd0 tx=0x7f73780054a0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 shutdown_connections 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd72e10e0 0x560cd72e34d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72d59d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3938027874 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72d5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f736f7fe640 0 client.28009 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:05.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 >> 172.21.15.28:0/3938027874 conn(0x560cd72c6a50 msgr2=0x560cd72c5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 shutdown_connections 2024-03-22T02:23:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 1 -- 172.21.15.28:0/3938027874 wait complete. 2024-03-22T02:23:05.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.016+0000 7f7398fa3740 20 client.28009 trim_cache size 0 max 16384 2024-03-22T02:23:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 Processor -- start 2024-03-22T02:23:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 -- start start 2024-03-22T02:23:05.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72dc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72d1040 0x560cd72c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73901ad130 con 0x560cd72cb4e0 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f73901aaa00 con 0x560cd72d1040 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7398fa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73901a8850 con 0x560cd72cbed0 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52114/0 (socket says 172.21.15.28:52114) 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 -- 172.21.15.28:0/3177261870 learned_addr learned my addr 172.21.15.28:0/3177261870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f738ffff640 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72dc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72d1040 0x560cd72c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 -- 172.21.15.28:0/3177261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 msgr2=0x560cd72dc9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72dc9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 -- 172.21.15.28:0/3177261870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72d1040 msgr2=0x560cd72c53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72d1040 0x560cd72c53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.035+0000 7f7394959640 1 -- 172.21.15.28:0/3177261870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd72e3ab0 con 0x560cd72cb4e0 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.036+0000 7f7394959640 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7388015010 tx=0x7f7388009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0c74448f194b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.036+0000 7f7385ffb640 1 -- 172.21.15.28:0/3177261870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388026620 con 0x560cd72cb4e0 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.036+0000 7f7385ffb640 1 -- 172.21.15.28:0/3177261870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73880267c0 con 0x560cd72cb4e0 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.036+0000 7f7385ffb640 1 -- 172.21.15.28:0/3177261870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388005880 con 0x560cd72cb4e0 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- 172.21.15.28:0/3177261870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 msgr2=0x560cd72d04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7388015010 tx=0x7f7388009500 comp rx=0 tx=0).stop 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- 172.21.15.28:0/3177261870 shutdown_connections 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72d1040 0x560cd72c53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72dc9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3177261870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72cb4e0 0x560cd72d04f0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- 172.21.15.28:0/3177261870 >> 172.21.15.28:0/3177261870 conn(0x560cd72c6a50 msgr2=0x560cd72ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- 172.21.15.28:0/3177261870 shutdown_connections 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- 172.21.15.28:0/3177261870 wait complete. 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- start start 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 Processor -- start 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72da520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72daa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73901a8ec0 con 0x560cd72d1040 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f73901aa330 con 0x560cd72cb4e0 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f73901a9590 con 0x560cd72cbed0 2024-03-22T02:23:05.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7398fa3740 10 client.0 _ll_register_callbacks cb 0x7f7388009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.037+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52122/0 (socket says 172.21.15.28:52122) 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 -- 172.21.15.28:0/3008343850 learned_addr learned my addr 172.21.15.28:0/3008343850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f738ffff640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72daa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72da520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 -- 172.21.15.28:0/3008343850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 msgr2=0x560cd72daa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72daa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 -- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 msgr2=0x560cd72da520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72da520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 -- 172.21.15.28:0/3008343850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd72dc060 con 0x560cd72d1040 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f738ffff640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72daa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:05.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7394959640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7388013640 tx=0x7f7388000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1fd2584fd63b1cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f736f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388003970 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f7388003b10 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73880267c0 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7388041050 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7398fa3740 10 client.18195 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.038+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560cd7347790 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73880267c0 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f73880226b0 con 0x560cd72d1040 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 10 client.18195 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 10 client.18195 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 20 client.18195 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:05.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd7348100 0x560cd734a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 4 client.18195 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:23:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560cd72378d0 con 0x560cd7348100 2024-03-22T02:23:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f738ffff640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd7348100 0x560cd734a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f738ffff640 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd7348100 0x560cd734a520 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311c6129b7ed082b server_cookie=c696091eae0c9a81 in_seq=0 out_seq=0 2024-03-22T02:23:05.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.039+0000 7f736f7fe640 10 client.18195 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:05.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:04 smithi028 ceph-mon[31916]: pgmap v644: 129 pgs: 129 active+clean; 40 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.4 MiB/s wr, 324 op/s 2024-03-22T02:23:05.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 930702 11.51 MB/sec execute 347 sec latency 259.788 ms 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.474+0000 7f736f7fe640 1 -- 172.21.15.28:0/3008343850 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560cd72378d0 con 0x560cd7348100 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f736f7fe640 10 client.18195 handle_command_reply: tid=0 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 client.18195 shutdown 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 msgr2=0x560cd72dba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7388013640 tx=0x7f7388000f40 comp rx=0 tx=0).stop 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 shutdown_connections 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560cd7348100 0x560cd734a520 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560cd72d1040 0x560cd72dba70 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560cd72cbed0 0x560cd72daa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 --2- 172.21.15.28:0/3008343850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560cd72cb4e0 0x560cd72da520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 >> 172.21.15.28:0/3008343850 conn(0x560cd72c6a50 msgr2=0x560cd72ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f736f7fe640 0 client.18195 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 shutdown_connections 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 1 -- 172.21.15.28:0/3008343850 wait complete. 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.475+0000 7f7398fa3740 20 client.18195 trim_cache size 0 max 16384 2024-03-22T02:23:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:23:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:23:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:23:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 msgr2=0x7f737c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 0x7f737c0a5480 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7380098890 tx=0x7f7380018420 comp rx=0 tx=0).stop 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 msgr2=0x7f7390198310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f738000e9f0 tx=0x7f738000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 shutdown_connections 2024-03-22T02:23:05.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f737c0a2fc0 0x7f737c0a5480 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7390103770 0x7f739019f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7390102db0 0x7f7390198310 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 --2- 172.21.15.28:0/2017631727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7390101bc0 0x7f7390197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 >> 172.21.15.28:0/2017631727 conn(0x7f73900fd150 msgr2=0x7f7390106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 shutdown_connections 2024-03-22T02:23:05.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:05.476+0000 7f73963bc640 1 -- 172.21.15.28:0/2017631727 wait complete. 2024-03-22T02:23:06.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 931857 11.49 MB/sec execute 348 sec latency 48.371 ms 2024-03-22T02:23:06.708 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:23:06.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- 172.21.15.28:0/3499724139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718102640 msgr2=0x7fd718108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 --2- 172.21.15.28:0/3499724139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718102640 0x7fd718108ed0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fd710009ca0 tx=0x7fd71002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- 172.21.15.28:0/3499724139 shutdown_connections 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 --2- 172.21.15.28:0/3499724139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718102640 0x7fd718108ed0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 --2- 172.21.15.28:0/3499724139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 0x7fd718102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 --2- 172.21.15.28:0/3499724139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718100a90 0x7fd718100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- 172.21.15.28:0/3499724139 >> 172.21.15.28:0/3499724139 conn(0x7fd7180fc020 msgr2=0x7fd7180fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- 172.21.15.28:0/3499724139 shutdown_connections 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- 172.21.15.28:0/3499724139 wait complete. 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 Processor -- start 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.816+0000 7fd71eac3640 1 -- start start 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 0x7fd71819cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718102640 0x7fd71819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd71810c6f0 con 0x7fd718100a90 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd71810c570 con 0x7fd718102640 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd71810c870 con 0x7fd718101c80 2024-03-22T02:23:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52152/0 (socket says 172.21.15.28:52152) 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 -- 172.21.15.28:0/2810520136 learned_addr learned my addr 172.21.15.28:0/2810520136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71d060640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718102640 0x7fd71819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd717fff640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 0x7fd71819cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 -- 172.21.15.28:0/2810520136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 msgr2=0x7fd71819cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 0x7fd71819cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 -- 172.21.15.28:0/2810520136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718102640 msgr2=0x7fd71819f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718102640 0x7fd71819f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 -- 172.21.15.28:0/2810520136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71819fab0 con 0x7fd718100a90 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9c0 tx=0x7fd70c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd715ffb640 1 -- 172.21.15.28:0/2810520136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c00cd80 con 0x7fd718100a90 2024-03-22T02:23:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd715ffb640 1 -- 172.21.15.28:0/2810520136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0035f0 con 0x7fd718100a90 2024-03-22T02:23:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71819fda0 con 0x7fd718100a90 2024-03-22T02:23:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.817+0000 7fd715ffb640 1 -- 172.21.15.28:0/2810520136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c010690 con 0x7fd718100a90 2024-03-22T02:23:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.818+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7181a0210 con 0x7fd718100a90 2024-03-22T02:23:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.819+0000 7fd715ffb640 1 -- 172.21.15.28:0/2810520136 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd70c0037a0 con 0x7fd718100a90 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd715ffb640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 0x7fd7080a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 Processor -- start 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 -- start start 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd717fff640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 0x7fd7080a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd715ffb640 1 -- 172.21.15.28:0/2810520136 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd70c0cfd60 con 0x7fd718100a90 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9d9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cacd9398c0 con 0x55cacd9d9f90 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cacd8e1c20 con 0x55cacd9d4430 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd721787740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cacd935870 con 0x55cacd9d4e20 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd71d060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd717fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9d9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41898/0 (socket says 172.21.15.28:41898) 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd71d060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52156/0 (socket says 172.21.15.28:52156) 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.820+0000 7fd71c85f640 1 -- 172.21.15.28:0/1964989076 learned_addr learned my addr 172.21.15.28:0/1964989076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 -- 172.21.15.28:0/1964989076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 msgr2=0x55cacd9d9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9d9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 -- 172.21.15.28:0/1964989076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 msgr2=0x55cacd9dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 -- 172.21.15.28:0/1964989076 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd906150 con 0x55cacd9d4430 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71d060640 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd717fff640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 0x7fd7080a53e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd718069630 tx=0x7fd7040158a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0a4500 tx=0x7fd70c09ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a69dd7e0ef1b4223 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd702ffd640 1 -- 172.21.15.28:0/1964989076 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0a0170 con 0x55cacd9d4430 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd702ffd640 1 -- 172.21.15.28:0/1964989076 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0a4a30 con 0x55cacd9d4430 2024-03-22T02:23:06.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.821+0000 7fd702ffd640 1 -- 172.21.15.28:0/1964989076 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c098690 con 0x55cacd9d4430 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- 172.21.15.28:0/1964989076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 msgr2=0x55cacd9d4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0a4500 tx=0x7fd70c09ba00 comp rx=0 tx=0).stop 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- 172.21.15.28:0/1964989076 shutdown_connections 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9d9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 --2- 172.21.15.28:0/1964989076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9d4850 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- 172.21.15.28:0/1964989076 >> 172.21.15.28:0/1964989076 conn(0x55cacd9cf9a0 msgr2=0x55cacd9cfda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- 172.21.15.28:0/1964989076 shutdown_connections 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- 172.21.15.28:0/1964989076 wait complete. 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 -- start start 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.822+0000 7fd721787740 1 Processor -- start 2024-03-22T02:23:06.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9de8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d9f90 0x55cacd9e5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cacd9dcee0 con 0x55cacd9d4430 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cacd8f5130 con 0x55cacd9d4e20 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cacd9dd060 con 0x55cacd9d9f90 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd721787740 10 client.0 _ll_register_callbacks cb 0x55cacd8f4b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd717fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9de8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd717fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41910/0 (socket says 172.21.15.28:41910) 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd717fff640 1 -- 172.21.15.28:0/2939921456 learned_addr learned my addr 172.21.15.28:0/2939921456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71d060640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d9f90 0x55cacd9e5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 -- 172.21.15.28:0/2939921456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d9f90 msgr2=0x55cacd9e5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d9f90 0x55cacd9e5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 -- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 msgr2=0x55cacd9dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 -- 172.21.15.28:0/2939921456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacd7177d0 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd717fff640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd71c85f640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9de8d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd70c09a570 tx=0x7fd70c003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2872a9d65e81ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c066060 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.823+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd70c09b470 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70c0a4a30 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd70c0ba070 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 10 client.18207 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cacd64d3f0 con 0x55cacd9d4430 2024-03-22T02:23:06.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0a4a30 con 0x55cacd9d4430 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fd70c09b110 con 0x55cacd9d4430 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 10 client.18207 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 10 client.18207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 20 client.18207 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacd9ea340 0x55cacd9ec730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 4 client.18207 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cacd946e00 con 0x55cacd9ea340 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.824+0000 7fd717fff640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacd9ea340 0x55cacd9ec730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.825+0000 7fd717fff640 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacd9ea340 0x55cacd9ec730 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f81b8410cc8cb08c server_cookie=d1bf96982b74b793 in_seq=0 out_seq=0 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.825+0000 7fd6f7fff640 10 client.18207 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.825+0000 7fd6f7fff640 1 -- 172.21.15.28:0/2939921456 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cacd946e00 con 0x55cacd9ea340 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.825+0000 7fd6f7fff640 10 client.18207 handle_command_reply: tid=0 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 client.18207 shutdown 2024-03-22T02:23:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 msgr2=0x55cacd9de8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9de8d0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fd70c09a570 tx=0x7fd70c003d10 comp rx=0 tx=0).stop 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 shutdown_connections 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacd9ea340 0x55cacd9ec730 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d9f90 0x55cacd9e5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4e20 0x55cacd9dee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd6f7fff640 0 client.18207 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 --2- 172.21.15.28:0/2939921456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9de8d0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.826+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 >> 172.21.15.28:0/2939921456 conn(0x55cacd9cf9a0 msgr2=0x55cacd9cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.827+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 shutdown_connections 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.827+0000 7fd721787740 1 -- 172.21.15.28:0/2939921456 wait complete. 2024-03-22T02:23:06.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.827+0000 7fd721787740 20 client.18207 trim_cache size 0 max 16384 2024-03-22T02:23:06.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.846+0000 7fd721787740 1 Processor -- start 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.846+0000 7fd721787740 1 -- start start 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.846+0000 7fd721787740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.846+0000 7fd721787740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd721787740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d9f90 0x55cacd9ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd721787740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181ad470 con 0x55cacd9d4430 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd721787740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181aad40 con 0x55cacd9d9f90 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd721787740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181a8b90 con 0x55cacd9d4e20 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52176/0 (socket says 172.21.15.28:52176) 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 -- 172.21.15.28:0/1406368515 learned_addr learned my addr 172.21.15.28:0/1406368515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71d060640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d9f90 0x55cacd9ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd717fff640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 -- 172.21.15.28:0/1406368515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 msgr2=0x55cacd9e5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 -- 172.21.15.28:0/1406368515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d9f90 msgr2=0x55cacd9ce8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d9f90 0x55cacd9ce8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 -- 172.21.15.28:0/1406368515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cacd60d0f0 con 0x55cacd9d4430 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd717fff640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd71c85f640 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0989f0 tx=0x7fd70c09bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68214f29255d3d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd7027fc640 1 -- 172.21.15.28:0/1406368515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c0a4a30 con 0x55cacd9d4430 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd7027fc640 1 -- 172.21.15.28:0/1406368515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0a0170 con 0x55cacd9d4430 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.847+0000 7fd7027fc640 1 -- 172.21.15.28:0/1406368515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c09b6d0 con 0x55cacd9d4430 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- 172.21.15.28:0/1406368515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 msgr2=0x55cacd9ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fd70c0989f0 tx=0x7fd70c09bf20 comp rx=0 tx=0).stop 2024-03-22T02:23:06.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- 172.21.15.28:0/1406368515 shutdown_connections 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d9f90 0x55cacd9ce8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 --2- 172.21.15.28:0/1406368515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d4430 0x55cacd9ce370 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- 172.21.15.28:0/1406368515 >> 172.21.15.28:0/1406368515 conn(0x55cacd9cf9a0 msgr2=0x55cacd9de4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- 172.21.15.28:0/1406368515 shutdown_connections 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- 172.21.15.28:0/1406368515 wait complete. 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 -- start start 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.848+0000 7fd721787740 1 Processor -- start 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9e3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181a9200 con 0x55cacd9d9f90 2024-03-22T02:23:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181aa670 con 0x55cacd9d4430 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7181a98d0 con 0x55cacd9d4e20 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd721787740 10 client.0 _ll_register_callbacks cb 0x7fd70c09c7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71d060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9e3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71d060640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52178/0 (socket says 172.21.15.28:52178) 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71d060640 1 -- 172.21.15.28:0/3414213125 learned_addr learned my addr 172.21.15.28:0/3414213125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd717fff640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 -- 172.21.15.28:0/3414213125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 msgr2=0x55cacd9e4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 -- 172.21.15.28:0/3414213125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 msgr2=0x55cacd9e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 -- 172.21.15.28:0/3414213125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cacd9e58e0 con 0x55cacd9d4430 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71d060640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:06.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd71c85f640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9e3df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55cacd9cf7e0 tx=0x7fd70c0983f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2284383022f97494 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.849+0000 7fd6f7fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd70c09b670 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd70c0a0170 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd70c097d70 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd70c0a4a30 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 10 client.28034 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55cacd9e58e0 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd70c0a0170 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fd70c0a1300 con 0x55cacd9d4430 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 10 client.28034 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 10 client.28034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 20 client.28034 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacda510d0 0x55cacda534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 4 client.28034 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:23:06.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cacd941150 con 0x55cacda510d0 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.850+0000 7fd717fff640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacda510d0 0x55cacda534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd6f7fff640 10 client.28034 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd717fff640 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacda510d0 0x55cacda534f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8d4a8cf5fde58f9 server_cookie=ed4fd32af225d0fd in_seq=0 out_seq=0 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd6f7fff640 1 -- 172.21.15.28:0/3414213125 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cacd941150 con 0x55cacda510d0 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd6f7fff640 10 client.28034 handle_command_reply: tid=0 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 client.28034 shutdown 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 msgr2=0x55cacd9e3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9e3df0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55cacd9cf7e0 tx=0x7fd70c0983f0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 shutdown_connections 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cacda510d0 0x55cacda534f0 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cacd9d9f90 0x55cacd9e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cacd9d4e20 0x55cacd9e4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 --2- 172.21.15.28:0/3414213125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cacd9d4430 0x55cacd9e3df0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd6f7fff640 0 client.28034 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 >> 172.21.15.28:0/3414213125 conn(0x55cacd9cf9a0 msgr2=0x55cacd9d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 shutdown_connections 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 1 -- 172.21.15.28:0/3414213125 wait complete. 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.851+0000 7fd721787740 20 client.28034 trim_cache size 0 max 16384 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:23:06.859 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:06.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 msgr2=0x7fd7080a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 0x7fd7080a53e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd718069630 tx=0x7fd7040158a0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 msgr2=0x7fd7180698c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fd70c00e9c0 tx=0x7fd70c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 shutdown_connections 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7080a2f20 0x7fd7080a53e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd718102640 0x7fd71819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd718101c80 0x7fd71819cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 --2- 172.21.15.28:0/2810520136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd718100a90 0x7fd7180698c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 >> 172.21.15.28:0/2810520136 conn(0x7fd7180fc020 msgr2=0x7fd718104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 shutdown_connections 2024-03-22T02:23:06.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.852+0000 7fd71eac3640 1 -- 172.21.15.28:0/2810520136 wait complete. 2024-03-22T02:23:06.862 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:06 smithi028 ceph-mon[31916]: pgmap v645: 129 pgs: 129 active+clean; 43 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2024-03-22T02:23:06.863 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:23:06.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:23:06.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- 172.21.15.28:0/3660232727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24100ac0 msgr2=0x7f6b24100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/3660232727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24100ac0 0x7f6b24100ee0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009900 tx=0x7f6b1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- 172.21.15.28:0/3660232727 shutdown_connections 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/3660232727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24102670 0x7f6b24108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/3660232727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 0x7f6b24102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/3660232727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24100ac0 0x7f6b24100ee0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- 172.21.15.28:0/3660232727 >> 172.21.15.28:0/3660232727 conn(0x7f6b240fc070 msgr2=0x7f6b240fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- 172.21.15.28:0/3660232727 shutdown_connections 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- 172.21.15.28:0/3660232727 wait complete. 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 Processor -- start 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.968+0000 7f6b2c35e640 1 -- start start 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24100ac0 0x7f6b24197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 0x7f6b24198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410c740 con 0x7f6b24102670 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410c5c0 con 0x7f6b24101cb0 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2410c8c0 con 0x7f6b24100ac0 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52210/0 (socket says 172.21.15.28:52210) 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 -- 172.21.15.28:0/1512693588 learned_addr learned my addr 172.21.15.28:0/1512693588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b298f9640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 0x7f6b24198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a0fa640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24100ac0 0x7f6b24197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 -- 172.21.15.28:0/1512693588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24100ac0 msgr2=0x7f6b24197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24100ac0 0x7f6b24197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 -- 172.21.15.28:0/1512693588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 msgr2=0x7f6b24198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:06.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 0x7f6b24198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 -- 172.21.15.28:0/1512693588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b2419fb20 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2a8fb640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00eb10 tx=0x7f6b1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c004b20 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c004cc0 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2419fe10 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c010760 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.969+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b241a0320 con 0x7f6b24102670 2024-03-22T02:23:06.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.970+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b241a05e0 con 0x7f6b24102670 2024-03-22T02:23:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.972+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6b1c005000 con 0x7f6b24102670 2024-03-22T02:23:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.972+0000 7f6b1b7fe640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 0x7f6b100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.972+0000 7f6b2a0fa640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 0x7f6b100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.972+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6b1c0d0180 con 0x7f6b24102670 2024-03-22T02:23:06.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.973+0000 7f6b2a0fa640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 0x7f6b100a5460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b14005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:06.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:06.976+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b1c09cba0 con 0x7f6b24102670 2024-03-22T02:23:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:06 smithi120 ceph-mon[36803]: pgmap v645: 129 pgs: 129 active+clean; 43 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2024-03-22T02:23:07.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:06 smithi083 ceph-mon[36340]: pgmap v645: 129 pgs: 129 active+clean; 43 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 320 op/s 2024-03-22T02:23:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.203+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b241a07b0 con 0x7f6b24102670 2024-03-22T02:23:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.204+0000 7f6b1b7fe640 1 -- 172.21.15.28:0/1512693588 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11867 (secure 0 0 0) 0x7f6b1c09c4a0 con 0x7f6b24102670 2024-03-22T02:23:07.205 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:23:07.205 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:22:55.371454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:23:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-03-22T02:23:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 msgr2=0x7f6b100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 0x7f6b100a5460 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6b14000c00 tx=0x7f6b14005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 msgr2=0x7f6b2419f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00eb10 tx=0x7f6b1c00c2d0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 shutdown_connections 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6b100a2fa0 0x7f6b100a5460 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b24102670 0x7f6b2419f4b0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6b24101cb0 0x7f6b24198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 --2- 172.21.15.28:0/1512693588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b24100ac0 0x7f6b24197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 >> 172.21.15.28:0/1512693588 conn(0x7f6b240fc070 msgr2=0x7f6b240fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 shutdown_connections 2024-03-22T02:23:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.206+0000 7f6b2c35e640 1 -- 172.21.15.28:0/1512693588 wait complete. 2024-03-22T02:23:07.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 -- 172.21.15.28:0/386756146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4102040 msgr2=0x7f4ff4102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 --2- 172.21.15.28:0/386756146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4102040 0x7f4ff4102460 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4009900 tx=0x7f4fe402bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 -- 172.21.15.28:0/386756146 shutdown_connections 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 --2- 172.21.15.28:0/386756146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4101730 0x7f4ff4108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 --2- 172.21.15.28:0/386756146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4100d70 0x7f4ff41011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 --2- 172.21.15.28:0/386756146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4102040 0x7f4ff4102460 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 -- 172.21.15.28:0/386756146 >> 172.21.15.28:0/386756146 conn(0x7f4ff40fb7a0 msgr2=0x7f4ff40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 -- 172.21.15.28:0/386756146 shutdown_connections 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 -- 172.21.15.28:0/386756146 wait complete. 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.328+0000 7f4ffa881640 1 Processor -- start 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 -- start start 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4101730 0x7f4ff4198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4102040 0x7f4ff419f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff410c710 con 0x7f4ff4100d70 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff410c590 con 0x7f4ff4101730 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ffa881640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff410c890 con 0x7f4ff4102040 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52230/0 (socket says 172.21.15.28:52230) 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 -- 172.21.15.28:0/1665604175 learned_addr learned my addr 172.21.15.28:0/1665604175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4101730 0x7f4ff4198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4102040 0x7f4ff419f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 -- 172.21.15.28:0/1665604175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4102040 msgr2=0x7f4ff419f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4102040 0x7f4ff419f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 -- 172.21.15.28:0/1665604175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4101730 msgr2=0x7f4ff4198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4101730 0x7f4ff4198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 -- 172.21.15.28:0/1665604175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff419fb00 con 0x7f4ff4100d70 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.329+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4000c00 tx=0x7f4fe402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.330+0000 7f4ff17fa640 1 -- 172.21.15.28:0/1665604175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4041070 con 0x7f4ff4100d70 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.330+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff419fd90 con 0x7f4ff4100d70 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.330+0000 7f4ff17fa640 1 -- 172.21.15.28:0/1665604175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fe4004450 con 0x7f4ff4100d70 2024-03-22T02:23:07.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.330+0000 7f4ff17fa640 1 -- 172.21.15.28:0/1665604175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4034440 con 0x7f4ff4100d70 2024-03-22T02:23:07.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.330+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff41a01c0 con 0x7f4ff4100d70 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ff17fa640 1 -- 172.21.15.28:0/1665604175 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4fe4044070 con 0x7f4ff4100d70 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 Processor -- start 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 -- start start 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd2620 0x55a399dd7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 0x55a399dd9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a399d328d0 con 0x55a399dd2620 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ff17fa640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 0x7f4fe00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a399cdae00 con 0x55a399dd7790 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.332+0000 7f4ffd468740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a399d2e880 con 0x55a399dd1c30 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd2620 0x55a399dd7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 0x55a399dd9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33938/0 (socket says 172.21.15.28:33938) 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 -- 172.21.15.28:0/2421682091 learned_addr learned my addr 172.21.15.28:0/2421682091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 0x7f4fe00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff17fa640 1 -- 172.21.15.28:0/1665604175 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4fe40f1280 con 0x7f4ff4100d70 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2421682091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 msgr2=0x55a399dd2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2421682091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd2620 msgr2=0x55a399dd7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd2620 0x55a399dd7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2421682091 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a399a463f0 con 0x55a399dd7790 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 0x55a399dd9c20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009bf0 tx=0x7f4fdc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db1fa24e073645cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 0x7f4fe00a53e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55a399dd2e80 tx=0x7f4fec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4fea7fc640 1 -- 172.21.15.28:0/2421682091 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc004050 con 0x55a399dd7790 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4fea7fc640 1 -- 172.21.15.28:0/2421682091 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fdc005500 con 0x55a399dd7790 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.333+0000 7f4fea7fc640 1 -- 172.21.15.28:0/2421682091 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc004dd0 con 0x55a399dd7790 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- 172.21.15.28:0/2421682091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 msgr2=0x55a399dd9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 0x55a399dd9c20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009bf0 tx=0x7f4fdc011810 comp rx=0 tx=0).stop 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- 172.21.15.28:0/2421682091 shutdown_connections 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd7790 0x55a399dd9c20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd2620 0x55a399dd7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2421682091 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd1c30 0x55a399dd2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- 172.21.15.28:0/2421682091 >> 172.21.15.28:0/2421682091 conn(0x55a399dc8d90 msgr2=0x55a399dc9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- 172.21.15.28:0/2421682091 shutdown_connections 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- 172.21.15.28:0/2421682091 wait complete. 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 -- start start 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.334+0000 7f4ffd468740 1 Processor -- start 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 0x55a399ddc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a399dda740 con 0x55a399dd7790 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a399cee300 con 0x55a399dd1c30 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a399dda8c0 con 0x55a399dd2620 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff8e1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 0x55a399ddc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41996/0 (socket says 172.21.15.28:41996) 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff3fff640 1 -- 172.21.15.28:0/2293391323 learned_addr learned my addr 172.21.15.28:0/2293391323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ffd468740 10 client.0 _ll_register_callbacks cb 0x55a399d29e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52248/0 (socket says 172.21.15.28:52248) 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2293391323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 msgr2=0x55a399ddc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2293391323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 msgr2=0x55a399de3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 -- 172.21.15.28:0/2293391323 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a399b107d0 con 0x55a399dd2620 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 0x55a399ddc610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc01b040 tx=0x7f4fdc004520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7118521b230d490e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.335+0000 7f4fd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc029070 con 0x55a399dd2620 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fdc005500 con 0x55a399dd2620 2024-03-22T02:23:07.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4fdc004d50 con 0x55a399dd2620 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4fdc035050 con 0x55a399dd2620 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 10 client.28051 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a399de69f0 con 0x55a399dd2620 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fdc005500 con 0x55a399dd2620 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f4fdc003c40 con 0x55a399dd2620 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 10 client.28051 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 10 client.28051 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 20 client.28051 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399de7f70 0x55a399dea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 4 client.28051 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.336+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a399d3fe10 con 0x55a399de7f70 2024-03-22T02:23:07.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.337+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399de7f70 0x55a399dea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.337+0000 7f4fd37fe640 10 client.28051 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.337+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399de7f70 0x55a399dea3f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1f8e6e2301882123 server_cookie=9428f2d04819454a in_seq=0 out_seq=0 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4fd37fe640 1 -- 172.21.15.28:0/2293391323 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a399d3fe10 con 0x55a399de7f70 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4fd37fe640 10 client.28051 handle_command_reply: tid=0 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 client.28051 shutdown 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 msgr2=0x55a399ddc610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 0x55a399ddc610 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc01b040 tx=0x7f4fdc004520 comp rx=0 tx=0).stop 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 shutdown_connections 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd7790 0x55a399de3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd2620 0x55a399ddc610 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399de7f70 0x55a399dea3f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2293391323 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd1c30 0x55a399ddc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 >> 172.21.15.28:0/2293391323 conn(0x55a399dc8d90 msgr2=0x55a399dc7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4fd37fe640 0 client.28051 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:07.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 shutdown_connections 2024-03-22T02:23:07.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 1 -- 172.21.15.28:0/2293391323 wait complete. 2024-03-22T02:23:07.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.338+0000 7f4ffd468740 20 client.28051 trim_cache size 0 max 16384 2024-03-22T02:23:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.351+0000 7f4ffd468740 1 Processor -- start 2024-03-22T02:23:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.351+0000 7f4ffd468740 1 -- start start 2024-03-22T02:23:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399dc7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41ab0c0 con 0x55a399dd1c30 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41a8990 con 0x55a399dd2620 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ffd468740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41a67e0 con 0x55a399dd7790 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52254/0 (socket says 172.21.15.28:52254) 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 -- 172.21.15.28:0/2149795595 learned_addr learned my addr 172.21.15.28:0/2149795595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399dc7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 -- 172.21.15.28:0/2149795595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 msgr2=0x55a399dc7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399dc7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 -- 172.21.15.28:0/2149795595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 msgr2=0x55a399de3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 -- 172.21.15.28:0/2149795595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a399dc33e0 con 0x55a399dd1c30 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4003630 tx=0x7f4fe40881a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eece1f6981abd816 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4fe9ffb640 1 -- 172.21.15.28:0/2149795595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe404b040 con 0x55a399dd1c30 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4fe9ffb640 1 -- 172.21.15.28:0/2149795595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fe40f10b0 con 0x55a399dd1c30 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.352+0000 7f4fe9ffb640 1 -- 172.21.15.28:0/2149795595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40c16f0 con 0x55a399dd1c30 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- 172.21.15.28:0/2149795595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 msgr2=0x55a399ddac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4003630 tx=0x7f4fe40881a0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- 172.21.15.28:0/2149795595 shutdown_connections 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399dc7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 --2- 172.21.15.28:0/2149795595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399ddac60 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- 172.21.15.28:0/2149795595 >> 172.21.15.28:0/2149795595 conn(0x55a399dc8d90 msgr2=0x55a399dc7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- 172.21.15.28:0/2149795595 shutdown_connections 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- 172.21.15.28:0/2149795595 wait complete. 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.353+0000 7f4ffd468740 1 -- start start 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 Processor -- start 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399de2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41a6e50 con 0x55a399dd1c30 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41a82c0 con 0x55a399dd2620 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff41a7520 con 0x55a399dd7790 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 10 client.0 _ll_register_callbacks cb 0x7f4fe4003f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52260/0 (socket says 172.21.15.28:52260) 2024-03-22T02:23:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 -- 172.21.15.28:0/179242119 learned_addr learned my addr 172.21.15.28:0/179242119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff8e1e640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399de2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 -- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 msgr2=0x55a399de2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399de2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 -- 172.21.15.28:0/179242119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 msgr2=0x55a399de1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 -- 172.21.15.28:0/179242119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a399de31e0 con 0x55a399dd1c30 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ff3fff640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b9ab0 tx=0x7f4fe4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acb0f568f8be69ae server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4fd37fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe404b040 con 0x55a399dd1c30 2024-03-22T02:23:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fe40f10b0 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.354+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4fe40b9cc0 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4fe40bd190 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 10 client.18237 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a399e4ebd0 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fe40f10b0 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f4fe40bd8b0 con 0x55a399dd1c30 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 10 client.18237 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 10 client.18237 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 20 client.18237 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399e4f330 0x55a399e51750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 4 client.18237 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a399d3a160 con 0x55a399e4f330 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.355+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399e4f330 0x55a399e51750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ff37fe640 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399e4f330 0x55a399e51750 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b69a8bef5e4aca3a server_cookie=5f5b966e6e756fda in_seq=0 out_seq=0 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4fd37fe640 10 client.18237 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4fd37fe640 1 -- 172.21.15.28:0/179242119 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a399d3a160 con 0x55a399e4f330 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4fd37fe640 10 client.18237 handle_command_reply: tid=0 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 client.18237 shutdown 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 msgr2=0x55a399de16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b9ab0 tx=0x7f4fe4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 shutdown_connections 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a399dd7790 0x55a399de2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55a399e4f330 0x55a399e51750 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a399dd2620 0x55a399de1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 --2- 172.21.15.28:0/179242119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a399dd1c30 0x55a399de16f0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4fd37fe640 0 client.18237 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 >> 172.21.15.28:0/179242119 conn(0x55a399dc8d90 msgr2=0x55a399dcc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 shutdown_connections 2024-03-22T02:23:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 1 -- 172.21.15.28:0/179242119 wait complete. 2024-03-22T02:23:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.356+0000 7f4ffd468740 20 client.18237 trim_cache size 0 max 16384 2024-03-22T02:23:07.365 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 msgr2=0x7f4fe00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 0x7f4fe00a53e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55a399dd2e80 tx=0x7f4fec009500 comp rx=0 tx=0).stop 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 msgr2=0x7f4ff4197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4fe4000c00 tx=0x7f4fe402c650 comp rx=0 tx=0).stop 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 shutdown_connections 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4fe00a2f20 0x7f4fe00a53e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4ff4102040 0x7f4ff419f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4ff4101730 0x7f4ff4198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 --2- 172.21.15.28:0/1665604175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ff4100d70 0x7f4ff4197ee0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 >> 172.21.15.28:0/1665604175 conn(0x7f4ff40fb7a0 msgr2=0x7f4ff40fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 shutdown_connections 2024-03-22T02:23:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.357+0000 7f4ffa881640 1 -- 172.21.15.28:0/1665604175 wait complete. 2024-03-22T02:23:07.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:23:07.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 934666 11.51 MB/sec execute 349 sec latency 65.837 ms 2024-03-22T02:23:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 -- 172.21.15.28:0/1816766384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44104fa0 msgr2=0x7efc44107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 --2- 172.21.15.28:0/1816766384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44104fa0 0x7efc44107390 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7efc2c009900 tx=0x7efc2c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:07.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 -- 172.21.15.28:0/1816766384 shutdown_connections 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 --2- 172.21.15.28:0/1816766384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc441078d0 0x7efc44109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 --2- 172.21.15.28:0/1816766384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44104fa0 0x7efc44107390 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 --2- 172.21.15.28:0/1816766384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc44102670 0x7efc44104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 -- 172.21.15.28:0/1816766384 >> 172.21.15.28:0/1816766384 conn(0x7efc440fc020 msgr2=0x7efc440fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 -- 172.21.15.28:0/1816766384 shutdown_connections 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.474+0000 7efc48f1e640 1 -- 172.21.15.28:0/1816766384 wait complete. 2024-03-22T02:23:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 Processor -- start 2024-03-22T02:23:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 -- start start 2024-03-22T02:23:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc44104fa0 0x7efc44100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc441078d0 0x7efc44100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410d800 con 0x7efc44102670 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410d680 con 0x7efc44104fa0 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc48f1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efc4410d980 con 0x7efc441078d0 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52278/0 (socket says 172.21.15.28:52278) 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc4259c640 1 -- 172.21.15.28:0/3583311606 learned_addr learned my addr 172.21.15.28:0/3583311606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc42d9d640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc441078d0 0x7efc44100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.475+0000 7efc39d9b640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc44104fa0 0x7efc44100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 -- 172.21.15.28:0/3583311606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc441078d0 msgr2=0x7efc44100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc441078d0 0x7efc44100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 -- 172.21.15.28:0/3583311606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc44104fa0 msgr2=0x7efc44100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc44104fa0 0x7efc44100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 -- 172.21.15.28:0/3583311606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc441013f0 con 0x7efc44102670 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc4259c640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7efc3400e9f0 tx=0x7efc3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc3bfff640 1 -- 172.21.15.28:0/3583311606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3400cd80 con 0x7efc44102670 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc441a1fc0 con 0x7efc44102670 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc3bfff640 1 -- 172.21.15.28:0/3583311606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc34005500 con 0x7efc44102670 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc3bfff640 1 -- 172.21.15.28:0/3583311606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc34010690 con 0x7efc44102670 2024-03-22T02:23:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.476+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc441a23d0 con 0x7efc44102670 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 Processor -- start 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc3bfff640 1 -- 172.21.15.28:0/3583311606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7efc340046c0 con 0x7efc44102670 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 -- start start 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4047fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 0x557f40484d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40487810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f403e48d0 con 0x557f40480210 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f4038ce00 con 0x557f40485380 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.479+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f403e0880 con 0x557f4047f820 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 0x557f40484d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40487810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc4259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 0x557f40484d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52286/0 (socket says 172.21.15.28:52286) 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4047fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc4259c640 1 -- 172.21.15.28:0/3951180925 learned_addr learned my addr 172.21.15.28:0/3951180925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 -- 172.21.15.28:0/3951180925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 msgr2=0x557f40487810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40487810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 -- 172.21.15.28:0/3951180925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 msgr2=0x557f40484d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 0x557f40484d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 -- 172.21.15.28:0/3951180925 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f400f83f0 con 0x557f4047f820 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc39d9b640 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40487810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc3bfff640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 0x7efc300a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc42d9d640 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4047fc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009f00 tx=0x7efc3c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a268271c3e36e06 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc4259c640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 0x7efc300a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc3bfff640 1 -- 172.21.15.28:0/3583311606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efc340cfc40 con 0x7efc44102670 2024-03-22T02:23:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc38d99640 1 -- 172.21.15.28:0/3951180925 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c004d20 con 0x557f4047f820 2024-03-22T02:23:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc38d99640 1 -- 172.21.15.28:0/3951180925 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc3c004030 con 0x557f4047f820 2024-03-22T02:23:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.480+0000 7efc38d99640 1 -- 172.21.15.28:0/3951180925 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c005260 con 0x557f4047f820 2024-03-22T02:23:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- 172.21.15.28:0/3951180925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 msgr2=0x557f4047fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4259c640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 0x7efc300a53e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efc3400e9c0 tx=0x7efc340d3720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4047fc40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009f00 tx=0x7efc3c011810 comp rx=0 tx=0).stop 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- 172.21.15.28:0/3951180925 shutdown_connections 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40487810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40480210 0x557f40484d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/3951180925 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4047fc40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- 172.21.15.28:0/3951180925 >> 172.21.15.28:0/3951180925 conn(0x557f4047ad90 msgr2=0x557f4047b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- 172.21.15.28:0/3951180925 shutdown_connections 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- 172.21.15.28:0/3951180925 wait complete. 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.481+0000 7efc4bbe4740 1 -- start start 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 Processor -- start 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40485380 0x557f404912b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557f40488330 con 0x557f4047f820 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557f403a0300 con 0x557f40480210 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557f404884b0 con 0x557f40485380 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc4bbe4740 10 client.0 _ll_register_callbacks cb 0x557f403dbe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52302/0 (socket says 172.21.15.28:52302) 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 -- 172.21.15.28:0/1148264968 learned_addr learned my addr 172.21.15.28:0/1148264968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc39d9b640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40485380 0x557f404912b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 -- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40485380 msgr2=0x557f404912b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40485380 0x557f404912b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 -- 172.21.15.28:0/1148264968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 msgr2=0x557f4048a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 -- 172.21.15.28:0/1148264968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f401c27d0 con 0x557f4047f820 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009270 tx=0x7efc3c004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30dcfce6b54bc38c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.482+0000 7efc29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:07.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c02d060 con 0x557f4047f820 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efc3c005240 con 0x557f4047f820 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc3c004030 con 0x557f4047f820 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7efc3c005440 con 0x557f4047f820 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 10 client.18249 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557f40494620 con 0x557f4047f820 2024-03-22T02:23:07.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc3c004030 con 0x557f4047f820 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7efc3c018070 con 0x557f4047f820 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 10 client.18249 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 10 client.18249 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 20 client.18249 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f40495ad0 0x557f40497ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 4 client.18249 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f403f1e10 con 0x557f40495ad0 2024-03-22T02:23:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.483+0000 7efc39d9b640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f40495ad0 0x557f40497ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.484+0000 7efc29ffb640 10 client.18249 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.484+0000 7efc39d9b640 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f40495ad0 0x557f40497ec0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5764ca9b56d183b1 server_cookie=c6487d3a690c7e4a in_seq=0 out_seq=0 2024-03-22T02:23:07.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc29ffb640 1 -- 172.21.15.28:0/1148264968 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f403f1e10 con 0x557f40495ad0 2024-03-22T02:23:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc29ffb640 10 client.18249 handle_command_reply: tid=0 2024-03-22T02:23:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 client.18249 shutdown 2024-03-22T02:23:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 msgr2=0x557f40489d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7efc3c009270 tx=0x7efc3c004870 comp rx=0 tx=0).stop 2024-03-22T02:23:07.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 shutdown_connections 2024-03-22T02:23:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f40495ad0 0x557f40497ec0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40485380 0x557f404912b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1148264968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40489d30 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc29ffb640 0 client.18249 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 >> 172.21.15.28:0/1148264968 conn(0x557f4047ad90 msgr2=0x557f40479fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 shutdown_connections 2024-03-22T02:23:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1148264968 wait complete. 2024-03-22T02:23:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.485+0000 7efc4bbe4740 20 client.18249 trim_cache size 0 max 16384 2024-03-22T02:23:07.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 Processor -- start 2024-03-22T02:23:07.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 -- start start 2024-03-22T02:23:07.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 0x557f40491240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40479c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc441af4e0 con 0x557f4047f820 2024-03-22T02:23:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc441acdb0 con 0x557f40485380 2024-03-22T02:23:07.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efc441aac00 con 0x557f40480210 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52312/0 (socket says 172.21.15.28:52312) 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.499+0000 7efc42d9d640 1 -- 172.21.15.28:0/1716378410 learned_addr learned my addr 172.21.15.28:0/1716378410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc39d9b640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 0x557f40491240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40479c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 -- 172.21.15.28:0/1716378410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 msgr2=0x557f40491240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 0x557f40491240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 -- 172.21.15.28:0/1716378410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 msgr2=0x557f40479c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40479c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 -- 172.21.15.28:0/1716378410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f40498230 con 0x557f4047f820 2024-03-22T02:23:07.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc39d9b640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 0x557f40491240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc42d9d640 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005d70 tx=0x7efc3c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e5bf85db6caee3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc2bfff640 1 -- 172.21.15.28:0/1716378410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c02d070 con 0x557f4047f820 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc2bfff640 1 -- 172.21.15.28:0/1716378410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc3c01f040 con 0x557f4047f820 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.500+0000 7efc2bfff640 1 -- 172.21.15.28:0/1716378410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c00fcf0 con 0x557f4047f820 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1716378410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 msgr2=0x557f40479740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005d70 tx=0x7efc3c014040 comp rx=0 tx=0).stop 2024-03-22T02:23:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1716378410 shutdown_connections 2024-03-22T02:23:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40485380 0x557f40479c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f40480210 0x557f40491240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/1716378410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f4047f820 0x557f40479740 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1716378410 >> 172.21.15.28:0/1716378410 conn(0x557f4047ad90 msgr2=0x557f40489940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1716378410 shutdown_connections 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- 172.21.15.28:0/1716378410 wait complete. 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- start start 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 Processor -- start 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4048ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ab270 con 0x557f40485380 2024-03-22T02:23:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ac6e0 con 0x557f40480210 2024-03-22T02:23:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7efc441ab940 con 0x557f4047f820 2024-03-22T02:23:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc4bbe4740 10 client.0 _ll_register_callbacks cb 0x7efc340d35f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc39d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc42d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52314/0 (socket says 172.21.15.28:52314) 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc42d9d640 1 -- 172.21.15.28:0/4221308417 learned_addr learned my addr 172.21.15.28:0/4221308417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.501+0000 7efc42d9d640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4048ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 -- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 msgr2=0x557f4048ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4048ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 -- 172.21.15.28:0/4221308417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 msgr2=0x557f4048f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 -- 172.21.15.28:0/4221308417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f404907d0 con 0x557f40485380 2024-03-22T02:23:07.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc42d9d640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005fb0 tx=0x7efc3c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4524392b87afea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3c02d040 con 0x557f40485380 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7efc3c002030 con 0x557f40485380 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc3c01f040 con 0x557f40485380 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7efc3c003fb0 con 0x557f40485380 2024-03-22T02:23:07.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc4bbe4740 10 client.18255 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557f404fc0e0 con 0x557f40485380 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.502+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc3c01f040 con 0x557f40485380 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7efc3c043020 con 0x557f40485380 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 10 client.18255 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 10 client.18255 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 20 client.18255 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f404fc7f0 0x557f404fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 4 client.18255 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f403ec160 con 0x557f404fc7f0 2024-03-22T02:23:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc39d9b640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f404fc7f0 0x557f404fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc39d9b640 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f404fc7f0 0x557f404fec10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ecb2d48f6e7aa4e9 server_cookie=b5d03a239bf3c072 in_seq=0 out_seq=0 2024-03-22T02:23:07.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc29ffb640 10 client.18255 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc29ffb640 1 -- 172.21.15.28:0/4221308417 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f403ec160 con 0x557f404fc7f0 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc29ffb640 10 client.18255 handle_command_reply: tid=0 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.503+0000 7efc4bbe4740 1 client.18255 shutdown 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 msgr2=0x557f404901e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7efc3c005fb0 tx=0x7efc3c005bd0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 shutdown_connections 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x557f404fc7f0 0x557f404fec10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557f40485380 0x557f404901e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557f40480210 0x557f4048f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc29ffb640 0 client.18255 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 --2- 172.21.15.28:0/4221308417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557f4047f820 0x557f4048ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 >> 172.21.15.28:0/4221308417 conn(0x557f4047ad90 msgr2=0x557f4047dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 shutdown_connections 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 1 -- 172.21.15.28:0/4221308417 wait complete. 2024-03-22T02:23:07.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc4bbe4740 20 client.18255 trim_cache size 0 max 16384 2024-03-22T02:23:07.540 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 msgr2=0x7efc300a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 0x7efc300a53e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efc3400e9c0 tx=0x7efc340d3720 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 msgr2=0x7efc44101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7efc3400e9f0 tx=0x7efc3400edf0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 shutdown_connections 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7efc300a2f20 0x7efc300a53e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efc441078d0 0x7efc44100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7efc44104fa0 0x7efc44100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 --2- 172.21.15.28:0/3583311606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc44102670 0x7efc44101fe0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 >> 172.21.15.28:0/3583311606 conn(0x7efc440fc020 msgr2=0x7efc44103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 shutdown_connections 2024-03-22T02:23:07.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.504+0000 7efc48f1e640 1 -- 172.21.15.28:0/3583311606 wait complete. 2024-03-22T02:23:07.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:23:07.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- 172.21.15.28:0/3069728081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00102d90 msgr2=0x7f1f00103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 --2- 172.21.15.28:0/3069728081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00102d90 0x7f1f00103210 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009900 tx=0x7f1ef402bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:07.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- 172.21.15.28:0/3069728081 shutdown_connections 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 --2- 172.21.15.28:0/3069728081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 0x7f1f00109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 --2- 172.21.15.28:0/3069728081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00102d90 0x7f1f00103210 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 --2- 172.21.15.28:0/3069728081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00101ba0 0x7f1f00101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- 172.21.15.28:0/3069728081 >> 172.21.15.28:0/3069728081 conn(0x7f1f000fd0f0 msgr2=0x7f1f000ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- 172.21.15.28:0/3069728081 shutdown_connections 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- 172.21.15.28:0/3069728081 wait complete. 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 Processor -- start 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.652+0000 7f1f08781640 1 -- start start 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00102d90 0x7f1f00198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 0x7f1f0019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0010d820 con 0x7f1f00101ba0 2024-03-22T02:23:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0010d6a0 con 0x7f1f00103750 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0010d9a0 con 0x7f1f00102d90 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f05d1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00102d90 0x7f1f00198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52336/0 (socket says 172.21.15.28:52336) 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 -- 172.21.15.28:0/346848947 learned_addr learned my addr 172.21.15.28:0/346848947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 0x7f1f0019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 -- 172.21.15.28:0/346848947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00102d90 msgr2=0x7f1f00198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00102d90 0x7f1f00198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 -- 172.21.15.28:0/346848947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 msgr2=0x7f1f0019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 0x7f1f0019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 -- 172.21.15.28:0/346848947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0019fb10 con 0x7f1f00101ba0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f0651d640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800e9c0 tx=0x7f1ef800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1eff7fe640 1 -- 172.21.15.28:0/346848947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef800cd80 con 0x7f1f00101ba0 2024-03-22T02:23:07.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0019fe00 con 0x7f1f00101ba0 2024-03-22T02:23:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.653+0000 7f1eff7fe640 1 -- 172.21.15.28:0/346848947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef80035f0 con 0x7f1f00101ba0 2024-03-22T02:23:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.654+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f001a0210 con 0x7f1f00101ba0 2024-03-22T02:23:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.654+0000 7f1eff7fe640 1 -- 172.21.15.28:0/346848947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8010630 con 0x7f1f00101ba0 2024-03-22T02:23:07.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.657+0000 7f1eff7fe640 1 -- 172.21.15.28:0/346848947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1ef80037a0 con 0x7f1f00101ba0 2024-03-22T02:23:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.658+0000 7f1eff7fe640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 0x7f1ee80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.659+0000 7f1eff7fe640 1 -- 172.21.15.28:0/346848947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1ef80cfca0 con 0x7f1f00101ba0 2024-03-22T02:23:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.659+0000 7f1f05d1c640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 0x7f1ee80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 Processor -- start 2024-03-22T02:23:07.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f05d1c640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 0x7f1ee80a53e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402c7b0 tx=0x7f1ef4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 -- start start 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbdc30 0x55985fbbe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbc3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55985fb1e8d0 con 0x55985fbc3790 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55985fac6e00 con 0x55985fbbdc30 2024-03-22T02:23:07.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.660+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55985fb1a880 con 0x55985fbbe620 2024-03-22T02:23:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52348/0 (socket says 172.21.15.28:52348) 2024-03-22T02:23:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1549975520 learned_addr learned my addr 172.21.15.28:0/1549975520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1549975520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 msgr2=0x55985fbc3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbc3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1549975520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbdc30 msgr2=0x55985fbbe050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbdc30 0x55985fbbe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.661+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1549975520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985f8323f0 con 0x55985fbc3790 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.662+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009fd0 tx=0x7f1ef00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a6dd3bc37264c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.663+0000 7f1efcff9640 1 -- 172.21.15.28:0/1549975520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0004d20 con 0x55985fbc3790 2024-03-22T02:23:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.663+0000 7f1efcff9640 1 -- 172.21.15.28:0/1549975520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0004050 con 0x55985fbc3790 2024-03-22T02:23:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.663+0000 7f1efcff9640 1 -- 172.21.15.28:0/1549975520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0005280 con 0x55985fbc3790 2024-03-22T02:23:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1549975520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 msgr2=0x55985fbc5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009fd0 tx=0x7f1ef00118d0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1549975520 shutdown_connections 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbc3790 0x55985fbc5c20 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbc3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1549975520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbdc30 0x55985fbbe050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1549975520 >> 172.21.15.28:0/1549975520 conn(0x55985fbb4d90 msgr2=0x55985fbb5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1549975520 shutdown_connections 2024-03-22T02:23:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.665+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1549975520 wait complete. 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 -- start start 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 Processor -- start 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbdc30 0x55985fbc8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbe620 0x55985fbc8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55985fbc6740 con 0x55985fbbe620 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55985fada300 con 0x55985fbc3790 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55985fbc68c0 con 0x55985fbbdc30 2024-03-22T02:23:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.666+0000 7f1f0b36d740 10 client.0 _ll_register_callbacks cb 0x55985fb15e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42120/0 (socket says 172.21.15.28:42120) 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1921518797 learned_addr learned my addr 172.21.15.28:0/1921518797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1921518797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbdc30 msgr2=0x55985fbc8230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbdc30 0x55985fbc8230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1921518797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbe620 msgr2=0x55985fbc8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbe620 0x55985fbc8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.667+0000 7f1f06d1e640 1 -- 172.21.15.28:0/1921518797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985f8fc7d0 con 0x55985fbc3790 2024-03-22T02:23:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.668+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ef001b040 tx=0x7f1ef0003c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f5d231a39e5242c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.668+0000 7f1ee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.669+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef0029070 con 0x55985fbc3790 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.669+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1ef000f4e0 con 0x55985fbc3790 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.669+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef0004050 con 0x55985fbc3790 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.670+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1ef0033050 con 0x55985fbc3790 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.670+0000 7f1f0b36d740 10 client.28085 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.670+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55985fbd2b20 con 0x55985fbc3790 2024-03-22T02:23:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.670+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef0004050 con 0x55985fbc3790 2024-03-22T02:23:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.670+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f1ef002e600 con 0x55985fbc3790 2024-03-22T02:23:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 10 client.28085 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 10 client.28085 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 20 client.28085 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fbd3fe0 0x55985fbd63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 4 client.28085 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55985fb2be10 con 0x55985fbd3fe0 2024-03-22T02:23:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.671+0000 7f1f0651d640 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fbd3fe0 0x55985fbd63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.673+0000 7f1ee5ffb640 10 client.28085 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.673+0000 7f1f0651d640 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fbd3fe0 0x55985fbd63f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daa1bf37f79e5ef0 server_cookie=e58a8110ba187ddb in_seq=0 out_seq=0 2024-03-22T02:23:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.676+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/1921518797 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55985fb2be10 con 0x55985fbd3fe0 2024-03-22T02:23:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.676+0000 7f1ee5ffb640 10 client.28085 handle_command_reply: tid=0 2024-03-22T02:23:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.676+0000 7f1f0b36d740 1 client.28085 shutdown 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 msgr2=0x55985fbcf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1ef001b040 tx=0x7f1ef0003c60 comp rx=0 tx=0).stop 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 shutdown_connections 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fbd3fe0 0x55985fbd63f0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbcf7b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbe620 0x55985fbc8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/1921518797 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbdc30 0x55985fbc8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1ee5ffb640 0 client.28085 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 >> 172.21.15.28:0/1921518797 conn(0x55985fbb4d90 msgr2=0x55985fbb3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 shutdown_connections 2024-03-22T02:23:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 1 -- 172.21.15.28:0/1921518797 wait complete. 2024-03-22T02:23:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.677+0000 7f1f0b36d740 20 client.28085 trim_cache size 0 max 16384 2024-03-22T02:23:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 Processor -- start 2024-03-22T02:23:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 -- start start 2024-03-22T02:23:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbcf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbdacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001ab260 con 0x55985fbbdc30 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001a8b30 con 0x55985fbc3790 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001a6980 con 0x55985fbbe620 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f05d1c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbcf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52380/0 (socket says 172.21.15.28:52380) 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f06d1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbdacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.701+0000 7f1f0651d640 1 -- 172.21.15.28:0/332740626 learned_addr learned my addr 172.21.15.28:0/332740626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 -- 172.21.15.28:0/332740626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 msgr2=0x55985fbcf770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbcf770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 -- 172.21.15.28:0/332740626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 msgr2=0x55985fbdacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbdacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 -- 172.21.15.28:0/332740626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55985fbbea00 con 0x55985fbbdc30 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1f0651d640 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c6a0 tx=0x7f1ef80097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28470bcbffcc679 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1ee7fff640 1 -- 172.21.15.28:0/332740626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef809bfc0 con 0x55985fbbdc30 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1ee7fff640 1 -- 172.21.15.28:0/332740626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef80a0110 con 0x55985fbbdc30 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.702+0000 7f1ee7fff640 1 -- 172.21.15.28:0/332740626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8098630 con 0x55985fbbdc30 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- 172.21.15.28:0/332740626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 msgr2=0x55985fbb8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800c6a0 tx=0x7f1ef80097b0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- 172.21.15.28:0/332740626 shutdown_connections 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbc3790 0x55985fbdacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbbe620 0x55985fbcf770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/332740626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbb8700 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- 172.21.15.28:0/332740626 >> 172.21.15.28:0/332740626 conn(0x55985fbb4d90 msgr2=0x55985fbb7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- 172.21.15.28:0/332740626 shutdown_connections 2024-03-22T02:23:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- 172.21.15.28:0/332740626 wait complete. 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- start start 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 Processor -- start 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbe620 0x55985fbcd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbc3790 0x55985fbce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001a6ff0 con 0x55985fbbdc30 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001a8460 con 0x55985fbbe620 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1f001a76c0 con 0x55985fbc3790 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0b36d740 10 client.0 _ll_register_callbacks cb 0x7f1ef0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0651d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52392/0 (socket says 172.21.15.28:52392) 2024-03-22T02:23:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f0651d640 1 -- 172.21.15.28:0/248376147 learned_addr learned my addr 172.21.15.28:0/248376147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f06d1e640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbc3790 0x55985fbce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.703+0000 7f1f05d1c640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbe620 0x55985fbcd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 -- 172.21.15.28:0/248376147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbc3790 msgr2=0x55985fbce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbc3790 0x55985fbce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 -- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbe620 msgr2=0x55985fbcd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbe620 0x55985fbcd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 -- 172.21.15.28:0/248376147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55985fbce980 con 0x55985fbbdc30 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0651d640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8098990 tx=0x7f1ef80056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2659b0aefff71 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1ee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8003b10 con 0x55985fbbdc30 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1ef8098630 con 0x55985fbbdc30 2024-03-22T02:23:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ef80a0110 con 0x55985fbbdc30 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1ef8096b30 con 0x55985fbbdc30 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0b36d740 10 client.18270 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55985fc3a330 con 0x55985fbbdc30 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.704+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ef80a0110 con 0x55985fbbdc30 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f1ef80b6020 con 0x55985fbbdc30 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 10 client.18270 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 10 client.18270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 20 client.18270 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fc3ab60 0x55985fc3cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 4 client.18270 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55985fb26160 con 0x55985fc3ab60 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f05d1c640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fc3ab60 0x55985fc3cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1f05d1c640 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fc3ab60 0x55985fc3cf80 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40bd880db3698568 server_cookie=954d766ecc016dc7 in_seq=0 out_seq=0 2024-03-22T02:23:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.705+0000 7f1ee5ffb640 10 client.18270 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1ee5ffb640 1 -- 172.21.15.28:0/248376147 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55985fb26160 con 0x55985fc3ab60 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1ee5ffb640 10 client.18270 handle_command_reply: tid=0 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 client.18270 shutdown 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 msgr2=0x55985fbcce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8098990 tx=0x7f1ef80056d0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 shutdown_connections 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55985fc3ab60 0x55985fc3cf80 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55985fbc3790 0x55985fbce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1ee5ffb640 0 client.18270 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55985fbbe620 0x55985fbcd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 --2- 172.21.15.28:0/248376147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55985fbbdc30 0x55985fbcce40 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 >> 172.21.15.28:0/248376147 conn(0x55985fbb4d90 msgr2=0x55985fbb8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 shutdown_connections 2024-03-22T02:23:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.706+0000 7f1f0b36d740 1 -- 172.21.15.28:0/248376147 wait complete. 2024-03-22T02:23:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f0b36d740 20 client.18270 trim_cache size 0 max 16384 2024-03-22T02:23:07.714 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 msgr2=0x7f1ee80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 0x7f1ee80a53e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1ef402c7b0 tx=0x7f1ef4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 msgr2=0x7f1f00197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800e9c0 tx=0x7f1ef800ee80 comp rx=0 tx=0).stop 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 shutdown_connections 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1ee80a2f20 0x7f1ee80a53e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1f00103750 0x7f1f0019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1f00102d90 0x7f1f00198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 --2- 172.21.15.28:0/346848947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f00101ba0 0x7f1f00197ef0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 >> 172.21.15.28:0/346848947 conn(0x7f1f000fd0f0 msgr2=0x7f1f001061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 shutdown_connections 2024-03-22T02:23:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:07.707+0000 7f1f08781640 1 -- 172.21.15.28:0/346848947 wait complete. 2024-03-22T02:23:07.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:07 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1512693588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:07.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:07 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1512693588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:08.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:07 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1512693588' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:08.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 936874 11.50 MB/sec execute 350 sec latency 87.227 ms 2024-03-22T02:23:08.719 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:23:08.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:23:08.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:23:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- 172.21.15.28:0/3385985181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 msgr2=0x7f52ac109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/3385985181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac109ce0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f52a4009ca0 tx=0x7f52a402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- 172.21.15.28:0/3385985181 shutdown_connections 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/3385985181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac109ce0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/3385985181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac104f60 0x7f52ac107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/3385985181 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac102630 0x7f52ac104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- 172.21.15.28:0/3385985181 >> 172.21.15.28:0/3385985181 conn(0x7f52ac0fc040 msgr2=0x7f52ac0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- 172.21.15.28:0/3385985181 shutdown_connections 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- 172.21.15.28:0/3385985181 wait complete. 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 Processor -- start 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.834+0000 7f52b1cc3640 1 -- start start 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 0x7f52ac197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac104f60 0x7f52ac1983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac10d850 con 0x7f52ac107890 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac10d6d0 con 0x7f52ac104f60 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac10d9d0 con 0x7f52ac102630 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 0x7f52ac197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 0x7f52ac197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34062/0 (socket says 172.21.15.28:34062) 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52ab7fe640 1 -- 172.21.15.28:0/2942690264 learned_addr learned my addr 172.21.15.28:0/2942690264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac104f60 0x7f52ac1983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 -- 172.21.15.28:0/2942690264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 msgr2=0x7f52ac197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 0x7f52ac197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 -- 172.21.15.28:0/2942690264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac104f60 msgr2=0x7f52ac1983e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac104f60 0x7f52ac1983e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 -- 172.21.15.28:0/2942690264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac19fac0 con 0x7f52ac107890 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52abfff640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac19f450 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f52a402c630 tx=0x7f52a4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52a8ff9640 1 -- 172.21.15.28:0/2942690264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a4041020 con 0x7f52ac107890 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52ac19fd50 con 0x7f52ac107890 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52a8ff9640 1 -- 172.21.15.28:0/2942690264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52a4034cd0 con 0x7f52ac107890 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52a8ff9640 1 -- 172.21.15.28:0/2942690264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a4039040 con 0x7f52ac107890 2024-03-22T02:23:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.835+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52ac1a01c0 con 0x7f52ac107890 2024-03-22T02:23:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.838+0000 7f52a8ff9640 1 -- 172.21.15.28:0/2942690264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f52a4044020 con 0x7f52ac107890 2024-03-22T02:23:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.838+0000 7f52a8ff9640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 0x7f52940a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.838+0000 7f52b4985740 1 Processor -- start 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52a8ff9640 1 -- 172.21.15.28:0/2942690264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f52a40f1560 con 0x7f52ac107890 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 0x7f52940a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 -- start start 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0c55b0 0x5624ae0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 0x5624ae0ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624ae029570 con 0x5624ae0ca720 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5624adfd1e10 con 0x5624ae0c4bc0 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52b4985740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624ae025520 con 0x5624ae0c55b0 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42162/0 (socket says 172.21.15.28:42162) 2024-03-22T02:23:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52ab7fe640 1 -- 172.21.15.28:0/1845310898 learned_addr learned my addr 172.21.15.28:0/1845310898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52abfff640 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 0x5624ae0ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.839+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 0x7f52940a53e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f529c009ce0 tx=0x7f529c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 -- 172.21.15.28:0/1845310898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0c55b0 msgr2=0x5624ae0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0c55b0 0x5624ae0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 -- 172.21.15.28:0/1845310898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 msgr2=0x5624ae0ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 0x5624ae0ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 -- 172.21.15.28:0/1845310898 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624ade057e0 con 0x5624ae0c4bc0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52abfff640 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 0x5624ae0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f529c009b30 tx=0x7f529c01c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d136159f285052 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f5291ffb640 1 -- 172.21.15.28:0/1845310898 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c005220 con 0x5624ae0c4bc0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f5291ffb640 1 -- 172.21.15.28:0/1845310898 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f529c01ce10 con 0x5624ae0c4bc0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.840+0000 7f5291ffb640 1 -- 172.21.15.28:0/1845310898 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c0248b0 con 0x5624ae0c4bc0 2024-03-22T02:23:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- 172.21.15.28:0/1845310898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 msgr2=0x5624ae0c4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f529c009b30 tx=0x7f529c01c000 comp rx=0 tx=0).stop 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- 172.21.15.28:0/1845310898 shutdown_connections 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0ca720 0x5624ae0ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0c55b0 0x5624ae0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 --2- 172.21.15.28:0/1845310898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c4fe0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- 172.21.15.28:0/1845310898 >> 172.21.15.28:0/1845310898 conn(0x5624ae0c0130 msgr2=0x5624ae0c0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- 172.21.15.28:0/1845310898 shutdown_connections 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- 172.21.15.28:0/1845310898 wait complete. 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 -- start start 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.841+0000 7f52b4985740 1 Processor -- start 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e7360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624ae0de6e0 con 0x5624ae0c4bc0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5624adfe5110 con 0x5624ae0c55b0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624ae0de860 con 0x5624ae0ca720 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 10 client.0 _ll_register_callbacks cb 0x5624ae021be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52440/0 (socket says 172.21.15.28:52440) 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42174/0 (socket says 172.21.15.28:42174) 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 -- 172.21.15.28:0/642562045 learned_addr learned my addr 172.21.15.28:0/642562045 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 -- 172.21.15.28:0/642562045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 msgr2=0x5624ae0e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 -- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 msgr2=0x5624ae0e0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 -- 172.21.15.28:0/642562045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624ade65f70 con 0x5624ae0c4bc0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52aaffd640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f529c00a010 tx=0x7f529c024c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b5c143c873155c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f528affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c033030 con 0x5624ae0c4bc0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f529c009040 con 0x5624ae0c4bc0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.842+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f529c01ce10 con 0x5624ae0c4bc0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f529c024e30 con 0x5624ae0c4bc0 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 10 client.18282 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5624adcfd170 con 0x5624ae0c4bc0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f529c01ce10 con 0x5624ae0c4bc0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f529c023070 con 0x5624ae0c4bc0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 10 client.18282 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 10 client.18282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 20 client.18282 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae0eb8f0 0x5624ae0edd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 4 client.18282 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5624ae036ab0 con 0x5624ae0eb8f0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.843+0000 7f52aaffd640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae0eb8f0 0x5624ae0edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.844+0000 7f52aaffd640 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae0eb8f0 0x5624ae0edd40 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eed0a239cb5659de server_cookie=a6e82c3c8dc27762 in_seq=0 out_seq=0 2024-03-22T02:23:08.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.844+0000 7f528affd640 10 client.18282 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f528affd640 1 -- 172.21.15.28:0/642562045 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5624ae036ab0 con 0x5624ae0eb8f0 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f528affd640 10 client.18282 handle_command_reply: tid=0 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 client.18282 shutdown 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 msgr2=0x5624ae0dfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f529c00a010 tx=0x7f529c024c40 comp rx=0 tx=0).stop 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 shutdown_connections 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae0eb8f0 0x5624ae0edd40 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f528affd640 0 client.18282 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 --2- 172.21.15.28:0/642562045 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0dfde0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 >> 172.21.15.28:0/642562045 conn(0x5624ae0c0130 msgr2=0x5624ae0bf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 shutdown_connections 2024-03-22T02:23:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 1 -- 172.21.15.28:0/642562045 wait complete. 2024-03-22T02:23:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.845+0000 7f52b4985740 20 client.18282 trim_cache size 0 max 16384 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 Processor -- start 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 -- start start 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0c6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1a4a50 con 0x5624ae0c55b0 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1a2320 con 0x5624ae0c4bc0 2024-03-22T02:23:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.866+0000 7f52b4985740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac073b30 con 0x5624ae0ca720 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34096/0 (socket says 172.21.15.28:34096) 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42176/0 (socket says 172.21.15.28:42176) 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52abfff640 1 -- 172.21.15.28:0/3144023283 learned_addr learned my addr 172.21.15.28:0/3144023283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52456/0 (socket says 172.21.15.28:52456) 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 -- 172.21.15.28:0/3144023283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 msgr2=0x5624ae0c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 -- 172.21.15.28:0/3144023283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 msgr2=0x5624ae0c3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 -- 172.21.15.28:0/3144023283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624ae0ba800 con 0x5624ae0c55b0 2024-03-22T02:23:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52aaffd640 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x5624ae0c59b0 tx=0x7f52a00123e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f0420f1742816d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52917fa640 1 -- 172.21.15.28:0/3144023283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a001f740 con 0x5624ae0c55b0 2024-03-22T02:23:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52917fa640 1 -- 172.21.15.28:0/3144023283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52a0005020 con 0x5624ae0c55b0 2024-03-22T02:23:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.867+0000 7f52917fa640 1 -- 172.21.15.28:0/3144023283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0004a90 con 0x5624ae0c55b0 2024-03-22T02:23:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 -- 172.21.15.28:0/3144023283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 msgr2=0x5624ae0c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x5624ae0c59b0 tx=0x7f52a00123e0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 -- 172.21.15.28:0/3144023283 shutdown_connections 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c55b0 0x5624ae0c6050 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 --2- 172.21.15.28:0/3144023283 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c4bc0 0x5624ae0c3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 -- 172.21.15.28:0/3144023283 >> 172.21.15.28:0/3144023283 conn(0x5624ae0c0130 msgr2=0x5624ae0c3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 -- 172.21.15.28:0/3144023283 shutdown_connections 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.868+0000 7f52b4985740 1 -- 172.21.15.28:0/3144023283 wait complete. 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 -- start start 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 Processor -- start 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1a07e0 con 0x5624ae0c4bc0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34112/0 (socket says 172.21.15.28:34112) 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52abfff640 1 -- 172.21.15.28:0/1015942665 learned_addr learned my addr 172.21.15.28:0/1015942665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1a1c50 con 0x5624ae0c55b0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac1a0eb0 con 0x5624ae0ca720 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.869+0000 7f52b4985740 10 client.0 _ll_register_callbacks cb 0x7f529c009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 -- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 msgr2=0x5624ae0e4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 -- 172.21.15.28:0/1015942665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 msgr2=0x5624ae0e3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0e3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 -- 172.21.15.28:0/1015942665 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624ae0e5730 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52abfff640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f52a4005cc0 tx=0x7f52a40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da5b059db232ea24 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f528affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a40c1a60 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f52a40bdb70 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52a0005020 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.870+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f52a40c6290 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 10 client.28102 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5624ae151c80 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52a0005020 con 0x5624ae0ca720 2024-03-22T02:23:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f52a403c030 con 0x5624ae0ca720 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 10 client.28102 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 10 client.28102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 20 client.28102 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae152490 0x5624ae1548b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 4 client.28102 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5624ae030e00 con 0x5624ae152490 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.871+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae152490 0x5624ae1548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.872+0000 7f528affd640 10 client.28102 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.872+0000 7f52ab7fe640 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae152490 0x5624ae1548b0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc8520e7de55adc0 server_cookie=48c1fb2dcb9c34e1 in_seq=0 out_seq=0 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f528affd640 1 -- 172.21.15.28:0/1015942665 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5624ae030e00 con 0x5624ae152490 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f528affd640 10 client.28102 handle_command_reply: tid=0 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 client.28102 shutdown 2024-03-22T02:23:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 msgr2=0x5624ae0e5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f52a4005cc0 tx=0x7f52a40095c0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 shutdown_connections 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624ae152490 0x5624ae1548b0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624ae0ca720 0x5624ae0e5140 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624ae0c55b0 0x5624ae0e4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f528affd640 0 client.28102 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 --2- 172.21.15.28:0/1015942665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624ae0c4bc0 0x5624ae0e3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 >> 172.21.15.28:0/1015942665 conn(0x5624ae0c0130 msgr2=0x5624ae0c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 shutdown_connections 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 1 -- 172.21.15.28:0/1015942665 wait complete. 2024-03-22T02:23:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.873+0000 7f52b4985740 20 client.28102 trim_cache size 0 max 16384 2024-03-22T02:23:08.881 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:23:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "493f8b6e-453b-4416-827c-f7dad843bd43", 2024-03-22T02:23:08.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:23:08.881 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 msgr2=0x7f52940a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 0x7f52940a53e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f529c009ce0 tx=0x7f529c005f90 comp rx=0 tx=0).stop 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 msgr2=0x7f52ac19f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac19f450 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f52a402c630 tx=0x7f52a4032f20 comp rx=0 tx=0).stop 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 shutdown_connections 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f52940a2f20 0x7f52940a53e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52ac107890 0x7f52ac19f450 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f52ac104f60 0x7f52ac1983e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 --2- 172.21.15.28:0/2942690264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f52ac102630 0x7f52ac197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 >> 172.21.15.28:0/2942690264 conn(0x7f52ac0fc040 msgr2=0x7f52ac102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 shutdown_connections 2024-03-22T02:23:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:08.874+0000 7f52b1cc3640 1 -- 172.21.15.28:0/2942690264 wait complete. 2024-03-22T02:23:08.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:08 smithi028 ceph-mon[31916]: pgmap v646: 129 pgs: 129 active+clean; 42 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-03-22T02:23:08.886 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:23:08.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:23:08.898 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:08 smithi120 ceph-mon[36803]: pgmap v646: 129 pgs: 129 active+clean; 42 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-03-22T02:23:08.915 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:08 smithi083 ceph-mon[36340]: pgmap v646: 129 pgs: 129 active+clean; 42 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 2.1 MiB/s wr, 216 op/s 2024-03-22T02:23:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 -- 172.21.15.28:0/2262070551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184101400 msgr2=0x7f6184101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2262070551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184101400 0x7f6184101880 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f617400aa10 tx=0x7f617402cf70 comp rx=0 tx=0).stop 2024-03-22T02:23:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 -- 172.21.15.28:0/2262070551 shutdown_connections 2024-03-22T02:23:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2262070551 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 0x7f6184108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2262070551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184101400 0x7f6184101880 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2262070551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184100210 0x7f6184100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.001+0000 7f618bf4b640 1 -- 172.21.15.28:0/2262070551 >> 172.21.15.28:0/2262070551 conn(0x7f61840fb780 msgr2=0x7f61840fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- 172.21.15.28:0/2262070551 shutdown_connections 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- 172.21.15.28:0/2262070551 wait complete. 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 Processor -- start 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- start start 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 0x7f6184196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 0x7f618419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f618410b620 con 0x7f6184100210 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f618410b4a0 con 0x7f6184101dc0 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618bf4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f618410b7a0 con 0x7f6184101400 2024-03-22T02:23:09.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52468/0 (socket says 172.21.15.28:52468) 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f6189ce7640 1 -- 172.21.15.28:0/2586843350 learned_addr learned my addr 172.21.15.28:0/2586843350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f61894e6640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 0x7f6184196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.002+0000 7f618a4e8640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 0x7f618419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 -- 172.21.15.28:0/2586843350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 msgr2=0x7f6184196230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 0x7f6184196230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 -- 172.21.15.28:0/2586843350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 msgr2=0x7f618419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 0x7f618419d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 -- 172.21.15.28:0/2586843350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618419d900 con 0x7f6184100210 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f61894e6640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 0x7f6184196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:09.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f6189ce7640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f618000eb70 tx=0x7f618000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f617affd640 1 -- 172.21.15.28:0/2586843350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180003b70 con 0x7f6184100210 2024-03-22T02:23:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f617affd640 1 -- 172.21.15.28:0/2586843350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f618000ee70 con 0x7f6184100210 2024-03-22T02:23:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f617affd640 1 -- 172.21.15.28:0/2586843350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180010830 con 0x7f6184100210 2024-03-22T02:23:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618419dbf0 con 0x7f6184100210 2024-03-22T02:23:09.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.003+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f618419e0f0 con 0x7f6184100210 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.006+0000 7f617affd640 1 -- 172.21.15.28:0/2586843350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6180004020 con 0x7f6184100210 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f617affd640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 0x7f61700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f617affd640 1 -- 172.21.15.28:0/2586843350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f61800cfef0 con 0x7f6184100210 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f61894e6640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 0x7f61700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 Processor -- start 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 -- start start 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e280d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f61894e6640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 0x7f61700a53e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6174004d40 tx=0x7f6174005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e280df90 0x55c4e2810420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4e276d8c0 con 0x55c4e280df90 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2715c20 con 0x55c4e2808e20 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618ec0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2769870 con 0x55c4e2808430 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f61894e6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e280d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f618a4e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e280df90 0x55c4e2810420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34140/0 (socket says 172.21.15.28:34140) 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 -- 172.21.15.28:0/3668522336 learned_addr learned my addr 172.21.15.28:0/3668522336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.007+0000 7f61894e6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e280d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42208/0 (socket says 172.21.15.28:42208) 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 -- 172.21.15.28:0/3668522336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 msgr2=0x55c4e280d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e280d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 -- 172.21.15.28:0/3668522336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e280df90 msgr2=0x55c4e2810420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e280df90 0x55c4e2810420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 -- 172.21.15.28:0/3668522336 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e273a150 con 0x55c4e2808430 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f6189ce7640 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6180098b50 tx=0x7f618009c0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89db4bae1120433f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f616ffff640 1 -- 172.21.15.28:0/3668522336 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61800b5070 con 0x55c4e2808430 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f616ffff640 1 -- 172.21.15.28:0/3668522336 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f618009c910 con 0x55c4e2808430 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.008+0000 7f616ffff640 1 -- 172.21.15.28:0/3668522336 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61800987f0 con 0x55c4e2808430 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- 172.21.15.28:0/3668522336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 msgr2=0x55c4e2808850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6180098b50 tx=0x7f618009c0a0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- 172.21.15.28:0/3668522336 shutdown_connections 2024-03-22T02:23:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e280df90 0x55c4e2810420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e280d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 --2- 172.21.15.28:0/3668522336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2808850 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- 172.21.15.28:0/3668522336 >> 172.21.15.28:0/3668522336 conn(0x55c4e28039a0 msgr2=0x55c4e2803da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- 172.21.15.28:0/3668522336 shutdown_connections 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- 172.21.15.28:0/3668522336 wait complete. 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 -- start start 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.009+0000 7f618ec0b740 1 Processor -- start 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e28129b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2819f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2810f40 con 0x55c4e2808e20 2024-03-22T02:23:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c4e2729130 con 0x55c4e280df90 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c4e28110c0 con 0x55c4e2808430 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618ec0b740 10 client.0 _ll_register_callbacks cb 0x55c4e2728b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52474/0 (socket says 172.21.15.28:52474) 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e28129b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 -- 172.21.15.28:0/775880261 learned_addr learned my addr 172.21.15.28:0/775880261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f618a4e8640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2819f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 -- 172.21.15.28:0/775880261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 msgr2=0x55c4e28129b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e28129b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 -- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 msgr2=0x55c4e2819f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2819f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 -- 172.21.15.28:0/775880261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4e254b7d0 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f61894e6640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f617400a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7012abd3f38c04f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.010+0000 7f616d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174051020 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f6174004310 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f618009c910 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f617405c050 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 10 client.18300 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c4e24813f0 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f618009c910 con 0x55c4e2808e20 2024-03-22T02:23:09.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f617400b040 con 0x55c4e2808e20 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 10 client.18300 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 10 client.18300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 20 client.18300 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e281e860 0x55c4e2820cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 4 client.18300 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4e277ae00 con 0x55c4e281e860 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.011+0000 7f6189ce7640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e281e860 0x55c4e2820cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.012+0000 7f616d7fa640 10 client.18300 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.012+0000 7f6189ce7640 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e281e860 0x55c4e2820cd0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c020a41d3d6d9cf2 server_cookie=d7203935ac9a98c4 in_seq=0 out_seq=0 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f616d7fa640 1 -- 172.21.15.28:0/775880261 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4e277ae00 con 0x55c4e281e860 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f616d7fa640 10 client.18300 handle_command_reply: tid=0 2024-03-22T02:23:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 client.18300 shutdown 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 msgr2=0x55c4e2812ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f6174037040 tx=0x7f617400a370 comp rx=0 tx=0).stop 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 shutdown_connections 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e281e860 0x55c4e2820cd0 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2819f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2812ef0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f616d7fa640 0 client.18300 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 --2- 172.21.15.28:0/775880261 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e28129b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 >> 172.21.15.28:0/775880261 conn(0x55c4e28039a0 msgr2=0x55c4e2802c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 shutdown_connections 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 1 -- 172.21.15.28:0/775880261 wait complete. 2024-03-22T02:23:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.013+0000 7f618ec0b740 20 client.18300 trim_cache size 0 max 16384 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 Processor -- start 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 -- start start 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 0x55c4e2807310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e2825590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61841ab090 con 0x55c4e2808430 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61841a8960 con 0x55c4e2808e20 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f618ec0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f61841a67b0 con 0x55c4e280df90 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.033+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 0x55c4e2807310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 0x55c4e2807310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52476/0 (socket says 172.21.15.28:52476) 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34160/0 (socket says 172.21.15.28:34160) 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f6189ce7640 1 -- 172.21.15.28:0/2809841097 learned_addr learned my addr 172.21.15.28:0/2809841097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f61894e6640 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e2825590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 -- 172.21.15.28:0/2809841097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 msgr2=0x55c4e2825590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e2825590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 -- 172.21.15.28:0/2809841097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 msgr2=0x55c4e2807310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 0x55c4e2807310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 -- 172.21.15.28:0/2809841097 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4e24410f0 con 0x55c4e280df90 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f61894e6640 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e2825590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f618a4e8640 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f617c009e30 tx=0x7f617c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8931c1df9e99c160 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f616f7fe640 1 -- 172.21.15.28:0/2809841097 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c022070 con 0x55c4e280df90 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f616f7fe640 1 -- 172.21.15.28:0/2809841097 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617c0035f0 con 0x55c4e280df90 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.034+0000 7f616f7fe640 1 -- 172.21.15.28:0/2809841097 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c01d6c0 con 0x55c4e280df90 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- 172.21.15.28:0/2809841097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 msgr2=0x55c4e2825ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f617c009e30 tx=0x7f617c005f90 comp rx=0 tx=0).stop 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- 172.21.15.28:0/2809841097 shutdown_connections 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e280df90 0x55c4e2825ad0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e2808e20 0x55c4e2825590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 --2- 172.21.15.28:0/2809841097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808430 0x55c4e2807310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- 172.21.15.28:0/2809841097 >> 172.21.15.28:0/2809841097 conn(0x55c4e28039a0 msgr2=0x55c4e2806c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- 172.21.15.28:0/2809841097 shutdown_connections 2024-03-22T02:23:09.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- 172.21.15.28:0/2809841097 wait complete. 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 -- start start 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.035+0000 7f618ec0b740 1 Processor -- start 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2887660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f61841a6e20 con 0x55c4e2808e20 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f61841a8290 con 0x55c4e280df90 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f61841a74f0 con 0x55c4e2808430 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618ec0b740 10 client.0 _ll_register_callbacks cb 0x7f6174004790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52490/0 (socket says 172.21.15.28:52490) 2024-03-22T02:23:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 -- 172.21.15.28:0/1105019725 learned_addr learned my addr 172.21.15.28:0/1105019725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f6189ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34174/0 (socket says 172.21.15.28:34174) 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f618a4e8640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2887660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 -- 172.21.15.28:0/1105019725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 msgr2=0x55c4e2880150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 -- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 msgr2=0x55c4e2887660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2887660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 -- 172.21.15.28:0/1105019725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4e2887c50 con 0x55c4e2808e20 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f6189ce7640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f61894e6640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f617400a340 tx=0x7f61740058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd4aabb9ed84f04 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.036+0000 7f616d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6174065060 con 0x55c4e2808e20 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f617c0035f0 con 0x55c4e2808e20 2024-03-22T02:23:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f61740359a0 con 0x55c4e2808e20 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f617406c050 con 0x55c4e2808e20 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 10 client.18306 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c4e2888f90 con 0x55c4e2808e20 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f617c0035f0 con 0x55c4e2808e20 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f61740037e0 con 0x55c4e2808e20 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 10 client.18306 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 10 client.18306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 20 client.18306 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e2889900 0x55c4e288bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 4 client.18306 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c4e2775150 con 0x55c4e2889900 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.037+0000 7f6189ce7640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e2889900 0x55c4e288bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f616d7fa640 10 client.18306 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f6189ce7640 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e2889900 0x55c4e288bd20 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7164e20fe6d9452f server_cookie=1a2dea1e1013632d in_seq=0 out_seq=0 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f616d7fa640 1 -- 172.21.15.28:0/1105019725 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c4e2775150 con 0x55c4e2889900 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f616d7fa640 10 client.18306 handle_command_reply: tid=0 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 client.18306 shutdown 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 msgr2=0x55c4e2880690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f617400a340 tx=0x7f61740058c0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 shutdown_connections 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c4e2889900 0x55c4e288bd20 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c4e280df90 0x55c4e2887660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4e2808e20 0x55c4e2880690 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 --2- 172.21.15.28:0/1105019725 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c4e2808430 0x55c4e2880150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f616d7fa640 0 client.18306 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 >> 172.21.15.28:0/1105019725 conn(0x55c4e28039a0 msgr2=0x55c4e2807490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 shutdown_connections 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 1 -- 172.21.15.28:0/1105019725 wait complete. 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.038+0000 7f618ec0b740 20 client.18306 trim_cache size 0 max 16384 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "493f8b6e-453b-4416-827c-f7dad843bd43": { 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "493f8b6e-453b-4416-827c-f7dad843bd43", 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:23:09.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:23:09.048 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 msgr2=0x7f61700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 0x7f61700a53e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f6174004d40 tx=0x7f6174005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 msgr2=0x7f6184195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f618000eb70 tx=0x7f618000c6a0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 shutdown_connections 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f61700a2f20 0x7f61700a53e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6184101dc0 0x7f618419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6184101400 0x7f6184196230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 --2- 172.21.15.28:0/2586843350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6184100210 0x7f6184195cf0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 >> 172.21.15.28:0/2586843350 conn(0x7f61840fb780 msgr2=0x7f6184104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 shutdown_connections 2024-03-22T02:23:09.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:09.039+0000 7f618bf4b640 1 -- 172.21.15.28:0/2586843350 wait complete. 2024-03-22T02:23:09.050 INFO:tasks.cephfs.filesystem:scrub status for tag:493f8b6e-453b-4416-827c-f7dad843bd43 - {'path': '/', 'tag': '493f8b6e-453b-4416-827c-f7dad843bd43', 'options': 'recursive,force'} 2024-03-22T02:23:09.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 938745 11.50 MB/sec execute 351 sec latency 12.231 ms 2024-03-22T02:23:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:09 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:23:09.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:09 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:09 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:23:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:09 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:10.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:09 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:23:10.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:09 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:10.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 941376 11.51 MB/sec execute 352 sec latency 98.009 ms 2024-03-22T02:23:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:10 smithi120 ceph-mon[36803]: pgmap v647: 129 pgs: 129 active+clean; 39 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.9 MiB/s wr, 349 op/s 2024-03-22T02:23:10.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:10 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:23:10.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:10 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:10.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:10 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:23:10.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:10 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:10 smithi083 ceph-mon[36340]: pgmap v647: 129 pgs: 129 active+clean; 39 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.9 MiB/s wr, 349 op/s 2024-03-22T02:23:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:10 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:23:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:10 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:10 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:23:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:10 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:10 smithi028 ceph-mon[31916]: pgmap v647: 129 pgs: 129 active+clean; 39 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 2.9 MiB/s wr, 349 op/s 2024-03-22T02:23:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:10 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:23:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:10 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:10 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:23:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:10 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:11.428 INFO:tasks.workunit.client.0.smithi028.stdout: 1 942552 11.49 MB/sec execute 353 sec latency 103.925 ms 2024-03-22T02:23:12.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 945366 11.51 MB/sec execute 354 sec latency 12.258 ms 2024-03-22T02:23:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:12 smithi120 ceph-mon[36803]: pgmap v648: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 2.7 MiB/s wr, 379 op/s 2024-03-22T02:23:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:12 smithi083 ceph-mon[36340]: pgmap v648: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 2.7 MiB/s wr, 379 op/s 2024-03-22T02:23:13.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:12 smithi028 ceph-mon[31916]: pgmap v648: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 2.7 MiB/s wr, 379 op/s 2024-03-22T02:23:13.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 947474 11.50 MB/sec execute 355 sec latency 47.475 ms 2024-03-22T02:23:13.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:14.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:14.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:23:14.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 949261 11.50 MB/sec execute 356 sec latency 29.841 ms 2024-03-22T02:23:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:14 smithi120 ceph-mon[36803]: pgmap v649: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.7 MiB/s wr, 396 op/s 2024-03-22T02:23:15.040 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:14 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:15.041 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:14 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:15.041 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:14 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:23:15.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:14 smithi028 ceph-mon[31916]: pgmap v649: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.7 MiB/s wr, 396 op/s 2024-03-22T02:23:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:14 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:14 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:14 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:23:15.073 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:14 smithi083 ceph-mon[36340]: pgmap v649: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.7 MiB/s wr, 396 op/s 2024-03-22T02:23:15.073 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:14 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:23:15.073 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:14 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:23:15.073 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:14 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:23:15.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 952023 11.51 MB/sec execute 357 sec latency 12.804 ms 2024-03-22T02:23:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:15 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:23:15.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:15 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:23:15.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:15 smithi120 ceph-mon[36803]: pgmap v650: 129 pgs: 129 active+clean; 36 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-03-22T02:23:16.042 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:15 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:23:16.042 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:15 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:23:16.042 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:15 smithi083 ceph-mon[36340]: pgmap v650: 129 pgs: 129 active+clean; 36 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-03-22T02:23:16.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:15 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:23:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:15 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:23:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:15 smithi028 ceph-mon[31916]: pgmap v650: 129 pgs: 129 active+clean; 36 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 3.2 MiB/s wr, 430 op/s 2024-03-22T02:23:16.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 952918 11.49 MB/sec execute 358 sec latency 58.429 ms 2024-03-22T02:23:17.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 955743 11.51 MB/sec execute 359 sec latency 16.960 ms 2024-03-22T02:23:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:17 smithi028 ceph-mon[31916]: pgmap v651: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 2.7 MiB/s wr, 389 op/s 2024-03-22T02:23:18.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:17 smithi120 ceph-mon[36803]: pgmap v651: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 2.7 MiB/s wr, 389 op/s 2024-03-22T02:23:18.291 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:17 smithi083 ceph-mon[36340]: pgmap v651: 129 pgs: 129 active+clean; 37 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 2.7 MiB/s wr, 389 op/s 2024-03-22T02:23:18.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 957660 11.49 MB/sec execute 360 sec latency 30.011 ms 2024-03-22T02:23:18.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- 172.21.15.28:0/3118301901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 msgr2=0x7f81f0104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/3118301901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0104a20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009ca0 tx=0x7f81e802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- 172.21.15.28:0/3118301901 shutdown_connections 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/3118301901 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 0x7f81f0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/3118301901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 0x7f81f0107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/3118301901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0104a20 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- 172.21.15.28:0/3118301901 >> 172.21.15.28:0/3118301901 conn(0x7f81f00fc020 msgr2=0x7f81f00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- 172.21.15.28:0/3118301901 shutdown_connections 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- 172.21.15.28:0/3118301901 wait complete. 2024-03-22T02:23:18.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 Processor -- start 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.570+0000 7f81f6bd7640 1 -- start start 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 0x7f81f01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 0x7f81f019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010d860 con 0x7f81f0102630 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010d6e0 con 0x7f81f0107890 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81f010d9e0 con 0x7f81f0104f60 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51278/0 (socket says 172.21.15.28:51278) 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 -- 172.21.15.28:0/1105573729 learned_addr learned my addr 172.21.15.28:0/1105573729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81effff640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 0x7f81f01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f5174640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 0x7f81f019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 -- 172.21.15.28:0/1105573729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 msgr2=0x7f81f01983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 0x7f81f01983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 -- 172.21.15.28:0/1105573729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 msgr2=0x7f81f019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 0x7f81f019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 -- 172.21.15.28:0/1105573729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f019fa90 con 0x7f81f0102630 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f4973640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009dc0 tx=0x7f81e8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81edffb640 1 -- 172.21.15.28:0/1105573729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8041020 con 0x7f81f0102630 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f019fd20 con 0x7f81f0102630 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81edffb640 1 -- 172.21.15.28:0/1105573729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e8034cd0 con 0x7f81f0102630 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.571+0000 7f81edffb640 1 -- 172.21.15.28:0/1105573729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e8039040 con 0x7f81f0102630 2024-03-22T02:23:18.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.572+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f01a01c0 con 0x7f81f0102630 2024-03-22T02:23:18.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.574+0000 7f81edffb640 1 -- 172.21.15.28:0/1105573729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f81e8044020 con 0x7f81f0102630 2024-03-22T02:23:18.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.574+0000 7f81f97c0740 1 Processor -- start 2024-03-22T02:23:18.582 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.574+0000 7f81f97c0740 1 -- start start 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.574+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f6fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f97c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e13ecb040 con 0x563e13f6b2e0 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f97c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563e13e739e0 con 0x563e13f6a8f0 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f97c0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563e13ec6ff0 con 0x563e13f70450 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f6fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44004/0 (socket says 172.21.15.28:44004) 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f6fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51288/0 (socket says 172.21.15.28:51288) 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 -- 172.21.15.28:0/1040796293 learned_addr learned my addr 172.21.15.28:0/1040796293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:18.583 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81effff640 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81edffb640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 0x7f81d80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 0x7f81d80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 -- 172.21.15.28:0/1040796293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 msgr2=0x563e13f728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 -- 172.21.15.28:0/1040796293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 msgr2=0x563e13f6fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f6fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 -- 172.21.15.28:0/1040796293 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e13bdf210 con 0x563e13f6a8f0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81effff640 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81edffb640 1 -- 172.21.15.28:0/1105573729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f81e80f1640 con 0x7f81f0102630 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81e00138b0 tx=0x7f81e0013d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b19b0053a81789b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81f5174640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 0x7f81d80a5480 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f81e001e000 tx=0x7f81e0020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.575+0000 7f81d6ffd640 1 -- 172.21.15.28:0/1040796293 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e0012280 con 0x563e13f6a8f0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81d6ffd640 1 -- 172.21.15.28:0/1040796293 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e0023070 con 0x563e13f6a8f0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81d6ffd640 1 -- 172.21.15.28:0/1040796293 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00035f0 con 0x563e13f6a8f0 2024-03-22T02:23:18.584 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 -- 172.21.15.28:0/1040796293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 msgr2=0x563e13f6ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81e00138b0 tx=0x7f81e0013d70 comp rx=0 tx=0).stop 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 -- 172.21.15.28:0/1040796293 shutdown_connections 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f6fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 --2- 172.21.15.28:0/1040796293 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6a8f0 0x563e13f6ad10 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 -- 172.21.15.28:0/1040796293 >> 172.21.15.28:0/1040796293 conn(0x563e13f61a50 msgr2=0x563e13f61e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 -- 172.21.15.28:0/1040796293 shutdown_connections 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.576+0000 7f81f97c0740 1 -- 172.21.15.28:0/1040796293 wait complete. 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 -- start start 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 Processor -- start 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.585 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f7bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e13f733a0 con 0x563e13f6a8f0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563e13e86db0 con 0x563e13f6b2e0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563e13f73520 con 0x563e13f70450 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f97c0740 10 client.0 _ll_register_callbacks cb 0x563e13bdea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f4973640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51304/0 (socket says 172.21.15.28:51304) 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44012/0 (socket says 172.21.15.28:44012) 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f4973640 1 -- 172.21.15.28:0/3721990088 learned_addr learned my addr 172.21.15.28:0/3721990088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.577+0000 7f81f5174640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f7bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 -- 172.21.15.28:0/3721990088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 msgr2=0x563e13f7bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f7bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 -- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 msgr2=0x563e13f74da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 -- 172.21.15.28:0/3721990088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e13cab620 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81effff640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f4973640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f81e80b9dc0 tx=0x7f81e80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7ae8c18ca3afd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80bd4a0 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f81e80b96b0 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81e0023070 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f81e80bd2c0 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 10 client.18318 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e13f7f150 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e0023070 con 0x563e13f6a8f0 2024-03-22T02:23:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f81e803c030 con 0x563e13f6a8f0 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 10 client.18318 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 10 client.18318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 20 client.18318 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13f80810 0x563e13f82c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.578+0000 7f81f97c0740 4 client.18318 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.579+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e13ed8580 con 0x563e13f80810 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.579+0000 7f81effff640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13f80810 0x563e13f82c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.579+0000 7f81effff640 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13f80810 0x563e13f82c40 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53081e962800f520 server_cookie=e9cd726936f28f4f in_seq=0 out_seq=0 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.579+0000 7f81cffff640 10 client.18318 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81cffff640 1 -- 172.21.15.28:0/3721990088 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563e13ed8580 con 0x563e13f80810 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81cffff640 10 client.18318 handle_command_reply: tid=0 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 client.18318 shutdown 2024-03-22T02:23:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 msgr2=0x563e13f74860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f81e80b9dc0 tx=0x7f81e80095c0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 shutdown_connections 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13f80810 0x563e13f82c40 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f70450 0x563e13f7bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f74da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81cffff640 0 client.18318 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3721990088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6a8f0 0x563e13f74860 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 >> 172.21.15.28:0/3721990088 conn(0x563e13f61a50 msgr2=0x563e13f60c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 shutdown_connections 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 1 -- 172.21.15.28:0/3721990088 wait complete. 2024-03-22T02:23:18.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.580+0000 7f81f97c0740 20 client.18318 trim_cache size 0 max 16384 2024-03-22T02:23:18.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.595+0000 7f81f97c0740 1 Processor -- start 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.595+0000 7f81f97c0740 1 -- start start 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f7bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01aafa0 con 0x563e13f70450 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a8870 con 0x563e13f6b2e0 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f97c0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a66c0 con 0x563e13f6a8f0 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f5174640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81effff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f7bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59880/0 (socket says 172.21.15.28:59880) 2024-03-22T02:23:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f5174640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51318/0 (socket says 172.21.15.28:51318) 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 -- 172.21.15.28:0/2970279364 learned_addr learned my addr 172.21.15.28:0/2970279364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 -- 172.21.15.28:0/2970279364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 msgr2=0x563e13f7bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f7bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 -- 172.21.15.28:0/2970279364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 msgr2=0x563e13f60900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 -- 172.21.15.28:0/2970279364 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e13f83290 con 0x563e13f6a8f0 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f5174640 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81effff640 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f7bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81f4973640 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81e80bbc10 tx=0x7f81e80bbe70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f96fccc9d0206263 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.596+0000 7f81d67fc640 1 -- 172.21.15.28:0/2970279364 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80bd4a0 con 0x563e13f6a8f0 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81d67fc640 1 -- 172.21.15.28:0/2970279364 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e80be060 con 0x563e13f6a8f0 2024-03-22T02:23:18.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81d67fc640 1 -- 172.21.15.28:0/2970279364 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e80f1480 con 0x563e13f6a8f0 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 -- 172.21.15.28:0/2970279364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 msgr2=0x563e13f603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f81e80bbc10 tx=0x7f81e80bbe70 comp rx=0 tx=0).stop 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 -- 172.21.15.28:0/2970279364 shutdown_connections 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f70450 0x563e13f60900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f6b2e0 0x563e13f7bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 --2- 172.21.15.28:0/2970279364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f603c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 -- 172.21.15.28:0/2970279364 >> 172.21.15.28:0/2970279364 conn(0x563e13f61a50 msgr2=0x563e13f65540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 -- 172.21.15.28:0/2970279364 shutdown_connections 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.597+0000 7f81f97c0740 1 -- 172.21.15.28:0/2970279364 wait complete. 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 -- start start 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 Processor -- start 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f79770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f70450 0x563e13f7ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a6d30 con 0x563e13f6b2e0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a81a0 con 0x563e13f70450 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f81f01a7400 con 0x563e13f6a8f0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f97c0740 10 client.0 _ll_register_callbacks cb 0x7f81e8003970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f4973640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f79770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51320/0 (socket says 172.21.15.28:51320) 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 -- 172.21.15.28:0/3291705530 learned_addr learned my addr 172.21.15.28:0/3291705530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f5174640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f70450 0x563e13f7ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 -- 172.21.15.28:0/3291705530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 msgr2=0x563e13f79770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f79770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 -- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f70450 msgr2=0x563e13f7ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f70450 0x563e13f7ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81effff640 1 -- 172.21.15.28:0/3291705530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e13f7b240 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.598+0000 7f81f4973640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f79770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81effff640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f81e4012580 tx=0x7f81e4012c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b6d1e66a41a9118 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e401fa40 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f81e401fbe0 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81e80be060 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f81e4031050 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 10 client.18324 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563e13fe6ec0 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81e80be060 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f81e4004a00 con 0x563e13f6b2e0 2024-03-22T02:23:18.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 10 client.18324 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 10 client.18324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 20 client.18324 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13fe78f0 0x563e13fe9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 4 client.18324 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563e13ed28d0 con 0x563e13fe78f0 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.599+0000 7f81f4973640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13fe78f0 0x563e13fe9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.600+0000 7f81cffff640 10 client.18324 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:18.606 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:18.600+0000 7f81f4973640 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13fe78f0 0x563e13fe9d10 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf137475303a9ba7 server_cookie=65af52ed9c1f31b3 in_seq=0 out_seq=0 2024-03-22T02:23:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81cffff640 1 -- 172.21.15.28:0/3291705530 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563e13ed28d0 con 0x563e13fe78f0 2024-03-22T02:23:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81cffff640 10 client.18324 handle_command_reply: tid=0 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 client.18324 shutdown 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 msgr2=0x563e13f79cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f81e4012580 tx=0x7f81e4012c80 comp rx=0 tx=0).stop 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 shutdown_connections 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563e13fe78f0 0x563e13fe9d10 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563e13f70450 0x563e13f7ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81cffff640 0 client.18324 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e13f6b2e0 0x563e13f79cb0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 --2- 172.21.15.28:0/3291705530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563e13f6a8f0 0x563e13f79770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 >> 172.21.15.28:0/3291705530 conn(0x563e13f61a50 msgr2=0x563e13f6bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.000+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 shutdown_connections 2024-03-22T02:23:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f97c0740 1 -- 172.21.15.28:0/3291705530 wait complete. 2024-03-22T02:23:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f97c0740 20 client.18324 trim_cache size 0 max 16384 2024-03-22T02:23:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:23:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:23:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:23:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 msgr2=0x7f81d80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 0x7f81d80a5480 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f81e001e000 tx=0x7f81e0020000 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 msgr2=0x7f81f0197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009dc0 tx=0x7f81e8032f20 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 shutdown_connections 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f81d80a2fc0 0x7f81d80a5480 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f81f0107890 0x7f81f019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f81f0104f60 0x7f81f01983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 --2- 172.21.15.28:0/1105573729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f0102630 0x7f81f0197e70 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.001+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 >> 172.21.15.28:0/1105573729 conn(0x7f81f00fc020 msgr2=0x7f81f0102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.002+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 shutdown_connections 2024-03-22T02:23:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:19.002+0000 7f81f6bd7640 1 -- 172.21.15.28:0/1105573729 wait complete. 2024-03-22T02:23:19.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 959542 11.50 MB/sec execute 361 sec latency 181.161 ms 2024-03-22T02:23:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:19 smithi028 ceph-mon[31916]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-03-22T02:23:20.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:19 smithi120 ceph-mon[36803]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-03-22T02:23:20.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:19 smithi083 ceph-mon[36340]: pgmap v652: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 3.4 MiB/s wr, 417 op/s 2024-03-22T02:23:20.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 961773 11.49 MB/sec execute 362 sec latency 29.257 ms 2024-03-22T02:23:21.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 963343 11.49 MB/sec execute 363 sec latency 96.269 ms 2024-03-22T02:23:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:21 smithi028 ceph-mon[31916]: pgmap v653: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.5 MiB/s wr, 292 op/s 2024-03-22T02:23:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:21 smithi120 ceph-mon[36803]: pgmap v653: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.5 MiB/s wr, 292 op/s 2024-03-22T02:23:22.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:21 smithi083 ceph-mon[36340]: pgmap v653: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 2.5 MiB/s wr, 292 op/s 2024-03-22T02:23:22.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 965813 11.49 MB/sec execute 364 sec latency 40.878 ms 2024-03-22T02:23:23.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 967908 11.49 MB/sec execute 365 sec latency 29.450 ms 2024-03-22T02:23:24.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:23 smithi028 ceph-mon[31916]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 313 op/s 2024-03-22T02:23:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:23 smithi120 ceph-mon[36803]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 313 op/s 2024-03-22T02:23:24.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:23 smithi083 ceph-mon[36340]: pgmap v654: 129 pgs: 129 active+clean; 38 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.7 MiB/s wr, 313 op/s 2024-03-22T02:23:24.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 970402 11.50 MB/sec execute 366 sec latency 21.768 ms 2024-03-22T02:23:25.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 973275 11.51 MB/sec execute 367 sec latency 29.139 ms 2024-03-22T02:23:26.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:25 smithi028 ceph-mon[31916]: pgmap v655: 129 pgs: 129 active+clean; 34 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-03-22T02:23:26.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:25 smithi120 ceph-mon[36803]: pgmap v655: 129 pgs: 129 active+clean; 34 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-03-22T02:23:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:25 smithi083 ceph-mon[36340]: pgmap v655: 129 pgs: 129 active+clean; 34 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 3.3 MiB/s wr, 400 op/s 2024-03-22T02:23:26.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 974572 11.49 MB/sec execute 368 sec latency 80.920 ms 2024-03-22T02:23:27.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 977133 11.51 MB/sec execute 369 sec latency 25.268 ms 2024-03-22T02:23:28.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:27 smithi028 ceph-mon[31916]: pgmap v656: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.7 MiB/s wr, 375 op/s 2024-03-22T02:23:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:27 smithi120 ceph-mon[36803]: pgmap v656: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.7 MiB/s wr, 375 op/s 2024-03-22T02:23:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:27 smithi083 ceph-mon[36340]: pgmap v656: 129 pgs: 129 active+clean; 35 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 2.7 MiB/s wr, 375 op/s 2024-03-22T02:23:28.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 979435 11.50 MB/sec execute 370 sec latency 27.571 ms 2024-03-22T02:23:29.429 INFO:tasks.workunit.client.0.smithi028.stdout: 1 981295 11.50 MB/sec execute 371 sec latency 35.295 ms 2024-03-22T02:23:30.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 984117 11.52 MB/sec execute 372 sec latency 14.456 ms 2024-03-22T02:23:30.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:30 smithi028 ceph-mon[31916]: pgmap v657: 129 pgs: 129 active+clean; 54 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 425 op/s 2024-03-22T02:23:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:30 smithi120 ceph-mon[36803]: pgmap v657: 129 pgs: 129 active+clean; 54 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 425 op/s 2024-03-22T02:23:30.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:30 smithi083 ceph-mon[36340]: pgmap v657: 129 pgs: 129 active+clean; 54 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 425 op/s 2024-03-22T02:23:31.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 985776 11.50 MB/sec execute 373 sec latency 28.781 ms 2024-03-22T02:23:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 -- 172.21.15.28:0/3827526176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 msgr2=0x7f32801073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 --2- 172.21.15.28:0/3827526176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f32801073b0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f326c009900 tx=0x7f326c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 -- 172.21.15.28:0/3827526176 shutdown_connections 2024-03-22T02:23:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 --2- 172.21.15.28:0/3827526176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 0x7f3280109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 --2- 172.21.15.28:0/3827526176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f32801073b0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 --2- 172.21.15.28:0/3827526176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 -- 172.21.15.28:0/3827526176 >> 172.21.15.28:0/3827526176 conn(0x7f32800fc020 msgr2=0x7f32800fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 -- 172.21.15.28:0/3827526176 shutdown_connections 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.094+0000 7f32870bb640 1 -- 172.21.15.28:0/3827526176 wait complete. 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 Processor -- start 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 -- start start 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 0x7f328019f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f328010d830 con 0x7f3280104fc0 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f328010d6b0 con 0x7f3280102690 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f32870bb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f328010d9b0 con 0x7f32801078f0 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49552/0 (socket says 172.21.15.28:49552) 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f3284e57640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 -- 172.21.15.28:0/2442045826 learned_addr learned my addr 172.21.15.28:0/2442045826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f3285658640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 0x7f328019f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 -- 172.21.15.28:0/2442045826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 msgr2=0x7f328019f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 0x7f328019f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 -- 172.21.15.28:0/2442045826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 msgr2=0x7f3280197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f327ffff640 1 -- 172.21.15.28:0/2442045826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f328019fa50 con 0x7f3280104fc0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.095+0000 7f3284e57640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f327ffff640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f326c000c00 tx=0x7f326c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f327dffb640 1 -- 172.21.15.28:0/2442045826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c041070 con 0x7f3280104fc0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f327dffb640 1 -- 172.21.15.28:0/2442045826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c004450 con 0x7f3280104fc0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f328019fce0 con 0x7f3280104fc0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f327dffb640 1 -- 172.21.15.28:0/2442045826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c034440 con 0x7f3280104fc0 2024-03-22T02:23:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.096+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32801a01d0 con 0x7f3280104fc0 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.098+0000 7f327dffb640 1 -- 172.21.15.28:0/2442045826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f326c044070 con 0x7f3280104fc0 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 Processor -- start 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 -- start start 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f327dffb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 0x7f32700a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 0x5565c889a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c889b040 0x5565c889d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565c87fa040 con 0x5565c8895ed0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565c87a29e0 con 0x5565c889b040 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.099+0000 7f3289d78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565c87f5ff0 con 0x5565c88954e0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 0x5565c889a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3284e57640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 0x7f32700a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3285658640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3285658640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51158/0 (socket says 172.21.15.28:51158) 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3285658640 1 -- 172.21.15.28:0/398336134 learned_addr learned my addr 172.21.15.28:0/398336134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3285658640 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c889b040 0x5565c889d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327dffb640 1 -- 172.21.15.28:0/2442045826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f326c0f1260 con 0x7f3280104fc0 2024-03-22T02:23:32.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 -- 172.21.15.28:0/398336134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 msgr2=0x5565c8895900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 -- 172.21.15.28:0/398336134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c889b040 msgr2=0x5565c889d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c889b040 0x5565c889d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 -- 172.21.15.28:0/398336134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c850e210 con 0x5565c8895ed0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3285658640 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f3284e57640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 0x7f32700a5500 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fc0 tx=0x7f3274005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f327ffff640 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 0x5565c889a9d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f326c0bd3c0 tx=0x7f326c0886b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eda9651461d968c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f326affd640 1 -- 172.21.15.28:0/398336134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c04b070 con 0x5565c8895ed0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f326affd640 1 -- 172.21.15.28:0/398336134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c004c80 con 0x5565c8895ed0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.100+0000 7f326affd640 1 -- 172.21.15.28:0/398336134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c0a80f0 con 0x5565c8895ed0 2024-03-22T02:23:32.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 -- 172.21.15.28:0/398336134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 msgr2=0x5565c889a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 0x5565c889a9d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f326c0bd3c0 tx=0x7f326c0886b0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 -- 172.21.15.28:0/398336134 shutdown_connections 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c889b040 0x5565c889d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8895ed0 0x5565c889a9d0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 --2- 172.21.15.28:0/398336134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c8895900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 -- 172.21.15.28:0/398336134 >> 172.21.15.28:0/398336134 conn(0x5565c8890a50 msgr2=0x5565c8890e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 -- 172.21.15.28:0/398336134 shutdown_connections 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.101+0000 7f3289d78740 1 -- 172.21.15.28:0/398336134 wait complete. 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 -- start start 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 Processor -- start 2024-03-22T02:23:32.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c889f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c889f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565c889ddb0 con 0x5565c889b040 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565c87b5db0 con 0x5565c8895ed0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565c889df30 con 0x5565c88954e0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3289d78740 10 client.0 _ll_register_callbacks cb 0x5565c850da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3284e57640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c889f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f327ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c889f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49578/0 (socket says 172.21.15.28:49578) 2024-03-22T02:23:32.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 -- 172.21.15.28:0/2255125712 learned_addr learned my addr 172.21.15.28:0/2255125712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 -- 172.21.15.28:0/2255125712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 msgr2=0x5565c889f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c889f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 -- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 msgr2=0x5565c889f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c889f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.102+0000 7f3285658640 1 -- 172.21.15.28:0/2255125712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c85da620 con 0x5565c889b040 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3285658640 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f3278003e40 tx=0x7f3278009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ce024186b5e763 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327801f440 con 0x5565c889b040 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f327801e3e0 con 0x5565c889b040 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f326c004c80 con 0x5565c889b040 2024-03-22T02:23:32.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f327801f5e0 con 0x5565c889b040 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 10 client.18336 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5565c88a9b60 con 0x5565c889b040 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f326c004c80 con 0x5565c889b040 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f327801d070 con 0x5565c889b040 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 10 client.18336 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 10 client.18336 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 20 client.18336 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c88ab2f0 0x5565c88ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 4 client.18336 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.103+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565c8807580 con 0x5565c88ab2f0 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.104+0000 7f3284e57640 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c88ab2f0 0x5565c88ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.104+0000 7f325ffff640 10 client.18336 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.104+0000 7f3284e57640 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c88ab2f0 0x5565c88ad770 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec7b9ff29e99184e server_cookie=53bbf59a63046aa6 in_seq=0 out_seq=0 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f325ffff640 1 -- 172.21.15.28:0/2255125712 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565c8807580 con 0x5565c88ab2f0 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f325ffff640 10 client.18336 handle_command_reply: tid=0 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 client.18336 shutdown 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 msgr2=0x5565c88a6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f3278003e40 tx=0x7f3278009f40 comp rx=0 tx=0).stop 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 shutdown_connections 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c88ab2f0 0x5565c88ad770 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c889b040 0x5565c88a6780 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c889f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f325ffff640 0 client.18336 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 --2- 172.21.15.28:0/2255125712 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c889f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 >> 172.21.15.28:0/2255125712 conn(0x5565c8890a50 msgr2=0x5565c888fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 shutdown_connections 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 1 -- 172.21.15.28:0/2255125712 wait complete. 2024-03-22T02:23:32.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.105+0000 7f3289d78740 20 client.18336 trim_cache size 0 max 16384 2024-03-22T02:23:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 Processor -- start 2024-03-22T02:23:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 -- start start 2024-03-22T02:23:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c88a6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c888f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32801aafe0 con 0x5565c8890a50 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f32801a88b0 con 0x5565c8895ed0 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3289d78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32801a6700 con 0x5565c88954e0 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f327ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c88a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3285658640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c888f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49584/0 (socket says 172.21.15.28:49584) 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 -- 172.21.15.28:0/2017019182 learned_addr learned my addr 172.21.15.28:0/2017019182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 -- 172.21.15.28:0/2017019182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 msgr2=0x5565c88a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c88a6740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 -- 172.21.15.28:0/2017019182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 msgr2=0x5565c888f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c888f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3284e57640 1 -- 172.21.15.28:0/2017019182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565c88addf0 con 0x5565c8890a50 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.121+0000 7f3285658640 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c888f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3284e57640 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x5565c88919c0 tx=0x7f3274014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44aaa72bace0e21b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f326a7fc640 1 -- 172.21.15.28:0/2017019182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3274029250 con 0x5565c8890a50 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f326a7fc640 1 -- 172.21.15.28:0/2017019182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32740041e0 con 0x5565c8890a50 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f326a7fc640 1 -- 172.21.15.28:0/2017019182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f327400fe40 con 0x5565c8890a50 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 -- 172.21.15.28:0/2017019182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 msgr2=0x5565c888f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x5565c88919c0 tx=0x7f3274014000 comp rx=0 tx=0).stop 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 -- 172.21.15.28:0/2017019182 shutdown_connections 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c888f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c88954e0 0x5565c88a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 --2- 172.21.15.28:0/2017019182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c8890a50 0x5565c888f3c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.122+0000 7f3289d78740 1 -- 172.21.15.28:0/2017019182 >> 172.21.15.28:0/2017019182 conn(0x5565c889ab40 msgr2=0x5565c8894540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- 172.21.15.28:0/2017019182 shutdown_connections 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- 172.21.15.28:0/2017019182 wait complete. 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- start start 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 Processor -- start 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c8890a50 0x5565c88a3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c88a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f32801a6d70 con 0x5565c88954e0 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f32801a81e0 con 0x5565c8895ed0 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f32801a7440 con 0x5565c8890a50 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3289d78740 10 client.0 _ll_register_callbacks cb 0x7f3278003d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:32.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f327ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f327ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49586/0 (socket says 172.21.15.28:49586) 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3284e57640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c8890a50 0x5565c88a3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f327ffff640 1 -- 172.21.15.28:0/2813628477 learned_addr learned my addr 172.21.15.28:0/2813628477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.123+0000 7f3285658640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c88a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 -- 172.21.15.28:0/2813628477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c8890a50 msgr2=0x5565c88a3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c8890a50 0x5565c88a3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 -- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 msgr2=0x5565c88a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c88a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 -- 172.21.15.28:0/2813628477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565c88917c0 con 0x5565c88954e0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f327ffff640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f326c0bd650 tx=0x7f326c0bd200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77eba5e84008428 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:32.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c04b040 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f326c0035f0 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32740041e0 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f326c0b9790 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 10 client.18342 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5565c89116b0 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32740041e0 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f326c0dd070 con 0x5565c88954e0 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 10 client.18342 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 10 client.18342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 20 client.18342 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c889b040 0x5565c8913d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:32.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 4 client.18342 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:23:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.124+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565c88018d0 con 0x5565c889b040 2024-03-22T02:23:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.125+0000 7f3284e57640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c889b040 0x5565c8913d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.125+0000 7f3284e57640 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c889b040 0x5565c8913d50 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bc95c78b2881b23 server_cookie=457cf7a788db9acb in_seq=0 out_seq=0 2024-03-22T02:23:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.125+0000 7f325ffff640 10 client.18342 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:32.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 987791 11.51 MB/sec execute 374 sec latency 195.604 ms 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.447+0000 7f325ffff640 1 -- 172.21.15.28:0/2813628477 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565c88018d0 con 0x5565c889b040 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.447+0000 7f325ffff640 10 client.18342 handle_command_reply: tid=0 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.447+0000 7f3289d78740 1 client.18342 shutdown 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 msgr2=0x5565c88a4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f326c0bd650 tx=0x7f326c0bd200 comp rx=0 tx=0).stop 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 shutdown_connections 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565c889b040 0x5565c8913d50 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565c8895ed0 0x5565c88a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.449 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565c88954e0 0x5565c88a4480 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f325ffff640 0 client.18342 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 --2- 172.21.15.28:0/2813628477 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565c8890a50 0x5565c88a3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 >> 172.21.15.28:0/2813628477 conn(0x5565c889ab40 msgr2=0x5565c8896970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 shutdown_connections 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 1 -- 172.21.15.28:0/2813628477 wait complete. 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.448+0000 7f3289d78740 20 client.18342 trim_cache size 0 max 16384 2024-03-22T02:23:32.450 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:23:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:23:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:23:32.451 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 msgr2=0x7f32700a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 0x7f32700a5500 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fc0 tx=0x7f3274005f90 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 msgr2=0x7f3280198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f326c000c00 tx=0x7f326c02c640 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 shutdown_connections 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f32700a3040 0x7f32700a5500 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32801078f0 0x7f328019f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3280104fc0 0x7f3280198370 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 --2- 172.21.15.28:0/2442045826 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3280102690 0x7f3280197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:32.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 >> 172.21.15.28:0/2442045826 conn(0x7f32800fc020 msgr2=0x7f3280102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 shutdown_connections 2024-03-22T02:23:32.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:32.449+0000 7f32870bb640 1 -- 172.21.15.28:0/2442045826 wait complete. 2024-03-22T02:23:32.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:32 smithi028 ceph-mon[31916]: pgmap v658: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-22T02:23:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:32 smithi120 ceph-mon[36803]: pgmap v658: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-22T02:23:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:32 smithi083 ceph-mon[36340]: pgmap v658: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 383 op/s 2024-03-22T02:23:33.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 989784 11.49 MB/sec execute 375 sec latency 28.105 ms 2024-03-22T02:23:34.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 991833 11.50 MB/sec execute 376 sec latency 31.225 ms 2024-03-22T02:23:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:34 smithi120 ceph-mon[36803]: pgmap v659: 129 pgs: 129 active+clean; 60 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:23:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:34 smithi083 ceph-mon[36340]: pgmap v659: 129 pgs: 129 active+clean; 60 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:23:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:34 smithi028 ceph-mon[31916]: pgmap v659: 129 pgs: 129 active+clean; 60 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:23:35.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 994594 11.51 MB/sec execute 377 sec latency 37.181 ms 2024-03-22T02:23:36.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 995661 11.49 MB/sec execute 378 sec latency 97.693 ms 2024-03-22T02:23:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:36 smithi120 ceph-mon[36803]: pgmap v660: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 411 op/s 2024-03-22T02:23:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:36 smithi083 ceph-mon[36340]: pgmap v660: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 411 op/s 2024-03-22T02:23:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:36 smithi028 ceph-mon[31916]: pgmap v660: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 4.3 MiB/s wr, 411 op/s 2024-03-22T02:23:37.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 998406 11.51 MB/sec execute 379 sec latency 25.122 ms 2024-03-22T02:23:38.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1000184 11.49 MB/sec execute 380 sec latency 46.225 ms 2024-03-22T02:23:38.677 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:38 smithi083 ceph-mon[36340]: pgmap v661: 129 pgs: 129 active+clean; 53 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 314 op/s 2024-03-22T02:23:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:38 smithi120 ceph-mon[36803]: pgmap v661: 129 pgs: 129 active+clean; 53 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 314 op/s 2024-03-22T02:23:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:38 smithi028 ceph-mon[31916]: pgmap v661: 129 pgs: 129 active+clean; 53 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 314 op/s 2024-03-22T02:23:39.051 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:23:39.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- 172.21.15.28:0/610925858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174100970 msgr2=0x7fb174100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- 172.21.15.28:0/610925858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174100970 0x7fb174100d90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb170009900 tx=0x7fb17002bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- 172.21.15.28:0/610925858 shutdown_connections 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- 172.21.15.28:0/610925858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174102520 0x7fb174108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- 172.21.15.28:0/610925858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 0x7fb174101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- 172.21.15.28:0/610925858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174100970 0x7fb174100d90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- 172.21.15.28:0/610925858 >> 172.21.15.28:0/610925858 conn(0x7fb1740fbf00 msgr2=0x7fb1740fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- 172.21.15.28:0/610925858 shutdown_connections 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- 172.21.15.28:0/610925858 wait complete. 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 Processor -- start 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- start start 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174100970 0x7fb174197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 0x7fb1741983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb17410c5e0 con 0x7fb174102520 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb17410c460 con 0x7fb174100970 2024-03-22T02:23:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.160+0000 7fb17a980640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb17410c760 con 0x7fb174101b60 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49608/0 (socket says 172.21.15.28:49608) 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 -- 172.21.15.28:0/3941521648 learned_addr learned my addr 172.21.15.28:0/3941521648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17997e640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174100970 0x7fb174197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17917d640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 0x7fb1741983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 -- 172.21.15.28:0/3941521648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 msgr2=0x7fb1741983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 0x7fb1741983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 -- 172.21.15.28:0/3941521648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174100970 msgr2=0x7fb174197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174100970 0x7fb174197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 -- 172.21.15.28:0/3941521648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17419fa30 con 0x7fb174102520 2024-03-22T02:23:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a17f640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00efd0 tx=0x7fb16c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb166ffd640 1 -- 172.21.15.28:0/3941521648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c003b70 con 0x7fb174102520 2024-03-22T02:23:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb166ffd640 1 -- 172.21.15.28:0/3941521648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16c003d10 con 0x7fb174102520 2024-03-22T02:23:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17419fd20 con 0x7fb174102520 2024-03-22T02:23:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb166ffd640 1 -- 172.21.15.28:0/3941521648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0109f0 con 0x7fb174102520 2024-03-22T02:23:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.161+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1741a0210 con 0x7fb174102520 2024-03-22T02:23:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.163+0000 7fb166ffd640 1 -- 172.21.15.28:0/3941521648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb16c004020 con 0x7fb174102520 2024-03-22T02:23:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.164+0000 7fb166ffd640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 0x7fb1600a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.164+0000 7fb17e7c8740 1 Processor -- start 2024-03-22T02:23:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.164+0000 7fb166ffd640 1 -- 172.21.15.28:0/3941521648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb16c0cfb90 con 0x7fb174102520 2024-03-22T02:23:39.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.164+0000 7fb17997e640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 0x7fb1600a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.164+0000 7fb17e7c8740 1 -- start start 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeafda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 0x55ddaeb02b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddaea628c0 con 0x55ddaeafd620 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ddaea0ac20 con 0x55ddaeb03180 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddaea5e870 con 0x55ddaeafe010 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17917d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 0x55ddaeb02b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51632/0 (socket says 172.21.15.28:51632) 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17917d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 0x55ddaeb02b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51228/0 (socket says 172.21.15.28:51228) 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeafda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 -- 172.21.15.28:0/332070391 learned_addr learned my addr 172.21.15.28:0/332070391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17997e640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 0x7fb1600a5460 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb170000c00 tx=0x7fb170005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 -- 172.21.15.28:0/332070391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 msgr2=0x55ddaeb02b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 0x55ddaeb02b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 -- 172.21.15.28:0/332070391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 msgr2=0x55ddaeafda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeafda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 -- 172.21.15.28:0/332070391 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddaea2f150 con 0x55ddaeb03180 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.165+0000 7fb17a17f640 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0a4390 tx=0x7fb16c0104c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b87c40a2e935b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb15ffff640 1 -- 172.21.15.28:0/332070391 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c0bc0c0 con 0x55ddaeb03180 2024-03-22T02:23:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb15ffff640 1 -- 172.21.15.28:0/332070391 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16c09beb0 con 0x55ddaeb03180 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb15ffff640 1 -- 172.21.15.28:0/332070391 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16c004310 con 0x55ddaeb03180 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 -- 172.21.15.28:0/332070391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 msgr2=0x55ddaeb05610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb16c0a4390 tx=0x7fb16c0104c0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 -- 172.21.15.28:0/332070391 shutdown_connections 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb05610 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafe010 0x55ddaeb02b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/332070391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeafda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 -- 172.21.15.28:0/332070391 >> 172.21.15.28:0/332070391 conn(0x55ddaeaf8b90 msgr2=0x55ddaeaf8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 -- 172.21.15.28:0/332070391 shutdown_connections 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.166+0000 7fb17e7c8740 1 -- 172.21.15.28:0/332070391 wait complete. 2024-03-22T02:23:39.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 -- start start 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 Processor -- start 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddaeb060d0 con 0x55ddaeafe010 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ddaea1e130 con 0x55ddaeafd620 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ddaeb06250 con 0x55ddaeb03180 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17997e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49626/0 (socket says 172.21.15.28:49626) 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17917d640 1 -- 172.21.15.28:0/487333713 learned_addr learned my addr 172.21.15.28:0/487333713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17997e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51642/0 (socket says 172.21.15.28:51642) 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.167+0000 7fb17e7c8740 10 client.0 _ll_register_callbacks cb 0x55ddaea1db00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 -- 172.21.15.28:0/487333713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 msgr2=0x55ddaeb0f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 -- 172.21.15.28:0/487333713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 msgr2=0x55ddaeb07fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 -- 172.21.15.28:0/487333713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddae8407d0 con 0x55ddaeafd620 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17917d640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:39.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17997e640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb17002c170 tx=0x7fb170009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71c5655465eb44df server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb15d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700046c0 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb16c09beb0 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb170004860 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb170004a60 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17e7c8740 10 client.28175 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ddae7763f0 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.168+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb16c09beb0 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb170034440 con 0x55ddaeafd620 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 10 client.28175 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 10 client.28175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 20 client.28175 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb13870 0x55ddaeb15d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 4 client.28175 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddaea6fe00 con 0x55ddaeb13870 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17917d640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb13870 0x55ddaeb15d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb17917d640 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb13870 0x55ddaeb15d00 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3ec0fa32e55b14 server_cookie=abffc0e797a6f709 in_seq=0 out_seq=0 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.169+0000 7fb15d7fa640 10 client.28175 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.170+0000 7fb15d7fa640 1 -- 172.21.15.28:0/487333713 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddaea6fe00 con 0x55ddaeb13870 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.170+0000 7fb15d7fa640 10 client.28175 handle_command_reply: tid=0 2024-03-22T02:23:39.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.170+0000 7fb17e7c8740 1 client.28175 shutdown 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 msgr2=0x55ddaeb07aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb17002c170 tx=0x7fb170009260 comp rx=0 tx=0).stop 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 shutdown_connections 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb13870 0x55ddaeb15d00 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb07fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb15d7fa640 0 client.28175 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/487333713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafd620 0x55ddaeb07aa0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 >> 172.21.15.28:0/487333713 conn(0x55ddaeaf8b90 msgr2=0x55ddaeaf7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 shutdown_connections 2024-03-22T02:23:39.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 1 -- 172.21.15.28:0/487333713 wait complete. 2024-03-22T02:23:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.171+0000 7fb17e7c8740 20 client.28175 trim_cache size 0 max 16384 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.189+0000 7fb17e7c8740 1 Processor -- start 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.189+0000 7fb17e7c8740 1 -- start start 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafd620 0x55ddaeb076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb07bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741ad140 con 0x55ddaeafe010 2024-03-22T02:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741aaa10 con 0x55ddaeb03180 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741a8860 con 0x55ddaeafd620 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49636/0 (socket says 172.21.15.28:49636) 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17997e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafd620 0x55ddaeb076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 -- 172.21.15.28:0/2529563513 learned_addr learned my addr 172.21.15.28:0/2529563513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17a17f640 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb07bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 -- 172.21.15.28:0/2529563513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafd620 msgr2=0x55ddaeb076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafd620 0x55ddaeb076b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 -- 172.21.15.28:0/2529563513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 msgr2=0x55ddaeb07bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb07bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 -- 172.21.15.28:0/2529563513 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddae7360f0 con 0x55ddaeafe010 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17a17f640 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb07bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb17917d640 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x55ddaeb16540 tx=0x7fb168005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90b59a72c2b19dc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb15f7fe640 1 -- 172.21.15.28:0/2529563513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168004310 con 0x55ddaeafe010 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb15f7fe640 1 -- 172.21.15.28:0/2529563513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1680044b0 con 0x55ddaeafe010 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.190+0000 7fb15f7fe640 1 -- 172.21.15.28:0/2529563513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb168003e10 con 0x55ddaeafe010 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- 172.21.15.28:0/2529563513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 msgr2=0x55ddaeb0efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x55ddaeb16540 tx=0x7fb168005f90 comp rx=0 tx=0).stop 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- 172.21.15.28:0/2529563513 shutdown_connections 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeb03180 0x55ddaeb07bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafe010 0x55ddaeb0efe0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/2529563513 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeafd620 0x55ddaeb076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- 172.21.15.28:0/2529563513 >> 172.21.15.28:0/2529563513 conn(0x55ddaeaf8b90 msgr2=0x55ddaeaf8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- 172.21.15.28:0/2529563513 shutdown_connections 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- 172.21.15.28:0/2529563513 wait complete. 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.191+0000 7fb17e7c8740 1 -- start start 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 Processor -- start 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafe010 0x55ddaeb0c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741a8ed0 con 0x55ddaeafd620 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741aa340 con 0x55ddaeafe010 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1741a95a0 con 0x55ddaeb03180 2024-03-22T02:23:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17e7c8740 10 client.0 _ll_register_callbacks cb 0x7fb170004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17917d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafe010 0x55ddaeb0c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49648/0 (socket says 172.21.15.28:49648) 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17a17f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51250/0 (socket says 172.21.15.28:51250) 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 -- 172.21.15.28:0/174655052 learned_addr learned my addr 172.21.15.28:0/174655052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 -- 172.21.15.28:0/174655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 msgr2=0x55ddaeb0d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 -- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafe010 msgr2=0x55ddaeb0c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafe010 0x55ddaeb0c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17997e640 1 -- 172.21.15.28:0/174655052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddaeb0df40 con 0x55ddaeafd620 2024-03-22T02:23:39.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.192+0000 7fb17a17f640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17997e640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb170004650 tx=0x7fb170004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d22459ab58731d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb170034640 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb170005410 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1680044b0 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb170005610 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 10 client.18360 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ddaeb0f250 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1680044b0 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb170003b60 con 0x55ddaeafd620 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 10 client.18360 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 10 client.18360 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:39.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 20 client.18360 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb7a150 0x55ddaeb7c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 4 client.18360 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.193+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ddaea6a150 con 0x55ddaeb7a150 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.194+0000 7fb17917d640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb7a150 0x55ddaeb7c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.194+0000 7fb15d7fa640 10 client.18360 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.194+0000 7fb17917d640 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb7a150 0x55ddaeb7c570 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae3f34a83639f07c server_cookie=dbb7e2331beb2f41 in_seq=0 out_seq=0 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb15d7fa640 1 -- 172.21.15.28:0/174655052 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ddaea6a150 con 0x55ddaeb7a150 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb15d7fa640 10 client.18360 handle_command_reply: tid=0 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 client.18360 shutdown 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 msgr2=0x55ddaeb0c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fb170004650 tx=0x7fb170004970 comp rx=0 tx=0).stop 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 shutdown_connections 2024-03-22T02:23:39.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ddaeb7a150 0x55ddaeb7c570 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ddaeb03180 0x55ddaeb0d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ddaeafe010 0x55ddaeb0c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb15d7fa640 0 client.18360 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 --2- 172.21.15.28:0/174655052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddaeafd620 0x55ddaeb0c400 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 >> 172.21.15.28:0/174655052 conn(0x55ddaeaf8b90 msgr2=0x55ddaeb076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 shutdown_connections 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 1 -- 172.21.15.28:0/174655052 wait complete. 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17e7c8740 20 client.18360 trim_cache size 0 max 16384 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:23:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:23:39.204 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:39.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 msgr2=0x7fb1600a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 0x7fb1600a5460 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb170000c00 tx=0x7fb170005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 msgr2=0x7fb17419f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.195+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00efd0 tx=0x7fb16c00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 shutdown_connections 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1600a2fa0 0x7fb1600a5460 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb174102520 0x7fb17419f3c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb174101b60 0x7fb1741983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 --2- 172.21.15.28:0/3941521648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb174100970 0x7fb174197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 >> 172.21.15.28:0/3941521648 conn(0x7fb1740fbf00 msgr2=0x7fb1740fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 shutdown_connections 2024-03-22T02:23:39.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.196+0000 7fb17a980640 1 -- 172.21.15.28:0/3941521648 wait complete. 2024-03-22T02:23:39.207 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:23:39.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 -- 172.21.15.28:0/1516504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764101ca0 msgr2=0x7fd764102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 --2- 172.21.15.28:0/1516504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764101ca0 0x7fd764102120 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd758009900 tx=0x7fd75802bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 -- 172.21.15.28:0/1516504428 shutdown_connections 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 --2- 172.21.15.28:0/1516504428 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764102660 0x7fd764108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 --2- 172.21.15.28:0/1516504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764101ca0 0x7fd764102120 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 --2- 172.21.15.28:0/1516504428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764100ab0 0x7fd764100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 -- 172.21.15.28:0/1516504428 >> 172.21.15.28:0/1516504428 conn(0x7fd7640fc040 msgr2=0x7fd7640fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 -- 172.21.15.28:0/1516504428 shutdown_connections 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 -- 172.21.15.28:0/1516504428 wait complete. 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.311+0000 7fd76c532640 1 Processor -- start 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 -- start start 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764101ca0 0x7fd76419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764102660 0x7fd76419f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c730 con 0x7fd764100ab0 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c5b0 con 0x7fd764101ca0 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd76410c8b0 con 0x7fd764102660 2024-03-22T02:23:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49662/0 (socket says 172.21.15.28:49662) 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 -- 172.21.15.28:0/1326019668 learned_addr learned my addr 172.21.15.28:0/1326019668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd769acd640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764101ca0 0x7fd76419cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76aacf640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764102660 0x7fd76419f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 -- 172.21.15.28:0/1326019668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764102660 msgr2=0x7fd76419f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764102660 0x7fd76419f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 -- 172.21.15.28:0/1326019668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764101ca0 msgr2=0x7fd76419cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764101ca0 0x7fd76419cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 -- 172.21.15.28:0/1326019668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76419fb20 con 0x7fd764100ab0 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76a2ce640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00e9c0 tx=0x7fd75c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00cd80 con 0x7fd764100ab0 2024-03-22T02:23:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd76419fdb0 con 0x7fd764100ab0 2024-03-22T02:23:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.312+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd75c0035f0 con 0x7fd764100ab0 2024-03-22T02:23:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.313+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd75c010690 con 0x7fd764100ab0 2024-03-22T02:23:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.313+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7641a01c0 con 0x7fd764100ab0 2024-03-22T02:23:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.314+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd764107e20 con 0x7fd764100ab0 2024-03-22T02:23:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.315+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd75c0037a0 con 0x7fd764100ab0 2024-03-22T02:23:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.315+0000 7fd7637fe640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 0x7fd74c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.315+0000 7fd769acd640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 0x7fd74c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.315+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd75c0cfb60 con 0x7fd764100ab0 2024-03-22T02:23:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.316+0000 7fd769acd640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 0x7fd74c0a53e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd75802c7b0 tx=0x7fd758005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.320+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd75c09c580 con 0x7fd764100ab0 2024-03-22T02:23:39.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1002529 11.50 MB/sec execute 381 sec latency 30.433 ms 2024-03-22T02:23:39.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.559+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd764062860 con 0x7fd764100ab0 2024-03-22T02:23:39.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.560+0000 7fd7637fe640 1 -- 172.21.15.28:0/1326019668 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11867 (secure 0 0 0) 0x7fd75c09be80 con 0x7fd764100ab0 2024-03-22T02:23:39.561 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:23:39.561 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:22:55.371454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:23:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 msgr2=0x7fd74c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 0x7fd74c0a53e0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd75802c7b0 tx=0x7fd758005fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 msgr2=0x7fd76406ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fd75c00e9c0 tx=0x7fd75c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 shutdown_connections 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd74c0a2f20 0x7fd74c0a53e0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd764102660 0x7fd76419f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd764101ca0 0x7fd76419cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 --2- 172.21.15.28:0/1326019668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd764100ab0 0x7fd76406ad90 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 >> 172.21.15.28:0/1326019668 conn(0x7fd7640fc040 msgr2=0x7fd764104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 shutdown_connections 2024-03-22T02:23:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.563+0000 7fd76c532640 1 -- 172.21.15.28:0/1326019668 wait complete. 2024-03-22T02:23:39.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- 172.21.15.28:0/1665764006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc103770 msgr2=0x7fe9fc10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 --2- 172.21.15.28:0/1665764006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc103770 0x7fe9fc10a000 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009900 tx=0x7fe9f402bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- 172.21.15.28:0/1665764006 shutdown_connections 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 --2- 172.21.15.28:0/1665764006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc103770 0x7fe9fc10a000 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 --2- 172.21.15.28:0/1665764006 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 0x7fe9fc103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 --2- 172.21.15.28:0/1665764006 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- 172.21.15.28:0/1665764006 >> 172.21.15.28:0/1665764006 conn(0x7fe9fc0fd130 msgr2=0x7fe9fc0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- 172.21.15.28:0/1665764006 shutdown_connections 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- 172.21.15.28:0/1665764006 wait complete. 2024-03-22T02:23:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 Processor -- start 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.689+0000 7fea0445c640 1 -- start start 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 0x7fe9fc1983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc103770 0x7fe9fc19f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10d840 con 0x7fe9fc101bc0 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10d6c0 con 0x7fe9fc102db0 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10d9c0 con 0x7fe9fc103770 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49672/0 (socket says 172.21.15.28:49672) 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 -- 172.21.15.28:0/3600240568 learned_addr learned my addr 172.21.15.28:0/3600240568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea019f7640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 0x7fe9fc1983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea029f9640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc103770 0x7fe9fc19f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 -- 172.21.15.28:0/3600240568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc103770 msgr2=0x7fe9fc19f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc103770 0x7fe9fc19f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 -- 172.21.15.28:0/3600240568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 msgr2=0x7fe9fc1983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 0x7fe9fc1983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 -- 172.21.15.28:0/3600240568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc19faa0 con 0x7fe9fc101bc0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea021f8640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800eb70 tx=0x7fe9f800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fe9f37fe640 1 -- 172.21.15.28:0/3600240568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8003b70 con 0x7fe9fc101bc0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fe9f37fe640 1 -- 172.21.15.28:0/3600240568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f800ee70 con 0x7fe9fc101bc0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc19fd90 con 0x7fe9fc101bc0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fe9f37fe640 1 -- 172.21.15.28:0/3600240568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8010830 con 0x7fe9fc101bc0 2024-03-22T02:23:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.690+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc1a01d0 con 0x7fe9fc101bc0 2024-03-22T02:23:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.693+0000 7fe9f37fe640 1 -- 172.21.15.28:0/3600240568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe9f8004020 con 0x7fe9fc101bc0 2024-03-22T02:23:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.693+0000 7fea07041740 1 Processor -- start 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.693+0000 7fea07041740 1 -- start start 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.693+0000 7fe9f37fe640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 0x7fe9e80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22d5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eea22328d0 con 0x55eea22d3380 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eea21dae00 con 0x55eea22cd820 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea07041740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eea222e880 con 0x55eea22ce210 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea029f9640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 0x7fe9e80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fe9f37fe640 1 -- 172.21.15.28:0/3600240568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe9f80cff10 con 0x7fe9fc101bc0 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea029f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22d5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea021f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea029f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22d5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49676/0 (socket says 172.21.15.28:49676) 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51288/0 (socket says 172.21.15.28:51288) 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea029f9640 1 -- 172.21.15.28:0/1836319112 learned_addr learned my addr 172.21.15.28:0/1836319112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 -- 172.21.15.28:0/1836319112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 msgr2=0x55eea22cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 -- 172.21.15.28:0/1836319112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 msgr2=0x55eea22d5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22d5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 -- 172.21.15.28:0/1836319112 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea1f463f0 con 0x55eea22ce210 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea029f9640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 0x7fe9e80a53e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009230 tx=0x7fe9f402cc20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.694+0000 7fea019f7640 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009bf0 tx=0x7fe9ec011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10658eedb57f2b9f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fe9f0bfe640 1 -- 172.21.15.28:0/1836319112 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004050 con 0x55eea22ce210 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fe9f0bfe640 1 -- 172.21.15.28:0/1836319112 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec005500 con 0x55eea22ce210 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fe9f0bfe640 1 -- 172.21.15.28:0/1836319112 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec004dd0 con 0x55eea22ce210 2024-03-22T02:23:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 -- 172.21.15.28:0/1836319112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 msgr2=0x55eea22d2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009bf0 tx=0x7fe9ec011810 comp rx=0 tx=0).stop 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 -- 172.21.15.28:0/1836319112 shutdown_connections 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22d5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d2d10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 --2- 172.21.15.28:0/1836319112 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.695+0000 7fea07041740 1 -- 172.21.15.28:0/1836319112 >> 172.21.15.28:0/1836319112 conn(0x55eea22c8d90 msgr2=0x55eea22c9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- 172.21.15.28:0/1836319112 shutdown_connections 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- 172.21.15.28:0/1836319112 wait complete. 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- start start 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 Processor -- start 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22d7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eea22d6330 con 0x55eea22d3380 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eea21ee300 con 0x55eea22cd820 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eea22d64b0 con 0x55eea22ce210 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea07041740 10 client.0 _ll_register_callbacks cb 0x55eea2229e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea029f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea021f8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22d7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea029f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49682/0 (socket says 172.21.15.28:49682) 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea029f9640 1 -- 172.21.15.28:0/4217655233 learned_addr learned my addr 172.21.15.28:0/4217655233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.696+0000 7fea019f7640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 -- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 msgr2=0x55eea22d8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 -- 172.21.15.28:0/4217655233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 msgr2=0x55eea22d7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22d7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 -- 172.21.15.28:0/4217655233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea20107d0 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea029f9640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005270 tx=0x7fe9f402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ceaa0ef0c2d8375 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4051020 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe9f4004ad0 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9ec005500 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe9f405c050 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea07041740 10 client.18378 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55eea22e2720 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9ec005500 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fe9f4042440 con 0x55eea22d3380 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.697+0000 7fea07041740 10 client.18378 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea07041740 10 client.18378 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:23:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea07041740 20 client.18378 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea22e3eb0 0x55eea22e6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea07041740 4 client.18378 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eea223fe10 con 0x55eea22e3eb0 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea021f8640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea22e3eb0 0x55eea22e6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fe9e5ffb640 10 client.18378 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.698+0000 7fea021f8640 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea22e3eb0 0x55eea22e6370 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=578b64ef22f31c21 server_cookie=a02f21e26bf922d3 in_seq=0 out_seq=0 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/4217655233 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eea223fe10 con 0x55eea22e3eb0 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fe9e5ffb640 10 client.18378 handle_command_reply: tid=0 2024-03-22T02:23:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 client.18378 shutdown 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 msgr2=0x55eea22df340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4005270 tx=0x7fe9f402bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 shutdown_connections 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22d3380 0x55eea22df340 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22ce210 0x55eea22d8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea22e3eb0 0x55eea22e6370 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 --2- 172.21.15.28:0/4217655233 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22cd820 0x55eea22d7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fe9e5ffb640 0 client.18378 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 >> 172.21.15.28:0/4217655233 conn(0x55eea22c8d90 msgr2=0x55eea22c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 shutdown_connections 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 1 -- 172.21.15.28:0/4217655233 wait complete. 2024-03-22T02:23:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.699+0000 7fea07041740 20 client.18378 trim_cache size 0 max 16384 2024-03-22T02:23:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 Processor -- start 2024-03-22T02:23:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 -- start start 2024-03-22T02:23:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22df300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1ad1b0 con 0x55eea22cd820 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1aaa80 con 0x55eea22ce210 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea07041740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1a88d0 con 0x55eea22d3380 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49688/0 (socket says 172.21.15.28:49688) 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 -- 172.21.15.28:0/415946885 learned_addr learned my addr 172.21.15.28:0/415946885 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea019f7640 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22df300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea029f9640 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 -- 172.21.15.28:0/415946885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 msgr2=0x55eea22c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 -- 172.21.15.28:0/415946885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 msgr2=0x55eea22df300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22df300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.713+0000 7fea021f8640 1 -- 172.21.15.28:0/415946885 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eea22d65e0 con 0x55eea22cd820 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea021f8640 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8098690 tx=0x7fe9f80a0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b0b1ea5c1f259b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fe9e7fff640 1 -- 172.21.15.28:0/415946885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80a4c40 con 0x55eea22cd820 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fe9e7fff640 1 -- 172.21.15.28:0/415946885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f809a2d0 con 0x55eea22cd820 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fe9e7fff640 1 -- 172.21.15.28:0/415946885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8004310 con 0x55eea22cd820 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 -- 172.21.15.28:0/415946885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 msgr2=0x55eea22c7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8098690 tx=0x7fe9f80a0040 comp rx=0 tx=0).stop 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 -- 172.21.15.28:0/415946885 shutdown_connections 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22df300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 --2- 172.21.15.28:0/415946885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22c7740 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.714+0000 7fea07041740 1 -- 172.21.15.28:0/415946885 >> 172.21.15.28:0/415946885 conn(0x55eea22c8d90 msgr2=0x55eea22d6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- 172.21.15.28:0/415946885 shutdown_connections 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- 172.21.15.28:0/415946885 wait complete. 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- start start 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 Processor -- start 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22dd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22de640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1a8f40 con 0x55eea22cd820 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1aa3b0 con 0x55eea22ce210 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc1a9610 con 0x55eea22d3380 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea07041740 10 client.0 _ll_register_callbacks cb 0x7fe9f80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49696/0 (socket says 172.21.15.28:49696) 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea019f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 -- 172.21.15.28:0/597899870 learned_addr learned my addr 172.21.15.28:0/597899870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea029f9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22de640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 -- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 msgr2=0x55eea22de640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22de640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 -- 172.21.15.28:0/597899870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 msgr2=0x55eea22dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.715+0000 7fea021f8640 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22dd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea021f8640 1 -- 172.21.15.28:0/597899870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eea22dec30 con 0x55eea22cd820 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea021f8640 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80cd970 tx=0x7fe9f809c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1263f688f80db31e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f80a4c40 con 0x55eea22cd820 2024-03-22T02:23:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fe9f8004310 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9f809a2d0 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fe9f809c490 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 10 client.18384 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55eea2353180 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9f809a2d0 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fe9f809b680 con 0x55eea22cd820 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 10 client.18384 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 10 client.18384 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 20 client.18384 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea2353a20 0x55eea2355e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 4 client.18384 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.716+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eea223a160 con 0x55eea2353a20 2024-03-22T02:23:39.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea019f7640 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea2353a20 0x55eea2355e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fe9e5ffb640 10 client.18384 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea019f7640 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea2353a20 0x55eea2355e40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f0484d0c747c612 server_cookie=272d7913981aa59f in_seq=0 out_seq=0 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fe9e5ffb640 1 -- 172.21.15.28:0/597899870 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eea223a160 con 0x55eea2353a20 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fe9e5ffb640 10 client.18384 handle_command_reply: tid=0 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 client.18384 shutdown 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 msgr2=0x55eea22dd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80cd970 tx=0x7fe9f809c3b0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 shutdown_connections 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eea22d3380 0x55eea22de640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55eea2353a20 0x55eea2355e40 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eea22ce210 0x55eea22dd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 --2- 172.21.15.28:0/597899870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eea22cd820 0x55eea22dd0f0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fe9e5ffb640 0 client.18384 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:23:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 >> 172.21.15.28:0/597899870 conn(0x55eea22c8d90 msgr2=0x55eea22cbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 shutdown_connections 2024-03-22T02:23:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 1 -- 172.21.15.28:0/597899870 wait complete. 2024-03-22T02:23:39.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.717+0000 7fea07041740 20 client.18384 trim_cache size 0 max 16384 2024-03-22T02:23:39.725 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 msgr2=0x7fe9e80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 0x7fe9e80a53e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe9f4009230 tx=0x7fe9f402cc20 comp rx=0 tx=0).stop 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 msgr2=0x7fe9fc197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800eb70 tx=0x7fe9f800c6a0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 shutdown_connections 2024-03-22T02:23:39.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9e80a2f20 0x7fe9e80a53e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9fc103770 0x7fe9fc19f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9fc102db0 0x7fe9fc1983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 --2- 172.21.15.28:0/3600240568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9fc101bc0 0x7fe9fc197e80 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 >> 172.21.15.28:0/3600240568 conn(0x7fe9fc0fd130 msgr2=0x7fe9fc1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 shutdown_connections 2024-03-22T02:23:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.718+0000 7fea0445c640 1 -- 172.21.15.28:0/3600240568 wait complete. 2024-03-22T02:23:39.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:23:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 -- 172.21.15.28:0/1621524822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 msgr2=0x7fee081073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 --2- 172.21.15.28:0/1621524822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee081073b0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009900 tx=0x7fedf802bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 -- 172.21.15.28:0/1621524822 shutdown_connections 2024-03-22T02:23:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 --2- 172.21.15.28:0/1621524822 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee081078f0 0x7fee08109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 --2- 172.21.15.28:0/1621524822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee081073b0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 --2- 172.21.15.28:0/1621524822 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee08102690 0x7fee08104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.858+0000 7fee0f36b640 1 -- 172.21.15.28:0/1621524822 >> 172.21.15.28:0/1621524822 conn(0x7fee080fc020 msgr2=0x7fee080fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- 172.21.15.28:0/1621524822 shutdown_connections 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- 172.21.15.28:0/1621524822 wait complete. 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 Processor -- start 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- start start 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee08198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee081078f0 0x7fee0819f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810d810 con 0x7fee08104fc0 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810d690 con 0x7fee081078f0 2024-03-22T02:23:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0f36b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee0810d990 con 0x7fee08102690 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.859+0000 7fee0d107640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51310/0 (socket says 172.21.15.28:51310) 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 -- 172.21.15.28:0/4286547857 learned_addr learned my addr 172.21.15.28:0/4286547857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0c906640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee08198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d908640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee081078f0 0x7fee0819f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 -- 172.21.15.28:0/4286547857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee081078f0 msgr2=0x7fee0819f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee081078f0 0x7fee0819f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 -- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 msgr2=0x7fee08198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee08198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 -- 172.21.15.28:0/4286547857 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0819fb30 con 0x7fee08102690 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0d107640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee00009f50 tx=0x7fee0000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fedfe7fc640 1 -- 172.21.15.28:0/4286547857 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00020070 con 0x7fee08102690 2024-03-22T02:23:39.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee081004e0 con 0x7fee08102690 2024-03-22T02:23:39.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fedfe7fc640 1 -- 172.21.15.28:0/4286547857 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00013d00 con 0x7fee08102690 2024-03-22T02:23:39.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fedfe7fc640 1 -- 172.21.15.28:0/4286547857 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0001bbb0 con 0x7fee08102690 2024-03-22T02:23:39.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.860+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee08100a50 con 0x7fee08102690 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fedfe7fc640 1 -- 172.21.15.28:0/4286547857 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fee00023070 con 0x7fee08102690 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 Processor -- start 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 -- start start 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f90d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f93810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621e2ef08d0 con 0x5621e2f8b820 2024-03-22T02:23:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621e2e98e00 con 0x5621e2f8c210 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.863+0000 7fee1202a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621e2eec880 con 0x5621e2f91380 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49728/0 (socket says 172.21.15.28:49728) 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 -- 172.21.15.28:0/3478391004 learned_addr learned my addr 172.21.15.28:0/3478391004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d107640 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f90d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0c906640 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f93810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fedfe7fc640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 0x7fedf40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 -- 172.21.15.28:0/3478391004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 msgr2=0x5621e2f93810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f93810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 -- 172.21.15.28:0/3478391004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 msgr2=0x5621e2f90d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f90d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 -- 172.21.15.28:0/3478391004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621e2c043f0 con 0x5621e2f8b820 2024-03-22T02:23:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fedfe7fc640 1 -- 172.21.15.28:0/4286547857 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fee000d3930 con 0x7fee08102690 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 0x7fedf40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fee0d908640 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fee04011590 tx=0x7fee04011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961a0d9db8822f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fedf37fe640 1 -- 172.21.15.28:0/3478391004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04017050 con 0x5621e2f8b820 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fedf37fe640 1 -- 172.21.15.28:0/3478391004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0400fd20 con 0x5621e2f8b820 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.864+0000 7fedf37fe640 1 -- 172.21.15.28:0/3478391004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0400ea80 con 0x5621e2f8b820 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- 172.21.15.28:0/3478391004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 msgr2=0x5621e2f8bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee0d908640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 0x7fedf40a53e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fee04022000 tx=0x7fee04023000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fee04011590 tx=0x7fee04011a50 comp rx=0 tx=0).stop 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- 172.21.15.28:0/3478391004 shutdown_connections 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f93810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f90d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 --2- 172.21.15.28:0/3478391004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8bc40 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- 172.21.15.28:0/3478391004 >> 172.21.15.28:0/3478391004 conn(0x5621e2f86d90 msgr2=0x5621e2f87190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- 172.21.15.28:0/3478391004 shutdown_connections 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- 172.21.15.28:0/3478391004 wait complete. 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.865+0000 7fee1202a740 1 -- start start 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 Processor -- start 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 0x5621e2f96250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 0x5621e2f9d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621e2f94330 con 0x5621e2f8b820 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621e2eac300 con 0x5621e2f91380 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621e2f944b0 con 0x5621e2f8c210 2024-03-22T02:23:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee1202a740 10 client.0 _ll_register_callbacks cb 0x5621e2ee7e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d107640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d107640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49736/0 (socket says 172.21.15.28:49736) 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d107640 1 -- 172.21.15.28:0/66376895 learned_addr learned my addr 172.21.15.28:0/66376895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 0x5621e2f96250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d908640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 0x5621e2f9d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 -- 172.21.15.28:0/66376895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 msgr2=0x5621e2f9d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 0x5621e2f9d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 -- 172.21.15.28:0/66376895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 msgr2=0x5621e2f95d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0c906640 1 -- 172.21.15.28:0/66376895 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621e2cce7d0 con 0x5621e2f8c210 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d908640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 0x5621e2f9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.866+0000 7fee0d107640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee0c906640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 0x5621e2f96250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fedf802c170 tx=0x7fedf8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83de849efc3205c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf8005d50 con 0x5621e2f8c210 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee0400fd20 con 0x5621e2f8c210 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fedf8005230 con 0x5621e2f8c210 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fedf804f050 con 0x5621e2f8c210 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 10 client.28210 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5621e2fa0670 con 0x5621e2f8c210 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee0400fd20 con 0x5621e2f8c210 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fedf8003950 con 0x5621e2f8c210 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 10 client.28210 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 10 client.28210 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 20 client.28210 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e2fa1d20 0x5621e2fa41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 4 client.28210 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.867+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621e2efde10 con 0x5621e2fa1d20 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.868+0000 7fee0d107640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e2fa1d20 0x5621e2fa41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.868+0000 7fedf0ff9640 10 client.28210 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.868+0000 7fee0d107640 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e2fa1d20 0x5621e2fa41e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a42577849f000f5d server_cookie=3031665035a337db in_seq=0 out_seq=0 2024-03-22T02:23:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fedf0ff9640 1 -- 172.21.15.28:0/66376895 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621e2efde10 con 0x5621e2fa1d20 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fedf0ff9640 10 client.28210 handle_command_reply: tid=0 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 client.28210 shutdown 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 msgr2=0x5621e2f96250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 0x5621e2f96250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fedf802c170 tx=0x7fedf8004270 comp rx=0 tx=0).stop 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 shutdown_connections 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e2fa1d20 0x5621e2fa41e0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f91380 0x5621e2f9d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f8c210 0x5621e2f96250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 --2- 172.21.15.28:0/66376895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f95d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fedf0ff9640 0 client.28210 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 >> 172.21.15.28:0/66376895 conn(0x5621e2f86d90 msgr2=0x5621e2f85fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 shutdown_connections 2024-03-22T02:23:39.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 1 -- 172.21.15.28:0/66376895 wait complete. 2024-03-22T02:23:39.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.869+0000 7fee1202a740 20 client.28210 trim_cache size 0 max 16384 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 Processor -- start 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 -- start start 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f9d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f8dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee081aaff0 con 0x5621e2f8b820 2024-03-22T02:23:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee081a88c0 con 0x5621e2f8c210 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee1202a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee081a6710 con 0x5621e2f91380 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49748/0 (socket says 172.21.15.28:49748) 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 -- 172.21.15.28:0/665530313 learned_addr learned my addr 172.21.15.28:0/665530313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d908640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f8dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 -- 172.21.15.28:0/665530313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 msgr2=0x5621e2f8dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f8dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 -- 172.21.15.28:0/665530313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 msgr2=0x5621e2f9d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0c906640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f9d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f9d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 -- 172.21.15.28:0/665530313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621e2f8b3e0 con 0x5621e2f8b820 2024-03-22T02:23:39.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d908640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0c906640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f9d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.883+0000 7fee0d107640 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fee000097d0 tx=0x7fee000d5e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84339c179fcd7871 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fedf2ffd640 1 -- 172.21.15.28:0/665530313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee000a8660 con 0x5621e2f8b820 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fedf2ffd640 1 -- 172.21.15.28:0/665530313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee000a3da0 con 0x5621e2f8b820 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fedf2ffd640 1 -- 172.21.15.28:0/665530313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee000a01c0 con 0x5621e2f8b820 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- 172.21.15.28:0/665530313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 msgr2=0x5621e2f8d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fee000097d0 tx=0x7fee000d5e40 comp rx=0 tx=0).stop 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- 172.21.15.28:0/665530313 shutdown_connections 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8c210 0x5621e2f9d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 --2- 172.21.15.28:0/665530313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8b820 0x5621e2f8d6d0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- 172.21.15.28:0/665530313 >> 172.21.15.28:0/665530313 conn(0x5621e2f86d90 msgr2=0x5621e2f95d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- 172.21.15.28:0/665530313 shutdown_connections 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- 172.21.15.28:0/665530313 wait complete. 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.884+0000 7fee1202a740 1 -- start start 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 Processor -- start 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 0x5621e2f9a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f9bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee081a6d80 con 0x5621e2f8c210 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fee081a81f0 con 0x5621e2f8b820 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fee081a7450 con 0x5621e2f91380 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee1202a740 10 client.0 _ll_register_callbacks cb 0x7fedf8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49756/0 (socket says 172.21.15.28:49756) 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 -- 172.21.15.28:0/158303201 learned_addr learned my addr 172.21.15.28:0/158303201 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0d908640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f9bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0d107640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 0x5621e2f9a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 -- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 msgr2=0x5621e2f9bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f9bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 -- 172.21.15.28:0/158303201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 msgr2=0x5621e2f9a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 0x5621e2f9a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 -- 172.21.15.28:0/158303201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621e2f9c270 con 0x5621e2f8c210 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0d107640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 0x5621e2f9a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fee0c906640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fedf802cfd0 tx=0x7fedf8003fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8178bcc6b582753a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.885+0000 7fedf0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf8005d50 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee000a3da0 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fedf80035f0 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fedf8003810 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 10 client.18402 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5621e2f9d4e0 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee000a3da0 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fedf8034600 con 0x5621e2f8c210 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 10 client.18402 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 10 client.18402 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:23:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 20 client.18402 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e3008750 0x5621e300ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 4 client.18402 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621e2ef8160 con 0x5621e3008750 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.886+0000 7fee0d107640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e3008750 0x5621e300ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee0d107640 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e3008750 0x5621e300ab70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=97d2e041cbb9005b server_cookie=f2ef5b12b4ced7d2 in_seq=0 out_seq=0 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fedf0ff9640 10 client.18402 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fedf0ff9640 1 -- 172.21.15.28:0/158303201 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621e2ef8160 con 0x5621e3008750 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fedf0ff9640 10 client.18402 handle_command_reply: tid=0 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 client.18402 shutdown 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 msgr2=0x5621e2f9ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fedf802cfd0 tx=0x7fedf8003fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 shutdown_connections 2024-03-22T02:23:39.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5621e3008750 0x5621e300ab70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621e2f91380 0x5621e2f9bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fedf0ff9640 0 client.18402 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621e2f8c210 0x5621e2f9ac70 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 --2- 172.21.15.28:0/158303201 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621e2f8b820 0x5621e2f9a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 >> 172.21.15.28:0/158303201 conn(0x5621e2f86d90 msgr2=0x5621e2f95cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 shutdown_connections 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 1 -- 172.21.15.28:0/158303201 wait complete. 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.887+0000 7fee1202a740 20 client.18402 trim_cache size 0 max 16384 2024-03-22T02:23:39.897 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 msgr2=0x7fedf40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 0x7fedf40a53e0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fee04022000 tx=0x7fee04023000 comp rx=0 tx=0).stop 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 msgr2=0x7fee08197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fee00009f50 tx=0x7fee0000ee50 comp rx=0 tx=0).stop 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 shutdown_connections 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fedf40a2f20 0x7fedf40a53e0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fee081078f0 0x7fee0819f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee08104fc0 0x7fee08198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 --2- 172.21.15.28:0/4286547857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fee08102690 0x7fee08197f10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 >> 172.21.15.28:0/4286547857 conn(0x7fee080fc020 msgr2=0x7fee08103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 shutdown_connections 2024-03-22T02:23:39.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:39.888+0000 7fee0f36b640 1 -- 172.21.15.28:0/4286547857 wait complete. 2024-03-22T02:23:39.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:23:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 -- 172.21.15.28:0/10474262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64102640 msgr2=0x7fcc64108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 --2- 172.21.15.28:0/10474262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64102640 0x7fcc64108ed0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009ca0 tx=0x7fcc5c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 -- 172.21.15.28:0/10474262 shutdown_connections 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 --2- 172.21.15.28:0/10474262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64102640 0x7fcc64108ed0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 --2- 172.21.15.28:0/10474262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64101c80 0x7fcc64102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 --2- 172.21.15.28:0/10474262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 -- 172.21.15.28:0/10474262 >> 172.21.15.28:0/10474262 conn(0x7fcc640fc020 msgr2=0x7fcc640fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 -- 172.21.15.28:0/10474262 shutdown_connections 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.008+0000 7fcc6b281640 1 -- 172.21.15.28:0/10474262 wait complete. 2024-03-22T02:23:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 Processor -- start 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 -- start start 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 0x7fcc641983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6410c710 con 0x7fcc64101c80 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6410c590 con 0x7fcc64102640 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6b281640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6410c890 con 0x7fcc64100a90 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6901d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 0x7fcc641983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51792/0 (socket says 172.21.15.28:51792) 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6981e640 1 -- 172.21.15.28:0/768068730 learned_addr learned my addr 172.21.15.28:0/768068730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6901d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51366/0 (socket says 172.21.15.28:51366) 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 -- 172.21.15.28:0/768068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 msgr2=0x7fcc64197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 -- 172.21.15.28:0/768068730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 msgr2=0x7fcc6419f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6881c640 1 -- 172.21.15.28:0/768068730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc6419fad0 con 0x7fcc64101c80 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6901d640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.009+0000 7fcc6981e640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc6881c640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 0x7fcc641983f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400e9f0 tx=0x7fcc5400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc5a7fc640 1 -- 172.21.15.28:0/768068730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5400ccc0 con 0x7fcc64101c80 2024-03-22T02:23:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6419fdc0 con 0x7fcc64101c80 2024-03-22T02:23:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc5a7fc640 1 -- 172.21.15.28:0/768068730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc54010910 con 0x7fcc64101c80 2024-03-22T02:23:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc5a7fc640 1 -- 172.21.15.28:0/768068730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc54018960 con 0x7fcc64101c80 2024-03-22T02:23:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.010+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc641a0290 con 0x7fcc64101c80 2024-03-22T02:23:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.013+0000 7fcc5a7fc640 1 -- 172.21.15.28:0/768068730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fcc54010ac0 con 0x7fcc64101c80 2024-03-22T02:23:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 Processor -- start 2024-03-22T02:23:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 -- start start 2024-03-22T02:23:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d59d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d5c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563977cb98d0 con 0x563977d54820 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563977c61e00 con 0x563977d5a380 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6de69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563977cb5880 con 0x563977d55210 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49802/0 (socket says 172.21.15.28:49802) 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6981e640 1 -- 172.21.15.28:0/3485245084 learned_addr learned my addr 172.21.15.28:0/3485245084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:40.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6881c640 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d5c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc5a7fc640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 0x7fcc500a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.014+0000 7fcc6901d640 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d59d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 -- 172.21.15.28:0/3485245084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 msgr2=0x563977d59d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d59d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 -- 172.21.15.28:0/3485245084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 msgr2=0x563977d5c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d5c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 -- 172.21.15.28:0/3485245084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639779cd3f0 con 0x563977d54820 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc5a7fc640 1 -- 172.21.15.28:0/768068730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcc540cfb60 con 0x7fcc64101c80 2024-03-22T02:23:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 0x7fcc500a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc6981e640 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0098b0 tx=0x7fcc5c0098e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29718baa154b0af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc4f7fe640 1 -- 172.21.15.28:0/3485245084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c039040 con 0x563977d54820 2024-03-22T02:23:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc4f7fe640 1 -- 172.21.15.28:0/3485245084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c0347f0 con 0x563977d54820 2024-03-22T02:23:40.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.015+0000 7fcc4f7fe640 1 -- 172.21.15.28:0/3485245084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c004460 con 0x563977d54820 2024-03-22T02:23:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 -- 172.21.15.28:0/3485245084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 msgr2=0x563977d54c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0098b0 tx=0x7fcc5c0098e0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6981e640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 0x7fcc500a53e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c04a000 tx=0x7fcc5c045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 -- 172.21.15.28:0/3485245084 shutdown_connections 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d5c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d59d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 --2- 172.21.15.28:0/3485245084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d54820 0x563977d54c40 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 -- 172.21.15.28:0/3485245084 >> 172.21.15.28:0/3485245084 conn(0x563977d4fd90 msgr2=0x563977d50190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 -- 172.21.15.28:0/3485245084 shutdown_connections 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.016+0000 7fcc6de69740 1 -- 172.21.15.28:0/3485245084 wait complete. 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 -- start start 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 Processor -- start 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d67af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563977d5d330 con 0x563977d5a380 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563977c75300 con 0x563977d54820 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563977d5d4b0 con 0x563977d55210 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6de69740 10 client.0 _ll_register_callbacks cb 0x563977cb0e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49804/0 (socket says 172.21.15.28:49804) 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6881c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d67af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6981e640 1 -- 172.21.15.28:0/811003638 learned_addr learned my addr 172.21.15.28:0/811003638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.017+0000 7fcc6901d640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d675b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 -- 172.21.15.28:0/811003638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 msgr2=0x563977d67af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d67af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 -- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 msgr2=0x563977d675b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d675b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 -- 172.21.15.28:0/811003638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563977a977d0 con 0x563977d5a380 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6981e640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c037040 tx=0x7fcc5c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daab08da3afbcf73 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c039040 con 0x563977d5a380 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc5c0347f0 con 0x563977d5a380 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcc5c0349a0 con 0x563977d5a380 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcc5c064050 con 0x563977d5a380 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6de69740 10 client.18414 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563977d71e30 con 0x563977d5a380 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.018+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c0347f0 con 0x563977d5a380 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcc5c03f040 con 0x563977d5a380 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 10 client.18414 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 10 client.18414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 20 client.18414 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977d73180 0x563977d75610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 4 client.18414 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563977cc6e10 con 0x563977d73180 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6901d640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977d73180 0x563977d75610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc4cff9640 10 client.18414 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.019+0000 7fcc6901d640 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977d73180 0x563977d75610 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2a1d11c70cbd2c1 server_cookie=17456b283fe70cbb in_seq=0 out_seq=0 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc4cff9640 1 -- 172.21.15.28:0/811003638 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563977cc6e10 con 0x563977d73180 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc4cff9640 10 client.18414 handle_command_reply: tid=0 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 client.18414 shutdown 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 msgr2=0x563977d6eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c037040 tx=0x7fcc5c009360 comp rx=0 tx=0).stop 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 shutdown_connections 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977d73180 0x563977d75610 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d6eac0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d67af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 --2- 172.21.15.28:0/811003638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d675b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 >> 172.21.15.28:0/811003638 conn(0x563977d4fd90 msgr2=0x563977d4efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc4cff9640 0 client.18414 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 shutdown_connections 2024-03-22T02:23:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 1 -- 172.21.15.28:0/811003638 wait complete. 2024-03-22T02:23:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.021+0000 7fcc6de69740 20 client.18414 trim_cache size 0 max 16384 2024-03-22T02:23:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 Processor -- start 2024-03-22T02:23:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 -- start start 2024-03-22T02:23:40.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d4e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d6ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641ab120 con 0x563977d5a380 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641a89f0 con 0x563977d54820 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6de69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641a6840 con 0x563977d55210 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6881c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d6ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6901d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d4e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6981e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49806/0 (socket says 172.21.15.28:49806) 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.035+0000 7fcc6981e640 1 -- 172.21.15.28:0/889718543 learned_addr learned my addr 172.21.15.28:0/889718543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 -- 172.21.15.28:0/889718543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 msgr2=0x563977d6ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d6ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 -- 172.21.15.28:0/889718543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 msgr2=0x563977d4e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d4e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 -- 172.21.15.28:0/889718543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563977d4a3e0 con 0x563977d5a380 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc6981e640 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c032590 tx=0x7fcc5c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524b614fdc976c23 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc4effd640 1 -- 172.21.15.28:0/889718543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c039070 con 0x563977d5a380 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc4effd640 1 -- 172.21.15.28:0/889718543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c05b070 con 0x563977d5a380 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.036+0000 7fcc4effd640 1 -- 172.21.15.28:0/889718543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c034a50 con 0x563977d5a380 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- 172.21.15.28:0/889718543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 msgr2=0x563977d4ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c032590 tx=0x7fcc5c0095c0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- 172.21.15.28:0/889718543 shutdown_connections 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d5a380 0x563977d4ec80 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d55210 0x563977d6ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- 172.21.15.28:0/889718543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d54820 0x563977d4e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- 172.21.15.28:0/889718543 >> 172.21.15.28:0/889718543 conn(0x563977d4fd90 msgr2=0x563977d675b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- 172.21.15.28:0/889718543 shutdown_connections 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- 172.21.15.28:0/889718543 wait complete. 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- start start 2024-03-22T02:23:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 Processor -- start 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 0x563977d6c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d6d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641a6eb0 con 0x563977d55210 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641a8320 con 0x563977d5a380 2024-03-22T02:23:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641a7580 con 0x563977d54820 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.037+0000 7fcc6de69740 10 client.0 _ll_register_callbacks cb 0x7fcc54098000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6901d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 0x563977d6c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49812/0 (socket says 172.21.15.28:49812) 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 -- 172.21.15.28:0/1268744525 learned_addr learned my addr 172.21.15.28:0/1268744525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6981e640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d6d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 -- 172.21.15.28:0/1268744525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 msgr2=0x563977d6c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 0x563977d6c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 -- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 msgr2=0x563977d6d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d6d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 -- 172.21.15.28:0/1268744525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563977d6de90 con 0x563977d55210 2024-03-22T02:23:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6901d640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 0x563977d6c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6881c640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400e9c0 tx=0x7fcc540982d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f40de279a6fc5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc4cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc540bc0b0 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcc54097c80 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc5c05b070 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcc5409bf30 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6de69740 10 client.18420 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.038+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x563977dd9750 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc5c05b070 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcc5409b510 con 0x563977d55210 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 10 client.18420 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:40.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 10 client.18420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 20 client.18420 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977dda100 0x563977ddc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 4 client.18420 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563977cc1160 con 0x563977dda100 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6901d640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977dda100 0x563977ddc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc6901d640 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977dda100 0x563977ddc520 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc39234825962150 server_cookie=29bbd6caaf366424 in_seq=0 out_seq=0 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.039+0000 7fcc4cff9640 10 client.18420 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc4cff9640 1 -- 172.21.15.28:0/1268744525 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563977cc1160 con 0x563977dda100 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc4cff9640 10 client.18420 handle_command_reply: tid=0 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 client.18420 shutdown 2024-03-22T02:23:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 msgr2=0x563977d6c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400e9c0 tx=0x7fcc540982d0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 shutdown_connections 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563977dda100 0x563977ddc520 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563977d5a380 0x563977d6d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563977d55210 0x563977d6c8e0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc4cff9640 0 client.18420 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 --2- 172.21.15.28:0/1268744525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563977d54820 0x563977d6c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 >> 172.21.15.28:0/1268744525 conn(0x563977d4fd90 msgr2=0x563977d52ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 shutdown_connections 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 1 -- 172.21.15.28:0/1268744525 wait complete. 2024-03-22T02:23:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.041+0000 7fcc6de69740 20 client.18420 trim_cache size 0 max 16384 2024-03-22T02:23:40.049 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 msgr2=0x7fcc500a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 0x7fcc500a53e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c04a000 tx=0x7fcc5c045000 comp rx=0 tx=0).stop 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 msgr2=0x7fcc641983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 0x7fcc641983f0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc5400e9f0 tx=0x7fcc5400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 shutdown_connections 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcc500a2f20 0x7fcc500a53e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcc64102640 0x7fcc6419f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc64101c80 0x7fcc641983f0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 --2- 172.21.15.28:0/768068730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcc64100a90 0x7fcc64197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 >> 172.21.15.28:0/768068730 conn(0x7fcc640fc020 msgr2=0x7fcc640fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 shutdown_connections 2024-03-22T02:23:40.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:40.042+0000 7fcc6b281640 1 -- 172.21.15.28:0/768068730 wait complete. 2024-03-22T02:23:40.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1005075 11.50 MB/sec execute 382 sec latency 16.623 ms 2024-03-22T02:23:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:40 smithi120 ceph-mon[36803]: pgmap v662: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:23:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:40 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1326019668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:40 smithi083 ceph-mon[36340]: pgmap v662: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:23:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:40 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1326019668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:40 smithi028 ceph-mon[31916]: pgmap v662: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:23:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:40 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1326019668' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:23:41.054 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:23:41.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:23:41.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:23:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 -- 172.21.15.28:0/1766119630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc101780 msgr2=0x7fcbcc101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1766119630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc101780 0x7fcbcc101c00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4009ca0 tx=0x7fcbc402c180 comp rx=0 tx=0).stop 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 -- 172.21.15.28:0/1766119630 shutdown_connections 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1766119630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc101780 0x7fcbcc101c00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1766119630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc100de0 0x7fcbcc101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1766119630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100480 0x7fcbcc1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 -- 172.21.15.28:0/1766119630 >> 172.21.15.28:0/1766119630 conn(0x7fcbcc0fc020 msgr2=0x7fcbcc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 -- 172.21.15.28:0/1766119630 shutdown_connections 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.205+0000 7fcbd131e640 1 -- 172.21.15.28:0/1766119630 wait complete. 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 Processor -- start 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 -- start start 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100de0 0x7fcbcc19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc101780 0x7fcbcc1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10f9f0 con 0x7fcbcc100480 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10f870 con 0x7fcbcc101780 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbd131e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc10fb70 con 0x7fcbcc100de0 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbcaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbcaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49824/0 (socket says 172.21.15.28:49824) 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1609462393 learned_addr learned my addr 172.21.15.28:0/1609462393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100de0 0x7fcbcc19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.206+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc101780 0x7fcbcc1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1609462393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100de0 msgr2=0x7fcbcc19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100de0 0x7fcbcc19a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1609462393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc101780 msgr2=0x7fcbcc1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc101780 0x7fcbcc1a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1609462393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc1a1cb0 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00e9f0 tx=0x7fcbbc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbc8ff9640 1 -- 172.21.15.28:0/1609462393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc00cd80 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbc8ff9640 1 -- 172.21.15.28:0/1609462393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc010910 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc1a1fa0 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbc8ff9640 1 -- 172.21.15.28:0/1609462393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc018930 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.207+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc1a2400 con 0x7fcbcc100480 2024-03-22T02:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.209+0000 7fcbc8ff9640 1 -- 172.21.15.28:0/1609462393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fcbbc010ac0 con 0x7fcbcc100480 2024-03-22T02:23:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 Processor -- start 2024-03-22T02:23:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 -- start start 2024-03-22T02:23:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbc8ff9640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 0x7fcbb80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 0x7fcbb80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4867bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbc8ff9640 1 -- 172.21.15.28:0/1609462393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcbbc0cfb10 con 0x7fcbcc100480 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 0x55ae48683bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae485e0570 con 0x55ae48681720 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae48588e10 con 0x55ae4867c5b0 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.210+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae485dc520 con 0x55ae4867bbc0 2024-03-22T02:23:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51836/0 (socket says 172.21.15.28:51836) 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4867bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc3fff640 1 -- 172.21.15.28:0/1157561824 learned_addr learned my addr 172.21.15.28:0/1157561824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 0x55ae48683bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 0x7fcbb80a53e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4005e30 tx=0x7fcbc4032480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1157561824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 msgr2=0x55ae486810b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1157561824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 msgr2=0x55ae48683bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 0x55ae48683bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 -- 172.21.15.28:0/1157561824 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae483bc7e0 con 0x55ae4867bbc0 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 0x55ae48683bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4867bfe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc09be10 tx=0x7fcbbc09be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=154aec657ea09d55 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc17fa640 1 -- 172.21.15.28:0/1157561824 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc0a47e0 con 0x55ae4867bbc0 2024-03-22T02:23:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc17fa640 1 -- 172.21.15.28:0/1157561824 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc09a180 con 0x55ae4867bbc0 2024-03-22T02:23:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.211+0000 7fcbc17fa640 1 -- 172.21.15.28:0/1157561824 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc097bd0 con 0x55ae4867bbc0 2024-03-22T02:23:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1157561824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 msgr2=0x55ae4867bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4867bfe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc09be10 tx=0x7fcbbc09be80 comp rx=0 tx=0).stop 2024-03-22T02:23:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1157561824 shutdown_connections 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae48681720 0x55ae48683bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867c5b0 0x55ae486810b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1157561824 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4867bfe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1157561824 >> 172.21.15.28:0/1157561824 conn(0x55ae48677130 msgr2=0x55ae48677530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1157561824 shutdown_connections 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1157561824 wait complete. 2024-03-22T02:23:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 -- start start 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.212+0000 7fcbd3fe8740 1 Processor -- start 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 0x55ae48685e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae48681720 0x55ae4868d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae486846d0 con 0x55ae4867c5b0 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae4859c110 con 0x55ae4867bbc0 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae48684850 con 0x55ae48681720 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbd3fe8740 10 client.0 _ll_register_callbacks cb 0x55ae485d8be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 0x55ae48685e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49828/0 (socket says 172.21.15.28:49828) 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbc3fff640 1 -- 172.21.15.28:0/967566838 learned_addr learned my addr 172.21.15.28:0/967566838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae48681720 0x55ae4868d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 -- 172.21.15.28:0/967566838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae48681720 msgr2=0x55ae4868d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae48681720 0x55ae4868d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 -- 172.21.15.28:0/967566838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 msgr2=0x55ae48686380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 -- 172.21.15.28:0/967566838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae4841cf70 con 0x55ae4867bbc0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 0x55ae48685e40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00c6a0 tx=0x7fcbbc00cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26e5134cf718a674 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.213+0000 7fcbb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbbc0b4070 con 0x55ae4867bbc0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcbbc010430 con 0x55ae4867bbc0 2024-03-22T02:23:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbbc09a180 con 0x55ae4867bbc0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcbbc09b5e0 con 0x55ae4867bbc0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 10 client.28250 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae482b4170 con 0x55ae4867bbc0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbbc09a180 con 0x55ae4867bbc0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcbbc09ad40 con 0x55ae4867bbc0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 10 client.28250 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 10 client.28250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 20 client.28250 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486918b0 0x55ae48693ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 4 client.28250 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae485edab0 con 0x55ae486918b0 2024-03-22T02:23:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.214+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486918b0 0x55ae48693ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.215+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486918b0 0x55ae48693ca0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2662d3892ad0eb46 server_cookie=8f8b107c27a496a0 in_seq=0 out_seq=0 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.215+0000 7fcbb27fc640 10 client.28250 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbb27fc640 1 -- 172.21.15.28:0/967566838 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae485edab0 con 0x55ae486918b0 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbb27fc640 10 client.28250 handle_command_reply: tid=0 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 client.28250 shutdown 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 msgr2=0x55ae48685e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 0x55ae48685e40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00c6a0 tx=0x7fcbbc00cf90 comp rx=0 tx=0).stop 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 shutdown_connections 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486918b0 0x55ae48693ca0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae48681720 0x55ae4868d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae48686380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbb27fc640 0 client.28250 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/967566838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae4867bbc0 0x55ae48685e40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 >> 172.21.15.28:0/967566838 conn(0x55ae48677130 msgr2=0x55ae486763f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 shutdown_connections 2024-03-22T02:23:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/967566838 wait complete. 2024-03-22T02:23:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.216+0000 7fcbd3fe8740 20 client.28250 trim_cache size 0 max 16384 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.233+0000 7fcbd3fe8740 1 Processor -- start 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.233+0000 7fcbd3fe8740 1 -- start start 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae486860c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4867cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1af510 con 0x55ae4867c5b0 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1acde0 con 0x55ae48681720 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1aac30 con 0x55ae4867bbc0 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49834/0 (socket says 172.21.15.28:49834) 2024-03-22T02:23:41.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 -- 172.21.15.28:0/1704554431 learned_addr learned my addr 172.21.15.28:0/1704554431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae486860c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4867cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 -- 172.21.15.28:0/1704554431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 msgr2=0x55ae486860c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae486860c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 -- 172.21.15.28:0/1704554431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 msgr2=0x55ae4867cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4867cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 -- 172.21.15.28:0/1704554431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae48684070 con 0x55ae4867c5b0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae486860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafc744ebc80d8d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc0ff9640 1 -- 172.21.15.28:0/1704554431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb4020310 con 0x55ae4867c5b0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc0ff9640 1 -- 172.21.15.28:0/1704554431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb4010430 con 0x55ae4867c5b0 2024-03-22T02:23:41.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.234+0000 7fcbc0ff9640 1 -- 172.21.15.28:0/1704554431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb401e680 con 0x55ae4867c5b0 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1704554431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 msgr2=0x55ae4868d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4005f90 comp rx=0 tx=0).stop 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1704554431 shutdown_connections 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4867cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868d380 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/1704554431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae486860c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1704554431 >> 172.21.15.28:0/1704554431 conn(0x55ae48677130 msgr2=0x55ae4867ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1704554431 shutdown_connections 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/1704554431 wait complete. 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 -- start start 2024-03-22T02:23:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.235+0000 7fcbd3fe8740 1 Processor -- start 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4868aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4868bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ab2a0 con 0x55ae4867c5b0 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ac710 con 0x55ae48681720 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc1ab970 con 0x55ae4867bbc0 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 10 client.0 _ll_register_callbacks cb 0x7fcbc4045030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49838/0 (socket says 172.21.15.28:49838) 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 -- 172.21.15.28:0/55702716 learned_addr learned my addr 172.21.15.28:0/55702716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4868aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbcb7fe640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4868bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 -- 172.21.15.28:0/55702716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 msgr2=0x55ae4868aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4868aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 -- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 msgr2=0x55ae4868bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4868bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 -- 172.21.15.28:0/55702716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae4868c4c0 con 0x55ae4867c5b0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4868aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbc3fff640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40058c0 tx=0x7fcbb40120e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c189c0d370f35a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbb27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402d070 con 0x55ae4867c5b0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbb4010430 con 0x55ae4867c5b0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.236+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcbb4005040 con 0x55ae4867c5b0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcbb4005240 con 0x55ae4867c5b0 2024-03-22T02:23:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 10 client.18438 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae486f7a90 con 0x55ae4867c5b0 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbb4010430 con 0x55ae4867c5b0 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcbb4004700 con 0x55ae4867c5b0 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 10 client.18438 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 10 client.18438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 20 client.18438 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486f8350 0x55ae486fa770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 4 client.18438 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ae485e7e00 con 0x55ae486f8350 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486f8350 0x55ae486fa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.237+0000 7fcbcaffd640 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486f8350 0x55ae486fa770 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=118557144c1c870 server_cookie=a04aa705090e880e in_seq=0 out_seq=0 2024-03-22T02:23:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.238+0000 7fcbb27fc640 10 client.18438 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.257+0000 7fcbb27fc640 1 -- 172.21.15.28:0/55702716 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ae485e7e00 con 0x55ae486f8350 2024-03-22T02:23:41.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.257+0000 7fcbb27fc640 10 client.18438 handle_command_reply: tid=0 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.257+0000 7fcbd3fe8740 1 client.18438 shutdown 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 msgr2=0x55ae4868afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40058c0 tx=0x7fcbb40120e0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 shutdown_connections 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ae486f8350 0x55ae486fa770 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae48681720 0x55ae4868bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae4867c5b0 0x55ae4868afa0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 --2- 172.21.15.28:0/55702716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae4867bbc0 0x55ae4868aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 >> 172.21.15.28:0/55702716 conn(0x55ae48677130 msgr2=0x55ae48675b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbb27fc640 0 client.18438 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 shutdown_connections 2024-03-22T02:23:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 1 -- 172.21.15.28:0/55702716 wait complete. 2024-03-22T02:23:41.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd3fe8740 20 client.18438 trim_cache size 0 max 16384 2024-03-22T02:23:41.262 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:41.262 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:23:41.262 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d02f8e8f-e1e0-4670-be68-f8177e8de901", 2024-03-22T02:23:41.262 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:23:41.263 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 msgr2=0x7fcbb80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 0x7fcbb80a53e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4005e30 tx=0x7fcbc4032480 comp rx=0 tx=0).stop 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 msgr2=0x7fcbcc19a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc00e9f0 tx=0x7fcbbc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 shutdown_connections 2024-03-22T02:23:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcbb80a2f20 0x7fcbb80a53e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcbcc101780 0x7fcbcc1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbcc100de0 0x7fcbcc19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 --2- 172.21.15.28:0/1609462393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbcc100480 0x7fcbcc19a090 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.258+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 >> 172.21.15.28:0/1609462393 conn(0x7fcbcc0fc020 msgr2=0x7fcbcc0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.259+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 shutdown_connections 2024-03-22T02:23:41.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.259+0000 7fcbd131e640 1 -- 172.21.15.28:0/1609462393 wait complete. 2024-03-22T02:23:41.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:23:41.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1287147709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 msgr2=0x7fd99c102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1287147709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c102100 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fd990009900 tx=0x7fd99002bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1287147709 shutdown_connections 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1287147709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c102640 0x7fd99c108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1287147709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c102100 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1287147709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c100a90 0x7fd99c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1287147709 >> 172.21.15.28:0/1287147709 conn(0x7fd99c0fc020 msgr2=0x7fd99c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1287147709 shutdown_connections 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1287147709 wait complete. 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 Processor -- start 2024-03-22T02:23:41.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.383+0000 7fd9a24fb640 1 -- start start 2024-03-22T02:23:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c100a90 0x7fd99c069880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c102640 0x7fd99c19f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10c710 con 0x7fd99c101c80 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10c590 con 0x7fd99c100a90 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a24fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c10c890 con 0x7fd99c102640 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c100a90 0x7fd99c069880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49864/0 (socket says 172.21.15.28:49864) 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 -- 172.21.15.28:0/1751252832 learned_addr learned my addr 172.21.15.28:0/1751252832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c102640 0x7fd99c19f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 -- 172.21.15.28:0/1751252832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c102640 msgr2=0x7fd99c19f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c102640 0x7fd99c19f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 -- 172.21.15.28:0/1751252832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c100a90 msgr2=0x7fd99c069880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c100a90 0x7fd99c069880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 -- 172.21.15.28:0/1751252832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99c19fb30 con 0x7fd99c101c80 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.384+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd990009230 tx=0x7fd99002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.385+0000 7fd9997fa640 1 -- 172.21.15.28:0/1751252832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990041070 con 0x7fd99c101c80 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.385+0000 7fd9997fa640 1 -- 172.21.15.28:0/1751252832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd990004450 con 0x7fd99c101c80 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.385+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99c19fdc0 con 0x7fd99c101c80 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.385+0000 7fd9997fa640 1 -- 172.21.15.28:0/1751252832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990003e20 con 0x7fd99c101c80 2024-03-22T02:23:41.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.385+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99c1a0270 con 0x7fd99c101c80 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.387+0000 7fd9997fa640 1 -- 172.21.15.28:0/1751252832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd990044070 con 0x7fd99c101c80 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9997fa640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 0x7fd9840a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 Processor -- start 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 -- start start 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99bfff640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 0x7fd9840a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9997fa640 1 -- 172.21.15.28:0/1751252832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd9900f12a0 con 0x7fd99c101c80 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b53850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559470ab88c0 con 0x559470b58f90 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559470a60c20 con 0x559470b53e20 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a51be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559470ab4870 con 0x559470b53430 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49874/0 (socket says 172.21.15.28:49874) 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51892/0 (socket says 172.21.15.28:51892) 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 -- 172.21.15.28:0/2567264877 learned_addr learned my addr 172.21.15.28:0/2567264877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99bfff640 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b53850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99bfff640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 0x7fd9840a5460 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd99c101ae0 tx=0x7fd98c011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 -- 172.21.15.28:0/2567264877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 msgr2=0x559470b53850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b53850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 -- 172.21.15.28:0/2567264877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 msgr2=0x559470b58920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd9a0a98640 1 -- 172.21.15.28:0/2567264877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559470a85150 con 0x559470b58f90 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.388+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd994009fd0 tx=0x7fd9940118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bcf84c59b25a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9827fc640 1 -- 172.21.15.28:0/2567264877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994004d20 con 0x559470b58f90 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9827fc640 1 -- 172.21.15.28:0/2567264877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994004050 con 0x559470b58f90 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9827fc640 1 -- 172.21.15.28:0/2567264877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994005280 con 0x559470b58f90 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 -- 172.21.15.28:0/2567264877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 msgr2=0x559470b5b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fd994009fd0 tx=0x7fd9940118d0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 -- 172.21.15.28:0/2567264877 shutdown_connections 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5b420 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b58920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 --2- 172.21.15.28:0/2567264877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b53850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.389+0000 7fd9a51be740 1 -- 172.21.15.28:0/2567264877 >> 172.21.15.28:0/2567264877 conn(0x559470b4e9a0 msgr2=0x559470b4eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- 172.21.15.28:0/2567264877 shutdown_connections 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- 172.21.15.28:0/2567264877 wait complete. 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- start start 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 Processor -- start 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 0x559470b64e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559470b5bf40 con 0x559470b53e20 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559470a74130 con 0x559470b53430 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559470b5c0c0 con 0x559470b58f90 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a51be740 10 client.0 _ll_register_callbacks cb 0x559470a73b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 0x559470b64e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49890/0 (socket says 172.21.15.28:49890) 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.390+0000 7fd99b7fe640 1 -- 172.21.15.28:0/1075319741 learned_addr learned my addr 172.21.15.28:0/1075319741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 -- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53430 msgr2=0x559470b5d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 -- 172.21.15.28:0/1075319741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 msgr2=0x559470b5de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 -- 172.21.15.28:0/1075319741 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594708967d0 con 0x559470b58f90 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 0x559470b64e40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd99401b040 tx=0x7fd994003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a75584d521b9c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd97b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994029020 con 0x559470b58f90 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd99400f4e0 con 0x559470b58f90 2024-03-22T02:23:41.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd994004050 con 0x559470b58f90 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd994034050 con 0x559470b58f90 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a51be740 10 client.28264 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.391+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5594707cc3f0 con 0x559470b58f90 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994004050 con 0x559470b58f90 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fd994025570 con 0x559470b58f90 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 10 client.28264 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 10 client.28264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 20 client.28264 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470b695c0 0x559470b6ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 4 client.28264 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559470ac5e00 con 0x559470b695c0 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd99bfff640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470b695c0 0x559470b6ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd97b7fe640 10 client.28264 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.392+0000 7fd99bfff640 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470b695c0 0x559470b6ba00 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec2de34840954f35 server_cookie=480ea31551b03394 in_seq=0 out_seq=0 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd97b7fe640 1 -- 172.21.15.28:0/1075319741 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559470ac5e00 con 0x559470b695c0 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd97b7fe640 10 client.28264 handle_command_reply: tid=0 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 client.28264 shutdown 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 msgr2=0x559470b64e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 0x559470b64e40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd99401b040 tx=0x7fd994003c60 comp rx=0 tx=0).stop 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 shutdown_connections 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470b695c0 0x559470b6ba00 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b58f90 0x559470b64e40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b53e20 0x559470b5de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd97b7fe640 0 client.28264 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 --2- 172.21.15.28:0/1075319741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.394+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 >> 172.21.15.28:0/1075319741 conn(0x559470b4e9a0 msgr2=0x559470b4dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.395+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 shutdown_connections 2024-03-22T02:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.395+0000 7fd9a51be740 1 -- 172.21.15.28:0/1075319741 wait complete. 2024-03-22T02:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.395+0000 7fd9a51be740 20 client.28264 trim_cache size 0 max 16384 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 Processor -- start 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 -- start start 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b64e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1ad2d0 con 0x559470b58f90 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1aaba0 con 0x559470b53e20 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.413+0000 7fd9a51be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1a89f0 con 0x559470b53430 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49900/0 (socket says 172.21.15.28:49900) 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51486/0 (socket says 172.21.15.28:51486) 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 -- 172.21.15.28:0/3876060795 learned_addr learned my addr 172.21.15.28:0/3876060795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b64e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 -- 172.21.15.28:0/3876060795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 msgr2=0x559470b5d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 -- 172.21.15.28:0/3876060795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 msgr2=0x559470b64e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b64e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 -- 172.21.15.28:0/3876060795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55947078c0f0 con 0x559470b58f90 2024-03-22T02:23:41.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd994006010 tx=0x7fd994011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbff5a4c0308d297 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd981ffb640 1 -- 172.21.15.28:0/3876060795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994029040 con 0x559470b58f90 2024-03-22T02:23:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd981ffb640 1 -- 172.21.15.28:0/3876060795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994004000 con 0x559470b58f90 2024-03-22T02:23:41.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.414+0000 7fd981ffb640 1 -- 172.21.15.28:0/3876060795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994019760 con 0x559470b58f90 2024-03-22T02:23:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 -- 172.21.15.28:0/3876060795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 msgr2=0x559470b5de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd994006010 tx=0x7fd994011dd0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 -- 172.21.15.28:0/3876060795 shutdown_connections 2024-03-22T02:23:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b5de00 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b64e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 --2- 172.21.15.28:0/3876060795 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b5d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 -- 172.21.15.28:0/3876060795 >> 172.21.15.28:0/3876060795 conn(0x559470b4e9a0 msgr2=0x559470b55320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 -- 172.21.15.28:0/3876060795 shutdown_connections 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.415+0000 7fd9a51be740 1 -- 172.21.15.28:0/3876060795 wait complete. 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 -- start start 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 Processor -- start 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b62020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b62560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1a9060 con 0x559470b58f90 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1aa4d0 con 0x559470b53e20 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd99c1a9730 con 0x559470b53430 2024-03-22T02:23:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a51be740 10 client.0 _ll_register_callbacks cb 0x7fd994009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b62020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a0a98640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49912/0 (socket says 172.21.15.28:49912) 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd99bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b62020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51492/0 (socket says 172.21.15.28:51492) 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.416+0000 7fd9a0a98640 1 -- 172.21.15.28:0/735085786 learned_addr learned my addr 172.21.15.28:0/735085786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd99b7fe640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b62560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 -- 172.21.15.28:0/735085786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 msgr2=0x559470b62020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b62020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 -- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 msgr2=0x559470b62560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b62560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 -- 172.21.15.28:0/735085786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559470b63b60 con 0x559470b58f90 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a0a98640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd994025bd0 tx=0x7fd994009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b377be1b26e78a15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd97b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd994045060 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd9940194b0 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd994004000 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd99404c050 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a51be740 10 client.18456 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559470b64e70 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.417+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd994004000 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fd99400f9a0 con 0x559470b58f90 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 10 client.18456 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 10 client.18456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 20 client.18456 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470bcfe30 0x559470bd2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:41.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 4 client.18456 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559470ac0150 con 0x559470bcfe30 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd99bfff640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470bcfe30 0x559470bd2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd97b7fe640 10 client.18456 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.418+0000 7fd99bfff640 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470bcfe30 0x559470bd2250 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32f13c792bae1d87 server_cookie=c7cf0634b208061e in_seq=0 out_seq=0 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd97b7fe640 1 -- 172.21.15.28:0/735085786 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559470ac0150 con 0x559470bcfe30 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd97b7fe640 10 client.18456 handle_command_reply: tid=0 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 client.18456 shutdown 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 msgr2=0x559470b63570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd994025bd0 tx=0x7fd994009630 comp rx=0 tx=0).stop 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 shutdown_connections 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559470bcfe30 0x559470bd2250 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559470b58f90 0x559470b63570 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559470b53e20 0x559470b62560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 --2- 172.21.15.28:0/735085786 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559470b53430 0x559470b62020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 >> 172.21.15.28:0/735085786 conn(0x559470b4e9a0 msgr2=0x559470b5d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd97b7fe640 0 client.18456 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 shutdown_connections 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 1 -- 172.21.15.28:0/735085786 wait complete. 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.419+0000 7fd9a51be740 20 client.18456 trim_cache size 0 max 16384 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "d02f8e8f-e1e0-4670-be68-f8177e8de901": { 2024-03-22T02:23:41.426 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:23:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d02f8e8f-e1e0-4670-be68-f8177e8de901", 2024-03-22T02:23:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:23:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:23:41.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:23:41.427 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 msgr2=0x7fd9840a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 0x7fd9840a5460 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd99c101ae0 tx=0x7fd98c011440 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 msgr2=0x7fd99c19cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fd990009230 tx=0x7fd99002c650 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 shutdown_connections 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd9840a2fa0 0x7fd9840a5460 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd99c102640 0x7fd99c19f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd99c101c80 0x7fd99c19cc30 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 --2- 172.21.15.28:0/1751252832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd99c100a90 0x7fd99c069880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 >> 172.21.15.28:0/1751252832 conn(0x7fd99c0fc020 msgr2=0x7fd99c105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 shutdown_connections 2024-03-22T02:23:41.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:23:41.420+0000 7fd9a24fb640 1 -- 172.21.15.28:0/1751252832 wait complete. 2024-03-22T02:23:41.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1006233 11.49 MB/sec execute 383 sec latency 50.302 ms 2024-03-22T02:23:41.432 INFO:tasks.cephfs.filesystem:scrub status for tag:d02f8e8f-e1e0-4670-be68-f8177e8de901 - {'path': '/', 'tag': 'd02f8e8f-e1e0-4670-be68-f8177e8de901', 'options': 'recursive,force'} 2024-03-22T02:23:42.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1008788 11.50 MB/sec execute 384 sec latency 26.263 ms 2024-03-22T02:23:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: pgmap v663: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:23:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:23:42.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:23:42.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:23:42.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:42 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: pgmap v663: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:42 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: pgmap v663: 129 pgs: 129 active+clean; 57 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 353 op/s 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:42 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:23:43.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1010121 11.48 MB/sec execute 385 sec latency 85.690 ms 2024-03-22T02:23:44.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1012751 11.50 MB/sec execute 386 sec latency 21.078 ms 2024-03-22T02:23:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:44 smithi120 ceph-mon[36803]: pgmap v664: 129 pgs: 129 active+clean; 58 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-22T02:23:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:44 smithi083 ceph-mon[36340]: pgmap v664: 129 pgs: 129 active+clean; 58 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-22T02:23:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:44 smithi028 ceph-mon[31916]: pgmap v664: 129 pgs: 129 active+clean; 58 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 349 op/s 2024-03-22T02:23:45.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1014482 11.48 MB/sec execute 387 sec latency 24.764 ms 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1539872158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0100ab0 msgr2=0x7fbbd0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1539872158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0100ab0 0x7fbbd0100ed0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009900 tx=0x7fbbc002bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1539872158 shutdown_connections 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1539872158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0102660 0x7fbbd0108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1539872158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 0x7fbbd0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1539872158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0100ab0 0x7fbbd0100ed0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1539872158 >> 172.21.15.28:0/1539872158 conn(0x7fbbd00fc020 msgr2=0x7fbbd00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1539872158 shutdown_connections 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.539+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1539872158 wait complete. 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 Processor -- start 2024-03-22T02:23:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 -- start start 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 0x7fbbd019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 0x7fbbd01a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c730 con 0x7fbbd0102660 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c5b0 con 0x7fbbd0101ca0 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd6b1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd010c8b0 con 0x7fbbd0100ab0 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 0x7fbbd01a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35184/0 (socket says 172.21.15.28:35184) 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 -- 172.21.15.28:0/1231668779 learned_addr learned my addr 172.21.15.28:0/1231668779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbcffff640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 0x7fbbd019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.543 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 -- 172.21.15.28:0/1231668779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 msgr2=0x7fbbd019a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 0x7fbbd019a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 -- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 msgr2=0x7fbbd01a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 0x7fbbd01a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd48ba640 1 -- 172.21.15.28:0/1231668779 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbd01a1c90 con 0x7fbbd0100ab0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.540+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 0x7fbbd01a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0000c00 tx=0x7fbbc002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbcdffb640 1 -- 172.21.15.28:0/1231668779 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0041070 con 0x7fbbd0100ab0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbcdffb640 1 -- 172.21.15.28:0/1231668779 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc0004450 con 0x7fbbd0100ab0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbd01a1f20 con 0x7fbbd0100ab0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbcdffb640 1 -- 172.21.15.28:0/1231668779 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0034440 con 0x7fbbd0100ab0 2024-03-22T02:23:45.544 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.541+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbd01a2320 con 0x7fbbd0100ab0 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 Processor -- start 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 -- start start 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbcdffb640 1 -- 172.21.15.28:0/1231668779 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fbbc0044070 con 0x7fbbd0100ab0 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643de929040 con 0x5643de9c4ed0 2024-03-22T02:23:45.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5643de8d19e0 con 0x5643de9ca040 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5643de924ff0 con 0x5643de9c44e0 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd48ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd48ba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55468/0 (socket says 172.21.15.28:55468) 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd48ba640 1 -- 172.21.15.28:0/3274886829 learned_addr learned my addr 172.21.15.28:0/3274886829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbcffff640 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 -- 172.21.15.28:0/3274886829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 msgr2=0x5643de9cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 -- 172.21.15.28:0/3274886829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 msgr2=0x5643de9c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd50bb640 1 -- 172.21.15.28:0/3274886829 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643de63d210 con 0x5643de9c44e0 2024-03-22T02:23:45.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.544+0000 7fbbcdffb640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 0x7fbbb80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c4900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005fd0 tx=0x7fbbc8011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34e211eff887b8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 0x7fbbb80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbcdffb640 1 -- 172.21.15.28:0/1231668779 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbbc00f12d0 con 0x7fbbd0100ab0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbb6ffd640 1 -- 172.21.15.28:0/3274886829 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8004d20 con 0x5643de9c44e0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbb6ffd640 1 -- 172.21.15.28:0/3274886829 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc800f9a0 con 0x5643de9c44e0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.545+0000 7fbbb6ffd640 1 -- 172.21.15.28:0/3274886829 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc8004510 con 0x5643de9c44e0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- 172.21.15.28:0/3274886829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 msgr2=0x5643de9c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c4900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005fd0 tx=0x7fbbc8011bf0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd48ba640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 0x7fbbb80a5480 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0005ab0 tx=0x7fbbc00f4db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- 172.21.15.28:0/3274886829 shutdown_connections 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c4ed0 0x5643de9c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/3274886829 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c4900 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- 172.21.15.28:0/3274886829 >> 172.21.15.28:0/3274886829 conn(0x5643de9bfa50 msgr2=0x5643de9bfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- 172.21.15.28:0/3274886829 shutdown_connections 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- 172.21.15.28:0/3274886829 wait complete. 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- start start 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 Processor -- start 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9d5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5643de9ccff0 con 0x5643de9c44e0 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5643de8e4db0 con 0x5643de9ca040 2024-03-22T02:23:45.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.546+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5643de9cd170 con 0x5643de9c4ed0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd97e5740 10 client.0 _ll_register_callbacks cb 0x5643de63ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35206/0 (socket says 172.21.15.28:35206) 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9ce4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55474/0 (socket says 172.21.15.28:55474) 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 -- 172.21.15.28:0/670014306 learned_addr learned my addr 172.21.15.28:0/670014306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9d5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 -- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 msgr2=0x5643de9d5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9d5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 -- 172.21.15.28:0/670014306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 msgr2=0x5643de9ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 -- 172.21.15.28:0/670014306 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643de709620 con 0x5643de9c4ed0 2024-03-22T02:23:45.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbcffff640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4012700 tx=0x7fbbc4012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0bac7992a23bfb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbaffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc400f8c0 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbbc400fa60 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbc800f9a0 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbbc4025050 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd97e5740 10 client.28282 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.547+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5643de9d8eb0 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc800f9a0 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fbbc4010430 con 0x5643de9c4ed0 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 10 client.28282 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 10 client.28282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 20 client.28282 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:45.556 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643de9da580 0x5643de9dc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 4 client.28282 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643de936580 con 0x5643de9da580 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643de9da580 0x5643de9dc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbaffff640 10 client.28282 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.548+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643de9da580 0x5643de9dc9d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a254dd2659cfbe2 server_cookie=d0bf652ec81db333 in_seq=0 out_seq=0 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbaffff640 1 -- 172.21.15.28:0/670014306 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5643de936580 con 0x5643de9da580 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbaffff640 10 client.28282 handle_command_reply: tid=0 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 client.28282 shutdown 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 msgr2=0x5643de9cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4012700 tx=0x7fbbc4012bc0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 shutdown_connections 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643de9da580 0x5643de9dc9d0 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9ca040 0x5643de9d5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c4ed0 0x5643de9cea30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/670014306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbaffff640 0 client.28282 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 >> 172.21.15.28:0/670014306 conn(0x5643de9bfa50 msgr2=0x5643de9bec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 shutdown_connections 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 1 -- 172.21.15.28:0/670014306 wait complete. 2024-03-22T02:23:45.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.549+0000 7fbbd97e5740 20 client.28282 trim_cache size 0 max 16384 2024-03-22T02:23:45.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 Processor -- start 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 -- start start 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01af450 con 0x5643de9ca040 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01acd20 con 0x5643de9c4ed0 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01aab70 con 0x5643de9c44e0 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55488/0 (socket says 172.21.15.28:55488) 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 -- 172.21.15.28:0/1583860248 learned_addr learned my addr 172.21.15.28:0/1583860248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbcffff640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 -- 172.21.15.28:0/1583860248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 msgr2=0x5643de9c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 -- 172.21.15.28:0/1583860248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 msgr2=0x5643de9d5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbd50bb640 1 -- 172.21.15.28:0/1583860248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643de9dd050 con 0x5643de9ca040 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.565+0000 7fbbcffff640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005cc0 tx=0x7fbbc8009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133599f57b4776d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbb67fc640 1 -- 172.21.15.28:0/1583860248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc800f9a0 con 0x5643de9ca040 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbb67fc640 1 -- 172.21.15.28:0/1583860248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc8003cb0 con 0x5643de9ca040 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbb67fc640 1 -- 172.21.15.28:0/1583860248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc802a620 con 0x5643de9ca040 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 -- 172.21.15.28:0/1583860248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 msgr2=0x5643de9e12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8005cc0 tx=0x7fbbc8009fd0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 -- 172.21.15.28:0/1583860248 shutdown_connections 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9ca040 0x5643de9e12d0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/1583860248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9c44e0 0x5643de9c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 -- 172.21.15.28:0/1583860248 >> 172.21.15.28:0/1583860248 conn(0x5643de9bfa50 msgr2=0x5643de9c5970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.566+0000 7fbbd97e5740 1 -- 172.21.15.28:0/1583860248 shutdown_connections 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 -- 172.21.15.28:0/1583860248 wait complete. 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 -- start start 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 Processor -- start 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9ca040 0x5643de9d4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01ab1e0 con 0x5643de9c44e0 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01ac650 con 0x5643de9c4ed0 2024-03-22T02:23:45.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbbd01ab8b0 con 0x5643de9ca040 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd97e5740 10 client.0 _ll_register_callbacks cb 0x7fbbc4009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd50bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55492/0 (socket says 172.21.15.28:55492) 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd50bb640 1 -- 172.21.15.28:0/909396451 learned_addr learned my addr 172.21.15.28:0/909396451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbcffff640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.567+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9ca040 0x5643de9d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 -- 172.21.15.28:0/909396451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9ca040 msgr2=0x5643de9d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9ca040 0x5643de9d4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 -- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 msgr2=0x5643de9d3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 -- 172.21.15.28:0/909396451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643de9c04c0 con 0x5643de9c44e0 2024-03-22T02:23:45.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd50bb640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8011730 tx=0x7fbbc800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef4852f32d953d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc80054d0 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbc8003cb0 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fbbc8005670 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fbbc8038460 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd97e5740 10 client.18474 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5643de9c4120 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbc8003cb0 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.568+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fbbc8018070 con 0x5643de9c44e0 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 10 client.18474 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 10 client.18474 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 20 client.18474 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:45.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643dea40d50 0x5643dea43140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 4 client.18474 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:23:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5643de9bb2b0 con 0x5643dea40d50 2024-03-22T02:23:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbcffff640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643dea40d50 0x5643dea43140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbcffff640 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643dea40d50 0x5643dea43140 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4db0706a3e8812ed server_cookie=4022d07fdac2cbbd in_seq=0 out_seq=0 2024-03-22T02:23:45.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.569+0000 7fbbaffff640 10 client.18474 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:45.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbaffff640 1 -- 172.21.15.28:0/909396451 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5643de9bb2b0 con 0x5643dea40d50 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbaffff640 10 client.18474 handle_command_reply: tid=0 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 client.18474 shutdown 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 msgr2=0x5643de9d32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7fbbc8011730 tx=0x7fbbc800f790 comp rx=0 tx=0).stop 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 shutdown_connections 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5643dea40d50 0x5643dea43140 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5643de9ca040 0x5643de9d4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbaffff640 0 client.18474 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5643de9c4ed0 0x5643de9d3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 --2- 172.21.15.28:0/909396451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5643de9c44e0 0x5643de9d32e0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 >> 172.21.15.28:0/909396451 conn(0x5643de9bfa50 msgr2=0x5643de9c6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 shutdown_connections 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 1 -- 172.21.15.28:0/909396451 wait complete. 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.884+0000 7fbbd97e5740 20 client.18474 trim_cache size 0 max 16384 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:23:45.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 msgr2=0x7fbbb80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 0x7fbbb80a5480 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0005ab0 tx=0x7fbbc00f4db0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 msgr2=0x7fbbd019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0000c00 tx=0x7fbbc002c640 comp rx=0 tx=0).stop 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 shutdown_connections 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbbb80a2fc0 0x7fbbb80a5480 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbd0102660 0x7fbbd01a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbbd0101ca0 0x7fbbd019a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 --2- 172.21.15.28:0/1231668779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbbd0100ab0 0x7fbbd019a070 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 >> 172.21.15.28:0/1231668779 conn(0x7fbbd00fc020 msgr2=0x7fbbd00fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 shutdown_connections 2024-03-22T02:23:45.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:45.885+0000 7fbbd6b1e640 1 -- 172.21.15.28:0/1231668779 wait complete. 2024-03-22T02:23:46.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1016030 11.48 MB/sec execute 388 sec latency 182.125 ms 2024-03-22T02:23:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:46 smithi120 ceph-mon[36803]: pgmap v665: 129 pgs: 129 active+clean; 62 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:23:46.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:46 smithi083 ceph-mon[36340]: pgmap v665: 129 pgs: 129 active+clean; 62 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:23:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:46 smithi028 ceph-mon[31916]: pgmap v665: 129 pgs: 129 active+clean; 62 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:23:47.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1016924 11.46 MB/sec execute 389 sec latency 161.485 ms 2024-03-22T02:23:48.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1019713 11.48 MB/sec execute 390 sec latency 11.568 ms 2024-03-22T02:23:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:48 smithi120 ceph-mon[36803]: pgmap v666: 129 pgs: 129 active+clean; 61 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-03-22T02:23:48.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:48 smithi083 ceph-mon[36340]: pgmap v666: 129 pgs: 129 active+clean; 61 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-03-22T02:23:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:48 smithi028 ceph-mon[31916]: pgmap v666: 129 pgs: 129 active+clean; 61 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-03-22T02:23:49.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1021459 11.46 MB/sec execute 391 sec latency 31.148 ms 2024-03-22T02:23:50.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1023680 11.47 MB/sec execute 392 sec latency 10.752 ms 2024-03-22T02:23:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:50 smithi120 ceph-mon[36803]: pgmap v667: 129 pgs: 129 active+clean; 53 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-03-22T02:23:50.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:50 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:23:50.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:50 smithi083 ceph-mon[36340]: pgmap v667: 129 pgs: 129 active+clean; 53 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-03-22T02:23:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:50 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:23:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:50 smithi028 ceph-mon[31916]: pgmap v667: 129 pgs: 129 active+clean; 53 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 381 op/s 2024-03-22T02:23:50.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:50 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:23:51.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1025606 11.46 MB/sec execute 393 sec latency 50.715 ms 2024-03-22T02:23:51.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:51 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:23:51.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:51 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:23:51.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:51 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:23:52.430 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1027236 11.46 MB/sec execute 394 sec latency 67.119 ms 2024-03-22T02:23:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:52 smithi120 ceph-mon[36803]: pgmap v668: 129 pgs: 129 active+clean; 54 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-22T02:23:52.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:52 smithi083 ceph-mon[36340]: pgmap v668: 129 pgs: 129 active+clean; 54 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-22T02:23:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:52 smithi028 ceph-mon[31916]: pgmap v668: 129 pgs: 129 active+clean; 54 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 347 op/s 2024-03-22T02:23:53.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1029666 11.46 MB/sec execute 395 sec latency 30.313 ms 2024-03-22T02:23:54.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1031272 11.45 MB/sec execute 396 sec latency 28.163 ms 2024-03-22T02:23:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:54 smithi120 ceph-mon[36803]: pgmap v669: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 383 op/s 2024-03-22T02:23:54.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:54 smithi083 ceph-mon[36340]: pgmap v669: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 383 op/s 2024-03-22T02:23:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:54 smithi028 ceph-mon[31916]: pgmap v669: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.7 MiB/s wr, 383 op/s 2024-03-22T02:23:55.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1034000 11.47 MB/sec execute 397 sec latency 12.983 ms 2024-03-22T02:23:56.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1035387 11.45 MB/sec execute 398 sec latency 54.837 ms 2024-03-22T02:23:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:56 smithi120 ceph-mon[36803]: pgmap v670: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-03-22T02:23:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:56 smithi083 ceph-mon[36340]: pgmap v670: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-03-22T02:23:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:56 smithi028 ceph-mon[31916]: pgmap v670: 129 pgs: 129 active+clean; 57 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 420 op/s 2024-03-22T02:23:57.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1037481 11.46 MB/sec execute 399 sec latency 110.639 ms 2024-03-22T02:23:58.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1039211 11.44 MB/sec execute 400 sec latency 37.008 ms 2024-03-22T02:23:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:23:58 smithi120 ceph-mon[36803]: pgmap v671: 129 pgs: 129 active+clean; 59 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 353 op/s 2024-03-22T02:23:58.764 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:23:58 smithi083 ceph-mon[36340]: pgmap v671: 129 pgs: 129 active+clean; 59 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 353 op/s 2024-03-22T02:23:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:23:58 smithi028 ceph-mon[31916]: pgmap v671: 129 pgs: 129 active+clean; 59 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 353 op/s 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.975+0000 7fdb5220e640 1 -- 172.21.15.28:0/1616967245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c104fa0 msgr2=0x7fdb4c107390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.975+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1616967245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c107390 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009900 tx=0x7fdb4002bc30 comp rx=0 tx=0).stop 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 -- 172.21.15.28:0/1616967245 shutdown_connections 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1616967245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 0x7fdb4c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1616967245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c107390 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1616967245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c102670 0x7fdb4c104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 -- 172.21.15.28:0/1616967245 >> 172.21.15.28:0/1616967245 conn(0x7fdb4c0fc020 msgr2=0x7fdb4c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 -- 172.21.15.28:0/1616967245 shutdown_connections 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 -- 172.21.15.28:0/1616967245 wait complete. 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 Processor -- start 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 -- start start 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.976+0000 7fdb5220e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 0x7fdb4c1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c10d830 con 0x7fdb4c102670 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c10d6b0 con 0x7fdb4c104fa0 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c10d9b0 con 0x7fdb4c1078d0 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43244/0 (socket says 172.21.15.28:43244) 2024-03-22T02:23:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/1949942577 learned_addr learned my addr 172.21.15.28:0/1949942577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 0x7fdb4c1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4affd640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/1949942577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 msgr2=0x7fdb4c1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 0x7fdb4c1a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/1949942577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c104fa0 msgr2=0x7fdb4c19a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c19a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/1949942577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4c1a1ca0 con 0x7fdb4c102670 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00e9f0 tx=0x7fdb3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb48ff9640 1 -- 172.21.15.28:0/1949942577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00cd80 con 0x7fdb4c102670 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4c1a1f90 con 0x7fdb4c102670 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb48ff9640 1 -- 172.21.15.28:0/1949942577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c010910 con 0x7fdb4c102670 2024-03-22T02:23:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb48ff9640 1 -- 172.21.15.28:0/1949942577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0188f0 con 0x7fdb4c102670 2024-03-22T02:23:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.977+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4c1a2400 con 0x7fdb4c102670 2024-03-22T02:23:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb48ff9640 1 -- 172.21.15.28:0/1949942577 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdb3c010ac0 con 0x7fdb4c102670 2024-03-22T02:23:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 Processor -- start 2024-03-22T02:23:58.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 -- start start 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a91900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a994d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5618d59f6040 con 0x5618d5a914e0 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618d599e9e0 con 0x5618d5a97040 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb54df6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618d59f1ff0 con 0x5618d5a91ed0 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a91900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35574/0 (socket says 172.21.15.28:35574) 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3604007315 learned_addr learned my addr 172.21.15.28:0/3604007315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.980+0000 7fdb4affd640 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb48ff9640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 0x7fdb340a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3604007315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 msgr2=0x5618d5a994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3604007315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 msgr2=0x5618d5a91900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a91900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3604007315 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d570a210 con 0x5618d5a91ed0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 0x7fdb340a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb48ff9640 1 -- 172.21.15.28:0/1949942577 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fdb3c0cfb90 con 0x7fdb4c102670 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c068be0 tx=0x7fdb3c066b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8152a38aab9575bb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb31ffb640 1 -- 172.21.15.28:0/3604007315 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c09baa0 con 0x5618d5a91ed0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 0x7fdb340a5480 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009fd0 tx=0x7fdb44004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb31ffb640 1 -- 172.21.15.28:0/3604007315 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c0d4040 con 0x5618d5a91ed0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.981+0000 7fdb31ffb640 1 -- 172.21.15.28:0/3604007315 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c068c10 con 0x5618d5a91ed0 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- 172.21.15.28:0/3604007315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 msgr2=0x5618d5a969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c068be0 tx=0x7fdb3c066b70 comp rx=0 tx=0).stop 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- 172.21.15.28:0/3604007315 shutdown_connections 2024-03-22T02:23:58.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5a969d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3604007315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a91900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- 172.21.15.28:0/3604007315 >> 172.21.15.28:0/3604007315 conn(0x5618d5a8ca50 msgr2=0x5618d5a8ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- 172.21.15.28:0/3604007315 shutdown_connections 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- 172.21.15.28:0/3604007315 wait complete. 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.982+0000 7fdb54df6740 1 -- start start 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 Processor -- start 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 0x5618d5a9b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 0x5618d5aa2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5618d5a99f90 con 0x5618d5a91ed0 2024-03-22T02:23:58.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618d59b1db0 con 0x5618d5a914e0 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618d5a9a110 con 0x5618d5a97040 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb54df6740 10 client.0 _ll_register_callbacks cb 0x5618d5709a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 0x5618d5aa2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 0x5618d5a9b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43258/0 (socket says 172.21.15.28:43258) 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 -- 172.21.15.28:0/2855651103 learned_addr learned my addr 172.21.15.28:0/2855651103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 -- 172.21.15.28:0/2855651103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 msgr2=0x5618d5aa2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 0x5618d5aa2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 -- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 msgr2=0x5618d5a9b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 0x5618d5a9b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4affd640 1 -- 172.21.15.28:0/2855651103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d57d6620 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 0x5618d5aa2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.983+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 0x5618d5a9b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb4affd640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c069d30 tx=0x7fdb40004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd7bc3d31b1a98e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40005d50 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fdb40005230 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb3c0d4040 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdb4004f050 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 10 client.18486 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5618d5aa5d80 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c0d4040 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fdb40003710 con 0x5618d5a91ed0 2024-03-22T02:23:58.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 10 client.18486 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 10 client.18486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 20 client.18486 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5aa7450 0x5618d5aa9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 4 client.18486 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.984+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618d5a03580 con 0x5618d5aa7450 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.985+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5aa7450 0x5618d5aa9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.985+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5aa7450 0x5618d5aa9880 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217bb3a07aadb38a server_cookie=cea26481d462f806 in_seq=0 out_seq=0 2024-03-22T02:23:58.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.985+0000 7fdb2affd640 10 client.18486 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.986+0000 7fdb2affd640 1 -- 172.21.15.28:0/2855651103 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618d5a03580 con 0x5618d5aa7450 2024-03-22T02:23:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.986+0000 7fdb2affd640 10 client.18486 handle_command_reply: tid=0 2024-03-22T02:23:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 client.18486 shutdown 2024-03-22T02:23:58.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 msgr2=0x5618d5a9b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c069d30 tx=0x7fdb40004250 comp rx=0 tx=0).stop 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 shutdown_connections 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5aa7450 0x5618d5aa9880 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a97040 0x5618d5aa2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a91ed0 0x5618d5a9b910 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb2affd640 0 client.18486 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 --2- 172.21.15.28:0/2855651103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a914e0 0x5618d5a9b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 >> 172.21.15.28:0/2855651103 conn(0x5618d5a8ca50 msgr2=0x5618d5a8bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 shutdown_connections 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 1 -- 172.21.15.28:0/2855651103 wait complete. 2024-03-22T02:23:58.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:58.987+0000 7fdb54df6740 20 client.18486 trim_cache size 0 max 16384 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.002+0000 7fdb54df6740 1 Processor -- start 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.002+0000 7fdb54df6740 1 -- start start 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.002+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.002+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5aa2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a901f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb54df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1af3a0 con 0x5618d5a914e0 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb54df6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1acc70 con 0x5618d5a97040 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb54df6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1aaac0 con 0x5618d5a91ed0 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43264/0 (socket says 172.21.15.28:43264) 2024-03-22T02:23:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3059814553 learned_addr learned my addr 172.21.15.28:0/3059814553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4affd640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5aa2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a901f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3059814553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 msgr2=0x5618d5aa2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5aa2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3059814553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 msgr2=0x5618d5a901f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a901f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 -- 172.21.15.28:0/3059814553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618d5a9a4b0 con 0x5618d5a914e0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a901f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0cd2d0 tx=0x7fdb3c066b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b9b77dba94edcc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb317fa640 1 -- 172.21.15.28:0/3059814553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c068e70 con 0x5618d5a914e0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb317fa640 1 -- 172.21.15.28:0/3059814553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c06e070 con 0x5618d5a914e0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.003+0000 7fdb317fa640 1 -- 172.21.15.28:0/3059814553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c09b8c0 con 0x5618d5a914e0 2024-03-22T02:23:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- 172.21.15.28:0/3059814553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 msgr2=0x5618d5a8fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0cd2d0 tx=0x7fdb3c066b70 comp rx=0 tx=0).stop 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- 172.21.15.28:0/3059814553 shutdown_connections 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a97040 0x5618d5a901f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a91ed0 0x5618d5aa2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 --2- 172.21.15.28:0/3059814553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a914e0 0x5618d5a8fcb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- 172.21.15.28:0/3059814553 >> 172.21.15.28:0/3059814553 conn(0x5618d5a8ca50 msgr2=0x5618d5a96460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- 172.21.15.28:0/3059814553 shutdown_connections 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- 172.21.15.28:0/3059814553 wait complete. 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 -- start start 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.004+0000 7fdb54df6740 1 Processor -- start 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a914e0 0x5618d5a9faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a91ed0 0x5618d5aa0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1ab130 con 0x5618d5a97040 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1ac5a0 con 0x5618d5a91ed0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c1ab800 con 0x5618d5a914e0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 10 client.0 _ll_register_callbacks cb 0x7fdb40009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a914e0 0x5618d5a9faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43266/0 (socket says 172.21.15.28:43266) 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 -- 172.21.15.28:0/1428948364 learned_addr learned my addr 172.21.15.28:0/1428948364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4affd640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a91ed0 0x5618d5aa0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 -- 172.21.15.28:0/1428948364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a914e0 msgr2=0x5618d5a9faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a914e0 0x5618d5a9faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 -- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a91ed0 msgr2=0x5618d5aa0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a91ed0 0x5618d5aa0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 -- 172.21.15.28:0/1428948364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618d5aa1630 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb4bfff640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400ff70 tx=0x7fdb44004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bc7c33ca0ad5da server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb2affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4402f060 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb3c06e070 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.005+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fdb4401b830 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fdb44036050 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 10 client.18492 fetch_fsmap learned FSMap version 36 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5618d5aa2a60 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c06e070 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fdb4402aba0 con 0x5618d5a97040 2024-03-22T02:23:59.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 10 client.18492 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 10 client.18492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 20 client.18492 populate_metadata read hostname 'smithi028' 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5b0dde0 0x5618d5b10200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 4 client.18492 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5618d59fd8d0 con 0x5618d5b0dde0 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.006+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5b0dde0 0x5618d5b10200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.007+0000 7fdb2affd640 10 client.18492 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:59.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.007+0000 7fdb4b7fe640 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5b0dde0 0x5618d5b10200 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7672e3ed082f51 server_cookie=9bd53904a7d275d6 in_seq=0 out_seq=0 2024-03-22T02:23:59.300 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb2affd640 1 -- 172.21.15.28:0/1428948364 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5618d59fd8d0 con 0x5618d5b0dde0 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb2affd640 10 client.18492 handle_command_reply: tid=0 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb54df6740 1 client.18492 shutdown 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 msgr2=0x5618d5aa1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400ff70 tx=0x7fdb44004ab0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 shutdown_connections 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.298+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618d5b0dde0 0x5618d5b10200 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618d5a97040 0x5618d5aa1040 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618d5a91ed0 0x5618d5aa0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb2affd640 0 client.18492 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 --2- 172.21.15.28:0/1428948364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618d5a914e0 0x5618d5a9faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 >> 172.21.15.28:0/1428948364 conn(0x5618d5a8ca50 msgr2=0x5618d5a96080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:59.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 shutdown_connections 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 1 -- 172.21.15.28:0/1428948364 wait complete. 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb54df6740 20 client.18492 trim_cache size 0 max 16384 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:23:59.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 msgr2=0x7fdb340a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 0x7fdb340a5480 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009fd0 tx=0x7fdb44004a40 comp rx=0 tx=0).stop 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 msgr2=0x7fdb4c19a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00e9f0 tx=0x7fdb3c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 shutdown_connections 2024-03-22T02:23:59.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdb340a2fc0 0x7fdb340a5480 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdb4c1078d0 0x7fdb4c1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdb4c104fa0 0x7fdb4c19a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 --2- 172.21.15.28:0/1949942577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb4c102670 0x7fdb4c19a080 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 >> 172.21.15.28:0/1949942577 conn(0x7fdb4c0fc020 msgr2=0x7fdb4c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 shutdown_connections 2024-03-22T02:23:59.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:23:59.299+0000 7fdb5220e640 1 -- 172.21.15.28:0/1949942577 wait complete. 2024-03-22T02:23:59.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1041106 11.45 MB/sec execute 401 sec latency 123.432 ms 2024-03-22T02:24:00.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1043264 11.44 MB/sec execute 402 sec latency 17.079 ms 2024-03-22T02:24:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:00 smithi120 ceph-mon[36803]: pgmap v672: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:24:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:00 smithi083 ceph-mon[36340]: pgmap v672: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:24:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:00 smithi028 ceph-mon[31916]: pgmap v672: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.0 MiB/s wr, 362 op/s 2024-03-22T02:24:01.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1045003 11.44 MB/sec execute 403 sec latency 17.932 ms 2024-03-22T02:24:02.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1046770 11.43 MB/sec execute 404 sec latency 92.995 ms 2024-03-22T02:24:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:02 smithi120 ceph-mon[36803]: pgmap v673: 129 pgs: 129 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-03-22T02:24:02.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:02 smithi083 ceph-mon[36340]: pgmap v673: 129 pgs: 129 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-03-22T02:24:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:02 smithi028 ceph-mon[31916]: pgmap v673: 129 pgs: 129 active+clean; 62 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-03-22T02:24:03.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1048573 11.43 MB/sec execute 405 sec latency 45.880 ms 2024-03-22T02:24:04.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1050974 11.43 MB/sec execute 406 sec latency 91.940 ms 2024-03-22T02:24:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:04 smithi120 ceph-mon[36803]: pgmap v674: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:24:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:04 smithi083 ceph-mon[36340]: pgmap v674: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:24:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:04 smithi028 ceph-mon[31916]: pgmap v674: 129 pgs: 129 active+clean; 58 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-03-22T02:24:05.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1052445 11.42 MB/sec execute 407 sec latency 37.805 ms 2024-03-22T02:24:06.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1055150 11.44 MB/sec execute 408 sec latency 20.574 ms 2024-03-22T02:24:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:06 smithi120 ceph-mon[36803]: pgmap v675: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 402 op/s 2024-03-22T02:24:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:06 smithi083 ceph-mon[36340]: pgmap v675: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 402 op/s 2024-03-22T02:24:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:06 smithi028 ceph-mon[31916]: pgmap v675: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 402 op/s 2024-03-22T02:24:07.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1056114 11.42 MB/sec execute 409 sec latency 63.458 ms 2024-03-22T02:24:08.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1058732 11.43 MB/sec execute 410 sec latency 44.240 ms 2024-03-22T02:24:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:08 smithi120 ceph-mon[36803]: pgmap v676: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:08.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:08 smithi083 ceph-mon[36340]: pgmap v676: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:08 smithi028 ceph-mon[31916]: pgmap v676: 129 pgs: 129 active+clean; 56 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:09.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1060375 11.42 MB/sec execute 411 sec latency 48.508 ms 2024-03-22T02:24:10.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1062545 11.42 MB/sec execute 412 sec latency 15.541 ms 2024-03-22T02:24:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:10 smithi120 ceph-mon[36803]: pgmap v677: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-22T02:24:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:10 smithi083 ceph-mon[36340]: pgmap v677: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-22T02:24:10.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:10 smithi028 ceph-mon[31916]: pgmap v677: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.4 MiB/s wr, 420 op/s 2024-03-22T02:24:11.431 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1064647 11.41 MB/sec execute 413 sec latency 30.261 ms 2024-03-22T02:24:11.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:24:11.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:24:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 -- 172.21.15.28:0/1133322240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38100ad0 msgr2=0x7f1c38100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 --2- 172.21.15.28:0/1133322240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38100ad0 0x7f1c38100ef0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009900 tx=0x7f1c2802bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 -- 172.21.15.28:0/1133322240 shutdown_connections 2024-03-22T02:24:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 --2- 172.21.15.28:0/1133322240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38102680 0x7f1c38108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 --2- 172.21.15.28:0/1133322240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38101cc0 0x7f1c38102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 --2- 172.21.15.28:0/1133322240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38100ad0 0x7f1c38100ef0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 -- 172.21.15.28:0/1133322240 >> 172.21.15.28:0/1133322240 conn(0x7f1c380fc040 msgr2=0x7f1c380fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 -- 172.21.15.28:0/1133322240 shutdown_connections 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.545+0000 7f1c3d320640 1 -- 172.21.15.28:0/1133322240 wait complete. 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 Processor -- start 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 -- start start 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38102680 0x7f1c3819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3810c720 con 0x7f1c38102680 2024-03-22T02:24:11.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3810c5a0 con 0x7f1c38101cc0 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c3d320640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c3810c8a0 con 0x7f1c38100ad0 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51314/0 (socket says 172.21.15.28:51314) 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:35136/0 (socket says 172.21.15.28:35136) 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c36ffd640 1 -- 172.21.15.28:0/3854273867 learned_addr learned my addr 172.21.15.28:0/3854273867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.546+0000 7f1c377fe640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38102680 0x7f1c3819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 -- 172.21.15.28:0/3854273867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 msgr2=0x7f1c381984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 -- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38102680 msgr2=0x7f1c3819f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38102680 0x7f1c3819f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 -- 172.21.15.28:0/3854273867 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c38102f60 con 0x7f1c38100ad0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c28000c00 tx=0x7f1c28004960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c34ff9640 1 -- 172.21.15.28:0/3854273867 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28041070 con 0x7f1c38100ad0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c34ff9640 1 -- 172.21.15.28:0/3854273867 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c28004450 con 0x7f1c38100ad0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c381031f0 con 0x7f1c38100ad0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c34ff9640 1 -- 172.21.15.28:0/3854273867 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28003d90 con 0x7f1c38100ad0 2024-03-22T02:24:11.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.547+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c38103730 con 0x7f1c38100ad0 2024-03-22T02:24:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.549+0000 7f1c34ff9640 1 -- 172.21.15.28:0/3854273867 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1c28044070 con 0x7f1c38100ad0 2024-03-22T02:24:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 Processor -- start 2024-03-22T02:24:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 -- start start 2024-03-22T02:24:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 0x55a671795920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 0x55a671798420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a6716f58c0 con 0x55a671795f90 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a67169dc20 con 0x55a671790430 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a6716f1870 con 0x55a671790e20 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c34ff9640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 0x7f1c240a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 0x55a671798420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c377fe640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 0x7f1c240a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 0x55a671795920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c377fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:35140/0 (socket says 172.21.15.28:35140) 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.550+0000 7f1c377fe640 1 -- 172.21.15.28:0/4125354555 learned_addr learned my addr 172.21.15.28:0/4125354555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c34ff9640 1 -- 172.21.15.28:0/3854273867 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1c280f1210 con 0x7f1c38100ad0 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 -- 172.21.15.28:0/4125354555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 msgr2=0x55a671795920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 0x55a671795920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 -- 172.21.15.28:0/4125354555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 msgr2=0x55a671790850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 -- 172.21.15.28:0/4125354555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6716c2150 con 0x55a671795f90 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 0x55a671795920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c377fe640 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 0x55a671798420 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009fa0 tx=0x7f1c200118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1661870fe04e933c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c377fe640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 0x7f1c240a5460 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x55a671791680 tx=0x7f1c30009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2d7fa640 1 -- 172.21.15.28:0/4125354555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c20004020 con 0x55a671795f90 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2d7fa640 1 -- 172.21.15.28:0/4125354555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c200041c0 con 0x55a671795f90 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.551+0000 7f1c2d7fa640 1 -- 172.21.15.28:0/4125354555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2000f800 con 0x55a671795f90 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/4125354555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 msgr2=0x55a671798420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 0x55a671798420 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1c20009fa0 tx=0x7f1c200118d0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/4125354555 shutdown_connections 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671795f90 0x55a671798420 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790e20 0x55a671795920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/4125354555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a671790850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/4125354555 >> 172.21.15.28:0/4125354555 conn(0x55a67178b9a0 msgr2=0x55a67178bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/4125354555 shutdown_connections 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.552+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/4125354555 wait complete. 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 -- start start 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 Processor -- start 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a6717aa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a671798f40 con 0x55a671790e20 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a6716b1130 con 0x55a671795f90 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a6717990c0 con 0x55a671790430 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c3ffe0740 10 client.0 _ll_register_callbacks cb 0x55a6716b0b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32772/0 (socket says 172.21.15.28:32772) 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c2ffff640 1 -- 172.21.15.28:0/784783068 learned_addr learned my addr 172.21.15.28:0/784783068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51334/0 (socket says 172.21.15.28:51334) 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.553+0000 7f1c377fe640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a6717aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 -- 172.21.15.28:0/784783068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 msgr2=0x55a6717a3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 -- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 msgr2=0x55a6717aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a6717aa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 -- 172.21.15.28:0/784783068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6714d37d0 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c377fe640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a6717aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1c20005bc0 tx=0x7f1c20009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867f1a3e6b2de712 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2001d050 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c200041c0 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1c20003ac0 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1c20032050 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c3ffe0740 10 client.18504 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a6714093f0 con 0x55a671790e20 2024-03-22T02:24:11.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c200041c0 con 0x55a671790e20 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.554+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f1c2000f5a0 con 0x55a671790e20 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 10 client.18504 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 10 client.18504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 20 client.18504 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a6717aedf0 0x55a6717b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 4 client.18504 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a671702e00 con 0x55a6717aedf0 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a6717aedf0 0x55a6717b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c1e7fc640 10 client.18504 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.555+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a6717aedf0 0x55a6717b1260 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e89a72e44573a2a2 server_cookie=4e216419abe3a18 in_seq=0 out_seq=0 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/784783068 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a671702e00 con 0x55a6717aedf0 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c1e7fc640 10 client.18504 handle_command_reply: tid=0 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 client.18504 shutdown 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 msgr2=0x55a6717a36c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f1c20005bc0 tx=0x7f1c20009d20 comp rx=0 tx=0).stop 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 shutdown_connections 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a6717aedf0 0x55a6717b1260 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a6717aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a36c0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/784783068 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c1e7fc640 0 client.18504 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:11.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 >> 172.21.15.28:0/784783068 conn(0x55a67178b9a0 msgr2=0x55a67178ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 shutdown_connections 2024-03-22T02:24:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/784783068 wait complete. 2024-03-22T02:24:11.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.556+0000 7f1c3ffe0740 20 client.18504 trim_cache size 0 max 16384 2024-03-22T02:24:11.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 Processor -- start 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 -- start start 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a67178bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 0x55a671792360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381ab160 con 0x55a671790e20 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381a8a30 con 0x55a671790430 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c3ffe0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381a6880 con 0x55a671795f90 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c377fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 0x55a671792360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c36ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a67178bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32788/0 (socket says 172.21.15.28:32788) 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.575+0000 7f1c2ffff640 1 -- 172.21.15.28:0/844591255 learned_addr learned my addr 172.21.15.28:0/844591255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 -- 172.21.15.28:0/844591255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 msgr2=0x55a671792360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 0x55a671792360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 -- 172.21.15.28:0/844591255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 msgr2=0x55a6717aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 -- 172.21.15.28:0/844591255 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6713c90f0 con 0x55a671790430 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c377fe640 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 0x55a671792360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a67178bd80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c280bd460 tx=0x7f1c28090200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bfdac16e68b1e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c2cff9640 1 -- 172.21.15.28:0/844591255 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2804b040 con 0x55a671790430 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c2cff9640 1 -- 172.21.15.28:0/844591255 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c280bdbd0 con 0x55a671790430 2024-03-22T02:24:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.576+0000 7f1c2cff9640 1 -- 172.21.15.28:0/844591255 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c28004d10 con 0x55a671790430 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/844591255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 msgr2=0x55a67178bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a67178bd80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1c280bd460 tx=0x7f1c28090200 comp rx=0 tx=0).stop 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/844591255 shutdown_connections 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671795f90 0x55a671792360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/844591255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671790430 0x55a67178bd80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/844591255 >> 172.21.15.28:0/844591255 conn(0x55a67178b9a0 msgr2=0x55a671799460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/844591255 shutdown_connections 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/844591255 wait complete. 2024-03-22T02:24:11.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 -- start start 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.577+0000 7f1c3ffe0740 1 Processor -- start 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a671813150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32790/0 (socket says 172.21.15.28:32790) 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 -- 172.21.15.28:0/3916474462 learned_addr learned my addr 172.21.15.28:0/3916474462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381a6ef0 con 0x55a671790e20 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381a8360 con 0x55a671795f90 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c381a75c0 con 0x55a671790430 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c377fe640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a671813150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 -- 172.21.15.28:0/3916474462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 msgr2=0x55a6717a76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 -- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 msgr2=0x55a671813150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a671813150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 10 client.0 _ll_register_callbacks cb 0x7f1c20009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c2ffff640 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a671813740 con 0x55a671790e20 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.578+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c280bdbd0 con 0x55a671790e20 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c2ffff640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1c200095f0 tx=0x7f1c2000feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea4f2d65bf21d251 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c1e7fc640 10 client.18510 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/3916474462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c2001d040 con 0x55a671790e20 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/3916474462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f1c20003ac0 con 0x55a671790e20 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/3916474462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1c20003d40 con 0x55a671790e20 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 10 client.18510 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=36}) -- 0x55a6718148c0 con 0x55a671790e20 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/3916474462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f1c20005040 con 0x55a671790e20 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 10 client.18510 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 10 client.18510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 20 client.18510 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a671815030 0x55a671817450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 4 client.18510 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.579+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6716fd150 con 0x55a671815030 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a671815030 0x55a671817450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c1e7fc640 10 client.18510 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c36ffd640 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a671815030 0x55a671817450 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e374d3668808c15 server_cookie=7777c873d4351052 in_seq=0 out_seq=0 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c1e7fc640 1 -- 172.21.15.28:0/3916474462 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a6716fd150 con 0x55a671815030 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c1e7fc640 10 client.18510 handle_command_reply: tid=0 2024-03-22T02:24:11.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 client.18510 shutdown 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 msgr2=0x55a6717a7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f1c200095f0 tx=0x7f1c2000feb0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 shutdown_connections 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a671815030 0x55a671817450 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a671795f90 0x55a671813150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a671790e20 0x55a6717a7be0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 --2- 172.21.15.28:0/3916474462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a671790430 0x55a6717a76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 >> 172.21.15.28:0/3916474462 conn(0x55a67178b9a0 msgr2=0x55a67178bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.580+0000 7f1c1e7fc640 0 client.18510 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 shutdown_connections 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3ffe0740 1 -- 172.21.15.28:0/3916474462 wait complete. 2024-03-22T02:24:11.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3ffe0740 20 client.18510 trim_cache size 0 max 16384 2024-03-22T02:24:11.588 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:11.588 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:24:11.588 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:24:11.588 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 msgr2=0x7f1c240a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 0x7f1c240a5460 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x55a671791680 tx=0x7f1c30009500 comp rx=0 tx=0).stop 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 msgr2=0x7f1c38197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1c28000c00 tx=0x7f1c28004960 comp rx=0 tx=0).stop 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 shutdown_connections 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c240a2fa0 0x7f1c240a5460 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c38102680 0x7f1c3819f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c38101cc0 0x7f1c381984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 --2- 172.21.15.28:0/3854273867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c38100ad0 0x7f1c38197f80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 >> 172.21.15.28:0/3854273867 conn(0x7f1c380fc040 msgr2=0x7f1c38069ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 shutdown_connections 2024-03-22T02:24:11.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.581+0000 7f1c3d320640 1 -- 172.21.15.28:0/3854273867 wait complete. 2024-03-22T02:24:11.593 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:24:11.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 -- 172.21.15.28:0/519798167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70101be0 msgr2=0x7f0a70102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 --2- 172.21.15.28:0/519798167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70101be0 0x7f0a70102000 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009900 tx=0x7f0a6002bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 -- 172.21.15.28:0/519798167 shutdown_connections 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 --2- 172.21.15.28:0/519798167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 0x7f0a7010a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 --2- 172.21.15.28:0/519798167 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70102dd0 0x7f0a70103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 --2- 172.21.15.28:0/519798167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70101be0 0x7f0a70102000 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 -- 172.21.15.28:0/519798167 >> 172.21.15.28:0/519798167 conn(0x7f0a700fd130 msgr2=0x7f0a700ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 -- 172.21.15.28:0/519798167 shutdown_connections 2024-03-22T02:24:11.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.705+0000 7f0a756aa640 1 -- 172.21.15.28:0/519798167 wait complete. 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 Processor -- start 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 -- start start 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 0x7f0a7019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 0x7f0a701a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a70075640 con 0x7f0a70102dd0 2024-03-22T02:24:11.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a700754c0 con 0x7f0a70101be0 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.706+0000 7f0a756aa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0a700757c0 con 0x7f0a70103790 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56068/0 (socket says 172.21.15.28:56068) 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 0x7f0a7019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6e7fc640 1 -- 172.21.15.28:0/1351955858 learned_addr learned my addr 172.21.15.28:0/1351955858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6f7fe640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 0x7f0a701a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 -- 172.21.15.28:0/1351955858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 msgr2=0x7f0a701a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 0x7f0a701a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 -- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 msgr2=0x7f0a7019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 -- 172.21.15.28:0/1351955858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a701a1d10 con 0x7f0a70101be0 2024-03-22T02:24:11.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6e7fc640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a6effd640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 0x7f0a7019a0f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009230 tx=0x7f0a6002c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60043020 con 0x7f0a70101be0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a701a1fa0 con 0x7f0a70101be0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a600054e0 con 0x7f0a70101be0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a60003b60 con 0x7f0a70101be0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.707+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a701a2410 con 0x7f0a70101be0 2024-03-22T02:24:11.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.709+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a70108e30 con 0x7f0a70101be0 2024-03-22T02:24:11.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.710+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0a60046020 con 0x7f0a70101be0 2024-03-22T02:24:11.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.711+0000 7f0a57fff640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 0x7f0a580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:11.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.711+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0a600f1a50 con 0x7f0a70101be0 2024-03-22T02:24:11.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.711+0000 7f0a6e7fc640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 0x7f0a580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:11.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.711+0000 7f0a6e7fc640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 0x7f0a580a53e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0a7019b600 tx=0x7f0a64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:11.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:11.715+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a6004d020 con 0x7f0a70101be0 2024-03-22T02:24:12.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.008+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a70062790 con 0x7f0a70101be0 2024-03-22T02:24:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.009+0000 7f0a57fff640 1 -- 172.21.15.28:0/1351955858 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11867 (secure 0 0 0) 0x7f0a600be470 con 0x7f0a70101be0 2024-03-22T02:24:12.010 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:24:12.010 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:22:55.371454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:24:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 msgr2=0x7f0a580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 0x7f0a580a53e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0a7019b600 tx=0x7f0a64005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 msgr2=0x7f0a7019a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 0x7f0a7019a0f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0a60009230 tx=0x7f0a6002c5d0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 shutdown_connections 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a580a2f20 0x7f0a580a53e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a70103790 0x7f0a701a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a70102dd0 0x7f0a7019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 --2- 172.21.15.28:0/1351955858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a70101be0 0x7f0a7019a0f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 >> 172.21.15.28:0/1351955858 conn(0x7f0a700fd130 msgr2=0x7f0a70106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 shutdown_connections 2024-03-22T02:24:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.011+0000 7f0a756aa640 1 -- 172.21.15.28:0/1351955858 wait complete. 2024-03-22T02:24:12.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 -- 172.21.15.28:0/818489053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4107890 msgr2=0x7f00a4109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/818489053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4107890 0x7f00a4109ce0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f009c009ca0 tx=0x7f009c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 -- 172.21.15.28:0/818489053 shutdown_connections 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/818489053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4107890 0x7f00a4109ce0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/818489053 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4104f60 0x7f00a4107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/818489053 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4102630 0x7f00a4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 -- 172.21.15.28:0/818489053 >> 172.21.15.28:0/818489053 conn(0x7f00a40fc040 msgr2=0x7f00a40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 -- 172.21.15.28:0/818489053 shutdown_connections 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.130+0000 7f00aa0b5640 1 -- 172.21.15.28:0/818489053 wait complete. 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 Processor -- start 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 -- start start 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4104f60 0x7f00a41983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4107890 0x7f00a419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410d850 con 0x7f00a4102630 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410d6d0 con 0x7f00a4104f60 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00aa0b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00a410d9d0 con 0x7f00a4107890 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56072/0 (socket says 172.21.15.28:56072) 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00a37fe640 1 -- 172.21.15.28:0/105483134 learned_addr learned my addr 172.21.15.28:0/105483134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4104f60 0x7f00a41983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.131+0000 7f00a3fff640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4107890 0x7f00a419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 -- 172.21.15.28:0/105483134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4107890 msgr2=0x7f00a419f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4107890 0x7f00a419f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 -- 172.21.15.28:0/105483134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4104f60 msgr2=0x7f00a41983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4104f60 0x7f00a41983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 -- 172.21.15.28:0/105483134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a419fab0 con 0x7f00a4102630 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a37fe640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f009400e9f0 tx=0x7f009400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a0ff9640 1 -- 172.21.15.28:0/105483134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009400cd80 con 0x7f00a4102630 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a419fda0 con 0x7f00a4102630 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a0ff9640 1 -- 172.21.15.28:0/105483134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0094010910 con 0x7f00a4102630 2024-03-22T02:24:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00a0ff9640 1 -- 172.21.15.28:0/105483134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00940188f0 con 0x7f00a4102630 2024-03-22T02:24:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.132+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a41a0210 con 0x7f00a4102630 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.134+0000 7f00a0ff9640 1 -- 172.21.15.28:0/105483134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0094010ac0 con 0x7f00a4102630 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.134+0000 7f00acd7b740 1 Processor -- start 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.134+0000 7f00acd7b740 1 -- start start 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c0136210 0x5621c013ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c013d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621c009a8d0 con 0x5621c013b380 2024-03-22T02:24:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621c0042e00 con 0x5621c0135820 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00acd7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621c0096880 con 0x5621c0136210 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c013d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c0136210 0x5621c013ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46216/0 (socket says 172.21.15.28:46216) 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a3fff640 1 -- 172.21.15.28:0/4178607979 learned_addr learned my addr 172.21.15.28:0/4178607979 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a0ff9640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 0x7f008c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 -- 172.21.15.28:0/4178607979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c0136210 msgr2=0x5621c013ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c0136210 0x5621c013ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 -- 172.21.15.28:0/4178607979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 msgr2=0x5621c0135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 -- 172.21.15.28:0/4178607979 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621bfdae3f0 con 0x5621c013b380 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a3fff640 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a3fff640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 0x7f008c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a0ff9640 1 -- 172.21.15.28:0/105483134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f00940cfb90 con 0x7f00a4102630 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c013d810 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f00a406ad00 tx=0x7f0098011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30ba0e374b4553b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.135+0000 7f0089ffb640 1 -- 172.21.15.28:0/4178607979 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098004230 con 0x5621c013b380 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f0089ffb640 1 -- 172.21.15.28:0/4178607979 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0098009e10 con 0x5621c013b380 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f0089ffb640 1 -- 172.21.15.28:0/4178607979 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098004d70 con 0x5621c013b380 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 -- 172.21.15.28:0/4178607979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 msgr2=0x5621c013d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00a3fff640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 0x7f008c0a53e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f009c006010 tx=0x7f009c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c013d810 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f00a406ad00 tx=0x7f0098011810 comp rx=0 tx=0).stop 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 -- 172.21.15.28:0/4178607979 shutdown_connections 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c013d810 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c0136210 0x5621c013ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 --2- 172.21.15.28:0/4178607979 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 -- 172.21.15.28:0/4178607979 >> 172.21.15.28:0/4178607979 conn(0x5621c0130d90 msgr2=0x5621c0131190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 -- 172.21.15.28:0/4178607979 shutdown_connections 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.136+0000 7f00acd7b740 1 -- 172.21.15.28:0/4178607979 wait complete. 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 -- start start 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 Processor -- start 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0136210 0x5621c0140280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c01472c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621c013e330 con 0x5621c0135820 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5621c0056300 con 0x5621c0136210 2024-03-22T02:24:12.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5621c013e4b0 con 0x5621c013b380 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00acd7b740 10 client.0 _ll_register_callbacks cb 0x5621c0091e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56094/0 (socket says 172.21.15.28:56094) 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 -- 172.21.15.28:0/545866863 learned_addr learned my addr 172.21.15.28:0/545866863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0136210 0x5621c0140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a3fff640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c01472c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 -- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 msgr2=0x5621c01472c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c01472c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 -- 172.21.15.28:0/545866863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0136210 msgr2=0x5621c0140280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0136210 0x5621c0140280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.137+0000 7f00a37fe640 1 -- 172.21.15.28:0/545866863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621bfe787d0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00a37fe640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f009400e9c0 tx=0x7f0094098200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb96a3ef2f7273e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009409b8c0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0094018db0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0098009e10 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f00940bc0b0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 10 client.18528 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5621c014a6f0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0098009e10 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f0094097cb0 con 0x5621c0135820 2024-03-22T02:24:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 10 client.18528 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 10 client.18528 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 20 client.18528 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c014be10 0x5621c014e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 4 client.18528 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621c00a7e10 con 0x5621c014be10 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.138+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c014be10 0x5621c014e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.139+0000 7f0082ffd640 10 client.18528 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.139+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c014be10 0x5621c014e2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16b84584dcd78067 server_cookie=9a7103ca799e21ff in_seq=0 out_seq=0 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.139+0000 7f0082ffd640 1 -- 172.21.15.28:0/545866863 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5621c00a7e10 con 0x5621c014be10 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.139+0000 7f0082ffd640 10 client.18528 handle_command_reply: tid=0 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 client.18528 shutdown 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 msgr2=0x5621c013fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f009400e9c0 tx=0x7f0094098200 comp rx=0 tx=0).stop 2024-03-22T02:24:12.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 shutdown_connections 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c01472c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c014be10 0x5621c014e2d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0136210 0x5621c0140280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f0082ffd640 0 client.18528 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 --2- 172.21.15.28:0/545866863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0135820 0x5621c013fd40 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 >> 172.21.15.28:0/545866863 conn(0x5621c0130d90 msgr2=0x5621c012ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 shutdown_connections 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 1 -- 172.21.15.28:0/545866863 wait complete. 2024-03-22T02:24:12.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.140+0000 7f00acd7b740 20 client.18528 trim_cache size 0 max 16384 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 Processor -- start 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 -- start start 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c01401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c0134700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41ad230 con 0x5621c0136210 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41aab00 con 0x5621c0135820 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00acd7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41a8950 con 0x5621c013b380 2024-03-22T02:24:12.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56102/0 (socket says 172.21.15.28:56102) 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 -- 172.21.15.28:0/2822292919 learned_addr learned my addr 172.21.15.28:0/2822292919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a3fff640 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c0134700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a37fe640 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c01401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 -- 172.21.15.28:0/2822292919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 msgr2=0x5621c0134700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c0134700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 -- 172.21.15.28:0/2822292919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 msgr2=0x5621c01401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c01401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.154+0000 7f00a2ffd640 1 -- 172.21.15.28:0/2822292919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621c01365f0 con 0x5621c0136210 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f00a406ad00 tx=0x7f0098011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4485f6925d9307 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00897fa640 1 -- 172.21.15.28:0/2822292919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098030070 con 0x5621c0136210 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00897fa640 1 -- 172.21.15.28:0/2822292919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0098004280 con 0x5621c0136210 2024-03-22T02:24:12.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00897fa640 1 -- 172.21.15.28:0/2822292919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009801e040 con 0x5621c0136210 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 -- 172.21.15.28:0/2822292919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 msgr2=0x5621c0147250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f00a406ad00 tx=0x7f0098011dc0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 -- 172.21.15.28:0/2822292919 shutdown_connections 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c013b380 0x5621c0134700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c0136210 0x5621c0147250 secure :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f00a406ad00 tx=0x7f0098011dc0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 --2- 172.21.15.28:0/2822292919 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c01401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 -- 172.21.15.28:0/2822292919 >> 172.21.15.28:0/2822292919 conn(0x5621c0130d90 msgr2=0x5621c0133ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 -- 172.21.15.28:0/2822292919 shutdown_connections 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.155+0000 7f00acd7b740 1 -- 172.21.15.28:0/2822292919 wait complete. 2024-03-22T02:24:12.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 -- start start 2024-03-22T02:24:12.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 Processor -- start 2024-03-22T02:24:12.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0145090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c01465e0 0x5621c0146a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41a8fc0 con 0x5621c013b380 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41aa430 con 0x5621c0135820 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f00a41a9690 con 0x5621c01465e0 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00acd7b740 10 client.0 _ll_register_callbacks cb 0x7f00940d3360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00a37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00a2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56116/0 (socket says 172.21.15.28:56116) 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00a2ffd640 1 -- 172.21.15.28:0/1920250716 learned_addr learned my addr 172.21.15.28:0/1920250716 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.156+0000 7f00a3fff640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c01465e0 0x5621c0146a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 -- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c01465e0 msgr2=0x5621c0146a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c01465e0 0x5621c0146a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 -- 172.21.15.28:0/1920250716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 msgr2=0x5621c01455d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 -- 172.21.15.28:0/1920250716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621c0146fa0 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00a37fe640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0145090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f009409bef0 tx=0x7f0094098330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a6c6aea7e10cac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009409b8c0 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0094018d50 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0098004280 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f009409bd30 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 10 client.28352 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5621c01b2d00 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0098004280 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f00940a4b10 con 0x5621c0135820 2024-03-22T02:24:12.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 10 client.28352 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 10 client.28352 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 20 client.28352 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.157+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c01b3470 0x5621c01b5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 4 client.28352 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621c01372b0 con 0x5621c01b3470 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c01b3470 0x5621c01b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00a2ffd640 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c01b3470 0x5621c01b5890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7b57d9399bbe2e server_cookie=854f0295f72bd5a5 in_seq=0 out_seq=0 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f0082ffd640 10 client.28352 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f0082ffd640 1 -- 172.21.15.28:0/1920250716 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621c01372b0 con 0x5621c01b3470 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f0082ffd640 10 client.28352 handle_command_reply: tid=0 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 client.28352 shutdown 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 msgr2=0x5621c0145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0145090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f009409bef0 tx=0x7f0094098330 comp rx=0 tx=0).stop 2024-03-22T02:24:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 shutdown_connections 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5621c01465e0 0x5621c0146a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621c013b380 0x5621c01455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5621c01b3470 0x5621c01b5890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 --2- 172.21.15.28:0/1920250716 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5621c0135820 0x5621c0145090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 >> 172.21.15.28:0/1920250716 conn(0x5621c0130d90 msgr2=0x5621c0134880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f0082ffd640 0 client.28352 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 shutdown_connections 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 1 -- 172.21.15.28:0/1920250716 wait complete. 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.158+0000 7f00acd7b740 20 client.28352 trim_cache size 0 max 16384 2024-03-22T02:24:12.166 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 msgr2=0x7f008c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 0x7f008c0a53e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f009c006010 tx=0x7f009c02e040 comp rx=0 tx=0).stop 2024-03-22T02:24:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 msgr2=0x7f00a4197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f009400e9f0 tx=0x7f009400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 shutdown_connections 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f008c0a2f20 0x7f008c0a53e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00a4107890 0x7f00a419f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f00a4104f60 0x7f00a41983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 --2- 172.21.15.28:0/105483134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00a4102630 0x7f00a4197e90 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 >> 172.21.15.28:0/105483134 conn(0x7f00a40fc040 msgr2=0x7f00a4102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 shutdown_connections 2024-03-22T02:24:12.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.159+0000 7f00aa0b5640 1 -- 172.21.15.28:0/105483134 wait complete. 2024-03-22T02:24:12.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:24:12.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- 172.21.15.28:0/32485355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 msgr2=0x7f5ca0109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/32485355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0109cc0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5c98009ca0 tx=0x7f5c9802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- 172.21.15.28:0/32485355 shutdown_connections 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/32485355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0109cc0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/32485355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/32485355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0102670 0x7f5ca0104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- 172.21.15.28:0/32485355 >> 172.21.15.28:0/32485355 conn(0x7f5ca00fc020 msgr2=0x7f5ca00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- 172.21.15.28:0/32485355 shutdown_connections 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- 172.21.15.28:0/32485355 wait complete. 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 Processor -- start 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.276+0000 7f5ca66aa640 1 -- start start 2024-03-22T02:24:12.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 0x7f5ca0101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010d830 con 0x7f5ca01078d0 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010d6b0 con 0x7f5ca0104fa0 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca010d9b0 con 0x7f5ca0102670 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 0x7f5ca0101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56124/0 (socket says 172.21.15.28:56124) 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 -- 172.21.15.28:0/2810037869 learned_addr learned my addr 172.21.15.28:0/2810037869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 -- 172.21.15.28:0/2810037869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 msgr2=0x7f5ca0101fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 0x7f5ca0101fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 -- 172.21.15.28:0/2810037869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 msgr2=0x7f5ca0100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 -- 172.21.15.28:0/2810037869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca01013f0 con 0x7f5ca01078d0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 0x7f5ca0101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca4c47640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f5c98037040 tx=0x7f5c98032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9d7fa640 1 -- 172.21.15.28:0/2810037869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98041020 con 0x7f5ca01078d0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9d7fa640 1 -- 172.21.15.28:0/2810037869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c98034cd0 con 0x7f5ca01078d0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca01a1f70 con 0x7f5ca01078d0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.277+0000 7f5c9d7fa640 1 -- 172.21.15.28:0/2810037869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98039040 con 0x7f5ca01078d0 2024-03-22T02:24:12.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.278+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca01a23d0 con 0x7f5ca01078d0 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9d7fa640 1 -- 172.21.15.28:0/2810037869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5c98044020 con 0x7f5ca01078d0 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 Processor -- start 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 -- start start 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 0x55ae318a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 0x55ae318af810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae3180c8d0 con 0x55ae318ad380 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae317b4e00 con 0x55ae318a8210 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca9368740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae31808880 con 0x55ae318a7820 2024-03-22T02:24:12.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9d7fa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 0x7f5c880a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 0x55ae318af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46262/0 (socket says 172.21.15.28:46262) 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2633659228 learned_addr learned my addr 172.21.15.28:0/2633659228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5ca4c47640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 0x7f5c880a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.280+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 0x55ae318af810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56126/0 (socket says 172.21.15.28:56126) 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 0x55ae318a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9d7fa640 1 -- 172.21.15.28:0/2810037869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5c980f1610 con 0x7f5ca01078d0 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2633659228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 msgr2=0x55ae318a7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 0x55ae318a7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2633659228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 msgr2=0x55ae318af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 0x55ae318af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2633659228 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae315203f0 con 0x55ae318a8210 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 0x55ae318a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5c900115f0 tx=0x7f5c90011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6900d7e248b8c147 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5ca4c47640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 0x7f5c880a53e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b9e20 tx=0x7f5c980095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c867fc640 1 -- 172.21.15.28:0/2633659228 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9001e070 con 0x55ae318a8210 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c867fc640 1 -- 172.21.15.28:0/2633659228 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c9000fb20 con 0x55ae318a8210 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.281+0000 7f5c867fc640 1 -- 172.21.15.28:0/2633659228 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c90019bf0 con 0x55ae318a8210 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- 172.21.15.28:0/2633659228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 msgr2=0x55ae318acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5c900115f0 tx=0x7f5c90011ab0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- 172.21.15.28:0/2633659228 shutdown_connections 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318ad380 0x55ae318af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a8210 0x55ae318acd10 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2633659228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a7820 0x55ae318a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- 172.21.15.28:0/2633659228 >> 172.21.15.28:0/2633659228 conn(0x55ae318a2d90 msgr2=0x55ae318a3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- 172.21.15.28:0/2633659228 shutdown_connections 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- 172.21.15.28:0/2633659228 wait complete. 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 -- start start 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.282+0000 7f5ca9368740 1 Processor -- start 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318b9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ae318b0330 con 0x55ae318a8210 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ae317c8300 con 0x55ae318a7820 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ae318b04b0 con 0x55ae318ad380 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 10 client.0 _ll_register_callbacks cb 0x55ae31803e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46266/0 (socket says 172.21.15.28:46266) 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9f7fe640 1 -- 172.21.15.28:0/2566495819 learned_addr learned my addr 172.21.15.28:0/2566495819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 msgr2=0x55ae318b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2566495819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 msgr2=0x55ae318b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 -- 172.21.15.28:0/2566495819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae315ea7d0 con 0x55ae318a8210 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b22d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x55ae318a3d00 tx=0x7f5c9000f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4da0e3abc53c94a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9001e040 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5c90004040 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.283+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c9000fb20 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5c9002f050 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 10 client.18546 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae318bc680 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c9000fb20 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f5c9002a030 con 0x55ae318a8210 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 10 client.18546 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 10 client.18546 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 20 client.18546 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae318bda80 0x55ae318bfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 4 client.18546 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:12.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae31819e10 con 0x55ae318bda80 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.284+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae318bda80 0x55ae318bfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae318bda80 0x55ae318bfeb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9abb35f606bce55e server_cookie=a464b8610f79a4db in_seq=0 out_seq=0 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5c7f7fe640 10 client.18546 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/2566495819 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae31819e10 con 0x55ae318bda80 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5c7f7fe640 10 client.18546 handle_command_reply: tid=0 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 client.18546 shutdown 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 msgr2=0x55ae318b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b22d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x55ae318a3d00 tx=0x7f5c9000f8f0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 shutdown_connections 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae318bda80 0x55ae318bfeb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b22d0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5c7f7fe640 0 client.18546 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 --2- 172.21.15.28:0/2566495819 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.285+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 >> 172.21.15.28:0/2566495819 conn(0x55ae318a2d90 msgr2=0x55ae318a1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.286+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 shutdown_connections 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.286+0000 7f5ca9368740 1 -- 172.21.15.28:0/2566495819 wait complete. 2024-03-22T02:24:12.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.286+0000 7f5ca9368740 20 client.18546 trim_cache size 0 max 16384 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 Processor -- start 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 -- start start 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318a6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318c47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01af4c0 con 0x55ae318a8210 2024-03-22T02:24:12.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01acd90 con 0x55ae318a7820 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5ca9368740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01aabe0 con 0x55ae318ad380 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56142/0 (socket says 172.21.15.28:56142) 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 -- 172.21.15.28:0/4210967863 learned_addr learned my addr 172.21.15.28:0/4210967863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318c47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 -- 172.21.15.28:0/4210967863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 msgr2=0x55ae318c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318c47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 -- 172.21.15.28:0/4210967863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 msgr2=0x55ae318a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 -- 172.21.15.28:0/4210967863 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae318c0530 con 0x55ae318a8210 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.299+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f5c900035f0 tx=0x7f5c90005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219967fb3690329 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5c85ffb640 1 -- 172.21.15.28:0/4210967863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9001e070 con 0x55ae318a8210 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5c85ffb640 1 -- 172.21.15.28:0/4210967863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c90003ce0 con 0x55ae318a8210 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5c85ffb640 1 -- 172.21.15.28:0/4210967863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c900052f0 con 0x55ae318a8210 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- 172.21.15.28:0/4210967863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 msgr2=0x55ae318b92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f5c900035f0 tx=0x7f5c90005040 comp rx=0 tx=0).stop 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- 172.21.15.28:0/4210967863 shutdown_connections 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318ad380 0x55ae318c47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a8210 0x55ae318b92d0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 --2- 172.21.15.28:0/4210967863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318a7820 0x55ae318a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- 172.21.15.28:0/4210967863 >> 172.21.15.28:0/4210967863 conn(0x55ae318a2d90 msgr2=0x55ae318a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- 172.21.15.28:0/4210967863 shutdown_connections 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- 172.21.15.28:0/4210967863 wait complete. 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.300+0000 7f5ca9368740 1 -- start start 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 Processor -- start 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a8210 0x55ae318b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318ad380 0x55ae318b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01ab250 con 0x55ae318a7820 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01ac6c0 con 0x55ae318ad380 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca01ab920 con 0x55ae318a8210 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5ca9368740 10 client.0 _ll_register_callbacks cb 0x7f5c9001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56154/0 (socket says 172.21.15.28:56154) 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 -- 172.21.15.28:0/1453969869 learned_addr learned my addr 172.21.15.28:0/1453969869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a8210 0x55ae318b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318ad380 0x55ae318b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 -- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a8210 msgr2=0x55ae318b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a8210 0x55ae318b6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 -- 172.21.15.28:0/1453969869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318ad380 msgr2=0x55ae318b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318ad380 0x55ae318b7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.301+0000 7f5c9f7fe640 1 -- 172.21.15.28:0/1453969869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae318b80d0 con 0x55ae318a7820 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c9f7fe640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x55ae318c0730 tx=0x7f5c900056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df2def6304e6031 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9402d070 con 0x55ae318a7820 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c90003ce0 con 0x55ae318a7820 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5c9400fbd0 con 0x55ae318a7820 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5c94036050 con 0x55ae318a7820 2024-03-22T02:24:12.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 10 client.18552 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ae318b94a0 con 0x55ae318a7820 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c90003ce0 con 0x55ae318a7820 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f5c94030020 con 0x55ae318a7820 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 10 client.18552 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 10 client.18552 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 20 client.18552 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae31924460 0x55ae31926880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 4 client.18552 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ae318b1b80 con 0x55ae31924460 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.302+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae31924460 0x55ae31926880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5c9ffff640 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae31924460 0x55ae31926880 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f59c49ae78e56603 server_cookie=b1ae99e23539e4c6 in_seq=0 out_seq=0 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5c7f7fe640 10 client.18552 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:12.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5c7f7fe640 1 -- 172.21.15.28:0/1453969869 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ae318b1b80 con 0x55ae31924460 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5c7f7fe640 10 client.18552 handle_command_reply: tid=0 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 client.18552 shutdown 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 msgr2=0x55ae318b6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x55ae318c0730 tx=0x7f5c900056f0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 shutdown_connections 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ae31924460 0x55ae31926880 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ae318ad380 0x55ae318b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5c7f7fe640 0 client.18552 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ae318a8210 0x55ae318b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 --2- 172.21.15.28:0/1453969869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ae318a7820 0x55ae318b6600 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 >> 172.21.15.28:0/1453969869 conn(0x55ae318a2d90 msgr2=0x55ae318a6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 shutdown_connections 2024-03-22T02:24:12.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 1 -- 172.21.15.28:0/1453969869 wait complete. 2024-03-22T02:24:12.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.303+0000 7f5ca9368740 20 client.18552 trim_cache size 0 max 16384 2024-03-22T02:24:12.312 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 msgr2=0x7f5c880a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 0x7f5c880a53e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b9e20 tx=0x7f5c980095c0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 msgr2=0x7f5ca0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f5c98037040 tx=0x7f5c98032f40 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 shutdown_connections 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c880a2f20 0x7f5c880a53e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca01078d0 0x7f5ca0100b60 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5ca0104fa0 0x7f5ca0100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 --2- 172.21.15.28:0/2810037869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ca0102670 0x7f5ca0101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 >> 172.21.15.28:0/2810037869 conn(0x7f5ca00fc020 msgr2=0x7f5ca0102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 shutdown_connections 2024-03-22T02:24:12.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.304+0000 7f5ca66aa640 1 -- 172.21.15.28:0/2810037869 wait complete. 2024-03-22T02:24:12.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:24:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 -- 172.21.15.28:0/775282608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 msgr2=0x7fb5ac100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/775282608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac100d40 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009900 tx=0x7fb59c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 -- 172.21.15.28:0/775282608 shutdown_connections 2024-03-22T02:24:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/775282608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 0x7fb5ac108d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.399 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/775282608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/775282608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac100d40 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 -- 172.21.15.28:0/775282608 >> 172.21.15.28:0/775282608 conn(0x7fb5ac0fbe90 msgr2=0x7fb5ac0fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 -- 172.21.15.28:0/775282608 shutdown_connections 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.396+0000 7fb5b17a2640 1 -- 172.21.15.28:0/775282608 wait complete. 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 Processor -- start 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 -- start start 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 0x7fb5ac19f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.400 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10c5a0 con 0x7fb5ac100920 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10c420 con 0x7fb5ac101b10 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b17a2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac10c720 con 0x7fb5ac1024d0 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56172/0 (socket says 172.21.15.28:56172) 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5abfff640 1 -- 172.21.15.28:0/2321984520 learned_addr learned my addr 172.21.15.28:0/2321984520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.397+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 0x7fb5ac19f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2321984520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 msgr2=0x7fb5ac19f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 0x7fb5ac19f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 msgr2=0x7fb5ac197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.401 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2321984520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac19f970 con 0x7fb5ac101b10 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac198290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000e9f0 tx=0x7fb5a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5a97fa640 1 -- 172.21.15.28:0/2321984520 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a000ccc0 con 0x7fb5ac101b10 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac19fc60 con 0x7fb5ac101b10 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5a97fa640 1 -- 172.21.15.28:0/2321984520 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a0010910 con 0x7fb5ac101b10 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5a97fa640 1 -- 172.21.15.28:0/2321984520 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0018960 con 0x7fb5ac101b10 2024-03-22T02:24:12.402 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.398+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac1a01b0 con 0x7fb5ac101b10 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.400+0000 7fb5a97fa640 1 -- 172.21.15.28:0/2321984520 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb5a0010ac0 con 0x7fb5ac101b10 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5a97fa640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 0x7fb5940a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 Processor -- start 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 -- start start 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5a97fa640 1 -- 172.21.15.28:0/2321984520 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb5a00cfb60 con 0x7fb5ac101b10 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 0x7fb5940a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a69ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6a4b0 0x564fc6a6efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564fc69ca040 con 0x564fc6a6f620 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564fc69729e0 con 0x564fc6a69ac0 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564fc69c5ff0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b0fa1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5abfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a69ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b0fa1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56176/0 (socket says 172.21.15.28:56176) 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/2296244605 learned_addr learned my addr 172.21.15.28:0/2296244605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6a4b0 0x564fc6a6efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.401+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/2296244605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6a4b0 msgr2=0x564fc6a6efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6a4b0 0x564fc6a6efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/2296244605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 msgr2=0x564fc6a69ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a69ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.410 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/2296244605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fc66de210 con 0x564fc6a6f620 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 0x7fb5940a53d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb59c02c8e0 tx=0x7fb59c004200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009fd0 tx=0x7fb5a40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a6c5353725c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5927fc640 1 -- 172.21.15.28:0/2296244605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4004d20 con 0x564fc6a6f620 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5927fc640 1 -- 172.21.15.28:0/2296244605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a4004050 con 0x564fc6a6f620 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.402+0000 7fb5927fc640 1 -- 172.21.15.28:0/2296244605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4005280 con 0x564fc6a6f620 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2296244605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 msgr2=0x564fc6a71ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009fd0 tx=0x7fb5a40118d0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2296244605 shutdown_connections 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6f620 0x564fc6a71ab0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6a4b0 0x564fc6a6efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2296244605 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a69ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2296244605 >> 172.21.15.28:0/2296244605 conn(0x564fc6a60c20 msgr2=0x564fc6a61020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2296244605 shutdown_connections 2024-03-22T02:24:12.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2296244605 wait complete. 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 -- start start 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 Processor -- start 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.403+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a7af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564fc6a725d0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564fc6985db0 con 0x564fc6a6f620 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564fc6a72750 con 0x564fc6a69ac0 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b55ea740 10 client.0 _ll_register_callbacks cb 0x564fc66dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56178/0 (socket says 172.21.15.28:56178) 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2192918602 learned_addr learned my addr 172.21.15.28:0/2192918602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a7af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2192918602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 msgr2=0x564fc6a739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a739d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 msgr2=0x564fc6a7af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a7af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/2192918602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fc67aa620 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a7af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0097c10 tx=0x7fb5a00bc0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e333124acf232438 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb58b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a009b580 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a4004050 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.404+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb5a0010430 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb5a00a4890 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 10 client.18564 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564fc6a7e330 con 0x564fc6a6a4b0 2024-03-22T02:24:12.413 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a4004050 con 0x564fc6a6a4b0 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb5a009adf0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 10 client.18564 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 10 client.18564 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 20 client.18564 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6a7fa70 0x564fc6a81e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 4 client.18564 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564fc69d7580 con 0x564fc6a7fa70 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.405+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6a7fa70 0x564fc6a81e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.406+0000 7fb5abfff640 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6a7fa70 0x564fc6a81e60 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef69bb364f9b40d server_cookie=96334f5b2c732890 in_seq=0 out_seq=0 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.406+0000 7fb58b7fe640 10 client.18564 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb58b7fe640 1 -- 172.21.15.28:0/2192918602 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564fc69d7580 con 0x564fc6a7fa70 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb58b7fe640 10 client.18564 handle_command_reply: tid=0 2024-03-22T02:24:12.414 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 client.18564 shutdown 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 msgr2=0x564fc6a73f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0097c10 tx=0x7fb5a00bc0b0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 shutdown_connections 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6a7fa70 0x564fc6a81e60 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a7af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a73f10 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb58b7fe640 0 client.18564 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/2192918602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 >> 172.21.15.28:0/2192918602 conn(0x564fc6a60c20 msgr2=0x564fc6a5fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 shutdown_connections 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 1 -- 172.21.15.28:0/2192918602 wait complete. 2024-03-22T02:24:12.415 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.407+0000 7fb5b55ea740 20 client.18564 trim_cache size 0 max 16384 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.422+0000 7fb5b55ea740 1 Processor -- start 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.422+0000 7fb5b55ea740 1 -- start start 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.422+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a6b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.422+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.422+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 0x564fc6a86720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1aaf40 con 0x564fc6a6a4b0 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1a8810 con 0x564fc6a69ac0 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1a6660 con 0x564fc6a6f620 2024-03-22T02:24:12.424 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 0x564fc6a86720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56184/0 (socket says 172.21.15.28:56184) 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/548979750 learned_addr learned my addr 172.21.15.28:0/548979750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5abfff640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a6b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/548979750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 msgr2=0x564fc6a6b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a6b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/548979750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 msgr2=0x564fc6a7af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 -- 172.21.15.28:0/548979750 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564fc6a5bd30 con 0x564fc6a6f620 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5abfff640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a6b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 0x564fc6a86720 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a401b040 tx=0x7fb5a4005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1684d4467b3ca97 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb591ffb640 1 -- 172.21.15.28:0/548979750 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4003b70 con 0x564fc6a6f620 2024-03-22T02:24:12.425 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.423+0000 7fb591ffb640 1 -- 172.21.15.28:0/548979750 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a4003d10 con 0x564fc6a6f620 2024-03-22T02:24:12.426 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb591ffb640 1 -- 172.21.15.28:0/548979750 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400f9e0 con 0x564fc6a6f620 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 -- 172.21.15.28:0/548979750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 msgr2=0x564fc6a86720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 0x564fc6a86720 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5a401b040 tx=0x7fb5a4005c70 comp rx=0 tx=0).stop 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 -- 172.21.15.28:0/548979750 shutdown_connections 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a6f620 0x564fc6a86720 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a7af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/548979750 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a69ac0 0x564fc6a6b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 -- 172.21.15.28:0/548979750 >> 172.21.15.28:0/548979750 conn(0x564fc6a60c20 msgr2=0x564fc6a6af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 -- 172.21.15.28:0/548979750 shutdown_connections 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.424+0000 7fb5b55ea740 1 -- 172.21.15.28:0/548979750 wait complete. 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 -- start start 2024-03-22T02:24:12.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 Processor -- start 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a78690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a79be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1a6cd0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1a8140 con 0x564fc6a6f620 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1a73a0 con 0x564fc6a69ac0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b55ea740 10 client.0 _ll_register_callbacks cb 0x7fb5a009bd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56200/0 (socket says 172.21.15.28:56200) 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/3410336196 learned_addr learned my addr 172.21.15.28:0/3410336196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5abfff640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a78690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5b0fa1640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a79be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/3410336196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 msgr2=0x564fc6a78690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a78690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 msgr2=0x564fc6a79be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a79be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.425+0000 7fb5ab7fe640 1 -- 172.21.15.28:0/3410336196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564fc6a69440 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5ab7fe640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00a4a50 tx=0x7fb5a00bc1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73007213f882f3fd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00981d0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb5a009b380 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5a4003d10 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb5a00a1520 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 10 client.18570 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564fc6a7b1a0 con 0x564fc6a6a4b0 2024-03-22T02:24:12.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5a4003d10 con 0x564fc6a6a4b0 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb5a00b9070 con 0x564fc6a6a4b0 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 10 client.18570 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 10 client.18570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 20 client.18570 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6aeaa80 0x564fc6aecea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 4 client.18570 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.426+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564fc69d18d0 con 0x564fc6aeaa80 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.427+0000 7fb5abfff640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6aeaa80 0x564fc6aecea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.427+0000 7fb58b7fe640 10 client.18570 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.427+0000 7fb5abfff640 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6aeaa80 0x564fc6aecea0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45e2c83f40076d7e server_cookie=99391f39857fdbd5 in_seq=0 out_seq=0 2024-03-22T02:24:12.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1066221 11.41 MB/sec execute 414 sec latency 74.257 ms 2024-03-22T02:24:12.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.434+0000 7fb25bd63640 1 -- 172.21.15.28:0/2644768629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254100ab0 msgr2=0x7fb254100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.434+0000 7fb25bd63640 1 --2- 172.21.15.28:0/2644768629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254100ab0 0x7fb254100ed0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb250009900 tx=0x7fb25002bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- 172.21.15.28:0/2644768629 shutdown_connections 2024-03-22T02:24:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- 172.21.15.28:0/2644768629 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254102660 0x7fb254108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- 172.21.15.28:0/2644768629 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254101ca0 0x7fb254102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- 172.21.15.28:0/2644768629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254100ab0 0x7fb254100ed0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- 172.21.15.28:0/2644768629 >> 172.21.15.28:0/2644768629 conn(0x7fb2540fc040 msgr2=0x7fb2540fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- 172.21.15.28:0/2644768629 shutdown_connections 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- 172.21.15.28:0/2644768629 wait complete. 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 Processor -- start 2024-03-22T02:24:12.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- start start 2024-03-22T02:24:12.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254100ab0 0x7fb254196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254101ca0 0x7fb254197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb25410c780 con 0x7fb254102660 2024-03-22T02:24:12.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb25410c600 con 0x7fb254100ab0 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25bd63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb25410c900 con 0x7fb254101ca0 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25a300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25a300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56206/0 (socket says 172.21.15.28:56206) 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb25a300640 1 -- 172.21.15.28:0/442941463 learned_addr learned my addr 172.21.15.28:0/442941463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.435+0000 7fb259aff640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254100ab0 0x7fb254196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb2592fe640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254101ca0 0x7fb254197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 -- 172.21.15.28:0/442941463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254101ca0 msgr2=0x7fb254197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254101ca0 0x7fb254197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 -- 172.21.15.28:0/442941463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254100ab0 msgr2=0x7fb254196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254100ab0 0x7fb254196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 -- 172.21.15.28:0/442941463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25419ea10 con 0x7fb254102660 2024-03-22T02:24:12.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25a300640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00eca0 tx=0x7fb24c00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb24affd640 1 -- 172.21.15.28:0/442941463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c004b20 con 0x7fb254102660 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb254069f70 con 0x7fb254102660 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb24affd640 1 -- 172.21.15.28:0/442941463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb24c004cc0 con 0x7fb254102660 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb24affd640 1 -- 172.21.15.28:0/442941463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb24c017670 con 0x7fb254102660 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.436+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb25406a510 con 0x7fb254102660 2024-03-22T02:24:12.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.438+0000 7fb24affd640 1 -- 172.21.15.28:0/442941463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb24c005020 con 0x7fb254102660 2024-03-22T02:24:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 Processor -- start 2024-03-22T02:24:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 -- start start 2024-03-22T02:24:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 0x5617b8640c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b8645d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8646380 0x5617b8648810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617b85a58d0 con 0x5617b8641210 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5617b854de00 con 0x5617b8640820 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617b85a1880 con 0x5617b8646380 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb24affd640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 0x7fb2400a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb259aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b8645d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 0x5617b8640c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb259aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b8645d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56212/0 (socket says 172.21.15.28:56212) 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb259aff640 1 -- 172.21.15.28:0/2151059952 learned_addr learned my addr 172.21.15.28:0/2151059952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb2592fe640 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8646380 0x5617b8648810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.439+0000 7fb25a300640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 0x7fb2400a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb24affd640 1 -- 172.21.15.28:0/442941463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb24c0d1270 con 0x7fb254102660 2024-03-22T02:24:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 -- 172.21.15.28:0/2151059952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8646380 msgr2=0x5617b8648810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8646380 0x5617b8648810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 -- 172.21.15.28:0/2151059952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 msgr2=0x5617b8645d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b8645d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 -- 172.21.15.28:0/2151059952 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b82b93f0 con 0x5617b8640820 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb25a300640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 0x7fb2400a53e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb24c09d570 tx=0x7fb24c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb2592fe640 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 0x5617b8640c40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb244009730 tx=0x7fb2440096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc82589871071842 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb23ffff640 1 -- 172.21.15.28:0/2151059952 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244004050 con 0x5617b8640820 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb23ffff640 1 -- 172.21.15.28:0/2151059952 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb244005500 con 0x5617b8640820 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.440+0000 7fb23ffff640 1 -- 172.21.15.28:0/2151059952 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244004ce0 con 0x5617b8640820 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- 172.21.15.28:0/2151059952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 msgr2=0x5617b8640c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 0x5617b8640c40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb244009730 tx=0x7fb2440096c0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- 172.21.15.28:0/2151059952 shutdown_connections 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8646380 0x5617b8648810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b8645d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/2151059952 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8640820 0x5617b8640c40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- 172.21.15.28:0/2151059952 >> 172.21.15.28:0/2151059952 conn(0x5617b863bd90 msgr2=0x5617b863c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- 172.21.15.28:0/2151059952 shutdown_connections 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- 172.21.15.28:0/2151059952 wait complete. 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 -- start start 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.441+0000 7fb25ea1f740 1 Processor -- start 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b864ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b8652290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617b8649330 con 0x5617b8641210 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5617b8561300 con 0x5617b8646380 2024-03-22T02:24:12.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617b86494b0 con 0x5617b8640820 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 10 client.0 _ll_register_callbacks cb 0x5617b859ce30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b864ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56224/0 (socket says 172.21.15.28:56224) 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 -- 172.21.15.28:0/3774659672 learned_addr learned my addr 172.21.15.28:0/3774659672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b8652290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 -- 172.21.15.28:0/3774659672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 msgr2=0x5617b864ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b864ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 -- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 msgr2=0x5617b8652290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b8652290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 -- 172.21.15.28:0/3774659672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b83837d0 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb259aff640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fb250009230 tx=0x7fb250004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=260ffea9aff103d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb23d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb250005d50 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb250005040 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.442+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb244005500 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb250005240 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 10 client.18582 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5617b8655650 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb244005500 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb2500035f0 con 0x5617b8641210 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 10 client.18582 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 10 client.18582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:12.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 20 client.18582 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b8656d70 0x5617b86591d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 4 client.18582 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617b85b2e10 con 0x5617b8656d70 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.443+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b8656d70 0x5617b86591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.444+0000 7fb23d7fa640 10 client.18582 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.444+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b8656d70 0x5617b86591d0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ad201ec97d33d9 server_cookie=70e8349ef60e4f05 in_seq=0 out_seq=0 2024-03-22T02:24:12.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:12 smithi028 ceph-mon[31916]: pgmap v678: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:24:12.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:12 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1351955858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:12 smithi120 ceph-mon[36803]: pgmap v678: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:24:12.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1351955858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb58b7fe640 1 -- 172.21.15.28:0/3410336196 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564fc69d18d0 con 0x564fc6aeaa80 2024-03-22T02:24:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb58b7fe640 10 client.18570 handle_command_reply: tid=0 2024-03-22T02:24:12.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb5b55ea740 1 client.18570 shutdown 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 msgr2=0x564fc6a78bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fb5a00a4a50 tx=0x7fb5a00bc1e0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.750+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 shutdown_connections 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564fc6aeaa80 0x564fc6aecea0 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb58b7fe640 0 client.18570 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.752 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564fc6a6f620 0x564fc6a79be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3774659672 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617b85b2e10 con 0x5617b8656d70 2024-03-22T02:24:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564fc6a6a4b0 0x564fc6a78bd0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 --2- 172.21.15.28:0/3410336196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564fc6a69ac0 0x564fc6a78690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 >> 172.21.15.28:0/3410336196 conn(0x564fc6a60c20 msgr2=0x564fc6a6b790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb23d7fa640 10 client.18582 handle_command_reply: tid=0 2024-03-22T02:24:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 client.18582 shutdown 2024-03-22T02:24:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 shutdown_connections 2024-03-22T02:24:12.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 1 -- 172.21.15.28:0/3410336196 wait complete. 2024-03-22T02:24:12.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 msgr2=0x5617b864b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7fb250009230 tx=0x7fb250004270 comp rx=0 tx=0).stop 2024-03-22T02:24:12.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb5b55ea740 20 client.18570 trim_cache size 0 max 16384 2024-03-22T02:24:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 shutdown_connections 2024-03-22T02:24:12.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b8656d70 0x5617b86591d0 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b8652290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8641210 0x5617b864b250 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3774659672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b864ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 >> 172.21.15.28:0/3774659672 conn(0x5617b863bd90 msgr2=0x5617b863afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:24:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:24:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:24:12.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:24:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb23d7fa640 0 client.18582 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 shutdown_connections 2024-03-22T02:24:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3774659672 wait complete. 2024-03-22T02:24:12.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.751+0000 7fb25ea1f740 20 client.18582 trim_cache size 0 max 16384 2024-03-22T02:24:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 msgr2=0x7fb5940a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 0x7fb5940a53d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb59c02c8e0 tx=0x7fb59c004200 comp rx=0 tx=0).stop 2024-03-22T02:24:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 msgr2=0x7fb5ac198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac198290 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000e9f0 tx=0x7fb5a000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 shutdown_connections 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5940a2f10 0x7fb5940a53d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5ac1024d0 0x7fb5ac19f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb5ac101b10 0x7fb5ac198290 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 --2- 172.21.15.28:0/2321984520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5ac100920 0x7fb5ac197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 >> 172.21.15.28:0/2321984520 conn(0x7fb5ac0fbe90 msgr2=0x7fb5ac0fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 shutdown_connections 2024-03-22T02:24:12.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:12.752+0000 7fb5b17a2640 1 -- 172.21.15.28:0/2321984520 wait complete. 2024-03-22T02:24:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:12 smithi083 ceph-mon[36340]: pgmap v678: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.0 MiB/s wr, 402 op/s 2024-03-22T02:24:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:12 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1351955858' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.772+0000 7fb25ea1f740 1 Processor -- start 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 -- start start 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8641210 0x5617b8652250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b865da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541ab100 con 0x5617b8640820 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a89d0 con 0x5617b8646380 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a6820 con 0x5617b8641210 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56236/0 (socket says 172.21.15.28:56236) 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 -- 172.21.15.28:0/1177837422 learned_addr learned my addr 172.21.15.28:0/1177837422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb259aff640 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8641210 0x5617b8652250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b865da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 -- 172.21.15.28:0/1177837422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8641210 msgr2=0x5617b8652250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8641210 0x5617b8652250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 -- 172.21.15.28:0/1177837422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 msgr2=0x5617b865da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b865da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 -- 172.21.15.28:0/1177837422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b86459c0 con 0x5617b8640820 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.773+0000 7fb2592fe640 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x5617b8641a70 tx=0x7fb2440096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4996d1a272e0128e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb23f7fe640 1 -- 172.21.15.28:0/1177837422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244024030 con 0x5617b8640820 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb23f7fe640 1 -- 172.21.15.28:0/1177837422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb24400fc70 con 0x5617b8640820 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb23f7fe640 1 -- 172.21.15.28:0/1177837422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb244031070 con 0x5617b8640820 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 -- 172.21.15.28:0/1177837422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 msgr2=0x5617b863ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x5617b8641a70 tx=0x7fb2440096c0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 -- 172.21.15.28:0/1177837422 shutdown_connections 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8646380 0x5617b865da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8641210 0x5617b8652250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/1177837422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8640820 0x5617b863ae50 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.774+0000 7fb25ea1f740 1 -- 172.21.15.28:0/1177837422 >> 172.21.15.28:0/1177837422 conn(0x5617b863bd90 msgr2=0x5617b863a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- 172.21.15.28:0/1177837422 shutdown_connections 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- 172.21.15.28:0/1177837422 wait complete. 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- start start 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 Processor -- start 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b8650090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 0x5617b86505d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a6e90 con 0x5617b8646380 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a8300 con 0x5617b8641210 2024-03-22T02:24:12.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2541a7560 con 0x5617b8640820 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb25ea1f740 10 client.0 _ll_register_callbacks cb 0x7fb250009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb2592fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56246/0 (socket says 172.21.15.28:56246) 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb259aff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 0x5617b86505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.775+0000 7fb2592fe640 1 -- 172.21.15.28:0/3531979699 learned_addr learned my addr 172.21.15.28:0/3531979699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b8650090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 -- 172.21.15.28:0/3531979699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 msgr2=0x5617b8650090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b8650090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 -- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 msgr2=0x5617b86505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 0x5617b86505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 -- 172.21.15.28:0/3531979699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617b8651b80 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb259aff640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 0x5617b86505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb2592fe640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb244011750 tx=0x7fb244033040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6718b5e2f2396e9d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb23d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2440112b0 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb24400f790 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.776+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb24400fc70 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb24400f990 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 10 client.18588 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5617b86bd6d0 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb24400fc70 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fb24403d040 con 0x5617b8646380 2024-03-22T02:24:12.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 10 client.18588 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 10 client.18588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 20 client.18588 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b86bde80 0x5617b86c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 4 client.18588 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617b864ab00 con 0x5617b86bde80 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.777+0000 7fb259aff640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b86bde80 0x5617b86c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb23d7fa640 10 client.18588 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb259aff640 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b86bde80 0x5617b86c02a0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a383a56fa1c10d server_cookie=eaca2cafd3527c33 in_seq=0 out_seq=0 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb23d7fa640 1 -- 172.21.15.28:0/3531979699 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617b864ab00 con 0x5617b86bde80 2024-03-22T02:24:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb23d7fa640 10 client.18588 handle_command_reply: tid=0 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 client.18588 shutdown 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 msgr2=0x5617b8651590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb244011750 tx=0x7fb244033040 comp rx=0 tx=0).stop 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 shutdown_connections 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617b86bde80 0x5617b86c02a0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617b8646380 0x5617b8651590 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617b8641210 0x5617b86505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 --2- 172.21.15.28:0/3531979699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617b8640820 0x5617b8650090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb23d7fa640 0 client.18588 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 >> 172.21.15.28:0/3531979699 conn(0x5617b863bd90 msgr2=0x5617b863f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 shutdown_connections 2024-03-22T02:24:12.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 1 -- 172.21.15.28:0/3531979699 wait complete. 2024-03-22T02:24:12.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.778+0000 7fb25ea1f740 20 client.18588 trim_cache size 0 max 16384 2024-03-22T02:24:12.788 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 msgr2=0x7fb2400a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 0x7fb2400a53e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb24c09d570 tx=0x7fb24c000f40 comp rx=0 tx=0).stop 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 msgr2=0x7fb25419e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00eca0 tx=0x7fb24c00b690 comp rx=0 tx=0).stop 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 shutdown_connections 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2400a2f20 0x7fb2400a53e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb254102660 0x7fb25419e3a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb254101ca0 0x7fb254197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 --2- 172.21.15.28:0/442941463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb254100ab0 0x7fb254196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 >> 172.21.15.28:0/442941463 conn(0x7fb2540fc040 msgr2=0x7fb254105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 shutdown_connections 2024-03-22T02:24:12.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:12.779+0000 7fb25bd63640 1 -- 172.21.15.28:0/442941463 wait complete. 2024-03-22T02:24:13.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1067645 11.40 MB/sec execute 415 sec latency 41.789 ms 2024-03-22T02:24:13.792 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:24:13.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:24:13.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 -- 172.21.15.28:0/1741300184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8104fa0 msgr2=0x7f0ab8105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 --2- 172.21.15.28:0/1741300184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8104fa0 0x7f0ab8105420 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009900 tx=0x7f0aa402bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 -- 172.21.15.28:0/1741300184 shutdown_connections 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 --2- 172.21.15.28:0/1741300184 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8105960 0x7f0ab810c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 --2- 172.21.15.28:0/1741300184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8104fa0 0x7f0ab8105420 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 --2- 172.21.15.28:0/1741300184 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8103db0 0x7f0ab81041d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 -- 172.21.15.28:0/1741300184 >> 172.21.15.28:0/1741300184 conn(0x7f0ab80ff340 msgr2=0x7f0ab8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 -- 172.21.15.28:0/1741300184 shutdown_connections 2024-03-22T02:24:13.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.904+0000 7f0abeb93640 1 -- 172.21.15.28:0/1741300184 wait complete. 2024-03-22T02:24:13.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 Processor -- start 2024-03-22T02:24:13.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 -- start start 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 0x7f0ab819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 0x7f0ab81a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810fa30 con 0x7f0ab8103db0 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810f8b0 con 0x7f0ab8105960 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abeb93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab810fbb0 con 0x7f0ab8104fa0 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56264/0 (socket says 172.21.15.28:56264) 2024-03-22T02:24:13.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abc92f640 1 -- 172.21.15.28:0/2128405699 learned_addr learned my addr 172.21.15.28:0/2128405699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 0x7f0ab819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.905+0000 7f0abd130640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 0x7f0ab81a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 -- 172.21.15.28:0/2128405699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 msgr2=0x7f0ab819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 0x7f0ab819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 -- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 msgr2=0x7f0ab819a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 -- 172.21.15.28:0/2128405699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ab81a1cc0 con 0x7f0ab8105960 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abc92f640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 0x7f0ab819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abd130640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 0x7f0ab81a1650 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009fd0 tx=0x7f0ab000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/2128405699 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab000ce20 con 0x7f0ab8105960 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/2128405699 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab0005020 con 0x7f0ab8105960 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ab81a1fb0 con 0x7f0ab8105960 2024-03-22T02:24:13.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/2128405699 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0010720 con 0x7f0ab8105960 2024-03-22T02:24:13.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.906+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ab81a2470 con 0x7f0ab8105960 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.908+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/2128405699 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0ab0005540 con 0x7f0ab8105960 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 Processor -- start 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 -- start start 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ab5ffb640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 0x7f0aa80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf6cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf720b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5568ebed1570 con 0x5568ebf72720 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568ebe79e10 con 0x5568ebf6d5b0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ac177c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5568ebecd520 con 0x5568ebf6cbc0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ab5ffb640 1 -- 172.21.15.28:0/2128405699 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0ab00cfe90 con 0x7f0ab8105960 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 0x7f0aa80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf6cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56280/0 (socket says 172.21.15.28:56280) 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 -- 172.21.15.28:0/3624930434 learned_addr learned my addr 172.21.15.28:0/3624930434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf720b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 -- 172.21.15.28:0/3624930434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 msgr2=0x5568ebf6cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf6cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 -- 172.21.15.28:0/3624930434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 msgr2=0x5568ebf720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf720b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.909+0000 7f0abd130640 1 -- 172.21.15.28:0/3624930434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ebcad7e0 con 0x5568ebf72720 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0abd130640 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0098b20 tx=0x7f0ab0003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2094562552f7811 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 0x7f0aa80a53e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0aa402c7b0 tx=0x7f0aa40095e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0aa2ffd640 1 -- 172.21.15.28:0/3624930434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab009c150 con 0x5568ebf72720 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0aa2ffd640 1 -- 172.21.15.28:0/3624930434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00a02a0 con 0x5568ebf72720 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0aa2ffd640 1 -- 172.21.15.28:0/3624930434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00987c0 con 0x5568ebf72720 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 -- 172.21.15.28:0/3624930434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 msgr2=0x5568ebf74bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0098b20 tx=0x7f0ab0003c50 comp rx=0 tx=0).stop 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 -- 172.21.15.28:0/3624930434 shutdown_connections 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf74bb0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf720b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.910+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3624930434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf6cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- 172.21.15.28:0/3624930434 >> 172.21.15.28:0/3624930434 conn(0x5568ebf68130 msgr2=0x5568ebf68530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- 172.21.15.28:0/3624930434 shutdown_connections 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- 172.21.15.28:0/3624930434 wait complete. 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- start start 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 Processor -- start 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 0x5568ebf77320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 0x5568ebf7e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5568ebf756d0 con 0x5568ebf6cbc0 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5568ebe8d110 con 0x5568ebf72720 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5568ebf75850 con 0x5568ebf6d5b0 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ac177c740 10 client.0 _ll_register_callbacks cb 0x5568ebec9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56282/0 (socket says 172.21.15.28:56282) 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0abc92f640 1 -- 172.21.15.28:0/3065460912 learned_addr learned my addr 172.21.15.28:0/3065460912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0abd130640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 0x5568ebf7e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.911+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 0x5568ebf77320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 -- 172.21.15.28:0/3065460912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 msgr2=0x5568ebf77320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 0x5568ebf77320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 -- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 msgr2=0x5568ebf7e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 0x5568ebf7e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 -- 172.21.15.28:0/3065460912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ebd0df70 con 0x5568ebf6cbc0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abd130640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 0x5568ebf7e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 0x5568ebf77320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0abc92f640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0aac012d20 tx=0x7f0aac00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9ab7799ca04a76 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0a97fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aac004050 con 0x5568ebf6cbc0 2024-03-22T02:24:13.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0aac0041f0 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab00a02a0 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0aac025050 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0ac177c740 10 client.18600 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5568ebba5170 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.912+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00a02a0 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f0aac010430 con 0x5568ebf6cbc0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 10 client.18600 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 10 client.18600 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 20 client.18600 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebf82cf0 0x5568ebf85190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 4 client.18600 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568ebedeab0 con 0x5568ebf82cf0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebf82cf0 0x5568ebf85190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0a97fff640 10 client.18600 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.913+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebf82cf0 0x5568ebf85190 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c5b96539094f7c server_cookie=a2350d0a8904d81c in_seq=0 out_seq=0 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0a97fff640 1 -- 172.21.15.28:0/3065460912 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5568ebedeab0 con 0x5568ebf82cf0 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0a97fff640 10 client.18600 handle_command_reply: tid=0 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 client.18600 shutdown 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 msgr2=0x5568ebf76de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0aac012d20 tx=0x7f0aac00f790 comp rx=0 tx=0).stop 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 shutdown_connections 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebf82cf0 0x5568ebf85190 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf72720 0x5568ebf7e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6d5b0 0x5568ebf77320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3065460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf76de0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0a97fff640 0 client.18600 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:13.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 >> 172.21.15.28:0/3065460912 conn(0x5568ebf68130 msgr2=0x5568ebf673f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 shutdown_connections 2024-03-22T02:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 1 -- 172.21.15.28:0/3065460912 wait complete. 2024-03-22T02:24:13.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.914+0000 7f0ac177c740 20 client.18600 trim_cache size 0 max 16384 2024-03-22T02:24:13.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.932+0000 7f0ac177c740 1 Processor -- start 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.932+0000 7f0ac177c740 1 -- start start 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf6e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 0x5568ebf6e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81af3d0 con 0x5568ebf6cbc0 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81acca0 con 0x5568ebf6d5b0 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ac177c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81aaaf0 con 0x5568ebf72720 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56292/0 (socket says 172.21.15.28:56292) 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 0x5568ebf6e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abc92f640 1 -- 172.21.15.28:0/3911049689 learned_addr learned my addr 172.21.15.28:0/3911049689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:13.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf6e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 -- 172.21.15.28:0/3911049689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 msgr2=0x5568ebf6e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf6e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 -- 172.21.15.28:0/3911049689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 msgr2=0x5568ebf71de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 -- 172.21.15.28:0/3911049689 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568ebf68ea0 con 0x5568ebf72720 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abc92f640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0ab7fff640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf6e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0abd130640 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 0x5568ebf6e590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f80 tx=0x7f0ab009c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=310bd73e3c91e92d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0aa27fc640 1 -- 172.21.15.28:0/3911049689 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab009c150 con 0x5568ebf72720 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.933+0000 7f0aa27fc640 1 -- 172.21.15.28:0/3911049689 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00a4b60 con 0x5568ebf72720 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0aa27fc640 1 -- 172.21.15.28:0/3911049689 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00a46d0 con 0x5568ebf72720 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 -- 172.21.15.28:0/3911049689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 msgr2=0x5568ebf6e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 0x5568ebf6e590 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f80 tx=0x7f0ab009c990 comp rx=0 tx=0).stop 2024-03-22T02:24:13.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 -- 172.21.15.28:0/3911049689 shutdown_connections 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf72720 0x5568ebf6e590 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf6e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 --2- 172.21.15.28:0/3911049689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf6cbc0 0x5568ebf71de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 -- 172.21.15.28:0/3911049689 >> 172.21.15.28:0/3911049689 conn(0x5568ebf68130 msgr2=0x5568ebf6bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 -- 172.21.15.28:0/3911049689 shutdown_connections 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.934+0000 7f0ac177c740 1 -- 172.21.15.28:0/3911049689 wait complete. 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 -- start start 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 Processor -- start 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf7ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81ab160 con 0x5568ebf72720 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81ac5d0 con 0x5568ebf6d5b0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ab81ab830 con 0x5568ebf6cbc0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ac177c740 10 client.0 _ll_register_callbacks cb 0x7f0aac009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ab7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf7ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ab7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf7ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46464/0 (socket says 172.21.15.28:46464) 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56300/0 (socket says 172.21.15.28:56300) 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0ab7fff640 1 -- 172.21.15.28:0/762803832 learned_addr learned my addr 172.21.15.28:0/762803832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abc92f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37074/0 (socket says 172.21.15.28:37074) 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 -- 172.21.15.28:0/762803832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 msgr2=0x5568ebf7a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 -- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 msgr2=0x5568ebf7ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf7ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abd130640 1 -- 172.21.15.28:0/762803832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568ebf7c370 con 0x5568ebf72720 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.935+0000 7f0abc92f640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0abd130640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x5568ebedeb30 tx=0x7f0ab0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec6741392c43b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab007e020 con 0x5568ebf72720 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ab00a4b60 con 0x5568ebf72720 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0ab00bc2d0 con 0x5568ebf72720 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0ab00b5280 con 0x5568ebf72720 2024-03-22T02:24:13.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 10 client.18606 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5568ebf7d6b0 con 0x5568ebf72720 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ab00a4b60 con 0x5568ebf72720 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f0ab0085120 con 0x5568ebf72720 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 10 client.18606 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 10 client.18606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 20 client.18606 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebfe90d0 0x5568ebfeb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 4 client.18606 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.936+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5568ebed8e00 con 0x5568ebfe90d0 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.937+0000 7f0abc92f640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebfe90d0 0x5568ebfeb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.937+0000 7f0abc92f640 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebfe90d0 0x5568ebfeb4c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35611f4b211726bb server_cookie=a3abdbf4413e259b in_seq=0 out_seq=0 2024-03-22T02:24:13.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.937+0000 7f0a97fff640 10 client.18606 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0a97fff640 1 -- 172.21.15.28:0/762803832 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5568ebed8e00 con 0x5568ebfe90d0 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0a97fff640 10 client.18606 handle_command_reply: tid=0 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 client.18606 shutdown 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 msgr2=0x5568ebf7bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x5568ebedeb30 tx=0x7f0ab0005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 shutdown_connections 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5568ebfe90d0 0x5568ebfeb4c0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5568ebf72720 0x5568ebf7bd80 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5568ebf6d5b0 0x5568ebf7ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 --2- 172.21.15.28:0/762803832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5568ebf6cbc0 0x5568ebf7a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 >> 172.21.15.28:0/762803832 conn(0x5568ebf68130 msgr2=0x5568ebf6bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0a97fff640 0 client.18606 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.938+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 shutdown_connections 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0ac177c740 1 -- 172.21.15.28:0/762803832 wait complete. 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0ac177c740 20 client.18606 trim_cache size 0 max 16384 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2fa38424-ccf8-4ee7-92cd-375c6bf42a6f", 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:24:13.946 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 msgr2=0x7f0aa80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 0x7f0aa80a53e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0aa402c7b0 tx=0x7f0aa40095e0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 msgr2=0x7f0ab81a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:13.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 0x7f0ab81a1650 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009fd0 tx=0x7f0ab000ef10 comp rx=0 tx=0).stop 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 shutdown_connections 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0aa80a2f20 0x7f0aa80a53e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ab8105960 0x7f0ab81a1650 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ab8104fa0 0x7f0ab819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 --2- 172.21.15.28:0/2128405699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8103db0 0x7f0ab819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 >> 172.21.15.28:0/2128405699 conn(0x7f0ab80ff340 msgr2=0x7f0ab81082a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 shutdown_connections 2024-03-22T02:24:13.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:13.939+0000 7f0abeb93640 1 -- 172.21.15.28:0/2128405699 wait complete. 2024-03-22T02:24:13.951 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:24:13.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:24:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 -- 172.21.15.28:0/1047063025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 msgr2=0x7fcf801032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 --2- 172.21.15.28:0/1047063025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf801032c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009930 tx=0x7fcf7002bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 -- 172.21.15.28:0/1047063025 shutdown_connections 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 --2- 172.21.15.28:0/1047063025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf80103800 0x7fcf8010a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 --2- 172.21.15.28:0/1047063025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf801032c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 --2- 172.21.15.28:0/1047063025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf800ffe70 0x7fcf80100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 -- 172.21.15.28:0/1047063025 >> 172.21.15.28:0/1047063025 conn(0x7fcf800fb7a0 msgr2=0x7fcf800fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 -- 172.21.15.28:0/1047063025 shutdown_connections 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.058+0000 7fcf884b5640 1 -- 172.21.15.28:0/1047063025 wait complete. 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 Processor -- start 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 -- start start 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 0x7fcf80197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf80103800 0x7fcf8019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8010d830 con 0x7fcf80102e40 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8010d6b0 con 0x7fcf800ffe70 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf884b5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8010d9b0 con 0x7fcf80103800 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56324/0 (socket says 172.21.15.28:56324) 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 -- 172.21.15.28:0/3978042709 learned_addr learned my addr 172.21.15.28:0/3978042709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf86251640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 0x7fcf80197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf86a52640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf80103800 0x7fcf8019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 -- 172.21.15.28:0/3978042709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf80103800 msgr2=0x7fcf8019f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf80103800 0x7fcf8019f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 -- 172.21.15.28:0/3978042709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 msgr2=0x7fcf80197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.059+0000 7fcf85a50640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 0x7fcf80197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf85a50640 1 -- 172.21.15.28:0/3978042709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8019faf0 con 0x7fcf80102e40 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf86251640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 0x7fcf80197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf85a50640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000c00 tx=0x7fcf7002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf777fe640 1 -- 172.21.15.28:0/3978042709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70041070 con 0x7fcf80102e40 2024-03-22T02:24:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8019fd80 con 0x7fcf80102e40 2024-03-22T02:24:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf777fe640 1 -- 172.21.15.28:0/3978042709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf70004450 con 0x7fcf80102e40 2024-03-22T02:24:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf777fe640 1 -- 172.21.15.28:0/3978042709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf70034440 con 0x7fcf80102e40 2024-03-22T02:24:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.060+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf801a0210 con 0x7fcf80102e40 2024-03-22T02:24:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.062+0000 7fcf777fe640 1 -- 172.21.15.28:0/3978042709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fcf70044070 con 0x7fcf80102e40 2024-03-22T02:24:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 Processor -- start 2024-03-22T02:24:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 -- start start 2024-03-22T02:24:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf777fe640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 0x7fcf6c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c23b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c23e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c19b8c0 con 0x55ca9c236e20 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c143c20 con 0x55ca9c236430 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c197870 con 0x55ca9c23bf90 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf86a52640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 0x7fcf6c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c23b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf86251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf777fe640 1 -- 172.21.15.28:0/3978042709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcf700f12c0 con 0x7fcf80102e40 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf86a52640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c23e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf86251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46490/0 (socket says 172.21.15.28:46490) 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c23b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56334/0 (socket says 172.21.15.28:56334) 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.063+0000 7fcf86251640 1 -- 172.21.15.28:0/4133267062 learned_addr learned my addr 172.21.15.28:0/4133267062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 -- 172.21.15.28:0/4133267062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 msgr2=0x55ca9c23e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c23e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 -- 172.21.15.28:0/4133267062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 msgr2=0x55ca9c23b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c23b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 -- 172.21.15.28:0/4133267062 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9c168150 con 0x55ca9c236430 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86a52640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 0x7fcf6c0a53e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcf78005fd0 tx=0x7fcf78009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf86251640 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c0114c0 tx=0x7fcf7c011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a66eb89a447ae26c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf74bfe640 1 -- 172.21.15.28:0/4133267062 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c004020 con 0x55ca9c236430 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf74bfe640 1 -- 172.21.15.28:0/4133267062 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf7c005500 con 0x55ca9c236430 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.064+0000 7fcf74bfe640 1 -- 172.21.15.28:0/4133267062 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c004cb0 con 0x55ca9c236430 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- 172.21.15.28:0/4133267062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 msgr2=0x55ca9c236850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c0114c0 tx=0x7fcf7c011980 comp rx=0 tx=0).stop 2024-03-22T02:24:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- 172.21.15.28:0/4133267062 shutdown_connections 2024-03-22T02:24:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c23e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c23b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/4133267062 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c236850 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- 172.21.15.28:0/4133267062 >> 172.21.15.28:0/4133267062 conn(0x55ca9c2319a0 msgr2=0x55ca9c231da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- 172.21.15.28:0/4133267062 shutdown_connections 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- 172.21.15.28:0/4133267062 wait complete. 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- start start 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 Processor -- start 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236430 0x55ca9c240880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236e20 0x55ca9c240dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c23ef40 con 0x55ca9c23bf90 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c157130 con 0x55ca9c236e20 2024-03-22T02:24:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.065+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ca9c23f0c0 con 0x55ca9c236430 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf8b17b740 10 client.0 _ll_register_callbacks cb 0x55ca9c156b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56348/0 (socket says 172.21.15.28:56348) 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 -- 172.21.15.28:0/1694299831 learned_addr learned my addr 172.21.15.28:0/1694299831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86251640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236430 0x55ca9c240880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf85a50640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236e20 0x55ca9c240dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 -- 172.21.15.28:0/1694299831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236430 msgr2=0x55ca9c240880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236430 0x55ca9c240880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 -- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236e20 msgr2=0x55ca9c240dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236e20 0x55ca9c240dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 -- 172.21.15.28:0/1694299831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca9bf797d0 con 0x55ca9c23bf90 2024-03-22T02:24:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf86a52640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcf78013790 tx=0x7fcf7801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78e78abf6f2b5c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78004b20 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcf7801c040 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf7c005500 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcf78031050 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf8b17b740 10 client.18618 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca9beaf3f0 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.066+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf7c005500 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcf78028070 con 0x55ca9c23bf90 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 10 client.18618 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 10 client.18618 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 20 client.18618 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c24c6a0 0x55ca9c24eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 4 client.18618 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca9c1a8e00 con 0x55ca9c24c6a0 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf86251640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c24c6a0 0x55ca9c24eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf86251640 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c24c6a0 0x55ca9c24eb00 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fdbe9799e0c0222 server_cookie=c92edb3621bb5e76 in_seq=0 out_seq=0 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.067+0000 7fcf69ffb640 10 client.18618 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf69ffb640 1 -- 172.21.15.28:0/1694299831 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ca9c1a8e00 con 0x55ca9c24c6a0 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf69ffb640 10 client.18618 handle_command_reply: tid=0 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 client.18618 shutdown 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 msgr2=0x55ca9c247e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fcf78013790 tx=0x7fcf7801d000 comp rx=0 tx=0).stop 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 shutdown_connections 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c24c6a0 0x55ca9c24eb00 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247e00 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf69ffb640 0 client.18618 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236e20 0x55ca9c240dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/1694299831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236430 0x55ca9c240880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 >> 172.21.15.28:0/1694299831 conn(0x55ca9c2319a0 msgr2=0x55ca9c230c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 shutdown_connections 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.069+0000 7fcf8b17b740 1 -- 172.21.15.28:0/1694299831 wait complete. 2024-03-22T02:24:14.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.070+0000 7fcf8b17b740 20 client.18618 trim_cache size 0 max 16384 2024-03-22T02:24:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 Processor -- start 2024-03-22T02:24:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 -- start start 2024-03-22T02:24:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c238240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236e20 0x55ca9c238780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801ad2a0 con 0x55ca9c23bf90 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801aab70 con 0x55ca9c236430 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801a89c0 con 0x55ca9c236e20 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf86a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf86a52640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56360/0 (socket says 172.21.15.28:56360) 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf86a52640 1 -- 172.21.15.28:0/2928449404 learned_addr learned my addr 172.21.15.28:0/2928449404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf85a50640 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236e20 0x55ca9c238780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.087+0000 7fcf86251640 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c238240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 -- 172.21.15.28:0/2928449404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236e20 msgr2=0x55ca9c238780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236e20 0x55ca9c238780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 -- 172.21.15.28:0/2928449404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 msgr2=0x55ca9c238240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c238240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 -- 172.21.15.28:0/2928449404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca9be6f0f0 con 0x55ca9c23bf90 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf86a52640 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009fb0 tx=0x7fcf7800fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e72dddf34bef8ff server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf6bfff640 1 -- 172.21.15.28:0/2928449404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7803a070 con 0x55ca9c23bf90 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf6bfff640 1 -- 172.21.15.28:0/2928449404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78021070 con 0x55ca9c23bf90 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.088+0000 7fcf6bfff640 1 -- 172.21.15.28:0/2928449404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78004c30 con 0x55ca9c23bf90 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- 172.21.15.28:0/2928449404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 msgr2=0x55ca9c247dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009fb0 tx=0x7fcf7800fca0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- 172.21.15.28:0/2928449404 shutdown_connections 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c23bf90 0x55ca9c247dc0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c236e20 0x55ca9c238780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/2928449404 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c238240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- 172.21.15.28:0/2928449404 >> 172.21.15.28:0/2928449404 conn(0x55ca9c2319a0 msgr2=0x55ca9c231d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- 172.21.15.28:0/2928449404 shutdown_connections 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- 172.21.15.28:0/2928449404 wait complete. 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- start start 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 Processor -- start 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c247580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c245c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801a9030 con 0x55ca9c236e20 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801aa4a0 con 0x55ca9c236430 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fcf801a9700 con 0x55ca9c23bf90 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf8b17b740 10 client.0 _ll_register_callbacks cb 0x7fcf78005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf86251640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c247580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf85a50640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56368/0 (socket says 172.21.15.28:56368) 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf85a50640 1 -- 172.21.15.28:0/230195631 learned_addr learned my addr 172.21.15.28:0/230195631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.089+0000 7fcf86a52640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c245c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 -- 172.21.15.28:0/230195631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 msgr2=0x55ca9c245c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c245c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 -- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 msgr2=0x55ca9c247580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c247580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 -- 172.21.15.28:0/230195631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca9c246240 con 0x55ca9c236e20 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf86251640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c247580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf85a50640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcf700b9af0 tx=0x7fcf700ee880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f081ad41e13e701 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf7004b040 con 0x55ca9c236e20 2024-03-22T02:24:14.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fcf700a10d0 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf78021070 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fcf700f1120 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf8b17b740 10 client.18624 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55ca9c2b33e0 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf78021070 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7fcf700a15a0 con 0x55ca9c236e20 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.090+0000 7fcf8b17b740 10 client.18624 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 10 client.18624 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 20 client.18624 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c2b3d90 0x55ca9c2b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 4 client.18624 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ca9c1a3150 con 0x55ca9c2b3d90 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf86251640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c2b3d90 0x55ca9c2b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf86251640 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c2b3d90 0x55ca9c2b61b0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19e5c1ab0e84d8a5 server_cookie=73344e169115d46b in_seq=0 out_seq=0 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf69ffb640 10 client.18624 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf69ffb640 1 -- 172.21.15.28:0/230195631 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55ca9c1a3150 con 0x55ca9c2b3d90 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf69ffb640 10 client.18624 handle_command_reply: tid=0 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 client.18624 shutdown 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 msgr2=0x55ca9c247ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fcf700b9af0 tx=0x7fcf700ee880 comp rx=0 tx=0).stop 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 shutdown_connections 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ca9c2b3d90 0x55ca9c2b61b0 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ca9c23bf90 0x55ca9c245c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca9c236e20 0x55ca9c247ac0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 --2- 172.21.15.28:0/230195631 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ca9c236430 0x55ca9c247580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf69ffb640 0 client.18624 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:14.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 >> 172.21.15.28:0/230195631 conn(0x55ca9c2319a0 msgr2=0x55ca9c240880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 shutdown_connections 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.091+0000 7fcf8b17b740 1 -- 172.21.15.28:0/230195631 wait complete. 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf8b17b740 20 client.18624 trim_cache size 0 max 16384 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "2fa38424-ccf8-4ee7-92cd-375c6bf42a6f": { 2024-03-22T02:24:14.100 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "2fa38424-ccf8-4ee7-92cd-375c6bf42a6f", 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:14.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 msgr2=0x7fcf6c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 0x7fcf6c0a53e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcf78005fd0 tx=0x7fcf78009e00 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 msgr2=0x7fcf80198410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcf70000c00 tx=0x7fcf7002c650 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 shutdown_connections 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fcf6c0a2f20 0x7fcf6c0a53e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcf80103800 0x7fcf8019f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf80102e40 0x7fcf80198410 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 --2- 172.21.15.28:0/3978042709 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fcf800ffe70 0x7fcf80197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 >> 172.21.15.28:0/3978042709 conn(0x7fcf800fb7a0 msgr2=0x7fcf80106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 shutdown_connections 2024-03-22T02:24:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:14.092+0000 7fcf884b5640 1 -- 172.21.15.28:0/3978042709 wait complete. 2024-03-22T02:24:14.103 INFO:tasks.cephfs.filesystem:scrub status for tag:2fa38424-ccf8-4ee7-92cd-375c6bf42a6f - {'path': '/', 'tag': '2fa38424-ccf8-4ee7-92cd-375c6bf42a6f', 'options': 'recursive,force'} 2024-03-22T02:24:14.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1069744 11.40 MB/sec execute 416 sec latency 18.859 ms 2024-03-22T02:24:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:14 smithi120 ceph-mon[36803]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:24:14.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:14 smithi083 ceph-mon[36340]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:24:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:14 smithi028 ceph-mon[31916]: pgmap v679: 129 pgs: 129 active+clean; 60 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:24:15.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1071808 11.40 MB/sec execute 417 sec latency 13.708 ms 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:15 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:15.700 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.701 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:15 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:15.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:24:15.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:24:15.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:24:15.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:15.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:15 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:24:16.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1073457 11.40 MB/sec execute 418 sec latency 44.618 ms 2024-03-22T02:24:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:16 smithi120 ceph-mon[36803]: pgmap v680: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 458 op/s 2024-03-22T02:24:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:16 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:16 smithi083 ceph-mon[36340]: pgmap v680: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 458 op/s 2024-03-22T02:24:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:16 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:16 smithi028 ceph-mon[31916]: pgmap v680: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.6 MiB/s wr, 458 op/s 2024-03-22T02:24:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:16 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:17.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1075237 11.39 MB/sec execute 419 sec latency 105.023 ms 2024-03-22T02:24:17.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:17 smithi120 ceph-mon[36803]: pgmap v681: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-03-22T02:24:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:17.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:17 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:17 smithi083 ceph-mon[36340]: pgmap v681: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-03-22T02:24:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:17 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:17 smithi028 ceph-mon[31916]: pgmap v681: 129 pgs: 129 active+clean; 52 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 345 op/s 2024-03-22T02:24:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:17 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:18.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1076921 11.39 MB/sec execute 420 sec latency 36.785 ms 2024-03-22T02:24:18.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:18.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:18.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:19.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:19.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:24:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:24:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:24:19.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1079316 11.39 MB/sec execute 421 sec latency 30.940 ms 2024-03-22T02:24:19.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:19 smithi120 ceph-mon[36803]: pgmap v682: 129 pgs: 129 active+clean; 52 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-22T02:24:20.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:19 smithi083 ceph-mon[36340]: pgmap v682: 129 pgs: 129 active+clean; 52 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-22T02:24:20.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:19 smithi028 ceph-mon[31916]: pgmap v682: 129 pgs: 129 active+clean; 52 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 380 op/s 2024-03-22T02:24:20.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1080805 11.38 MB/sec execute 422 sec latency 31.935 ms 2024-03-22T02:24:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:20 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:24:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:20 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:24:21.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:20 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:24:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:20 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:24:21.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:20 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:24:21.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:20 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:24:21.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1083414 11.39 MB/sec execute 423 sec latency 21.047 ms 2024-03-22T02:24:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:21 smithi120 ceph-mon[36803]: pgmap v683: 129 pgs: 129 active+clean; 54 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-03-22T02:24:22.014 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:21 smithi083 ceph-mon[36340]: pgmap v683: 129 pgs: 129 active+clean; 54 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-03-22T02:24:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:21 smithi028 ceph-mon[31916]: pgmap v683: 129 pgs: 129 active+clean; 54 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-03-22T02:24:22.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1084575 11.37 MB/sec execute 424 sec latency 50.422 ms 2024-03-22T02:24:23.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1087051 11.38 MB/sec execute 425 sec latency 26.484 ms 2024-03-22T02:24:24.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:23 smithi028 ceph-mon[31916]: pgmap v684: 129 pgs: 129 active+clean; 55 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:24:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:23 smithi120 ceph-mon[36803]: pgmap v684: 129 pgs: 129 active+clean; 55 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:24:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:23 smithi083 ceph-mon[36340]: pgmap v684: 129 pgs: 129 active+clean; 55 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:24:24.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1088610 11.37 MB/sec execute 426 sec latency 68.043 ms 2024-03-22T02:24:25.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1090934 11.38 MB/sec execute 427 sec latency 11.705 ms 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 -- 172.21.15.28:0/3549819464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc104f60 msgr2=0x7f16cc107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 --2- 172.21.15.28:0/3549819464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc104f60 0x7f16cc107350 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009900 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 -- 172.21.15.28:0/3549819464 shutdown_connections 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 --2- 172.21.15.28:0/3549819464 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc107890 0x7f16cc109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 --2- 172.21.15.28:0/3549819464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc104f60 0x7f16cc107350 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 --2- 172.21.15.28:0/3549819464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 -- 172.21.15.28:0/3549819464 >> 172.21.15.28:0/3549819464 conn(0x7f16cc0fc020 msgr2=0x7f16cc0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 -- 172.21.15.28:0/3549819464 shutdown_connections 2024-03-22T02:24:25.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.848+0000 7f16d38af640 1 -- 172.21.15.28:0/3549819464 wait complete. 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 Processor -- start 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 -- start start 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 0x7f16cc198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d840 con 0x7f16cc107890 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d6c0 con 0x7f16cc104f60 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d38af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10d9c0 con 0x7f16cc102630 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d1e4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d164b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 0x7f16cc198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d1e4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38288/0 (socket says 172.21.15.28:38288) 2024-03-22T02:24:25.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d164b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:50246/0 (socket says 172.21.15.28:50246) 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d1e4c640 1 -- 172.21.15.28:0/3766730221 learned_addr learned my addr 172.21.15.28:0/3766730221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 -- 172.21.15.28:0/3766730221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 msgr2=0x7f16cc197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 -- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 msgr2=0x7f16cc19f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d0e4a640 1 -- 172.21.15.28:0/3766730221 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc19fa70 con 0x7f16cc104f60 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.849+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 0x7f16cc198390 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f16bc02c170 tx=0x7f16bc02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16c27fc640 1 -- 172.21.15.28:0/3766730221 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc041070 con 0x7f16cc104f60 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16c27fc640 1 -- 172.21.15.28:0/3766730221 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16bc004450 con 0x7f16cc104f60 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16cc19fd00 con 0x7f16cc104f60 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16c27fc640 1 -- 172.21.15.28:0/3766730221 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc003e50 con 0x7f16cc104f60 2024-03-22T02:24:25.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.850+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a01c0 con 0x7f16cc104f60 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.852+0000 7f16c27fc640 1 -- 172.21.15.28:0/3766730221 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f16bc044070 con 0x7f16cc104f60 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 Processor -- start 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 -- start start 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16c27fc640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 0x7f16b80a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 0x563074946de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563074947450 0x5630749498e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5630748a2040 con 0x563074947450 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56307484a9e0 con 0x5630749418f0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d6573740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56307489dff0 con 0x5630749422e0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16c27fc640 1 -- 172.21.15.28:0/3766730221 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f16bc0f12e0 con 0x7f16cc104f60 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 0x7f16b80a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d0e4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 0x563074946de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:60230/0 (socket says 172.21.15.28:60230) 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 -- 172.21.15.28:0/488523832 learned_addr learned my addr 172.21.15.28:0/488523832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d0e4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 0x563074946de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:50254/0 (socket says 172.21.15.28:50254) 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563074947450 0x5630749498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 -- 172.21.15.28:0/488523832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 msgr2=0x563074946de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 0x563074946de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 -- 172.21.15.28:0/488523832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563074947450 msgr2=0x5630749498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563074947450 0x5630749498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.853+0000 7f16d164b640 1 -- 172.21.15.28:0/488523832 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630745b6210 con 0x5630749418f0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.854+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 0x7f16b80a5620 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f16c40099a0 tx=0x7f16c400e700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.854+0000 7f16d164b640 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009d20 tx=0x7f16c8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7230a698e752309a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.854+0000 7f16b77fe640 1 -- 172.21.15.28:0/488523832 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c801e070 con 0x5630749418f0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.854+0000 7f16b77fe640 1 -- 172.21.15.28:0/488523832 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c8019b10 con 0x5630749418f0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.854+0000 7f16b77fe640 1 -- 172.21.15.28:0/488523832 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c800ea50 con 0x5630749418f0 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- 172.21.15.28:0/488523832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 msgr2=0x563074941d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009d20 tx=0x7f16c8011810 comp rx=0 tx=0).stop 2024-03-22T02:24:25.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- 172.21.15.28:0/488523832 shutdown_connections 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563074947450 0x5630749498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5630749422e0 0x563074946de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- 172.21.15.28:0/488523832 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x563074941d10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- 172.21.15.28:0/488523832 >> 172.21.15.28:0/488523832 conn(0x563074938a50 msgr2=0x563074938e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- 172.21.15.28:0/488523832 shutdown_connections 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- 172.21.15.28:0/488523832 wait complete. 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 -- start start 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 Processor -- start 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.855+0000 7f16d6573740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x56307494be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x563074952e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56307494a400 con 0x5630749418f0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56307485ddb0 con 0x5630749422e0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56307494a580 con 0x563074947450 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 10 client.0 _ll_register_callbacks cb 0x5630745b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38312/0 (socket says 172.21.15.28:38312) 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 -- 172.21.15.28:0/3413943960 learned_addr learned my addr 172.21.15.28:0/3413943960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x56307494be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x563074952e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 -- 172.21.15.28:0/3413943960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 msgr2=0x563074952e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x563074952e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 -- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 msgr2=0x56307494be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x56307494be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 -- 172.21.15.28:0/3413943960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563074682620 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d164b640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009cf0 tx=0x7f16c8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e688565be79dc933 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c801e050 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f16c8019430 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c8019b10 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16c8032050 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 10 client.18636 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.856+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5630749561a0 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c8019b10 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f16c8003680 con 0x5630749418f0 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 10 client.18636 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 10 client.18636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:25.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 20 client.18636 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563074957610 0x563074959a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 4 client.18636 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630748af580 con 0x563074957610 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563074957610 0x563074959a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563074957610 0x563074959a00 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20033d45694aa914 server_cookie=f88d105808279286 in_seq=0 out_seq=0 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.857+0000 7f16b4ff9640 10 client.18636 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.858+0000 7f16b4ff9640 1 -- 172.21.15.28:0/3413943960 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630748af580 con 0x563074957610 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.858+0000 7f16b4ff9640 10 client.18636 handle_command_reply: tid=0 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.858+0000 7f16d6573740 1 client.18636 shutdown 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.858+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 msgr2=0x56307494b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009cf0 tx=0x7f16c8004270 comp rx=0 tx=0).stop 2024-03-22T02:24:25.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 shutdown_connections 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563074957610 0x563074959a00 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x563074952e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x56307494be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16b4ff9640 0 client.18636 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 --2- 172.21.15.28:0/3413943960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307494b920 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 >> 172.21.15.28:0/3413943960 conn(0x563074938a50 msgr2=0x563074937c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 shutdown_connections 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 1 -- 172.21.15.28:0/3413943960 wait complete. 2024-03-22T02:24:25.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.859+0000 7f16d6573740 20 client.18636 trim_cache size 0 max 16384 2024-03-22T02:24:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 Processor -- start 2024-03-22T02:24:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 -- start start 2024-03-22T02:24:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x563074952df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x56307494abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1aafa0 con 0x5630749418f0 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a8870 con 0x5630749422e0 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d6573740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a66c0 con 0x563074947450 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38324/0 (socket says 172.21.15.28:38324) 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 -- 172.21.15.28:0/1684322455 learned_addr learned my addr 172.21.15.28:0/1684322455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d1e4c640 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x56307494abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x563074952df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 -- 172.21.15.28:0/1684322455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 msgr2=0x56307494abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x56307494abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 -- 172.21.15.28:0/1684322455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 msgr2=0x563074952df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x563074952df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 -- 172.21.15.28:0/1684322455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563074941cd0 con 0x5630749418f0 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.874+0000 7f16d164b640 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003aa0 tx=0x7f16c80195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d5ae77720f01e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16b6ffd640 1 -- 172.21.15.28:0/1684322455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8015070 con 0x5630749418f0 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16b6ffd640 1 -- 172.21.15.28:0/1684322455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c801f040 con 0x5630749418f0 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16b6ffd640 1 -- 172.21.15.28:0/1684322455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c801e070 con 0x5630749418f0 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 -- 172.21.15.28:0/1684322455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 msgr2=0x56307493c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f16c8003aa0 tx=0x7f16c80195c0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 -- 172.21.15.28:0/1684322455 shutdown_connections 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x56307494abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749422e0 0x563074952df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 --2- 172.21.15.28:0/1684322455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749418f0 0x56307493c780 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 -- 172.21.15.28:0/1684322455 >> 172.21.15.28:0/1684322455 conn(0x563074938a50 msgr2=0x5630749435e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 -- 172.21.15.28:0/1684322455 shutdown_connections 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.875+0000 7f16d6573740 1 -- 172.21.15.28:0/1684322455 wait complete. 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 -- start start 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 Processor -- start 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x5630749bb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a6d30 con 0x5630749422e0 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a81a0 con 0x5630749418f0 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a7400 con 0x563074947450 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d6573740 10 client.0 _ll_register_callbacks cb 0x7f16c801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:25.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d0e4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d164b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d0e4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38338/0 (socket says 172.21.15.28:38338) 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d164b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:60242/0 (socket says 172.21.15.28:60242) 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d0e4a640 1 -- 172.21.15.28:0/51085183 learned_addr learned my addr 172.21.15.28:0/51085183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.876+0000 7f16d1e4c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x5630749bb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 -- 172.21.15.28:0/51085183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 msgr2=0x5630749bb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x5630749bb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 -- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 msgr2=0x56307494fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 -- 172.21.15.28:0/51085183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630749bbef0 con 0x5630749422e0 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d164b640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:25.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d0e4a640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee970 tx=0x7f16bc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6498d5ba498da532 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc04b040 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f16bc0eea80 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c801f040 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f16bc0bd650 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 10 client.18642 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5630749bd270 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c801f040 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f16bc0b9cc0 con 0x5630749422e0 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 10 client.18642 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 10 client.18642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 20 client.18642 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:25.884 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5630749bdbd0 0x5630749bfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 4 client.18642 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:24:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.877+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5630748a98d0 con 0x5630749bdbd0 2024-03-22T02:24:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.878+0000 7f16d164b640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5630749bdbd0 0x5630749bfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.878+0000 7f16d164b640 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5630749bdbd0 0x5630749bfff0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=317b95a60b4dea0a server_cookie=7536a6d5d92d7c56 in_seq=0 out_seq=0 2024-03-22T02:24:25.885 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:25.878+0000 7f16b4ff9640 10 client.18642 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:26.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:25 smithi028 ceph-mon[31916]: pgmap v685: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 414 op/s 2024-03-22T02:24:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16b4ff9640 1 -- 172.21.15.28:0/51085183 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5630748a98d0 con 0x5630749bdbd0 2024-03-22T02:24:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16b4ff9640 10 client.18642 handle_command_reply: tid=0 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 client.18642 shutdown 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 msgr2=0x563074950420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0ee970 tx=0x7f16bc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 shutdown_connections 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5630749bdbd0 0x5630749bfff0 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563074947450 0x5630749bb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16b4ff9640 0 client.18642 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630749422e0 0x563074950420 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 --2- 172.21.15.28:0/51085183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5630749418f0 0x56307494fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 >> 172.21.15.28:0/51085183 conn(0x563074938a50 msgr2=0x563074942b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 shutdown_connections 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.158+0000 7f16d6573740 1 -- 172.21.15.28:0/51085183 wait complete. 2024-03-22T02:24:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d6573740 20 client.18642 trim_cache size 0 max 16384 2024-03-22T02:24:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:24:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:24:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:24:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 msgr2=0x7f16b80a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 0x7f16b80a5620 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f16c40099a0 tx=0x7f16c400e700 comp rx=0 tx=0).stop 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 msgr2=0x7f16cc198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 0x7f16cc198390 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f16bc02c170 tx=0x7f16bc02c650 comp rx=0 tx=0).stop 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 shutdown_connections 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b80a3160 0x7f16b80a5620 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc107890 0x7f16cc19f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc104f60 0x7f16cc198390 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 --2- 172.21.15.28:0/3766730221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc102630 0x7f16cc197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 >> 172.21.15.28:0/3766730221 conn(0x7f16cc0fc020 msgr2=0x7f16cc102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 shutdown_connections 2024-03-22T02:24:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:26.159+0000 7f16d38af640 1 -- 172.21.15.28:0/3766730221 wait complete. 2024-03-22T02:24:26.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:25 smithi120 ceph-mon[36803]: pgmap v685: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 414 op/s 2024-03-22T02:24:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:25 smithi083 ceph-mon[36340]: pgmap v685: 129 pgs: 129 active+clean; 59 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 414 op/s 2024-03-22T02:24:26.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1092440 11.37 MB/sec execute 428 sec latency 178.620 ms 2024-03-22T02:24:27.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1094486 11.37 MB/sec execute 429 sec latency 51.860 ms 2024-03-22T02:24:28.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:27 smithi028 ceph-mon[31916]: pgmap v686: 129 pgs: 129 active+clean; 59 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 263 op/s 2024-03-22T02:24:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:27 smithi120 ceph-mon[36803]: pgmap v686: 129 pgs: 129 active+clean; 59 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 263 op/s 2024-03-22T02:24:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:27 smithi083 ceph-mon[36340]: pgmap v686: 129 pgs: 129 active+clean; 59 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 263 op/s 2024-03-22T02:24:28.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1096489 11.36 MB/sec execute 430 sec latency 32.278 ms 2024-03-22T02:24:29.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1098381 11.36 MB/sec execute 431 sec latency 31.162 ms 2024-03-22T02:24:30.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:29 smithi028 ceph-mon[31916]: pgmap v687: 129 pgs: 129 active+clean; 55 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:24:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:29 smithi120 ceph-mon[36803]: pgmap v687: 129 pgs: 129 active+clean; 55 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:24:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:29 smithi083 ceph-mon[36340]: pgmap v687: 129 pgs: 129 active+clean; 55 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.9 MiB/s wr, 329 op/s 2024-03-22T02:24:30.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1100988 11.37 MB/sec execute 432 sec latency 22.234 ms 2024-03-22T02:24:31.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1102569 11.37 MB/sec execute 433 sec latency 55.335 ms 2024-03-22T02:24:32.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:31 smithi028 ceph-mon[31916]: pgmap v688: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:31 smithi120 ceph-mon[36803]: pgmap v688: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:31 smithi083 ceph-mon[36340]: pgmap v688: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.9 MiB/s wr, 366 op/s 2024-03-22T02:24:32.432 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1104853 11.37 MB/sec execute 434 sec latency 52.459 ms 2024-03-22T02:24:33.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1106200 11.36 MB/sec execute 435 sec latency 31.359 ms 2024-03-22T02:24:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:33 smithi028 ceph-mon[31916]: pgmap v689: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 330 op/s 2024-03-22T02:24:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:33 smithi120 ceph-mon[36803]: pgmap v689: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 330 op/s 2024-03-22T02:24:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:33 smithi083 ceph-mon[36340]: pgmap v689: 129 pgs: 129 active+clean; 54 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.4 MiB/s wr, 330 op/s 2024-03-22T02:24:34.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1108599 11.36 MB/sec execute 436 sec latency 15.745 ms 2024-03-22T02:24:35.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1110660 11.35 MB/sec execute 437 sec latency 39.874 ms 2024-03-22T02:24:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:35 smithi120 ceph-mon[36803]: pgmap v690: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2024-03-22T02:24:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:35 smithi083 ceph-mon[36340]: pgmap v690: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2024-03-22T02:24:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:35 smithi028 ceph-mon[31916]: pgmap v690: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.2 MiB/s wr, 401 op/s 2024-03-22T02:24:36.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1112470 11.36 MB/sec execute 438 sec latency 31.289 ms 2024-03-22T02:24:37.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1114782 11.36 MB/sec execute 439 sec latency 62.321 ms 2024-03-22T02:24:38.078 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:37 smithi083 ceph-mon[36340]: pgmap v691: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-03-22T02:24:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:37 smithi120 ceph-mon[36803]: pgmap v691: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-03-22T02:24:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:37 smithi028 ceph-mon[31916]: pgmap v691: 129 pgs: 129 active+clean; 57 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-03-22T02:24:38.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1116010 11.35 MB/sec execute 440 sec latency 46.891 ms 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 -- 172.21.15.28:0/2019142718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 msgr2=0x7f82a8101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/2019142718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a8101bb0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009ca0 tx=0x7f82a002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 -- 172.21.15.28:0/2019142718 shutdown_connections 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/2019142718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a8101bb0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/2019142718 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a81011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/2019142718 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 -- 172.21.15.28:0/2019142718 >> 172.21.15.28:0/2019142718 conn(0x7f82a80fc030 msgr2=0x7f82a80fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 -- 172.21.15.28:0/2019142718 shutdown_connections 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.255+0000 7f82ae9bf640 1 -- 172.21.15.28:0/2019142718 wait complete. 2024-03-22T02:24:39.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 Processor -- start 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 -- start start 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a819a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82a810fa30 con 0x7f82a8101730 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82a810f8b0 con 0x7f82a8100d90 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82ae9bf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82a810fbb0 con 0x7f82a8100430 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82acf5c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a819a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:47166/0 (socket says 172.21.15.28:47166) 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82a7fff640 1 -- 172.21.15.28:0/3705116150 learned_addr learned my addr 172.21.15.28:0/3705116150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.256+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46880/0 (socket says 172.21.15.28:46880) 2024-03-22T02:24:39.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 -- 172.21.15.28:0/3705116150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 msgr2=0x7f82a8199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 -- 172.21.15.28:0/3705116150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 msgr2=0x7f82a819a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a819a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 -- 172.21.15.28:0/3705116150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a81a1b70 con 0x7f82a8101730 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f82a002c630 tx=0x7f82a0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82a57fa640 1 -- 172.21.15.28:0/3705116150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0041020 con 0x7f82a8101730 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a81a1e00 con 0x7f82a8101730 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82a57fa640 1 -- 172.21.15.28:0/3705116150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82a0034cd0 con 0x7f82a8101730 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82a57fa640 1 -- 172.21.15.28:0/3705116150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0039040 con 0x7f82a8101730 2024-03-22T02:24:39.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.257+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a81a22c0 con 0x7f82a8101730 2024-03-22T02:24:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.259+0000 7f82a57fa640 1 -- 172.21.15.28:0/3705116150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f82a0044020 con 0x7f82a8101730 2024-03-22T02:24:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.259+0000 7f82b159c740 1 Processor -- start 2024-03-22T02:24:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.259+0000 7f82b159c740 1 -- start start 2024-03-22T02:24:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216cdd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559216c3d040 con 0x559216cde040 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559216be59e0 con 0x559216cd84e0 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82b159c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559216c38ff0 con 0x559216cd8ed0 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a57fa640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 0x7f82900a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216cdd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46894/0 (socket says 172.21.15.28:46894) 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 -- 172.21.15.28:0/865643188 learned_addr learned my addr 172.21.15.28:0/865643188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 0x7f82900a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a57fa640 1 -- 172.21.15.28:0/3705116150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f82a00f15e0 con 0x7f82a8101730 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 -- 172.21.15.28:0/865643188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 msgr2=0x559216cdd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216cdd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 -- 172.21.15.28:0/865643188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 msgr2=0x559216cd8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 -- 172.21.15.28:0/865643188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559216951210 con 0x559216cde040 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82a77fe640 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f829c009fd0 tx=0x7f829c009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50839d9c9fb3fdec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.260+0000 7f82acf5c640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 0x7f82900a5480 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f82a00bd8c0 tx=0x7f82a0003ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f828e7fc640 1 -- 172.21.15.28:0/865643188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c00fc50 con 0x559216cde040 2024-03-22T02:24:39.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f828e7fc640 1 -- 172.21.15.28:0/865643188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829c00fdf0 con 0x559216cde040 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f828e7fc640 1 -- 172.21.15.28:0/865643188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c01d5a0 con 0x559216cde040 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 -- 172.21.15.28:0/865643188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 msgr2=0x559216ce04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f829c009fd0 tx=0x7f829c009ba0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 -- 172.21.15.28:0/865643188 shutdown_connections 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce04d0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216cdd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 --2- 172.21.15.28:0/865643188 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 -- 172.21.15.28:0/865643188 >> 172.21.15.28:0/865643188 conn(0x559216cd3a50 msgr2=0x559216cd3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 -- 172.21.15.28:0/865643188 shutdown_connections 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.261+0000 7f82b159c740 1 -- 172.21.15.28:0/865643188 wait complete. 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 -- start start 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 Processor -- start 2024-03-22T02:24:39.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cde040 0x559216ce9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559216ce0ff0 con 0x559216cd8ed0 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559216bf8db0 con 0x559216cde040 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559216ce1170 con 0x559216cd84e0 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82b159c740 10 client.0 _ll_register_callbacks cb 0x559216950a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46910/0 (socket says 172.21.15.28:46910) 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82a7fff640 1 -- 172.21.15.28:0/3046468624 learned_addr learned my addr 172.21.15.28:0/3046468624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.262+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cde040 0x559216ce9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 -- 172.21.15.28:0/3046468624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 msgr2=0x559216ce2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 -- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cde040 msgr2=0x559216ce9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cde040 0x559216ce9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 -- 172.21.15.28:0/3046468624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559216a1d620 con 0x559216cd8ed0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8298012d20 tx=0x7f829800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464110a99a56dc28 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82877fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8298005020 con 0x559216cd8ed0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f82980051c0 con 0x559216cd8ed0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f829c00fdf0 con 0x559216cd8ed0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f82980053c0 con 0x559216cd8ed0 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82b159c740 10 client.18654 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559216ceceb0 con 0x559216cd8ed0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.263+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829c00fdf0 con 0x559216cd8ed0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f829801c470 con 0x559216cd8ed0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 10 client.18654 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 10 client.18654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 20 client.18654 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216cee5f0 0x559216cf0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 4 client.18654 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559216c4a580 con 0x559216cee5f0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216cee5f0 0x559216cf0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82877fe640 10 client.18654 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.264+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216cee5f0 0x559216cf0a50 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f9e60d5af8e4ca server_cookie=20c193110636dad0 in_seq=0 out_seq=0 2024-03-22T02:24:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82877fe640 1 -- 172.21.15.28:0/3046468624 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559216c4a580 con 0x559216cee5f0 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82877fe640 10 client.18654 handle_command_reply: tid=0 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 client.18654 shutdown 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 msgr2=0x559216ce2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f8298012d20 tx=0x7f829800f790 comp rx=0 tx=0).stop 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 shutdown_connections 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216cee5f0 0x559216cf0a50 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cde040 0x559216ce9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cd8ed0 0x559216ce2a50 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82877fe640 0 client.18654 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 --2- 172.21.15.28:0/3046468624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 >> 172.21.15.28:0/3046468624 conn(0x559216cd3a50 msgr2=0x559216cd2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.265+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 shutdown_connections 2024-03-22T02:24:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.266+0000 7f82b159c740 1 -- 172.21.15.28:0/3046468624 wait complete. 2024-03-22T02:24:39.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.266+0000 7f82b159c740 20 client.18654 trim_cache size 0 max 16384 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.281+0000 7f82b159c740 1 Processor -- start 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.281+0000 7f82b159c740 1 -- start start 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.281+0000 7f82b159c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.281+0000 7f82b159c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216ce9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82b159c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82b159c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81af350 con 0x559216cde040 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82b159c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81acc20 con 0x559216cd84e0 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82b159c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81aaa70 con 0x559216cd8ed0 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56528/0 (socket says 172.21.15.28:56528) 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 -- 172.21.15.28:0/4091691392 learned_addr learned my addr 172.21.15.28:0/4091691392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a7fff640 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216ce9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 -- 172.21.15.28:0/4091691392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 msgr2=0x559216ce9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216ce9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 -- 172.21.15.28:0/4091691392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 msgr2=0x559216cd73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 -- 172.21.15.28:0/4091691392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559216cd47c0 con 0x559216cde040 2024-03-22T02:24:39.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f82a77fe640 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce1b30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f829c011bb0 tx=0x7f829c00ffa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a9ca8611b97fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f828dffb640 1 -- 172.21.15.28:0/4091691392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c035070 con 0x559216cde040 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f828dffb640 1 -- 172.21.15.28:0/4091691392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829c00fdf0 con 0x559216cde040 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.282+0000 7f828dffb640 1 -- 172.21.15.28:0/4091691392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c004640 con 0x559216cde040 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- 172.21.15.28:0/4091691392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 msgr2=0x559216ce1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce1b30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f829c011bb0 tx=0x7f829c00ffa0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- 172.21.15.28:0/4091691392 shutdown_connections 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce1b30 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd8ed0 0x559216ce9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 --2- 172.21.15.28:0/4091691392 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd84e0 0x559216cd73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- 172.21.15.28:0/4091691392 >> 172.21.15.28:0/4091691392 conn(0x559216cd3a50 msgr2=0x559216cd6cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- 172.21.15.28:0/4091691392 shutdown_connections 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- 172.21.15.28:0/4091691392 wait complete. 2024-03-22T02:24:39.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.283+0000 7f82b159c740 1 -- start start 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 Processor -- start 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81ab0e0 con 0x559216cde040 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81ac550 con 0x559216cd8ed0 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f82a81ab7b0 con 0x559216cd84e0 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82b159c740 10 client.0 _ll_register_callbacks cb 0x7f8298012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56540/0 (socket says 172.21.15.28:56540) 2024-03-22T02:24:39.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 -- 172.21.15.28:0/3333716498 learned_addr learned my addr 172.21.15.28:0/3333716498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 -- 172.21.15.28:0/3333716498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 msgr2=0x559216ce7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 -- 172.21.15.28:0/3333716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 msgr2=0x559216ce86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a7fff640 1 -- 172.21.15.28:0/3333716498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559216ce8cb0 con 0x559216cd8ed0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.284+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82a7fff640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f829800a010 tx=0x7f82980128a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75b60921ad4ccad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829802d070 con 0x559216cd8ed0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8298009cf0 con 0x559216cd8ed0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f829c00fdf0 con 0x559216cd8ed0 2024-03-22T02:24:39.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f829800fe80 con 0x559216cd8ed0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 10 client.28478 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559216d54a60 con 0x559216cd8ed0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f829c00fdf0 con 0x559216cd8ed0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f82980295c0 con 0x559216cd8ed0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 10 client.28478 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 10 client.28478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 20 client.28478 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216d553e0 0x559216d57800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 4 client.28478 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.285+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559216ce2300 con 0x559216d553e0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.286+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216d553e0 0x559216d57800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:39.292 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.286+0000 7f82a77fe640 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216d553e0 0x559216d57800 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eebfe8a8378aff98 server_cookie=d83f18aecbe42b2 in_seq=0 out_seq=0 2024-03-22T02:24:39.293 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.286+0000 7f82877fe640 10 client.28478 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:39.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1118339 11.35 MB/sec execute 441 sec latency 96.988 ms 2024-03-22T02:24:39.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82877fe640 1 -- 172.21.15.28:0/3333716498 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559216ce2300 con 0x559216d553e0 2024-03-22T02:24:39.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82877fe640 10 client.28478 handle_command_reply: tid=0 2024-03-22T02:24:39.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 client.28478 shutdown 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 msgr2=0x559216ce77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f829800a010 tx=0x7f82980128a0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 shutdown_connections 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559216d553e0 0x559216d57800 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559216cde040 0x559216ce86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559216cd8ed0 0x559216ce77c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 --2- 172.21.15.28:0/3333716498 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559216cd84e0 0x559216ce7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82877fe640 0 client.28478 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 >> 172.21.15.28:0/3333716498 conn(0x559216cd3a50 msgr2=0x559216cd23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 shutdown_connections 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 1 -- 172.21.15.28:0/3333716498 wait complete. 2024-03-22T02:24:39.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.674+0000 7f82b159c740 20 client.28478 trim_cache size 0 max 16384 2024-03-22T02:24:39.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:24:39.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:24:39.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:24:39.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 msgr2=0x7f82900a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 0x7f82900a5480 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f82a00bd8c0 tx=0x7f82a0003ca0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 msgr2=0x7f82a81a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f82a002c630 tx=0x7f82a0032f20 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 shutdown_connections 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f82900a2fc0 0x7f82900a5480 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82a8101730 0x7f82a81a1500 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f82a8100d90 0x7f82a819a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 --2- 172.21.15.28:0/3705116150 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82a8100430 0x7f82a8199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:39.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 >> 172.21.15.28:0/3705116150 conn(0x7f82a80fc030 msgr2=0x7f82a80fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 shutdown_connections 2024-03-22T02:24:39.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:39.675+0000 7f82ae9bf640 1 -- 172.21.15.28:0/3705116150 wait complete. 2024-03-22T02:24:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:39 smithi120 ceph-mon[36803]: pgmap v692: 129 pgs: 129 active+clean; 56 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-22T02:24:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:39 smithi028 ceph-mon[31916]: pgmap v692: 129 pgs: 129 active+clean; 56 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-22T02:24:40.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1119476 11.34 MB/sec execute 442 sec latency 195.942 ms 2024-03-22T02:24:40.473 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:39 smithi083 ceph-mon[36340]: pgmap v692: 129 pgs: 129 active+clean; 56 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-03-22T02:24:41.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1122215 11.35 MB/sec execute 443 sec latency 23.175 ms 2024-03-22T02:24:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:41 smithi120 ceph-mon[36803]: pgmap v693: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-03-22T02:24:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:41 smithi083 ceph-mon[36340]: pgmap v693: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-03-22T02:24:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:41 smithi028 ceph-mon[31916]: pgmap v693: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 311 op/s 2024-03-22T02:24:42.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1123595 11.34 MB/sec execute 444 sec latency 37.216 ms 2024-03-22T02:24:43.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1125785 11.34 MB/sec execute 445 sec latency 91.977 ms 2024-03-22T02:24:44.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:24:44.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:24:44.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:43 smithi028 ceph-mon[31916]: pgmap v694: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-22T02:24:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/250084987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d881078d0 msgr2=0x7f8d88109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/250084987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d881078d0 0x7f8d88109cc0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f8d80009ca0 tx=0x7f8d8002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/250084987 shutdown_connections 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/250084987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d881078d0 0x7f8d88109cc0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/250084987 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d88104fa0 0x7f8d88107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/250084987 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 0x7f8d88104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/250084987 >> 172.21.15.28:0/250084987 conn(0x7f8d880fc020 msgr2=0x7f8d880fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/250084987 shutdown_connections 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/250084987 wait complete. 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.216+0000 7f8d8fd3e640 1 Processor -- start 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 -- start start 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 0x7f8d88197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d881078d0 0x7f8d8819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8810d830 con 0x7f8d88104fa0 2024-03-22T02:24:44.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8810d6b0 con 0x7f8d881078d0 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8810d9b0 con 0x7f8d88102670 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 0x7f8d88197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45372/0 (socket says 172.21.15.28:45372) 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1336212598 learned_addr learned my addr 172.21.15.28:0/1336212598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d881078d0 0x7f8d8819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1336212598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 msgr2=0x7f8d88197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 0x7f8d88197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1336212598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d881078d0 msgr2=0x7f8d8819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d881078d0 0x7f8d8819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1336212598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d8819fad0 con 0x7f8d88104fa0 2024-03-22T02:24:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e9c0 tx=0x7f8d7800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d7effd640 1 -- 172.21.15.28:0/1336212598 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800cd80 con 0x7f8d88104fa0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d7effd640 1 -- 172.21.15.28:0/1336212598 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d78005500 con 0x7f8d88104fa0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d881004c0 con 0x7f8d88104fa0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.217+0000 7f8d7effd640 1 -- 172.21.15.28:0/1336212598 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d78010690 con 0x7f8d88104fa0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.218+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d88100a10 con 0x7f8d88104fa0 2024-03-22T02:24:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.220+0000 7f8d7effd640 1 -- 172.21.15.28:0/1336212598 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8d780046c0 con 0x7f8d88104fa0 2024-03-22T02:24:44.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.220+0000 7f8d92a02740 1 Processor -- start 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 -- start start 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d7effd640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 0x7f8d740a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670a1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670a6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670a6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 0x7f8d740a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670a9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670a6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45388/0 (socket says 172.21.15.28:45388) 2024-03-22T02:24:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/2555190538 learned_addr learned my addr 172.21.15.28:0/2555190538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617670068c0 con 0x5617670a1e20 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561766faec20 con 0x5617670a1430 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561767002870 con 0x5617670a6f90 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d7effd640 1 -- 172.21.15.28:0/1336212598 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8d780cfb40 con 0x7f8d88104fa0 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670a9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 -- 172.21.15.28:0/2555190538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 msgr2=0x5617670a9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670a9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 -- 172.21.15.28:0/2555190538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 msgr2=0x5617670a6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670a6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.221+0000 7f8d8dada640 1 -- 172.21.15.28:0/2555190538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561766fd3150 con 0x5617670a1430 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.222+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 0x7f8d740a53e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8d80005e30 tx=0x7f8d8002ce40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.222+0000 7f8d8dada640 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670a1850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010d30 tx=0x7f8d84005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83806049b463c42d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.222+0000 7f8d73fff640 1 -- 172.21.15.28:0/2555190538 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84004250 con 0x5617670a1430 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.222+0000 7f8d73fff640 1 -- 172.21.15.28:0/2555190538 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d840043f0 con 0x5617670a1430 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.222+0000 7f8d73fff640 1 -- 172.21.15.28:0/2555190538 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8401a900 con 0x5617670a1430 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 msgr2=0x5617670a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670a1850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010d30 tx=0x7f8d84005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 shutdown_connections 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670a9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670a6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 --2- 172.21.15.28:0/2555190538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670a1850 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 >> 172.21.15.28:0/2555190538 conn(0x56176709c9a0 msgr2=0x56176709cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 shutdown_connections 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- 172.21.15.28:0/2555190538 wait complete. 2024-03-22T02:24:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 -- start start 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.223+0000 7f8d92a02740 1 Processor -- start 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617670a9f40 con 0x5617670a1e20 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561766fc2130 con 0x5617670a1430 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617670aa0c0 con 0x5617670a6f90 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 10 client.0 _ll_register_callbacks cb 0x561766fc1b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45394/0 (socket says 172.21.15.28:45394) 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/861378158 learned_addr learned my addr 172.21.15.28:0/861378158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/861378158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 msgr2=0x5617670b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 msgr2=0x5617670ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/861378158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561766de47d0 con 0x5617670a1e20 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8dada640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f8d7809bdf0 tx=0x7f8d780097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966aa9e38fc6ec18 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7809be60 con 0x5617670a1e20 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8d780bd3c0 con 0x5617670a1e20 2024-03-22T02:24:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.224+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d840043f0 con 0x5617670a1e20 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d7809ffb0 con 0x5617670a1e20 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 10 client.18672 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x561766d1a3f0 con 0x5617670a1e20 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d840043f0 con 0x5617670a1e20 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f8d780a43e0 con 0x5617670a1e20 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 10 client.18672 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 10 client.18672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 20 client.18672 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617670b76d0 0x5617670b9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 4 client.18672 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561767013e00 con 0x5617670b76d0 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.225+0000 7f8d8dada640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617670b76d0 0x5617670b9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d8dada640 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617670b76d0 0x5617670b9b60 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e8d005d5c19c20 server_cookie=2010534a9f57d739 in_seq=0 out_seq=0 2024-03-22T02:24:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d717fa640 10 client.18672 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d717fa640 1 -- 172.21.15.28:0/861378158 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561767013e00 con 0x5617670b76d0 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d717fa640 10 client.18672 handle_command_reply: tid=0 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d92a02740 1 client.18672 shutdown 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 msgr2=0x5617670abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.226+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f8d7809bdf0 tx=0x7f8d780097f0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 shutdown_connections 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5617670b76d0 0x5617670b9b60 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a6f90 0x5617670b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 --2- 172.21.15.28:0/861378158 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d717fa640 0 client.18672 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:44.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 >> 172.21.15.28:0/861378158 conn(0x56176709c9a0 msgr2=0x56176709bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 shutdown_connections 2024-03-22T02:24:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 1 -- 172.21.15.28:0/861378158 wait complete. 2024-03-22T02:24:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.227+0000 7f8d92a02740 20 client.18672 trim_cache size 0 max 16384 2024-03-22T02:24:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 Processor -- start 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 -- start start 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881ab0c0 con 0x5617670a1e20 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881a8990 con 0x5617670a6f90 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d92a02740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881a67e0 con 0x5617670a1430 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.247+0000 7f8d8d2d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45410/0 (socket says 172.21.15.28:45410) 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1780967519 learned_addr learned my addr 172.21.15.28:0/1780967519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1780967519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1430 msgr2=0x5617670ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1780967519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 msgr2=0x5617670b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 -- 172.21.15.28:0/1780967519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561766cda0f0 con 0x5617670a1e20 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f8d78005760 tx=0x7f8d78009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785cd5b7cc34b2e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d737fe640 1 -- 172.21.15.28:0/1780967519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d780a4870 con 0x5617670a1e20 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d737fe640 1 -- 172.21.15.28:0/1780967519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7801c1c0 con 0x5617670a1e20 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.248+0000 7f8d737fe640 1 -- 172.21.15.28:0/1780967519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d7809b2b0 con 0x5617670a1e20 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- 172.21.15.28:0/1780967519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 msgr2=0x5617670abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f8d78005760 tx=0x7f8d78009700 comp rx=0 tx=0).stop 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- 172.21.15.28:0/1780967519 shutdown_connections 2024-03-22T02:24:44.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1e20 0x5617670abdb0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1780967519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1430 0x5617670ab870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- 172.21.15.28:0/1780967519 >> 172.21.15.28:0/1780967519 conn(0x56176709c9a0 msgr2=0x5617670a2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- 172.21.15.28:0/1780967519 shutdown_connections 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- 172.21.15.28:0/1780967519 wait complete. 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 -- start start 2024-03-22T02:24:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.249+0000 7f8d92a02740 1 Processor -- start 2024-03-22T02:24:44.253 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:43 smithi120 ceph-mon[36803]: pgmap v694: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-22T02:24:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1e20 0x5617670b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881a6e50 con 0x5617670a1430 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881a82c0 con 0x5617670a6f90 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d881a7520 con 0x5617670a1e20 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 10 client.0 _ll_register_callbacks cb 0x7f8d7809bd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45418/0 (socket says 172.21.15.28:45418) 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 -- 172.21.15.28:0/1077335254 learned_addr learned my addr 172.21.15.28:0/1077335254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8e2db640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1e20 0x5617670b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 -- 172.21.15.28:0/1077335254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1e20 msgr2=0x5617670b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1e20 0x5617670b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 -- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 msgr2=0x5617670b1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b1c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 -- 172.21.15.28:0/1077335254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617670b21f0 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d8dada640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010d30 tx=0x7f8d84010930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf75ebc42fbe85e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d717fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8402fca0 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8d8402f290 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.250+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d7801c1c0 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f8d8403f050 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 10 client.18678 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56176711dca0 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d7801c1c0 con 0x5617670a1430 2024-03-22T02:24:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f8d84026430 con 0x5617670a1430 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 10 client.18678 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 10 client.18678 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 20 client.18678 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56176711e370 0x561767120790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 4 client.18678 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56176700e150 con 0x56176711e370 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56176711e370 0x561767120790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d717fa640 10 client.18678 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.251+0000 7f8d8d2d9640 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56176711e370 0x561767120790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9829dc9be80000b8 server_cookie=445de54ccf7bba56 in_seq=0 out_seq=0 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.252+0000 7f8d717fa640 1 -- 172.21.15.28:0/1077335254 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56176700e150 con 0x56176711e370 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.252+0000 7f8d717fa640 10 client.18678 handle_command_reply: tid=0 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.252+0000 7f8d92a02740 1 client.18678 shutdown 2024-03-22T02:24:44.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.252+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 msgr2=0x5617670b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.252+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f8d84010d30 tx=0x7f8d84010930 comp rx=0 tx=0).stop 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 shutdown_connections 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56176711e370 0x561767120790 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5617670a6f90 0x5617670b1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5617670a1e20 0x5617670b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 --2- 172.21.15.28:0/1077335254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617670a1430 0x5617670b06b0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 >> 172.21.15.28:0/1077335254 conn(0x56176709c9a0 msgr2=0x5617670a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d717fa640 0 client.18678 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 shutdown_connections 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 1 -- 172.21.15.28:0/1077335254 wait complete. 2024-03-22T02:24:44.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d92a02740 20 client.18678 trim_cache size 0 max 16384 2024-03-22T02:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:24:44.261 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 msgr2=0x7f8d740a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 0x7f8d740a53e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f8d80005e30 tx=0x7f8d8002ce40 comp rx=0 tx=0).stop 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 msgr2=0x7f8d881983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f8d7800e9c0 tx=0x7f8d7800ee80 comp rx=0 tx=0).stop 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 shutdown_connections 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8d740a2f20 0x7f8d740a53e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d881078d0 0x7f8d8819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d88104fa0 0x7f8d881983f0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 --2- 172.21.15.28:0/1336212598 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d88102670 0x7f8d88197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.253+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 >> 172.21.15.28:0/1336212598 conn(0x7f8d880fc020 msgr2=0x7f8d88102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.254+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 shutdown_connections 2024-03-22T02:24:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.254+0000 7f8d8fd3e640 1 -- 172.21.15.28:0/1336212598 wait complete. 2024-03-22T02:24:44.265 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:24:44.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:24:44.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:43 smithi083 ceph-mon[36340]: pgmap v694: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 258 op/s 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 -- 172.21.15.28:0/2801790723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 msgr2=0x7f0c6c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 --2- 172.21.15.28:0/2801790723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c1032f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c009860 tx=0x7f0c5c02bb70 comp rx=0 tx=0).stop 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 -- 172.21.15.28:0/2801790723 shutdown_connections 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 --2- 172.21.15.28:0/2801790723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c10a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 --2- 172.21.15.28:0/2801790723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c1032f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 --2- 172.21.15.28:0/2801790723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 0x7f0c6c1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 -- 172.21.15.28:0/2801790723 >> 172.21.15.28:0/2801790723 conn(0x7f0c6c0fb7d0 msgr2=0x7f0c6c0fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 -- 172.21.15.28:0/2801790723 shutdown_connections 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 -- 172.21.15.28:0/2801790723 wait complete. 2024-03-22T02:24:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.418+0000 7f0c7437a640 1 Processor -- start 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 -- start start 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 0x7f0c6c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10d880 con 0x7f0c6c103830 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10d700 con 0x7f0c6c102e70 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c7437a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6c10da00 con 0x7f0c6c0ffea0 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72116640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 0x7f0c6c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45434/0 (socket says 172.21.15.28:45434) 2024-03-22T02:24:44.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 -- 172.21.15.28:0/1630695634 learned_addr learned my addr 172.21.15.28:0/1630695634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c71915640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 -- 172.21.15.28:0/1630695634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 msgr2=0x7f0c6c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 0x7f0c6c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 -- 172.21.15.28:0/1630695634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 msgr2=0x7f0c6c198470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c198470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 -- 172.21.15.28:0/1630695634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6c19fb50 con 0x7f0c6c103830 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.419+0000 7f0c72917640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400efc0 tx=0x7f0c6400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.420+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c64003970 con 0x7f0c6c103830 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.420+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6c19fe40 con 0x7f0c6c103830 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.420+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c64004020 con 0x7f0c6c103830 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.420+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c6400b9f0 con 0x7f0c6c103830 2024-03-22T02:24:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.420+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c6c1a02b0 con 0x7f0c6c103830 2024-03-22T02:24:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.421+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c6c108fa0 con 0x7f0c6c103830 2024-03-22T02:24:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.422+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0c6401d020 con 0x7f0c6c103830 2024-03-22T02:24:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.423+0000 7f0c637fe640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 0x7f0c580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.423+0000 7f0c72116640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 0x7f0c580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.423+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0c640cf690 con 0x7f0c6c103830 2024-03-22T02:24:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.423+0000 7f0c72116640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 0x7f0c580a53e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c069fe0 tx=0x7f0c68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.428+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c6409c0b0 con 0x7f0c6c103830 2024-03-22T02:24:44.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1127198 11.33 MB/sec execute 446 sec latency 30.193 ms 2024-03-22T02:24:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.664+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c6c062790 con 0x7f0c6c103830 2024-03-22T02:24:44.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.665+0000 7f0c637fe640 1 -- 172.21.15.28:0/1630695634 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11867 (secure 0 0 0) 0x7f0c6409b9b0 con 0x7f0c6c103830 2024-03-22T02:24:44.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:24:44.666 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":36,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:22:55.371454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:24:44.666 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 36 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 msgr2=0x7f0c580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 0x7f0c580a53e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c069fe0 tx=0x7f0c68005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 msgr2=0x7f0c6c19f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400efc0 tx=0x7f0c6400c800 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 shutdown_connections 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0c580a2f20 0x7f0c580a53e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c6c103830 0x7f0c6c19f4e0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0c6c102e70 0x7f0c6c198470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 --2- 172.21.15.28:0/1630695634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c6c0ffea0 0x7f0c6c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 >> 172.21.15.28:0/1630695634 conn(0x7f0c6c0fb7d0 msgr2=0x7f0c6c106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 shutdown_connections 2024-03-22T02:24:44.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.667+0000 7f0c7437a640 1 -- 172.21.15.28:0/1630695634 wait complete. 2024-03-22T02:24:44.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:24:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 -- 172.21.15.28:0/1999390660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8104f40 msgr2=0x7f22a8107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 --2- 172.21.15.28:0/1999390660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8104f40 0x7f22a8107330 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f2298009900 tx=0x7f229802bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 -- 172.21.15.28:0/1999390660 shutdown_connections 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 --2- 172.21.15.28:0/1999390660 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8107870 0x7f22a8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 --2- 172.21.15.28:0/1999390660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8104f40 0x7f22a8107330 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 --2- 172.21.15.28:0/1999390660 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8102610 0x7f22a8104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.787+0000 7f22af2a4640 1 -- 172.21.15.28:0/1999390660 >> 172.21.15.28:0/1999390660 conn(0x7f22a80fc020 msgr2=0x7f22a80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- 172.21.15.28:0/1999390660 shutdown_connections 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- 172.21.15.28:0/1999390660 wait complete. 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 Processor -- start 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- start start 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8104f40 0x7f22a8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 0x7f22a819f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d820 con 0x7f22a8107870 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d6a0 con 0x7f22a8102610 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22af2a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22a810d9a0 con 0x7f22a8104f40 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22ad040640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22ad841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 0x7f22a819f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22ad040640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49136/0 (socket says 172.21.15.28:49136) 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22ad040640 1 -- 172.21.15.28:0/2811932711 learned_addr learned my addr 172.21.15.28:0/2811932711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.788+0000 7f22ac83f640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8104f40 0x7f22a8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 -- 172.21.15.28:0/2811932711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8104f40 msgr2=0x7f22a8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8104f40 0x7f22a8198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 -- 172.21.15.28:0/2811932711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 msgr2=0x7f22a8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 -- 172.21.15.28:0/2811932711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a819fb70 con 0x7f22a8107870 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad040640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22ad841640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 0x7f22a819f500 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f22a000ec50 tx=0x7f22a000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f229e7fc640 1 -- 172.21.15.28:0/2811932711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0003b70 con 0x7f22a8107870 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f229e7fc640 1 -- 172.21.15.28:0/2811932711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a0003d10 con 0x7f22a8107870 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a819fe60 con 0x7f22a8107870 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f229e7fc640 1 -- 172.21.15.28:0/2811932711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a0010890 con 0x7f22a8107870 2024-03-22T02:24:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.789+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a81a02d0 con 0x7f22a8107870 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.792+0000 7f229e7fc640 1 -- 172.21.15.28:0/2811932711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f22a0003fc0 con 0x7f22a8107870 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.792+0000 7f22b1f6d740 1 Processor -- start 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.792+0000 7f22b1f6d740 1 -- start start 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x5601506e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f229e7fc640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 0x7f22940a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506ebd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 0x5601506ee810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56015064b8d0 con 0x5601506e6820 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5601505f3e00 con 0x5601506e7210 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560150647880 con 0x5601506ec380 2024-03-22T02:24:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 0x5601506ee810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x5601506e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f229e7fc640 1 -- 172.21.15.28:0/2811932711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f22a00cfda0 con 0x7f22a8107870 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad040640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 0x7f22940a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x5601506e6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45454/0 (socket says 172.21.15.28:45454) 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ac83f640 1 -- 172.21.15.28:0/2517055609 learned_addr learned my addr 172.21.15.28:0/2517055609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ac83f640 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506ebd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 -- 172.21.15.28:0/2517055609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 msgr2=0x5601506ebd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506ebd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 -- 172.21.15.28:0/2517055609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 msgr2=0x5601506e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x5601506e6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 -- 172.21.15.28:0/2517055609 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56015035f3f0 con 0x5601506ec380 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad040640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 0x7f22940a53e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f22a4011500 tx=0x7f22a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.793+0000 7f22ad841640 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 0x5601506ee810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22a000cba0 tx=0x7f22a000ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f75ff7059d2907f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22937fe640 1 -- 172.21.15.28:0/2517055609 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a009ba20 con 0x5601506ec380 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22937fe640 1 -- 172.21.15.28:0/2517055609 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a0009820 con 0x5601506ec380 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22937fe640 1 -- 172.21.15.28:0/2517055609 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a00aed30 con 0x5601506ec380 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2517055609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 msgr2=0x5601506ee810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 0x5601506ee810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22a000cba0 tx=0x7f22a000ef50 comp rx=0 tx=0).stop 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2517055609 shutdown_connections 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506ec380 0x5601506ee810 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506ebd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2517055609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x5601506e6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2517055609 >> 172.21.15.28:0/2517055609 conn(0x5601506e1d90 msgr2=0x5601506e2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2517055609 shutdown_connections 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.794+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2517055609 wait complete. 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 -- start start 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 Processor -- start 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x5601506f8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601506ef330 con 0x5601506e7210 2024-03-22T02:24:44.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560150607300 con 0x5601506ec380 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5601506ef4b0 con 0x5601506e6820 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22b1f6d740 10 client.0 _ll_register_callbacks cb 0x560150642e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22ad841640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x5601506f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22ad040640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45466/0 (socket says 172.21.15.28:45466) 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.795+0000 7f22ac83f640 1 -- 172.21.15.28:0/2907685804 learned_addr learned my addr 172.21.15.28:0/2907685804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 -- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 msgr2=0x5601506f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 -- 172.21.15.28:0/2907685804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 msgr2=0x5601506f8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x5601506f8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 -- 172.21.15.28:0/2907685804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601504297d0 con 0x5601506e7210 2024-03-22T02:24:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ad841640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x5601506f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22ac83f640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2298036040 tx=0x7f2298005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3c2eac5347eb9c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2298048070 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2298005460 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22a0009820 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2298051050 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 10 client.18696 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5601506fb6f0 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a0009820 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f229804b020 con 0x5601506e7210 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 10 client.18696 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 10 client.18696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 20 client.18696 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5601506fce80 0x5601506ff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 4 client.18696 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.796+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560150658e10 con 0x5601506fce80 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.797+0000 7f22ad040640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5601506fce80 0x5601506ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.797+0000 7f22ad040640 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5601506fce80 0x5601506ff300 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=24ef494f9faf88aa server_cookie=f9624d2ae3eb215f in_seq=0 out_seq=0 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.797+0000 7f2290ff9640 10 client.18696 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f2290ff9640 1 -- 172.21.15.28:0/2907685804 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560150658e10 con 0x5601506fce80 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f2290ff9640 10 client.18696 handle_command_reply: tid=0 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 client.18696 shutdown 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 msgr2=0x5601506f12d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2298036040 tx=0x7f2298005890 comp rx=0 tx=0).stop 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 shutdown_connections 2024-03-22T02:24:44.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5601506fce80 0x5601506ff300 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x5601506f8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e7210 0x5601506f12d0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f2290ff9640 0 client.18696 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/2907685804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 >> 172.21.15.28:0/2907685804 conn(0x5601506e1d90 msgr2=0x5601506e0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 shutdown_connections 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 1 -- 172.21.15.28:0/2907685804 wait complete. 2024-03-22T02:24:44.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.798+0000 7f22b1f6d740 20 client.18696 trim_cache size 0 max 16384 2024-03-22T02:24:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.812+0000 7f22b1f6d740 1 Processor -- start 2024-03-22T02:24:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 -- start start 2024-03-22T02:24:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506e0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506e0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81ad2f0 con 0x5601506ec380 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81aabc0 con 0x5601506e7210 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81a8a10 con 0x5601506e6820 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad040640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506e0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506e0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45478/0 (socket says 172.21.15.28:45478) 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 -- 172.21.15.28:0/3138946123 learned_addr learned my addr 172.21.15.28:0/3138946123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506e0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49156/0 (socket says 172.21.15.28:49156) 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 -- 172.21.15.28:0/3138946123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 msgr2=0x5601506e0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506e0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 -- 172.21.15.28:0/3138946123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 msgr2=0x5601506e0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506e0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 -- 172.21.15.28:0/3138946123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601506e75f0 con 0x5601506ec380 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f22ad841640 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f22a0004700 tx=0x7f22a009bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f4225c062ad104 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.813+0000 7f2292ffd640 1 -- 172.21.15.28:0/3138946123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a00aef10 con 0x5601506ec380 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f2292ffd640 1 -- 172.21.15.28:0/3138946123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a0009820 con 0x5601506ec380 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f2292ffd640 1 -- 172.21.15.28:0/3138946123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a00a4ad0 con 0x5601506ec380 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 -- 172.21.15.28:0/3138946123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 msgr2=0x5601506e11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f22a0004700 tx=0x7f22a009bcc0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 -- 172.21.15.28:0/3138946123 shutdown_connections 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506ec380 0x5601506e11c0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506e7210 0x5601506e0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/3138946123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e6820 0x5601506e0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 -- 172.21.15.28:0/3138946123 >> 172.21.15.28:0/3138946123 conn(0x5601506e1d90 msgr2=0x5601506e8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 -- 172.21.15.28:0/3138946123 shutdown_connections 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.814+0000 7f22b1f6d740 1 -- 172.21.15.28:0/3138946123 wait complete. 2024-03-22T02:24:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 -- start start 2024-03-22T02:24:44.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 Processor -- start 2024-03-22T02:24:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 0x56015075ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x560150765f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81a9080 con 0x5601506e6820 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81aa4f0 con 0x5601506ec380 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f22a81a9750 con 0x5601506e7210 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22b1f6d740 10 client.0 _ll_register_callbacks cb 0x7f2298004420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22ad040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22ad040640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45482/0 (socket says 172.21.15.28:45482) 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22ac83f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 0x56015075ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22ad040640 1 -- 172.21.15.28:0/715172501 learned_addr learned my addr 172.21.15.28:0/715172501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.815+0000 7f22ad841640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x560150765f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 -- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 msgr2=0x56015075ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 0x56015075ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 -- 172.21.15.28:0/715172501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 msgr2=0x560150765f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x560150765f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 -- 172.21.15.28:0/715172501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560150766530 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ac83f640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 0x56015075ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22ad040640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x5601506ffb70 tx=0x7f22a4013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149668bb6b128f78 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22a402f060 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f22a400fcb0 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22a0009820 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f22a4036050 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22b1f6d740 10 client.18702 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x560150767900 con 0x5601506e6820 2024-03-22T02:24:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22a0009820 con 0x5601506e6820 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.816+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f22a4029450 con 0x5601506e6820 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 10 client.18702 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 10 client.18702 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 20 client.18702 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x560150768310 0x56015076a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 4 client.18702 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560150653160 con 0x560150768310 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22ac83f640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x560150768310 0x56015076a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22ac83f640 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x560150768310 0x56015076a730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1d119682e71aea56 server_cookie=163e744c63911c15 in_seq=0 out_seq=0 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f2290ff9640 10 client.18702 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f2290ff9640 1 -- 172.21.15.28:0/715172501 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560150653160 con 0x560150768310 2024-03-22T02:24:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f2290ff9640 10 client.18702 handle_command_reply: tid=0 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 client.18702 shutdown 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 msgr2=0x56015075e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x5601506ffb70 tx=0x7f22a4013040 comp rx=0 tx=0).stop 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 shutdown_connections 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x560150768310 0x56015076a730 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601506ec380 0x560150765f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f2290ff9640 0 client.18702 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601506e7210 0x56015075ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 --2- 172.21.15.28:0/715172501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601506e6820 0x56015075e9c0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.817+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 >> 172.21.15.28:0/715172501 conn(0x5601506e1d90 msgr2=0x5601506e4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 shutdown_connections 2024-03-22T02:24:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22b1f6d740 1 -- 172.21.15.28:0/715172501 wait complete. 2024-03-22T02:24:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22b1f6d740 20 client.18702 trim_cache size 0 max 16384 2024-03-22T02:24:44.826 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 msgr2=0x7f22940a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 0x7f22940a53e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f22a4011500 tx=0x7f22a4005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 msgr2=0x7f22a819f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 0x7f22a819f500 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f22a000ec50 tx=0x7f22a000c6a0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 shutdown_connections 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22940a2f20 0x7f22940a53e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22a8107870 0x7f22a819f500 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22a8104f40 0x7f22a8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 --2- 172.21.15.28:0/2811932711 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f22a8102610 0x7f22a8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 >> 172.21.15.28:0/2811932711 conn(0x7f22a80fc020 msgr2=0x7f22a8102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 shutdown_connections 2024-03-22T02:24:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.818+0000 7f22af2a4640 1 -- 172.21.15.28:0/2811932711 wait complete. 2024-03-22T02:24:44.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:24:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 -- 172.21.15.28:0/1176146058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 msgr2=0x7f1564100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 --2- 172.21.15.28:0/1176146058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564100ef0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f1554009900 tx=0x7f155402bc30 comp rx=0 tx=0).stop 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 -- 172.21.15.28:0/1176146058 shutdown_connections 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 --2- 172.21.15.28:0/1176146058 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 0x7f1564108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 --2- 172.21.15.28:0/1176146058 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 0x7f1564102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 --2- 172.21.15.28:0/1176146058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564100ef0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 -- 172.21.15.28:0/1176146058 >> 172.21.15.28:0/1176146058 conn(0x7f15640fc040 msgr2=0x7f15640fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 -- 172.21.15.28:0/1176146058 shutdown_connections 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.942+0000 7f156a593640 1 -- 172.21.15.28:0/1176146058 wait complete. 2024-03-22T02:24:44.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 Processor -- start 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 -- start start 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 0x7f15641984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 0x7f156419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f156410c720 con 0x7f1564100ad0 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f156410c5a0 con 0x7f1564101cc0 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f156a593640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f156410c8a0 con 0x7f1564102680 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45502/0 (socket says 172.21.15.28:45502) 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 -- 172.21.15.28:0/1175650696 learned_addr learned my addr 172.21.15.28:0/1175650696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f155bfff640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 0x7f15641984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1568b30640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 0x7f156419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 -- 172.21.15.28:0/1175650696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 msgr2=0x7f156419f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 0x7f156419f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 -- 172.21.15.28:0/1175650696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 msgr2=0x7f15641984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 0x7f15641984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.943+0000 7f1563fff640 1 -- 172.21.15.28:0/1175650696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f156419fba0 con 0x7f1564100ad0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f1563fff640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f1554000c00 tx=0x7f1554004a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f1561ffb640 1 -- 172.21.15.28:0/1175650696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554041070 con 0x7f1564100ad0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f156419fe30 con 0x7f1564100ad0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f1561ffb640 1 -- 172.21.15.28:0/1175650696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1554004450 con 0x7f1564100ad0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f1561ffb640 1 -- 172.21.15.28:0/1175650696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554034440 con 0x7f1564100ad0 2024-03-22T02:24:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.944+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15641a0240 con 0x7f1564100ad0 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.946+0000 7f1561ffb640 1 -- 172.21.15.28:0/1175650696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1554044070 con 0x7f1564100ad0 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 Processor -- start 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 -- start start 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1561ffb640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 0x7f15500a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838be580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585838158d0 con 0x5585838b9640 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585837bde00 con 0x5585838ba030 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f156d17d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558583811880 con 0x5585838bebf0 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1561ffb640 1 -- 172.21.15.28:0/1175650696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f15540f1310 con 0x7f1564100ad0 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1568b30640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 0x7f15500a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838be580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45512/0 (socket says 172.21.15.28:45512) 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1563fff640 1 -- 172.21.15.28:0/4002691913 learned_addr learned my addr 172.21.15.28:0/4002691913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1568b30640 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 -- 172.21.15.28:0/4002691913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 msgr2=0x5585838c1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 -- 172.21.15.28:0/4002691913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 msgr2=0x5585838b9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f155bfff640 1 -- 172.21.15.28:0/4002691913 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585835293f0 con 0x5585838ba030 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.947+0000 7f1563fff640 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.948+0000 7f1568b30640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 0x7f15500a53e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f155c005fd0 tx=0x7f155c004410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.948+0000 7f155bfff640 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838be580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f154c009bf0 tx=0x7f154c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdbbb5fee666e885 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.948+0000 7f155a7fc640 1 -- 172.21.15.28:0/4002691913 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c004050 con 0x5585838ba030 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.948+0000 7f155a7fc640 1 -- 172.21.15.28:0/4002691913 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c005500 con 0x5585838ba030 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.948+0000 7f155a7fc640 1 -- 172.21.15.28:0/4002691913 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0035f0 con 0x5585838ba030 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- 172.21.15.28:0/4002691913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 msgr2=0x5585838be580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838be580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f154c009bf0 tx=0x7f154c011810 comp rx=0 tx=0).stop 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- 172.21.15.28:0/4002691913 shutdown_connections 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838be580 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- 172.21.15.28:0/4002691913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- 172.21.15.28:0/4002691913 >> 172.21.15.28:0/4002691913 conn(0x5585838abd10 msgr2=0x5585838ac110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- 172.21.15.28:0/4002691913 shutdown_connections 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- 172.21.15.28:0/4002691913 wait complete. 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- start start 2024-03-22T02:24:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 Processor -- start 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585838c1ba0 con 0x5585838b9640 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.949+0000 7f156d17d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585837d1300 con 0x5585838ba030 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f156d17d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5585838c1d20 con 0x5585838bebf0 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f156d17d740 10 client.0 _ll_register_callbacks cb 0x55858380ce30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45516/0 (socket says 172.21.15.28:45516) 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 -- 172.21.15.28:0/1079228741 learned_addr learned my addr 172.21.15.28:0/1079228741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1568b30640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f155bfff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 -- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 msgr2=0x5585838caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 -- 172.21.15.28:0/1079228741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 msgr2=0x5585838c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 -- 172.21.15.28:0/1079228741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585835f37d0 con 0x5585838b9640 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f1563fff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f15540eea50 tx=0x7f15540bd220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1dceb2776677ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f15437fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155404b070 con 0x5585838b9640 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f15540bb9e0 con 0x5585838b9640 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f154c005500 con 0x5585838b9640 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f15540c1780 con 0x5585838b9640 2024-03-22T02:24:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f156d17d740 10 client.18714 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.950+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5585838cde60 con 0x5585838b9640 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f154c005500 con 0x5585838b9640 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f15540c5bb0 con 0x5585838b9640 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 10 client.18714 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 10 client.18714 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 20 client.18714 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x5585838d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 4 client.18714 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558583822e10 con 0x5585838cf480 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f155bfff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x5585838d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f15437fe640 10 client.18714 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.951+0000 7f155bfff640 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x5585838d18d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5541f93ee79ea743 server_cookie=f612e6ebcda552c4 in_seq=0 out_seq=0 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.952+0000 7f15437fe640 1 -- 172.21.15.28:0/1079228741 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558583822e10 con 0x5585838cf480 2024-03-22T02:24:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.952+0000 7f15437fe640 10 client.18714 handle_command_reply: tid=0 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.952+0000 7f156d17d740 1 client.18714 shutdown 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 msgr2=0x5585838c3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f15540eea50 tx=0x7f15540bd220 comp rx=0 tx=0).stop 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 shutdown_connections 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x5585838d18d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f15437fe640 0 client.18714 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838ba030 0x5585838c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 --2- 172.21.15.28:0/1079228741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838b9640 0x5585838c3570 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 >> 172.21.15.28:0/1079228741 conn(0x5585838abd10 msgr2=0x5585838b8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 shutdown_connections 2024-03-22T02:24:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 1 -- 172.21.15.28:0/1079228741 wait complete. 2024-03-22T02:24:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.953+0000 7f156d17d740 20 client.18714 trim_cache size 0 max 16384 2024-03-22T02:24:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 Processor -- start 2024-03-22T02:24:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 -- start start 2024-03-22T02:24:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838b9640 0x5585838c3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15641ab110 con 0x5585838ba030 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.975+0000 7f156d17d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15641a89e0 con 0x5585838bebf0 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.976+0000 7f156d17d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15641a6830 con 0x5585838b9640 2024-03-22T02:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.976+0000 7f1568b30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49218/0 (socket says 172.21.15.28:49218) 2024-03-22T02:24:44.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 -- 172.21.15.28:0/179268396 learned_addr learned my addr 172.21.15.28:0/179268396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 -- 172.21.15.28:0/179268396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838b9640 msgr2=0x5585838c3a10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:24:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838b9640 0x5585838c3a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 -- 172.21.15.28:0/179268396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 msgr2=0x5585838caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.977+0000 7f1568b30640 1 -- 172.21.15.28:0/179268396 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585838c1c40 con 0x5585838bebf0 2024-03-22T02:24:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.978+0000 7f1568b30640 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f155c0127e0 tx=0x7f155c01d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63123a5a7722db87 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.978+0000 7f1559ffb640 1 -- 172.21.15.28:0/179268396 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c020b80 con 0x5585838bebf0 2024-03-22T02:24:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.978+0000 7f1559ffb640 1 -- 172.21.15.28:0/179268396 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155c012e20 con 0x5585838bebf0 2024-03-22T02:24:44.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.978+0000 7f1559ffb640 1 -- 172.21.15.28:0/179268396 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c00f620 con 0x5585838bebf0 2024-03-22T02:24:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.979+0000 7f156d17d740 1 -- 172.21.15.28:0/179268396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 msgr2=0x5585838b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.979+0000 7f156d17d740 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f155c0127e0 tx=0x7f155c01d000 comp rx=0 tx=0).stop 2024-03-22T02:24:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 -- 172.21.15.28:0/179268396 shutdown_connections 2024-03-22T02:24:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838bebf0 0x5585838b8b50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 --2- 172.21.15.28:0/179268396 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838b9640 0x5585838c3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 -- 172.21.15.28:0/179268396 >> 172.21.15.28:0/179268396 conn(0x5585838abd10 msgr2=0x5585838b8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 -- 172.21.15.28:0/179268396 shutdown_connections 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 -- 172.21.15.28:0/179268396 wait complete. 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.980+0000 7f156d17d740 1 -- start start 2024-03-22T02:24:44.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 Processor -- start 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838b9640 0x5585838c8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15641a6ea0 con 0x5585838ba030 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f15641a8310 con 0x5585838b9640 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f15641a7570 con 0x5585838bebf0 2024-03-22T02:24:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.981+0000 7f156d17d740 10 client.0 _ll_register_callbacks cb 0x7f15540b9980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.982+0000 7f1563fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838b9640 0x5585838c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.982+0000 7f155bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.982+0000 7f155bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45544/0 (socket says 172.21.15.28:45544) 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.982+0000 7f155bfff640 1 -- 172.21.15.28:0/3450874850 learned_addr learned my addr 172.21.15.28:0/3450874850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.982+0000 7f1568b30640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 -- 172.21.15.28:0/3450874850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838b9640 msgr2=0x5585838c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838b9640 0x5585838c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 -- 172.21.15.28:0/3450874850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 msgr2=0x5585838c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 -- 172.21.15.28:0/3450874850 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585838c9d60 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f1568b30640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c9770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f155c009510 tx=0x7f155c004c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbd015966535a28e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.983+0000 7f15437fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.984+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c016030 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.984+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f155c020d90 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.985+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155c012e20 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.985+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f155c004e60 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.985+0000 7f156d17d740 10 client.28531 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.985+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x558583935810 con 0x5585838bebf0 2024-03-22T02:24:44.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155c012e20 con 0x5585838bebf0 2024-03-22T02:24:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f155c005310 con 0x5585838bebf0 2024-03-22T02:24:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 10 client.28531 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 10 client.28531 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:24:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 20 client.28531 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x558583938220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 4 client.28531 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.986+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55858381d160 con 0x5585838cf480 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.987+0000 7f1563fff640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x558583938220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.987+0000 7f1563fff640 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x558583938220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61cdd57919be4c8b server_cookie=d60307d2e9e4a801 in_seq=0 out_seq=0 2024-03-22T02:24:44.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.987+0000 7f15437fe640 10 client.28531 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f15437fe640 1 -- 172.21.15.28:0/3450874850 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55858381d160 con 0x5585838cf480 2024-03-22T02:24:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f15437fe640 10 client.28531 handle_command_reply: tid=0 2024-03-22T02:24:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 client.28531 shutdown 2024-03-22T02:24:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 msgr2=0x5585838c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c9770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f155c009510 tx=0x7f155c004c70 comp rx=0 tx=0).stop 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 shutdown_connections 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5585838cf480 0x558583938220 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585838bebf0 0x5585838c9770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585838ba030 0x5585838c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.988+0000 7f156d17d740 1 --2- 172.21.15.28:0/3450874850 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585838b9640 0x5585838c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.989+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 >> 172.21.15.28:0/3450874850 conn(0x5585838abd10 msgr2=0x5585838c3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.989+0000 7f15437fe640 0 client.28531 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.989+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 shutdown_connections 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.989+0000 7f156d17d740 1 -- 172.21.15.28:0/3450874850 wait complete. 2024-03-22T02:24:44.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.989+0000 7f156d17d740 20 client.28531 trim_cache size 0 max 16384 2024-03-22T02:24:44.991 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:44.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.990+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 msgr2=0x7f15500a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.990+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 0x7f15500a53e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f155c005fd0 tx=0x7f155c004410 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.990+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 msgr2=0x7f1564197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.990+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f1554000c00 tx=0x7f1554004a30 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 shutdown_connections 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f15500a2f20 0x7f15500a53e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1564102680 0x7f156419f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1564101cc0 0x7f15641984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 --2- 172.21.15.28:0/1175650696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564100ad0 0x7f1564197f80 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 >> 172.21.15.28:0/1175650696 conn(0x7f15640fc040 msgr2=0x7f1564069ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 shutdown_connections 2024-03-22T02:24:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:44.991+0000 7f156a593640 1 -- 172.21.15.28:0/1175650696 wait complete. 2024-03-22T02:24:45.006 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:44 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1630695634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:45.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 -- 172.21.15.28:0/1074155791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 msgr2=0x7f41b8108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/1074155791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b8108ee0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009ca0 tx=0x7f41b002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 -- 172.21.15.28:0/1074155791 shutdown_connections 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/1074155791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b8108ee0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/1074155791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8101c90 0x7f41b8102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/1074155791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8100aa0 0x7f41b8100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 -- 172.21.15.28:0/1074155791 >> 172.21.15.28:0/1074155791 conn(0x7f41b80fc030 msgr2=0x7f41b80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 -- 172.21.15.28:0/1074155791 shutdown_connections 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.120+0000 7f41bdaa4640 1 -- 172.21.15.28:0/1074155791 wait complete. 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 Processor -- start 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 -- start start 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8100aa0 0x7f41b8104a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8101c90 0x7f41b8103070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b810c720 con 0x7f41b8102650 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41b810c5a0 con 0x7f41b8101c90 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.121+0000 7f41bdaa4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41b810c8a0 con 0x7f41b8100aa0 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45556/0 (socket says 172.21.15.28:45556) 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 -- 172.21.15.28:0/2126357899 learned_addr learned my addr 172.21.15.28:0/2126357899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b77fe640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8100aa0 0x7f41b8104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8101c90 0x7f41b8103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 -- 172.21.15.28:0/2126357899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8100aa0 msgr2=0x7f41b8104a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8100aa0 0x7f41b8104a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 -- 172.21.15.28:0/2126357899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8101c90 msgr2=0x7f41b8103070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8101c90 0x7f41b8103070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 -- 172.21.15.28:0/2126357899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b8103e40 con 0x7f41b8102650 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b7fff640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f41b0037040 tx=0x7f41b0032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b4ff9640 1 -- 172.21.15.28:0/2126357899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0041020 con 0x7f41b8102650 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b81a1f80 con 0x7f41b8102650 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b4ff9640 1 -- 172.21.15.28:0/2126357899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41b0034cd0 con 0x7f41b8102650 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41b4ff9640 1 -- 172.21.15.28:0/2126357899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41b0039040 con 0x7f41b8102650 2024-03-22T02:24:45.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.122+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b81a23e0 con 0x7f41b8102650 2024-03-22T02:24:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.124+0000 7f41b4ff9640 1 -- 172.21.15.28:0/2126357899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f41b0044020 con 0x7f41b8102650 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 Processor -- start 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b4ff9640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 0x7f41a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 -- start start 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 0x7f41a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b4ff9640 1 -- 172.21.15.28:0/2126357899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f41b00f1600 con 0x7f41b8102650 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556686288d0 con 0x5556686c9380 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5556685d0e00 con 0x5556686c3820 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41c0769740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555668624880 con 0x5556686c4210 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45572/0 (socket says 172.21.15.28:45572) 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b7fff640 1 -- 172.21.15.28:0/2240662585 learned_addr learned my addr 172.21.15.28:0/2240662585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49256/0 (socket says 172.21.15.28:49256) 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.125+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 -- 172.21.15.28:0/2240662585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 msgr2=0x5556686c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 -- 172.21.15.28:0/2240662585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 msgr2=0x5556686cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 -- 172.21.15.28:0/2240662585 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55566833c3f0 con 0x5556686c3820 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b7fff640 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 0x7f41a00a53e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f41b81047a0 tx=0x7f41ac013830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f41b77fe640 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009b70 tx=0x7f41a8009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d16fab32aee5bb6a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f419dffb640 1 -- 172.21.15.28:0/2240662585 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8011e60 con 0x5556686c3820 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f419dffb640 1 -- 172.21.15.28:0/2240662585 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a8005020 con 0x5556686c3820 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.126+0000 7f419dffb640 1 -- 172.21.15.28:0/2240662585 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8004b20 con 0x5556686c3820 2024-03-22T02:24:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- 172.21.15.28:0/2240662585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 msgr2=0x5556686c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009b70 tx=0x7f41a8009f70 comp rx=0 tx=0).stop 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- 172.21.15.28:0/2240662585 shutdown_connections 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c9380 0x5556686cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 --2- 172.21.15.28:0/2240662585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c3820 0x5556686c3c40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- 172.21.15.28:0/2240662585 >> 172.21.15.28:0/2240662585 conn(0x5556686bed90 msgr2=0x5556686bf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- 172.21.15.28:0/2240662585 shutdown_connections 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- 172.21.15.28:0/2240662585 wait complete. 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 -- start start 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.127+0000 7f41c0769740 1 Processor -- start 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5556686cc330 con 0x5556686c4210 2024-03-22T02:24:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5556685e4300 con 0x5556686c9380 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5556686cc4b0 con 0x5556686c3820 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41c0769740 10 client.0 _ll_register_callbacks cb 0x55566861fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45584/0 (socket says 172.21.15.28:45584) 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 -- 172.21.15.28:0/4108625431 learned_addr learned my addr 172.21.15.28:0/4108625431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b7fff640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 -- 172.21.15.28:0/4108625431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 msgr2=0x5556686cdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686cdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 -- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 msgr2=0x5556686d5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 -- 172.21.15.28:0/4108625431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556684067d0 con 0x5556686c4210 2024-03-22T02:24:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009b70 tx=0x7f41ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e63698407fa307 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.128+0000 7f4196ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ac024070 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f41ac010bb0 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a8005020 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f41ac010df0 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 10 client.18729 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5556686d8670 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a8005020 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f41ac01f070 con 0x5556686c4210 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 10 client.18729 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 10 client.18729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 20 client.18729 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556686d9de0 0x5556686dc240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 4 client.18729 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.129+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555668635e10 con 0x5556686d9de0 2024-03-22T02:24:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.130+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556686d9de0 0x5556686dc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.130+0000 7f4196ffd640 10 client.18729 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.130+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556686d9de0 0x5556686dc240 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba33ab0043737bd5 server_cookie=186e2414ec4d87c8 in_seq=0 out_seq=0 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f4196ffd640 1 -- 172.21.15.28:0/4108625431 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555668635e10 con 0x5556686d9de0 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f4196ffd640 10 client.18729 handle_command_reply: tid=0 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 client.18729 shutdown 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 msgr2=0x5556686ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f41ac009b70 tx=0x7f41ac004730 comp rx=0 tx=0).stop 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 shutdown_connections 2024-03-22T02:24:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556686d9de0 0x5556686dc240 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686ce250 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f4196ffd640 0 client.18729 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 --2- 172.21.15.28:0/4108625431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686cdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 >> 172.21.15.28:0/4108625431 conn(0x5556686bed90 msgr2=0x5556686bdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 shutdown_connections 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 1 -- 172.21.15.28:0/4108625431 wait complete. 2024-03-22T02:24:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.131+0000 7f41c0769740 20 client.18729 trim_cache size 0 max 16384 2024-03-22T02:24:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 Processor -- start 2024-03-22T02:24:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 -- start start 2024-03-22T02:24:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686d5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686bd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81af4f0 con 0x5556686c3820 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81acdc0 con 0x5556686c9380 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41c0769740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81aac10 con 0x5556686c4210 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686d5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45600/0 (socket says 172.21.15.28:45600) 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b77fe640 1 -- 172.21.15.28:0/553629268 learned_addr learned my addr 172.21.15.28:0/553629268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b7fff640 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686bd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.145+0000 7f41b77fe640 1 -- 172.21.15.28:0/553629268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 msgr2=0x5556686d5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f41b77fe640 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686d5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f41b77fe640 1 -- 172.21.15.28:0/553629268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 msgr2=0x5556686bd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f41b77fe640 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686bd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f41b77fe640 1 -- 172.21.15.28:0/553629268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556686c33e0 con 0x5556686c3820 2024-03-22T02:24:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f41b77fe640 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009b70 tx=0x7f41a80097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60dfb6997b1cbed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f419d7fa640 1 -- 172.21.15.28:0/553629268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a801e040 con 0x5556686c3820 2024-03-22T02:24:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f419d7fa640 1 -- 172.21.15.28:0/553629268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a801f040 con 0x5556686c3820 2024-03-22T02:24:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.146+0000 7f419d7fa640 1 -- 172.21.15.28:0/553629268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8026730 con 0x5556686c3820 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- 172.21.15.28:0/553629268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 msgr2=0x5556686c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f41a8009b70 tx=0x7f41a80097a0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- 172.21.15.28:0/553629268 shutdown_connections 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686bd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c4210 0x5556686d5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- 172.21.15.28:0/553629268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c3820 0x5556686c4cd0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- 172.21.15.28:0/553629268 >> 172.21.15.28:0/553629268 conn(0x5556686bed90 msgr2=0x5556686c5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- 172.21.15.28:0/553629268 shutdown_connections 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- 172.21.15.28:0/553629268 wait complete. 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- start start 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 Processor -- start 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686d2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81ab280 con 0x5556686c4210 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81ac6f0 con 0x5556686c9380 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.147+0000 7f41c0769740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81ab950 con 0x5556686c3820 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41c0769740 10 client.0 _ll_register_callbacks cb 0x7f41ac005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686d2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49654/0 (socket says 172.21.15.28:49654) 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b7fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49296/0 (socket says 172.21.15.28:49296) 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 -- 172.21.15.28:0/4058977025 learned_addr learned my addr 172.21.15.28:0/4058977025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 -- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 msgr2=0x5556686d3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 -- 172.21.15.28:0/4058977025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 msgr2=0x5556686d2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686d2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 -- 172.21.15.28:0/4058977025 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556686d44f0 con 0x5556686c3820 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686d2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41b77fe640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011e00 tx=0x7f41a8011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f22347c60c03942 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f4196ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8004660 con 0x5556686c3820 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f41a8004800 con 0x5556686c3820 2024-03-22T02:24:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.148+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41a801f040 con 0x5556686c3820 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f41a8004a00 con 0x5556686c3820 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 10 client.28549 fetch_fsmap learned FSMap version 36 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5556687401a0 con 0x5556686c3820 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a801f040 con 0x5556686c3820 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 36) ==== 8350+0+0 (secure 0 0 0) 0x7f41a803d030 con 0x5556686c3820 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 10 client.28549 fetch_fsmap finished waiting for FSMap version 36 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 10 client.28549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 20 client.28549 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556687409f0 0x555668742e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 4 client.28549 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555668630160 con 0x5556687409f0 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.149+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556687409f0 0x555668742e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41b6ffd640 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556687409f0 0x555668742e10 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd54dafa26984b4f server_cookie=4edb31c3f5d50624 in_seq=0 out_seq=0 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f4196ffd640 10 client.28549 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f4196ffd640 1 -- 172.21.15.28:0/4058977025 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555668630160 con 0x5556687409f0 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f4196ffd640 10 client.28549 handle_command_reply: tid=0 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 client.28549 shutdown 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 msgr2=0x5556686d29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f41a8011e00 tx=0x7f41a8011c20 comp rx=0 tx=0).stop 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 shutdown_connections 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5556687409f0 0x555668742e10 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5556686c9380 0x5556686d3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5556686c4210 0x5556686d2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 --2- 172.21.15.28:0/4058977025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556686c3820 0x5556686d29b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f4196ffd640 0 client.28549 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 >> 172.21.15.28:0/4058977025 conn(0x5556686bed90 msgr2=0x5556686c1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 shutdown_connections 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 1 -- 172.21.15.28:0/4058977025 wait complete. 2024-03-22T02:24:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41c0769740 20 client.28549 trim_cache size 0 max 16384 2024-03-22T02:24:45.157 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:24:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.150+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 msgr2=0x7f41a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 0x7f41a00a53e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f41b81047a0 tx=0x7f41ac013830 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 msgr2=0x7f41b81035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f41b0037040 tx=0x7f41b0032f40 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 shutdown_connections 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41a00a2f20 0x7f41a00a53e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41b8102650 0x7f41b81035b0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f41b8101c90 0x7f41b8103070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 --2- 172.21.15.28:0/2126357899 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41b8100aa0 0x7f41b8104a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 >> 172.21.15.28:0/2126357899 conn(0x7f41b80fc030 msgr2=0x7f41b80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 shutdown_connections 2024-03-22T02:24:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:45.151+0000 7f41bdaa4640 1 -- 172.21.15.28:0/2126357899 wait complete. 2024-03-22T02:24:45.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:44 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1630695634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:45.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:44 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1630695634' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:24:45.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1129795 11.34 MB/sec execute 447 sec latency 17.394 ms 2024-03-22T02:24:46.162 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:24:46.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:24:46.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:24:46.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:45 smithi028 ceph-mon[31916]: pgmap v695: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 408 op/s 2024-03-22T02:24:46.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:45 smithi120 ceph-mon[36803]: pgmap v695: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 408 op/s 2024-03-22T02:24:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:45 smithi083 ceph-mon[36340]: pgmap v695: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 408 op/s 2024-03-22T02:24:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- 172.21.15.28:0/3034613182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8103750 msgr2=0x7faef8109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- 172.21.15.28:0/3034613182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8103750 0x7faef8109fe0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7faee8009ca0 tx=0x7faee802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- 172.21.15.28:0/3034613182 shutdown_connections 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- 172.21.15.28:0/3034613182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8103750 0x7faef8109fe0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- 172.21.15.28:0/3034613182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8102d90 0x7faef8103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- 172.21.15.28:0/3034613182 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- 172.21.15.28:0/3034613182 >> 172.21.15.28:0/3034613182 conn(0x7faef80fd0f0 msgr2=0x7faef80ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- 172.21.15.28:0/3034613182 shutdown_connections 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- 172.21.15.28:0/3034613182 wait complete. 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 Processor -- start 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 -- start start 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.276+0000 7faefd4eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8103750 0x7faef819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faef810d820 con 0x7faef8102d90 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faef810d6a0 con 0x7faef8101ba0 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faef810d9a0 con 0x7faef8103750 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 -- 172.21.15.28:0/1264380547 learned_addr learned my addr 172.21.15.28:0/1264380547 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef77fe640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8103750 0x7faef819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 -- 172.21.15.28:0/1264380547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8103750 msgr2=0x7faef819f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8103750 0x7faef819f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 -- 172.21.15.28:0/1264380547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 msgr2=0x7faef8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 -- 172.21.15.28:0/1264380547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef819fab0 con 0x7faef8102d90 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faef67fc640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9f0 tx=0x7faee000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faeeffff640 1 -- 172.21.15.28:0/1264380547 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee000cd80 con 0x7faef8102d90 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faeeffff640 1 -- 172.21.15.28:0/1264380547 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee0005500 con 0x7faef8102d90 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef819fda0 con 0x7faef8102d90 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faeeffff640 1 -- 172.21.15.28:0/1264380547 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0010690 con 0x7faef8102d90 2024-03-22T02:24:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.277+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef81a0210 con 0x7faef8102d90 2024-03-22T02:24:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.279+0000 7faeeffff640 1 -- 172.21.15.28:0/1264380547 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faee00046c0 con 0x7faef8102d90 2024-03-22T02:24:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 Processor -- start 2024-03-22T02:24:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 -- start start 2024-03-22T02:24:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe870fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe878bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592fe7d5570 con 0x5592fe8715b0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5592fe77de10 con 0x5592fe870bc0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faf001ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5592fe7d1520 con 0x5592fe876720 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faef77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe870fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45636/0 (socket says 172.21.15.28:45636) 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.280+0000 7faef6ffd640 1 -- 172.21.15.28:0/1446693032 learned_addr learned my addr 172.21.15.28:0/1446693032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faeeffff640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 0x7faee40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef67fc640 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe878bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef77fe640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 0x7faee40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 -- 172.21.15.28:0/1446693032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 msgr2=0x5592fe878bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe878bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 -- 172.21.15.28:0/1446693032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 msgr2=0x5592fe870fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe870fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 -- 172.21.15.28:0/1446693032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592fe5b17e0 con 0x5592fe8715b0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faeeffff640 1 -- 172.21.15.28:0/1264380547 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faee00cfbc0 con 0x7faef8102d90 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef77fe640 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe870fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7faef00114c0 tx=0x7faef0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319ea37f0585f5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faef77fe640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 0x7faee40a53e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faee8005e30 tx=0x7faee802c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faeed7fa640 1 -- 172.21.15.28:0/1446693032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0004020 con 0x5592fe8715b0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faeed7fa640 1 -- 172.21.15.28:0/1446693032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef0005500 con 0x5592fe8715b0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.281+0000 7faeed7fa640 1 -- 172.21.15.28:0/1446693032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef0004e10 con 0x5592fe8715b0 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 -- 172.21.15.28:0/1446693032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 msgr2=0x5592fe8760b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7faef00114c0 tx=0x7faef0011980 comp rx=0 tx=0).stop 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 -- 172.21.15.28:0/1446693032 shutdown_connections 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe878bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8760b0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 --2- 172.21.15.28:0/1446693032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe870fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 -- 172.21.15.28:0/1446693032 >> 172.21.15.28:0/1446693032 conn(0x5592fe86c130 msgr2=0x5592fe86c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 -- 172.21.15.28:0/1446693032 shutdown_connections 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.282+0000 7faf001ae740 1 -- 172.21.15.28:0/1446693032 wait complete. 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 -- start start 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 Processor -- start 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe882330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592fe8796d0 con 0x5592fe870bc0 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5592fe791110 con 0x5592fe8715b0 2024-03-22T02:24:46.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5592fe879850 con 0x5592fe876720 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faf001ae740 10 client.0 _ll_register_callbacks cb 0x5592fe7cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45642/0 (socket says 172.21.15.28:45642) 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 -- 172.21.15.28:0/2636228771 learned_addr learned my addr 172.21.15.28:0/2636228771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef77fe640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe882330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef67fc640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 -- 172.21.15.28:0/2636228771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 msgr2=0x5592fe882330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe882330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 -- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 msgr2=0x5592fe87b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.283+0000 7faef6ffd640 1 -- 172.21.15.28:0/2636228771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592fe611f70 con 0x5592fe870bc0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faef6ffd640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7faef0009530 tx=0x7faef0005140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725518871827e98b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef002f060 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faef0016040 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faef0005500 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7faef0036050 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 10 client.18747 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5592fe4a9170 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef0005500 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7faef000f6e0 con 0x5592fe870bc0 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 10 client.18747 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 10 client.18747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 20 client.18747 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8869a0 0x5592fe888e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 4 client.18747 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592fe7e2ab0 con 0x5592fe8869a0 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.284+0000 7faef67fc640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8869a0 0x5592fe888e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.285+0000 7faef67fc640 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8869a0 0x5592fe888e10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb04e95cf2d9cf17 server_cookie=ed5dc601197184e7 in_seq=0 out_seq=0 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.285+0000 7faede7fc640 10 client.18747 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faede7fc640 1 -- 172.21.15.28:0/2636228771 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5592fe7e2ab0 con 0x5592fe8869a0 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faede7fc640 10 client.18747 handle_command_reply: tid=0 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 client.18747 shutdown 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 msgr2=0x5592fe87adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7faef0009530 tx=0x7faef0005140 comp rx=0 tx=0).stop 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 shutdown_connections 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8869a0 0x5592fe888e10 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe882330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faede7fc640 0 client.18747 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 --2- 172.21.15.28:0/2636228771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87adb0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 >> 172.21.15.28:0/2636228771 conn(0x5592fe86c130 msgr2=0x5592fe86b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 shutdown_connections 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.286+0000 7faf001ae740 1 -- 172.21.15.28:0/2636228771 wait complete. 2024-03-22T02:24:46.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.287+0000 7faf001ae740 20 client.18747 trim_cache size 0 max 16384 2024-03-22T02:24:46.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 Processor -- start 2024-03-22T02:24:46.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 -- start start 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe871ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe8721e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faef81ab100 con 0x5592fe8715b0 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faef81a89d0 con 0x5592fe870bc0 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faf001ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faef81a6820 con 0x5592fe876720 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faef67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faef67fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45652/0 (socket says 172.21.15.28:45652) 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe871ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faef67fc640 1 -- 172.21.15.28:0/1865841555 learned_addr learned my addr 172.21.15.28:0/1865841555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.308+0000 7faef77fe640 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe8721e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 -- 172.21.15.28:0/1865841555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 msgr2=0x5592fe8721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe8721e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 -- 172.21.15.28:0/1865841555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 msgr2=0x5592fe871ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe871ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 -- 172.21.15.28:0/1865841555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592fe86cea0 con 0x5592fe8715b0 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe871ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faef67fc640 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7faee000cfd0 tx=0x7faee00bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bc5c25f59d81bd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faeecff9640 1 -- 172.21.15.28:0/1865841555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00af070 con 0x5592fe8715b0 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faeecff9640 1 -- 172.21.15.28:0/1865841555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee009c5e0 con 0x5592fe8715b0 2024-03-22T02:24:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.309+0000 7faeecff9640 1 -- 172.21.15.28:0/1865841555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0099f80 con 0x5592fe8715b0 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- 172.21.15.28:0/1865841555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 msgr2=0x5592fe8822f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7faee000cfd0 tx=0x7faee00bc040 comp rx=0 tx=0).stop 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- 172.21.15.28:0/1865841555 shutdown_connections 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe8721e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe8715b0 0x5592fe8822f0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 --2- 172.21.15.28:0/1865841555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe870bc0 0x5592fe871ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- 172.21.15.28:0/1865841555 >> 172.21.15.28:0/1865841555 conn(0x5592fe86c130 msgr2=0x5592fe86b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- 172.21.15.28:0/1865841555 shutdown_connections 2024-03-22T02:24:46.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- 172.21.15.28:0/1865841555 wait complete. 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 -- start start 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.310+0000 7faf001ae740 1 Processor -- start 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe880c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faef81a6e90 con 0x5592fe870bc0 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faef81a8300 con 0x5592fe8715b0 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faef81a7560 con 0x5592fe876720 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 10 client.0 _ll_register_callbacks cb 0x7faef0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45666/0 (socket says 172.21.15.28:45666) 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 -- 172.21.15.28:0/1711597895 learned_addr learned my addr 172.21.15.28:0/1711597895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef77fe640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe880c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef67fc640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 -- 172.21.15.28:0/1711597895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 msgr2=0x5592fe880c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe880c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 -- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 msgr2=0x5592fe87fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 -- 172.21.15.28:0/1711597895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592fe8701d0 con 0x5592fe870bc0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faef6ffd640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7faef0009250 tx=0x7faef000ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f58eb837060f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faede7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef002f070 con 0x5592fe870bc0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faef000f7c0 con 0x5592fe870bc0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faee009c5e0 con 0x5592fe870bc0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7faef0043050 con 0x5592fe870bc0 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.311+0000 7faf001ae740 10 client.18753 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:46.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5592fe8823b0 con 0x5592fe870bc0 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee009c5e0 con 0x5592fe870bc0 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7faef0021920 con 0x5592fe870bc0 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 10 client.18753 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 10 client.18753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 20 client.18753 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8ed230 0x5592fe8ef650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 4 client.18753 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5592fe7dce00 con 0x5592fe8ed230 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.312+0000 7faef67fc640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8ed230 0x5592fe8ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.313+0000 7faef67fc640 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8ed230 0x5592fe8ef650 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=245ec157005410c1 server_cookie=2789aaa1601c6164 in_seq=0 out_seq=0 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.313+0000 7faede7fc640 10 client.18753 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faede7fc640 1 -- 172.21.15.28:0/1711597895 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5592fe7dce00 con 0x5592fe8ed230 2024-03-22T02:24:46.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faede7fc640 10 client.18753 handle_command_reply: tid=0 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faf001ae740 1 client.18753 shutdown 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 msgr2=0x5592fe87f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7faef0009250 tx=0x7faef000ffd0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 shutdown_connections 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5592fe8ed230 0x5592fe8ef650 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.314+0000 7faede7fc640 0 client.18753 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5592fe876720 0x5592fe880c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5592fe8715b0 0x5592fe87fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 --2- 172.21.15.28:0/1711597895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592fe870bc0 0x5592fe87f750 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 >> 172.21.15.28:0/1711597895 conn(0x5592fe86c130 msgr2=0x5592fe87a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 shutdown_connections 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 1 -- 172.21.15.28:0/1711597895 wait complete. 2024-03-22T02:24:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faf001ae740 20 client.18753 trim_cache size 0 max 16384 2024-03-22T02:24:46.321 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:46.321 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:24:46.321 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "3b7ac9f4-4a7b-41c1-bc1e-563018db0bd3", 2024-03-22T02:24:46.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:24:46.321 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 msgr2=0x7faee40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 0x7faee40a53e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faee8005e30 tx=0x7faee802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 msgr2=0x7faef81983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7faee000e9f0 tx=0x7faee000edf0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 shutdown_connections 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faee40a2f20 0x7faee40a53e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faef8103750 0x7faef819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef8102d90 0x7faef81983d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 --2- 172.21.15.28:0/1264380547 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faef8101ba0 0x7faef8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 >> 172.21.15.28:0/1264380547 conn(0x7faef80fd0f0 msgr2=0x7faef8106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 shutdown_connections 2024-03-22T02:24:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.315+0000 7faefd4eb640 1 -- 172.21.15.28:0/1264380547 wait complete. 2024-03-22T02:24:46.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:24:46.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:24:46.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1131952 11.33 MB/sec execute 448 sec latency 27.257 ms 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 -- 172.21.15.28:0/3836814423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0107870 msgr2=0x7fafa0109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 --2- 172.21.15.28:0/3836814423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0107870 0x7fafa0109cc0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7faf98009ca0 tx=0x7faf9802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 -- 172.21.15.28:0/3836814423 shutdown_connections 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 --2- 172.21.15.28:0/3836814423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0107870 0x7fafa0109cc0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 --2- 172.21.15.28:0/3836814423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 0x7fafa0107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 --2- 172.21.15.28:0/3836814423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0102610 0x7fafa0104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 -- 172.21.15.28:0/3836814423 >> 172.21.15.28:0/3836814423 conn(0x7fafa00fc020 msgr2=0x7fafa00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 -- 172.21.15.28:0/3836814423 shutdown_connections 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.480+0000 7fafa5356640 1 -- 172.21.15.28:0/3836814423 wait complete. 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 Processor -- start 2024-03-22T02:24:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 -- start start 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 0x7fafa01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0107870 0x7fafa019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010d810 con 0x7fafa0102610 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010d690 con 0x7fafa0107870 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7fafa5356640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fafa010d990 con 0x7fafa0104f40 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45682/0 (socket says 172.21.15.28:45682) 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7faf9effd640 1 -- 172.21.15.28:0/1880133530 learned_addr learned my addr 172.21.15.28:0/1880133530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7faf967fc640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 0x7fafa01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.481+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0107870 0x7fafa019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 -- 172.21.15.28:0/1880133530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 msgr2=0x7fafa01983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 0x7fafa01983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 -- 172.21.15.28:0/1880133530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0107870 msgr2=0x7fafa019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0107870 0x7fafa019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 -- 172.21.15.28:0/1880133530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa019fad0 con 0x7fafa0102610 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9effd640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7faf9000e9f0 tx=0x7faf9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9cff9640 1 -- 172.21.15.28:0/1880133530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9000cd80 con 0x7fafa0102610 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9cff9640 1 -- 172.21.15.28:0/1880133530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf90010910 con 0x7fafa0102610 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa019fdc0 con 0x7fafa0102610 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7faf9cff9640 1 -- 172.21.15.28:0/1880133530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf90018990 con 0x7fafa0102610 2024-03-22T02:24:46.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.482+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa01a0230 con 0x7fafa0102610 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.484+0000 7faf9cff9640 1 -- 172.21.15.28:0/1880133530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faf90010ac0 con 0x7fafa0102610 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7faf9cff9640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 0x7faf8c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 Processor -- start 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7faf967fc640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 0x7faf8c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 -- start start 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7faf9cff9640 1 -- 172.21.15.28:0/1880133530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf900cfaf0 con 0x7fafa0102610 2024-03-22T02:24:46.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30e20 0x559238b35920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b38420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559238a958c0 con 0x559238b30430 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559238a3dc20 con 0x559238b30e20 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7fafa8019740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559238a91870 con 0x559238b35f90 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.485+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b38420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf967fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30e20 0x559238b35920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45690/0 (socket says 172.21.15.28:45690) 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9effd640 1 -- 172.21.15.28:0/2718310 learned_addr learned my addr 172.21.15.28:0/2718310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf967fc640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 0x7faf8c0a53e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fafa01993c0 tx=0x7faf88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 -- 172.21.15.28:0/2718310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30e20 msgr2=0x559238b35920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30e20 0x559238b35920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 -- 172.21.15.28:0/2718310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 msgr2=0x559238b30850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 -- 172.21.15.28:0/2718310 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559238a62150 con 0x559238b35f90 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9effd640 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b38420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf98037040 tx=0x7faf980323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77db3bbbd4f23b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf957fa640 1 -- 172.21.15.28:0/2718310 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98043070 con 0x559238b35f90 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf957fa640 1 -- 172.21.15.28:0/2718310 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf9802cca0 con 0x559238b35f90 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.486+0000 7faf957fa640 1 -- 172.21.15.28:0/2718310 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98039040 con 0x559238b35f90 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- 172.21.15.28:0/2718310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 msgr2=0x559238b38420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b38420 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf98037040 tx=0x7faf980323d0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- 172.21.15.28:0/2718310 shutdown_connections 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b38420 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30e20 0x559238b35920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 --2- 172.21.15.28:0/2718310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b30850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- 172.21.15.28:0/2718310 >> 172.21.15.28:0/2718310 conn(0x559238b2b9a0 msgr2=0x559238b2bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- 172.21.15.28:0/2718310 shutdown_connections 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- 172.21.15.28:0/2718310 wait complete. 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.487+0000 7fafa8019740 1 -- start start 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 Processor -- start 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 0x559238b3a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b41dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559238b38ee0 con 0x559238b30e20 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559238a51130 con 0x559238b35f90 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559238b39060 con 0x559238b30430 2024-03-22T02:24:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7fafa8019740 10 client.0 _ll_register_callbacks cb 0x559238a50b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 0x559238b3a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45692/0 (socket says 172.21.15.28:45692) 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf967fc640 1 -- 172.21.15.28:0/3705159564 learned_addr learned my addr 172.21.15.28:0/3705159564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b41dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 -- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 msgr2=0x559238b41dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b41dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 -- 172.21.15.28:0/3705159564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 msgr2=0x559238b3ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 -- 172.21.15.28:0/3705159564 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592388737d0 con 0x559238b30430 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf967fc640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf9effd640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 0x559238b3a8c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faf9009be50 tx=0x7faf9009b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c391b45197672d27 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.488+0000 7faf867fc640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf900bad80 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf90018b30 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf9802cca0 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7faf90018d30 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 10 client.28579 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5592387a93f0 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf9802cca0 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7faf90097d30 con 0x559238b30430 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 10 client.28579 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 10 client.28579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 20 client.28579 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238b46750 0x559238b48be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 4 client.28579 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559238aa2e00 con 0x559238b46750 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.489+0000 7faf967fc640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238b46750 0x559238b48be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.490+0000 7faf967fc640 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238b46750 0x559238b48be0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eca2633dffbd2ec9 server_cookie=da04777ace71eaad in_seq=0 out_seq=0 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.490+0000 7faf867fc640 10 client.28579 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.490+0000 7faf867fc640 1 -- 172.21.15.28:0/3705159564 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559238aa2e00 con 0x559238b46750 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.490+0000 7faf867fc640 10 client.28579 handle_command_reply: tid=0 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.490+0000 7fafa8019740 1 client.28579 shutdown 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 msgr2=0x559238b3a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 0x559238b3a8c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faf9009be50 tx=0x7faf9009b910 comp rx=0 tx=0).stop 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 shutdown_connections 2024-03-22T02:24:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238b46750 0x559238b48be0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b41dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7faf867fc640 0 client.28579 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 --2- 172.21.15.28:0/3705159564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30430 0x559238b3a8c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 >> 172.21.15.28:0/3705159564 conn(0x559238b2b9a0 msgr2=0x559238b2ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 shutdown_connections 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 1 -- 172.21.15.28:0/3705159564 wait complete. 2024-03-22T02:24:46.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.491+0000 7fafa8019740 20 client.28579 trim_cache size 0 max 16384 2024-03-22T02:24:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 Processor -- start 2024-03-22T02:24:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 -- start start 2024-03-22T02:24:46.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 0x559238b41d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01ad300 con 0x559238b30430 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01aabd0 con 0x559238b35f90 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7fafa8019740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01a8a20 con 0x559238b30e20 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf967fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 0x559238b41d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf9f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf967fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 0x559238b41d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49732/0 (socket says 172.21.15.28:49732) 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf9effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45698/0 (socket says 172.21.15.28:45698) 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.510+0000 7faf967fc640 1 -- 172.21.15.28:0/3974974163 learned_addr learned my addr 172.21.15.28:0/3974974163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 -- 172.21.15.28:0/3974974163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 msgr2=0x559238b41d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 0x559238b41d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 -- 172.21.15.28:0/3974974163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 msgr2=0x559238b3abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 -- 172.21.15.28:0/3974974163 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592387690f0 con 0x559238b35f90 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9effd640 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b2a370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faf98006010 tx=0x7faf9802c770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4971eefe88635bf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf94ff9640 1 -- 172.21.15.28:0/3974974163 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf9804c020 con 0x559238b35f90 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf94ff9640 1 -- 172.21.15.28:0/3974974163 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf98003970 con 0x559238b35f90 2024-03-22T02:24:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.511+0000 7faf94ff9640 1 -- 172.21.15.28:0/3974974163 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98039070 con 0x559238b35f90 2024-03-22T02:24:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- 172.21.15.28:0/3974974163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 msgr2=0x559238b2a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b2a370 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faf98006010 tx=0x7faf9802c770 comp rx=0 tx=0).stop 2024-03-22T02:24:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- 172.21.15.28:0/3974974163 shutdown_connections 2024-03-22T02:24:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b35f90 0x559238b2a370 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b30e20 0x559238b41d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- 172.21.15.28:0/3974974163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30430 0x559238b3abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- 172.21.15.28:0/3974974163 >> 172.21.15.28:0/3974974163 conn(0x559238b2b9a0 msgr2=0x559238b3a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- 172.21.15.28:0/3974974163 shutdown_connections 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- 172.21.15.28:0/3974974163 wait complete. 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 -- start start 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 Processor -- start 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30430 0x559238b3f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.512+0000 7fafa8019740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b40a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01a9090 con 0x559238b30e20 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01aa500 con 0x559238b30430 2024-03-22T02:24:46.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fafa01a9760 con 0x559238b35f90 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 10 client.0 _ll_register_callbacks cb 0x7faf900bc0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf9effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30430 0x559238b3f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45704/0 (socket says 172.21.15.28:45704) 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 -- 172.21.15.28:0/1753351024 learned_addr learned my addr 172.21.15.28:0/1753351024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf9f7fe640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b40a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 -- 172.21.15.28:0/1753351024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 msgr2=0x559238b40a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b40a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 -- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30430 msgr2=0x559238b3f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30430 0x559238b3f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 -- 172.21.15.28:0/1753351024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559238b41030 con 0x559238b30e20 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf967fc640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x559238b49450 tx=0x7faf88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458a8d828077fd27 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:46.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf867fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8802c6e0 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf98003970 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7faf88005020 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7faf8803c050 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 10 client.18771 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.513+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x559238bacc70 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf98003970 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7faf88028410 con 0x559238b30e20 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 10 client.18771 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 10 client.18771 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 20 client.18771 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238bad680 0x559238bafaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 4 client.18771 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559238a9d150 con 0x559238bad680 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7faf9effd640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238bad680 0x559238bafaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7faf867fc640 10 client.18771 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.514+0000 7faf9effd640 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238bad680 0x559238bafaa0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7133081d55e92d63 server_cookie=1f4f7deaba6a08b3 in_seq=0 out_seq=0 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.515+0000 7faf867fc640 1 -- 172.21.15.28:0/1753351024 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559238a9d150 con 0x559238bad680 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.515+0000 7faf867fc640 10 client.18771 handle_command_reply: tid=0 2024-03-22T02:24:46.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.515+0000 7fafa8019740 1 client.18771 shutdown 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 msgr2=0x559238b3fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x559238b49450 tx=0x7faf88004730 comp rx=0 tx=0).stop 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 shutdown_connections 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559238bad680 0x559238bafaa0 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559238b35f90 0x559238b40a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559238b30e20 0x559238b3fa80 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 --2- 172.21.15.28:0/1753351024 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559238b30430 0x559238b3f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 >> 172.21.15.28:0/1753351024 conn(0x559238b2b9a0 msgr2=0x559238b2a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7faf867fc640 0 client.18771 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 shutdown_connections 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 1 -- 172.21.15.28:0/1753351024 wait complete. 2024-03-22T02:24:46.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa8019740 20 client.18771 trim_cache size 0 max 16384 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "3b7ac9f4-4a7b-41c1-bc1e-563018db0bd3": { 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "3b7ac9f4-4a7b-41c1-bc1e-563018db0bd3", 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:24:46.524 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:24:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 msgr2=0x7faf8c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 0x7faf8c0a53e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fafa01993c0 tx=0x7faf88005f90 comp rx=0 tx=0).stop 2024-03-22T02:24:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 msgr2=0x7fafa0197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7faf9000e9f0 tx=0x7faf9000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 shutdown_connections 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf8c0a2f20 0x7faf8c0a53e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fafa0107870 0x7fafa019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fafa0104f40 0x7fafa01983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 --2- 172.21.15.28:0/1880133530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa0102610 0x7fafa0197eb0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 >> 172.21.15.28:0/1880133530 conn(0x7fafa00fc020 msgr2=0x7fafa0102fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.516+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 shutdown_connections 2024-03-22T02:24:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:24:46.517+0000 7fafa5356640 1 -- 172.21.15.28:0/1880133530 wait complete. 2024-03-22T02:24:46.528 INFO:tasks.cephfs.filesystem:scrub status for tag:3b7ac9f4-4a7b-41c1-bc1e-563018db0bd3 - {'path': '/', 'tag': '3b7ac9f4-4a7b-41c1-bc1e-563018db0bd3', 'options': 'recursive,force'} 2024-03-22T02:24:47.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:46 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:46 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:24:47.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:24:47.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:24:47.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:24:47.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:46 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:24:47.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1133787 11.33 MB/sec execute 449 sec latency 30.337 ms 2024-03-22T02:24:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:47 smithi120 ceph-mon[36803]: pgmap v696: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-03-22T02:24:48.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:47 smithi083 ceph-mon[36340]: pgmap v696: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-03-22T02:24:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:47 smithi028 ceph-mon[31916]: pgmap v696: 129 pgs: 129 active+clean; 57 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-03-22T02:24:48.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1135693 11.33 MB/sec execute 450 sec latency 72.285 ms 2024-03-22T02:24:49.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1137463 11.33 MB/sec execute 451 sec latency 28.819 ms 2024-03-22T02:24:50.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:49 smithi083 ceph-mon[36340]: pgmap v697: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 418 op/s 2024-03-22T02:24:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:49 smithi120 ceph-mon[36803]: pgmap v697: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 418 op/s 2024-03-22T02:24:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:49 smithi028 ceph-mon[31916]: pgmap v697: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.2 MiB/s wr, 418 op/s 2024-03-22T02:24:50.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1140288 11.34 MB/sec execute 452 sec latency 21.692 ms 2024-03-22T02:24:51.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1142498 11.33 MB/sec execute 453 sec latency 29.002 ms 2024-03-22T02:24:52.123 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:51 smithi028 ceph-mon[31916]: pgmap v698: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:24:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:51 smithi120 ceph-mon[36803]: pgmap v698: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:24:52.264 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:51 smithi083 ceph-mon[36340]: pgmap v698: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 402 op/s 2024-03-22T02:24:52.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1144213 11.33 MB/sec execute 454 sec latency 18.848 ms 2024-03-22T02:24:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 -- 172.21.15.28:0/1619990730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618100460 msgr2=0x7f4618102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 --2- 172.21.15.28:0/1619990730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618100460 0x7f4618102890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f460c009900 tx=0x7f460c02bc50 comp rx=0 tx=0).stop 2024-03-22T02:24:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 -- 172.21.15.28:0/1619990730 shutdown_connections 2024-03-22T02:24:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 --2- 172.21.15.28:0/1619990730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618105780 0x7f4618105c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 --2- 172.21.15.28:0/1619990730 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618102dd0 0x7f4618105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 --2- 172.21.15.28:0/1619990730 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618100460 0x7f4618102890 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.805+0000 7f461ce6f640 1 -- 172.21.15.28:0/1619990730 >> 172.21.15.28:0/1619990730 conn(0x7f46180fc060 msgr2=0x7f46180fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.806+0000 7f461ce6f640 1 -- 172.21.15.28:0/1619990730 shutdown_connections 2024-03-22T02:24:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.806+0000 7f461ce6f640 1 -- 172.21.15.28:0/1619990730 wait complete. 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.807+0000 7f461ce6f640 1 Processor -- start 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.807+0000 7f461ce6f640 1 -- start start 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618100460 0x7f4618110490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 0x7f461810eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46181074d0 con 0x7f4618102dd0 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4618107350 con 0x7f4618100460 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461ce6f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4618107650 con 0x7f4618105780 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f4615d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f4615d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45672/0 (socket says 172.21.15.28:45672) 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f4615d9b640 1 -- 172.21.15.28:0/4163034195 learned_addr learned my addr 172.21.15.28:0/4163034195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f461659c640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618100460 0x7f4618110490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.808+0000 7f4616d9d640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 0x7f461810eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 -- 172.21.15.28:0/4163034195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 msgr2=0x7f461810eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 0x7f461810eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 -- 172.21.15.28:0/4163034195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618100460 msgr2=0x7f4618110490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618100460 0x7f4618110490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 -- 172.21.15.28:0/4163034195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461810f1b0 con 0x7f4618102dd0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4616d9d640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 0x7f461810eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f4615d9b640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f461000ba50 tx=0x7f461000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f46077fe640 1 -- 172.21.15.28:0/4163034195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610019070 con 0x7f4618102dd0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461810f4a0 con 0x7f4618102dd0 2024-03-22T02:24:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.809+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4618119580 con 0x7f4618102dd0 2024-03-22T02:24:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.811+0000 7f46077fe640 1 -- 172.21.15.28:0/4163034195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f461000cc10 con 0x7f4618102dd0 2024-03-22T02:24:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.811+0000 7f46077fe640 1 -- 172.21.15.28:0/4163034195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461001d5f0 con 0x7f4618102dd0 2024-03-22T02:24:52.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.813+0000 7f46077fe640 1 -- 172.21.15.28:0/4163034195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f461001c070 con 0x7f4618102dd0 2024-03-22T02:24:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 Processor -- start 2024-03-22T02:24:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 -- start start 2024-03-22T02:24:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576ee89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee9040 0x55e576eeb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f4616d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f4616d9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34712/0 (socket says 172.21.15.28:34712) 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f4616d9d640 1 -- 172.21.15.28:0/1322680981 learned_addr learned my addr 172.21.15.28:0/1322680981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e576e48040 con 0x55e576ee3ed0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.814+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e576df09e0 con 0x55e576ee34e0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576ee89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e576e43ff0 con 0x55e576ee9040 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f46077fe640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 0x7f46000a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f4615d9b640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee9040 0x55e576eeb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f4616d9d640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 0x7f46000a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f46077fe640 1 -- 172.21.15.28:0/4163034195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f46100cfe70 con 0x7f4618102dd0 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 -- 172.21.15.28:0/1322680981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee9040 msgr2=0x55e576eeb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee9040 0x55e576eeb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 -- 172.21.15.28:0/1322680981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 msgr2=0x55e576ee3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 -- 172.21.15.28:0/1322680981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e576b5c210 con 0x55e576ee3ed0 2024-03-22T02:24:52.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f4616d9d640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.815+0000 7f461659c640 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576ee89d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f460c036040 tx=0x7f460c004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785e779422815224 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f4616d9d640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 0x7f46000a55d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4608006fd0 tx=0x7f46080046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f4604ff9640 1 -- 172.21.15.28:0/1322680981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c042070 con 0x55e576ee3ed0 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f4604ff9640 1 -- 172.21.15.28:0/1322680981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460c02cc60 con 0x55e576ee3ed0 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f4604ff9640 1 -- 172.21.15.28:0/1322680981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c005260 con 0x55e576ee3ed0 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 msgr2=0x55e576ee89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.816+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576ee89d0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f460c036040 tx=0x7f460c004f10 comp rx=0 tx=0).stop 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 shutdown_connections 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee9040 0x55e576eeb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576ee89d0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1322680981 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee34e0 0x55e576ee3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 >> 172.21.15.28:0/1322680981 conn(0x55e576edea50 msgr2=0x55e576edee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 shutdown_connections 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 -- 172.21.15.28:0/1322680981 wait complete. 2024-03-22T02:24:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.817+0000 7f461fa4c740 1 -- start start 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 Processor -- start 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee3ed0 0x55e576eeda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576ef4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e576eec030 con 0x55e576ee34e0 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e576e03db0 con 0x55e576ee9040 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e576eec1b0 con 0x55e576ee3ed0 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45688/0 (socket says 172.21.15.28:45688) 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 -- 172.21.15.28:0/2692288234 learned_addr learned my addr 172.21.15.28:0/2692288234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f4615d9b640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee3ed0 0x55e576eeda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f4616d9d640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576ef4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 10 client.0 _ll_register_callbacks cb 0x55e576b5ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 -- 172.21.15.28:0/2692288234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee3ed0 msgr2=0x55e576eeda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee3ed0 0x55e576eeda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 -- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 msgr2=0x55e576ef4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576ef4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461659c640 1 -- 172.21.15.28:0/2692288234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e576c28620 con 0x55e576ee34e0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f4616d9d640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576ef4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.818+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f460c02cc60 con 0x55e576ee34e0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461659c640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f460c009850 tx=0x7f460c005680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7892044992a20ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f45fdffb640 10 client.18780 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f45fdffb640 1 -- 172.21.15.28:0/2692288234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c04b070 con 0x55e576ee34e0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f45fdffb640 1 -- 172.21.15.28:0/2692288234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f460c050460 con 0x55e576ee34e0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f45fdffb640 1 -- 172.21.15.28:0/2692288234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f460c042040 con 0x55e576ee34e0 2024-03-22T02:24:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 10 client.18780 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=37}) -- 0x55e576ef7cd0 con 0x55e576ee34e0 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f45fdffb640 1 -- 172.21.15.28:0/2692288234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f460c050660 con 0x55e576ee34e0 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 10 client.18780 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 10 client.18780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 20 client.18780 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576ef9570 0x55e576efba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 4 client.18780 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.819+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e576e55580 con 0x55e576ef9570 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.820+0000 7f4615d9b640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576ef9570 0x55e576efba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.820+0000 7f4615d9b640 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576ef9570 0x55e576efba30 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87b3d0483dc4eef server_cookie=72232e4a7edd3e5b in_seq=0 out_seq=0 2024-03-22T02:24:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.820+0000 7f45fdffb640 10 client.18780 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.821+0000 7f45fdffb640 1 -- 172.21.15.28:0/2692288234 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e576e55580 con 0x55e576ef9570 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.821+0000 7f45fdffb640 10 client.18780 handle_command_reply: tid=0 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.821+0000 7f461fa4c740 1 client.18780 shutdown 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.821+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 msgr2=0x55e576eed4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.821+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f460c009850 tx=0x7f460c005680 comp rx=0 tx=0).stop 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 shutdown_connections 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576ef9570 0x55e576efba30 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576ef4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee3ed0 0x55e576eeda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2692288234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee34e0 0x55e576eed4c0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 >> 172.21.15.28:0/2692288234 conn(0x55e576edea50 msgr2=0x55e576eddc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f45fdffb640 0 client.18780 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 shutdown_connections 2024-03-22T02:24:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 1 -- 172.21.15.28:0/2692288234 wait complete. 2024-03-22T02:24:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.822+0000 7f461fa4c740 20 client.18780 trim_cache size 0 max 16384 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 Processor -- start 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 -- start start 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576ee23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee3ed0 0x55e576eecae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45f8012680 con 0x55e576ee9040 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45f800ff50 con 0x55e576ee3ed0 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461fa4c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45f800dda0 con 0x55e576ee34e0 2024-03-22T02:24:52.842 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4615d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee3ed0 0x55e576eecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45698/0 (socket says 172.21.15.28:45698) 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 -- 172.21.15.28:0/2537914367 learned_addr learned my addr 172.21.15.28:0/2537914367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f461659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576ee23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 -- 172.21.15.28:0/2537914367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 msgr2=0x55e576ee23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576ee23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 -- 172.21.15.28:0/2537914367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee3ed0 msgr2=0x55e576eecae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee3ed0 0x55e576eecae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 -- 172.21.15.28:0/2537914367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e576ee5770 con 0x55e576ee9040 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.837+0000 7f4616d9d640 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4608007140 tx=0x7f4608009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d25dedf4e5ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f45fffff640 1 -- 172.21.15.28:0/2537914367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460800f040 con 0x55e576ee9040 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f45fffff640 1 -- 172.21.15.28:0/2537914367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4608028070 con 0x55e576ee9040 2024-03-22T02:24:52.843 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f45fffff640 1 -- 172.21.15.28:0/2537914367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46080163a0 con 0x55e576ee9040 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 -- 172.21.15.28:0/2537914367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 msgr2=0x55e576eed020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4608007140 tx=0x7f4608009000 comp rx=0 tx=0).stop 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 -- 172.21.15.28:0/2537914367 shutdown_connections 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee9040 0x55e576eed020 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee3ed0 0x55e576eecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 --2- 172.21.15.28:0/2537914367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576ee23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 -- 172.21.15.28:0/2537914367 >> 172.21.15.28:0/2537914367 conn(0x55e576edea50 msgr2=0x55e576ee1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 -- 172.21.15.28:0/2537914367 shutdown_connections 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.838+0000 7f461fa4c740 1 -- 172.21.15.28:0/2537914367 wait complete. 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 -- start start 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 Processor -- start 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576f00880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576f5d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f4615d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45f800e410 con 0x55e576ee3ed0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f4615d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45706/0 (socket says 172.21.15.28:45706) 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f4615d9b640 1 -- 172.21.15.28:0/1436194671 learned_addr learned my addr 172.21.15.28:0/1436194671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576f00880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f45f800f880 con 0x55e576ee9040 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f45f800eae0 con 0x55e576ee34e0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f4616d9d640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576f5d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461fa4c740 10 client.0 _ll_register_callbacks cb 0x7f4608007270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 -- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 msgr2=0x55e576f5d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576f5d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 -- 172.21.15.28:0/1436194671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 msgr2=0x55e576f00dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:24:52.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f461659c640 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e576f5dfb0 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.839+0000 7f4615d9b640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461659c640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576f00880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e576edfa70 tx=0x7f460c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5570edad7c7d80a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c04b040 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f460c0048d0 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4608028070 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f460c004af0 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 10 client.28603 fetch_fsmap learned FSMap version 37 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e576f5f280 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f460c02cc60 con 0x55e576ee34e0 2024-03-22T02:24:52.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f460c047ad0 con 0x55e576ee34e0 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 10 client.28603 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 10 client.28603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.840+0000 7f461fa4c740 20 client.28603 populate_metadata read hostname 'smithi028' 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576f5fc40 0x55e576f62060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f461fa4c740 4 client.28603 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f4608028070 con 0x55e576f5fc40 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f4615d9b640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576f5fc40 0x55e576f62060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f4615d9b640 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576f5fc40 0x55e576f62060 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a9e8f9ed564e92 server_cookie=298293cc5160cad8 in_seq=0 out_seq=0 2024-03-22T02:24:52.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:52.841+0000 7f45fdffb640 10 client.28603 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f45fdffb640 1 -- 172.21.15.28:0/1436194671 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f4608028070 con 0x55e576f5fc40 2024-03-22T02:24:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f45fdffb640 10 client.28603 handle_command_reply: tid=0 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f461fa4c740 1 client.28603 shutdown 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 msgr2=0x55e576f00880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576f00880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x55e576edfa70 tx=0x7f460c000bc0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 shutdown_connections 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e576f5fc40 0x55e576f62060 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.251+0000 7f45fdffb640 0 client.28603 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e576ee9040 0x55e576f5d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e576ee3ed0 0x55e576f00dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 --2- 172.21.15.28:0/1436194671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e576ee34e0 0x55e576f00880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 >> 172.21.15.28:0/1436194671 conn(0x55e576edea50 msgr2=0x55e576ee2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 shutdown_connections 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 1 -- 172.21.15.28:0/1436194671 wait complete. 2024-03-22T02:24:53.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.252+0000 7f461fa4c740 20 client.28603 trim_cache size 0 max 16384 2024-03-22T02:24:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:24:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:24:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:24:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 msgr2=0x7f46000a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 0x7f46000a55d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f4608006fd0 tx=0x7f46080046c0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 msgr2=0x7f46181109d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f461000ba50 tx=0x7f461000bf10 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 shutdown_connections 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.253+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f46000a3110 0x7f46000a55d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4618105780 0x7f461810eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4618102dd0 0x7f46181109d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 --2- 172.21.15.28:0/4163034195 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4618100460 0x7f4618110490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 >> 172.21.15.28:0/4163034195 conn(0x7f46180fc060 msgr2=0x7f46180fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:24:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 shutdown_connections 2024-03-22T02:24:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:24:53.254+0000 7f461ce6f640 1 -- 172.21.15.28:0/4163034195 wait complete. 2024-03-22T02:24:53.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1144898 11.31 MB/sec execute 455 sec latency 304.146 ms 2024-03-22T02:24:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:53 smithi120 ceph-mon[36803]: pgmap v699: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-03-22T02:24:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:53 smithi083 ceph-mon[36340]: pgmap v699: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-03-22T02:24:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:53 smithi028 ceph-mon[31916]: pgmap v699: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.5 MiB/s wr, 347 op/s 2024-03-22T02:24:54.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1147367 11.32 MB/sec execute 456 sec latency 69.924 ms 2024-03-22T02:24:55.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:54 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:24:55.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:54 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:24:55.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:54 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:24:55.433 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1148901 11.31 MB/sec execute 457 sec latency 34.260 ms 2024-03-22T02:24:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:55 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:24:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:55 smithi120 ceph-mon[36803]: pgmap v700: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 452 op/s 2024-03-22T02:24:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:55 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:24:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:55 smithi083 ceph-mon[36340]: pgmap v700: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 452 op/s 2024-03-22T02:24:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:55 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:24:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:55 smithi028 ceph-mon[31916]: pgmap v700: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 452 op/s 2024-03-22T02:24:56.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1151301 11.32 MB/sec execute 458 sec latency 8.331 ms 2024-03-22T02:24:57.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1153260 11.31 MB/sec execute 459 sec latency 32.803 ms 2024-03-22T02:24:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:57 smithi120 ceph-mon[36803]: pgmap v701: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 332 op/s 2024-03-22T02:24:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:57 smithi083 ceph-mon[36340]: pgmap v701: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 332 op/s 2024-03-22T02:24:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:57 smithi028 ceph-mon[31916]: pgmap v701: 129 pgs: 129 active+clean; 62 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.3 MiB/s wr, 332 op/s 2024-03-22T02:24:58.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1154853 11.31 MB/sec execute 460 sec latency 45.459 ms 2024-03-22T02:24:59.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1157406 11.31 MB/sec execute 461 sec latency 28.690 ms 2024-03-22T02:25:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:24:59 smithi120 ceph-mon[36803]: pgmap v702: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-22T02:25:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:24:59 smithi083 ceph-mon[36340]: pgmap v702: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-22T02:25:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:24:59 smithi028 ceph-mon[31916]: pgmap v702: 129 pgs: 129 active+clean; 58 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 427 op/s 2024-03-22T02:25:00.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1158985 11.31 MB/sec execute 462 sec latency 52.769 ms 2024-03-22T02:25:01.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1161720 11.32 MB/sec execute 463 sec latency 16.306 ms 2024-03-22T02:25:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:01 smithi120 ceph-mon[36803]: pgmap v703: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-03-22T02:25:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:01 smithi083 ceph-mon[36340]: pgmap v703: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-03-22T02:25:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:01 smithi028 ceph-mon[31916]: pgmap v703: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-03-22T02:25:02.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1163843 11.31 MB/sec execute 464 sec latency 62.910 ms 2024-03-22T02:25:03.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1165484 11.31 MB/sec execute 465 sec latency 77.350 ms 2024-03-22T02:25:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:03 smithi120 ceph-mon[36803]: pgmap v704: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:25:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:03 smithi083 ceph-mon[36340]: pgmap v704: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:25:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:03 smithi028 ceph-mon[31916]: pgmap v704: 129 pgs: 129 active+clean; 59 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 342 op/s 2024-03-22T02:25:04.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1167940 11.32 MB/sec execute 466 sec latency 29.504 ms 2024-03-22T02:25:05.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1169557 11.31 MB/sec execute 467 sec latency 34.606 ms 2024-03-22T02:25:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:05 smithi120 ceph-mon[36803]: pgmap v705: 129 pgs: 129 active+clean; 67 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-03-22T02:25:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:05 smithi083 ceph-mon[36340]: pgmap v705: 129 pgs: 129 active+clean; 67 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-03-22T02:25:06.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:05 smithi028 ceph-mon[31916]: pgmap v705: 129 pgs: 129 active+clean; 67 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.3 MiB/s wr, 401 op/s 2024-03-22T02:25:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- 172.21.15.28:0/1605240542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4102660 msgr2=0x7fb2d4108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- 172.21.15.28:0/1605240542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4102660 0x7fb2d4108ef0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009ca0 tx=0x7fb2cc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- 172.21.15.28:0/1605240542 shutdown_connections 2024-03-22T02:25:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- 172.21.15.28:0/1605240542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4102660 0x7fb2d4108ef0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- 172.21.15.28:0/1605240542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4101ca0 0x7fb2d4102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- 172.21.15.28:0/1605240542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- 172.21.15.28:0/1605240542 >> 172.21.15.28:0/1605240542 conn(0x7fb2d40fc020 msgr2=0x7fb2d40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- 172.21.15.28:0/1605240542 shutdown_connections 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- 172.21.15.28:0/1605240542 wait complete. 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 Processor -- start 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- start start 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4101ca0 0x7fb2d4198310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4102660 0x7fb2d419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d410d810 con 0x7fb2d4100ab0 2024-03-22T02:25:06.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d410d690 con 0x7fb2d4102660 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.345+0000 7fb2dad63640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d410d990 con 0x7fb2d4101ca0 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59152/0 (socket says 172.21.15.28:59152) 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 -- 172.21.15.28:0/917467557 learned_addr learned my addr 172.21.15.28:0/917467557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4101ca0 0x7fb2d4198310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d9300640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4102660 0x7fb2d419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 -- 172.21.15.28:0/917467557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4101ca0 msgr2=0x7fb2d4198310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4101ca0 0x7fb2d4198310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 -- 172.21.15.28:0/917467557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4102660 msgr2=0x7fb2d419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4102660 0x7fb2d419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 -- 172.21.15.28:0/917467557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d419f9f0 con 0x7fb2d4100ab0 2024-03-22T02:25:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d950 tx=0x7fb2c400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d1ffb640 1 -- 172.21.15.28:0/917467557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c4017020 con 0x7fb2d4100ab0 2024-03-22T02:25:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d419fce0 con 0x7fb2d4100ab0 2024-03-22T02:25:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d1ffb640 1 -- 172.21.15.28:0/917467557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c4009c30 con 0x7fb2d4100ab0 2024-03-22T02:25:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.346+0000 7fb2d1ffb640 1 -- 172.21.15.28:0/917467557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c400f740 con 0x7fb2d4100ab0 2024-03-22T02:25:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.347+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d41a0180 con 0x7fb2d4100ab0 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d1ffb640 1 -- 172.21.15.28:0/917467557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb2c401a020 con 0x7fb2d4100ab0 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 Processor -- start 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 -- start start 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db1c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db1d040 0x55864db1f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d1ffb640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 0x7fb2bc0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55864da7c040 con 0x55864db17ed0 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55864da249e0 con 0x55864db1d040 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55864da77ff0 con 0x55864db174e0 2024-03-22T02:25:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db1c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d9300640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d9300640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:48756/0 (socket says 172.21.15.28:48756) 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d9300640 1 -- 172.21.15.28:0/3180566260 learned_addr learned my addr 172.21.15.28:0/3180566260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 0x7fb2bc0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d1ffb640 1 -- 172.21.15.28:0/917467557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb2c40cf170 con 0x7fb2d4100ab0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.349+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db1d040 0x55864db1f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 -- 172.21.15.28:0/3180566260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db1d040 msgr2=0x55864db1f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db1d040 0x55864db1f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 -- 172.21.15.28:0/3180566260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 msgr2=0x55864db1c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db1c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 -- 172.21.15.28:0/3180566260 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864d790210 con 0x55864db174e0 2024-03-22T02:25:06.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db1c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 0x7fb2bc0a5480 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb2d41992e0 tx=0x7fb2c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0324c0 tx=0x7fb2cc02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcdbd76fd6ce0969 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2baffd640 1 -- 172.21.15.28:0/3180566260 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc039040 con 0x55864db174e0 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2baffd640 1 -- 172.21.15.28:0/3180566260 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc003970 con 0x55864db174e0 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.350+0000 7fb2baffd640 1 -- 172.21.15.28:0/3180566260 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc004290 con 0x55864db174e0 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3180566260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 msgr2=0x55864db17900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0324c0 tx=0x7fb2cc02e040 comp rx=0 tx=0).stop 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3180566260 shutdown_connections 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db1d040 0x55864db1f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db1c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3180566260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db17900 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3180566260 >> 172.21.15.28:0/3180566260 conn(0x55864db12a50 msgr2=0x55864db12e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3180566260 shutdown_connections 2024-03-22T02:25:06.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3180566260 wait complete. 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 -- start start 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 Processor -- start 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db21390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db218d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.351+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55864db1ff30 con 0x55864db1d040 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55864da37db0 con 0x55864db17ed0 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55864db200b0 con 0x55864db174e0 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 10 client.0 _ll_register_callbacks cb 0x55864d78fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59166/0 (socket says 172.21.15.28:59166) 2024-03-22T02:25:06.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db21390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 -- 172.21.15.28:0/3398796506 learned_addr learned my addr 172.21.15.28:0/3398796506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db218d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 -- 172.21.15.28:0/3398796506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 msgr2=0x55864db21390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db21390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 -- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 msgr2=0x55864db218d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db218d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 -- 172.21.15.28:0/3398796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864d85c620 con 0x55864db1d040 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db21390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2d9300640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc02c700 tx=0x7fb2cc032a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a53e054e80baa69 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc039040 con 0x55864db1d040 2024-03-22T02:25:06.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb2cc034650 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2cc003970 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb2cc05d050 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 10 client.18798 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.352+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55864db2bc80 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc003970 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb2cc04e070 con 0x55864db1d040 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 10 client.18798 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 10 client.18798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 20 client.18798 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db2cfd0 0x55864db2f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 4 client.18798 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55864da89580 con 0x55864db2cfd0 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db2cfd0 0x55864db2f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2b3fff640 10 client.18798 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.353+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db2cfd0 0x55864db2f3c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa7cc917c2fd3c03 server_cookie=11e5845b9f1368f5 in_seq=0 out_seq=0 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2b3fff640 1 -- 172.21.15.28:0/3398796506 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55864da89580 con 0x55864db2cfd0 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2b3fff640 10 client.18798 handle_command_reply: tid=0 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 client.18798 shutdown 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 msgr2=0x55864db28910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc02c700 tx=0x7fb2cc032a20 comp rx=0 tx=0).stop 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 shutdown_connections 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db2cfd0 0x55864db2f3c0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db1d040 0x55864db28910 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2b3fff640 0 client.18798 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db218d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/3398796506 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db174e0 0x55864db21390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 >> 172.21.15.28:0/3398796506 conn(0x55864db12a50 msgr2=0x55864db11c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 shutdown_connections 2024-03-22T02:25:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 1 -- 172.21.15.28:0/3398796506 wait complete. 2024-03-22T02:25:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.354+0000 7fb2dda1f740 20 client.18798 trim_cache size 0 max 16384 2024-03-22T02:25:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 Processor -- start 2024-03-22T02:25:06.375 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 -- start start 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db11e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41ad1e0 con 0x55864db174e0 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41aaab0 con 0x55864db17ed0 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41a8900 con 0x55864db1d040 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.369+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59180/0 (socket says 172.21.15.28:59180) 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 -- 172.21.15.28:0/1556211004 learned_addr learned my addr 172.21.15.28:0/1556211004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d9300640 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db11e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 -- 172.21.15.28:0/1556211004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 msgr2=0x55864db11e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db11e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 -- 172.21.15.28:0/1556211004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 msgr2=0x55864db11900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db11900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 -- 172.21.15.28:0/1556211004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55864db201e0 con 0x55864db174e0 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40978f0 tx=0x7fb2c406d110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c49f4078540bf56 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2ba7fc640 1 -- 172.21.15.28:0/1556211004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c40a3ea0 con 0x55864db174e0 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2ba7fc640 1 -- 172.21.15.28:0/1556211004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c4099530 con 0x55864db174e0 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.370+0000 7fb2ba7fc640 1 -- 172.21.15.28:0/1556211004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c409f3c0 con 0x55864db174e0 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1556211004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 msgr2=0x55864db113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fb2c40978f0 tx=0x7fb2c406d110 comp rx=0 tx=0).stop 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1556211004 shutdown_connections 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db11e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db17ed0 0x55864db11900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1556211004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db174e0 0x55864db113c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1556211004 >> 172.21.15.28:0/1556211004 conn(0x55864db12a50 msgr2=0x55864db15cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1556211004 shutdown_connections 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1556211004 wait complete. 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 -- start start 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.371+0000 7fb2dda1f740 1 Processor -- start 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db174e0 0x55864db8e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db95bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41a8f70 con 0x55864db17ed0 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41aa3e0 con 0x55864db174e0 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d41a9640 con 0x55864db1d040 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 10 client.0 _ll_register_callbacks cb 0x7fb2c4010d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d8aff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db174e0 0x55864db8e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59196/0 (socket says 172.21.15.28:59196) 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d9300640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db95bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 -- 172.21.15.28:0/1700031574 learned_addr learned my addr 172.21.15.28:0/1700031574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 -- 172.21.15.28:0/1700031574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 msgr2=0x55864db95bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db95bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 -- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db174e0 msgr2=0x55864db8e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db174e0 0x55864db8e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 -- 172.21.15.28:0/1700031574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55864db961c0 con 0x55864db17ed0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2d3fff640 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8011810 tx=0x7fb2c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77caa24951a048de server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c801f8b0 con 0x55864db17ed0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fb2c801fa50 con 0x55864db17ed0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c4099530 con 0x55864db17ed0 2024-03-22T02:25:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb2c801fc50 con 0x55864db17ed0 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 10 client.18804 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.372+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55864db97540 con 0x55864db17ed0 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c4099530 con 0x55864db17ed0 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb2c8004aa0 con 0x55864db17ed0 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 10 client.18804 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 10 client.18804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 20 client.18804 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db97f90 0x55864db9a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 4 client.18804 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55864da838d0 con 0x55864db97f90 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db97f90 0x55864db9a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2b3fff640 10 client.18804 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.373+0000 7fb2d8aff640 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db97f90 0x55864db9a3b0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1f7276deaa17389 server_cookie=bc4d38e9f27606a in_seq=0 out_seq=0 2024-03-22T02:25:06.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1172395 11.32 MB/sec execute 468 sec latency 38.839 ms 2024-03-22T02:25:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2b3fff640 1 -- 172.21.15.28:0/1700031574 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55864da838d0 con 0x55864db97f90 2024-03-22T02:25:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2b3fff640 10 client.18804 handle_command_reply: tid=0 2024-03-22T02:25:06.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 client.18804 shutdown 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 msgr2=0x55864db8ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8011810 tx=0x7fb2c8004730 comp rx=0 tx=0).stop 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 shutdown_connections 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55864db97f90 0x55864db9a3b0 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55864db1d040 0x55864db95bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55864db17ed0 0x55864db8ec00 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2b3fff640 0 client.18804 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 --2- 172.21.15.28:0/1700031574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55864db174e0 0x55864db8e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 >> 172.21.15.28:0/1700031574 conn(0x55864db12a50 msgr2=0x55864db187c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 shutdown_connections 2024-03-22T02:25:06.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.717+0000 7fb2dda1f740 1 -- 172.21.15.28:0/1700031574 wait complete. 2024-03-22T02:25:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dda1f740 20 client.18804 trim_cache size 0 max 16384 2024-03-22T02:25:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:25:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:25:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:25:06.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:25:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 msgr2=0x7fb2bc0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 0x7fb2bc0a5480 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb2d41992e0 tx=0x7fb2c8005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 msgr2=0x7fb2d4197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fb2c400d950 tx=0x7fb2c400de10 comp rx=0 tx=0).stop 2024-03-22T02:25:06.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 shutdown_connections 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2bc0a2fc0 0x7fb2bc0a5480 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb2d4102660 0x7fb2d419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2d4101ca0 0x7fb2d4198310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 --2- 172.21.15.28:0/917467557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2d4100ab0 0x7fb2d4197dd0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.718+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 >> 172.21.15.28:0/917467557 conn(0x7fb2d40fc020 msgr2=0x7fb2d4104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.719+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 shutdown_connections 2024-03-22T02:25:06.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:06.719+0000 7fb2dad63640 1 -- 172.21.15.28:0/917467557 wait complete. 2024-03-22T02:25:07.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1173615 11.31 MB/sec execute 469 sec latency 260.650 ms 2024-03-22T02:25:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:07 smithi120 ceph-mon[36803]: pgmap v706: 129 pgs: 129 active+clean; 69 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-22T02:25:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:07 smithi083 ceph-mon[36340]: pgmap v706: 129 pgs: 129 active+clean; 69 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-22T02:25:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:07 smithi028 ceph-mon[31916]: pgmap v706: 129 pgs: 129 active+clean; 69 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 305 op/s 2024-03-22T02:25:08.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1175909 11.31 MB/sec execute 470 sec latency 66.258 ms 2024-03-22T02:25:09.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1177651 11.30 MB/sec execute 471 sec latency 43.085 ms 2024-03-22T02:25:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:09 smithi120 ceph-mon[36803]: pgmap v707: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 381 op/s 2024-03-22T02:25:10.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:09 smithi083 ceph-mon[36340]: pgmap v707: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 381 op/s 2024-03-22T02:25:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:09 smithi028 ceph-mon[31916]: pgmap v707: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.3 MiB/s wr, 381 op/s 2024-03-22T02:25:10.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1179867 11.31 MB/sec execute 472 sec latency 27.430 ms 2024-03-22T02:25:11.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1182721 11.32 MB/sec execute 473 sec latency 19.416 ms 2024-03-22T02:25:12.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:11 smithi120 ceph-mon[36803]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 326 op/s 2024-03-22T02:25:12.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:11 smithi083 ceph-mon[36340]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 326 op/s 2024-03-22T02:25:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:11 smithi028 ceph-mon[31916]: pgmap v708: 129 pgs: 129 active+clean; 60 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 326 op/s 2024-03-22T02:25:12.434 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1184254 11.31 MB/sec execute 474 sec latency 29.068 ms 2024-03-22T02:25:13.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1186531 11.32 MB/sec execute 475 sec latency 123.137 ms 2024-03-22T02:25:14.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:13 smithi120 ceph-mon[36803]: pgmap v709: 129 pgs: 129 active+clean; 63 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 372 op/s 2024-03-22T02:25:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:13 smithi083 ceph-mon[36340]: pgmap v709: 129 pgs: 129 active+clean; 63 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 372 op/s 2024-03-22T02:25:14.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:13 smithi028 ceph-mon[31916]: pgmap v709: 129 pgs: 129 active+clean; 63 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.7 MiB/s wr, 372 op/s 2024-03-22T02:25:14.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1188101 11.30 MB/sec execute 476 sec latency 35.793 ms 2024-03-22T02:25:15.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1190315 11.31 MB/sec execute 477 sec latency 20.732 ms 2024-03-22T02:25:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:15 smithi120 ceph-mon[36803]: pgmap v710: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 440 op/s 2024-03-22T02:25:16.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:15 smithi083 ceph-mon[36340]: pgmap v710: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 440 op/s 2024-03-22T02:25:16.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:15 smithi028 ceph-mon[31916]: pgmap v710: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.5 MiB/s wr, 440 op/s 2024-03-22T02:25:16.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1192715 11.31 MB/sec execute 478 sec latency 25.149 ms 2024-03-22T02:25:16.530 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:25:16.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 -- 172.21.15.28:0/1017860292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc105020 msgr2=0x7fc5fc107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 --2- 172.21.15.28:0/1017860292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc105020 0x7fc5fc107410 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0009900 tx=0x7fc5f002bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 -- 172.21.15.28:0/1017860292 shutdown_connections 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 --2- 172.21.15.28:0/1017860292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc107950 0x7fc5fc109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 --2- 172.21.15.28:0/1017860292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc105020 0x7fc5fc107410 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 --2- 172.21.15.28:0/1017860292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 -- 172.21.15.28:0/1017860292 >> 172.21.15.28:0/1017860292 conn(0x7fc5fc0fc040 msgr2=0x7fc5fc0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 -- 172.21.15.28:0/1017860292 shutdown_connections 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.642+0000 7fc600a9d640 1 -- 172.21.15.28:0/1017860292 wait complete. 2024-03-22T02:25:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 Processor -- start 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 -- start start 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 0x7fc5fc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc107950 0x7fc5fc19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc10d830 con 0x7fc5fc1026f0 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc10d6b0 con 0x7fc5fc105020 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc600a9d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc10d9b0 con 0x7fc5fc107950 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37678/0 (socket says 172.21.15.28:37678) 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc5fa59c640 1 -- 172.21.15.28:0/519867518 learned_addr learned my addr 172.21.15.28:0/519867518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 0x7fc5fc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.643+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc107950 0x7fc5fc19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/519867518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc107950 msgr2=0x7fc5fc19f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc107950 0x7fc5fc19f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 msgr2=0x7fc5fc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/519867518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5fc19fb20 con 0x7fc5fc105020 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 0x7fc5fc198440 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc5f002c170 tx=0x7fc5f002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5eb7fe640 1 -- 172.21.15.28:0/519867518 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0041070 con 0x7fc5fc105020 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5eb7fe640 1 -- 172.21.15.28:0/519867518 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5f0004450 con 0x7fc5fc105020 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5fc19fdb0 con 0x7fc5fc105020 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc5eb7fe640 1 -- 172.21.15.28:0/519867518 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f0003e50 con 0x7fc5fc105020 2024-03-22T02:25:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.644+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5fc1a0210 con 0x7fc5fc105020 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.646+0000 7fc5eb7fe640 1 -- 172.21.15.28:0/519867518 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc5f0044070 con 0x7fc5fc105020 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.646+0000 7fc603766740 1 Processor -- start 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 -- start start 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326e1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326e9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e326428c0 con 0x558e326e1840 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558e325eac20 con 0x558e326e73a0 2024-03-22T02:25:16.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc603766740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558e3263e870 con 0x558e326e2230 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326e1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5eb7fe640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 0x7fc5e40a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5f9d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326e9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45846/0 (socket says 172.21.15.28:45846) 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 -- 172.21.15.28:0/616524638 learned_addr learned my addr 172.21.15.28:0/616524638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 0x7fc5e40a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 -- 172.21.15.28:0/616524638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 msgr2=0x558e326e9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326e9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 -- 172.21.15.28:0/616524638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 msgr2=0x558e326e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326e1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5fa59c640 1 -- 172.21.15.28:0/616524638 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e3260f150 con 0x558e326e2230 2024-03-22T02:25:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.647+0000 7fc5eb7fe640 1 -- 172.21.15.28:0/519867518 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc5f00f12c0 con 0x7fc5fc105020 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.648+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0115f0 tx=0x7fc5ec011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf91b1dd6e249438 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.648+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 0x7fc5e40a5580 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x558e326e2a90 tx=0x7fc5f4004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.648+0000 7fc5e8ff9640 1 -- 172.21.15.28:0/616524638 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec004b20 con 0x558e326e2230 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.648+0000 7fc5e8ff9640 1 -- 172.21.15.28:0/616524638 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ec004cc0 con 0x558e326e2230 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.648+0000 7fc5e8ff9640 1 -- 172.21.15.28:0/616524638 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec005530 con 0x558e326e2230 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- 172.21.15.28:0/616524638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 msgr2=0x558e326e6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0115f0 tx=0x7fc5ec011ab0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- 172.21.15.28:0/616524638 shutdown_connections 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326e9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326e6d30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- 172.21.15.28:0/616524638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326e1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- 172.21.15.28:0/616524638 >> 172.21.15.28:0/616524638 conn(0x558e326d89a0 msgr2=0x558e326d8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- 172.21.15.28:0/616524638 shutdown_connections 2024-03-22T02:25:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- 172.21.15.28:0/616524638 wait complete. 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- start start 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 Processor -- start 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 0x558e326ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e73a0 0x558e326f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558e326ea350 con 0x558e326e2230 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558e325fe130 con 0x558e326e1840 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558e326ea4d0 con 0x558e326e73a0 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.649+0000 7fc603766740 10 client.0 _ll_register_callbacks cb 0x558e325fdb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 0x558e326ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37696/0 (socket says 172.21.15.28:37696) 2024-03-22T02:25:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/388937224 learned_addr learned my addr 172.21.15.28:0/388937224 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e73a0 0x558e326f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/388937224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e73a0 msgr2=0x558e326f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e73a0 0x558e326f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 msgr2=0x558e326ebc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 0x558e326ebc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 -- 172.21.15.28:0/388937224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e324207d0 con 0x558e326e2230 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 0x558e326ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00b9a90 tx=0x7fc5f0049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb85808a05786cc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5f004b040 con 0x558e326e2230 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc5f00d6490 con 0x558e326e2230 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ec004cc0 con 0x558e326e2230 2024-03-22T02:25:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc5f00a8070 con 0x558e326e2230 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc603766740 10 client.18816 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.650+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558e323563f0 con 0x558e326e2230 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ec004cc0 con 0x558e326e2230 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc5f00dd040 con 0x558e326e2230 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 10 client.18816 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 10 client.18816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 20 client.18816 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e326f79e0 0x558e326f9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 4 client.18816 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e3264fe00 con 0x558e326f79e0 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e326f79e0 0x558e326f9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e326f79e0 0x558e326f9e50 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998c6b20bd6b4396 server_cookie=edd2ca331c6a51d3 in_seq=0 out_seq=0 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.651+0000 7fc5e1ffb640 10 client.18816 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/388937224 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558e3264fe00 con 0x558e326f79e0 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc5e1ffb640 10 client.18816 handle_command_reply: tid=0 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 client.18816 shutdown 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 msgr2=0x558e326ec1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00b9a90 tx=0x7fc5f0049000 comp rx=0 tx=0).stop 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 shutdown_connections 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e326f79e0 0x558e326f9e50 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e73a0 0x558e326f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e2230 0x558e326ec1a0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc5e1ffb640 0 client.18816 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 --2- 172.21.15.28:0/388937224 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e1840 0x558e326ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 >> 172.21.15.28:0/388937224 conn(0x558e326d89a0 msgr2=0x558e326d7c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 shutdown_connections 2024-03-22T02:25:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 1 -- 172.21.15.28:0/388937224 wait complete. 2024-03-22T02:25:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.652+0000 7fc603766740 20 client.18816 trim_cache size 0 max 16384 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.671+0000 7fc603766740 1 Processor -- start 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.671+0000 7fc603766740 1 -- start start 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326fe720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1ab020 con 0x558e326e1840 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1a88f0 con 0x558e326e73a0 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc603766740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1a6740 con 0x558e326e2230 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37708/0 (socket says 172.21.15.28:37708) 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1928673520 learned_addr learned my addr 172.21.15.28:0/1928673520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1928673520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 msgr2=0x558e326f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1928673520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 msgr2=0x558e326fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326fe720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1928673520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e323160f0 con 0x558e326e1840 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec01b040 tx=0x7fc5ec005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6ab9f72a05e36b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5e3fff640 1 -- 172.21.15.28:0/1928673520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec02f070 con 0x558e326e1840 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5e3fff640 1 -- 172.21.15.28:0/1928673520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ec004cc0 con 0x558e326e1840 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.672+0000 7fc5e3fff640 1 -- 172.21.15.28:0/1928673520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec029500 con 0x558e326e1840 2024-03-22T02:25:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 -- 172.21.15.28:0/1928673520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 msgr2=0x558e326dc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec01b040 tx=0x7fc5ec005bd0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 -- 172.21.15.28:0/1928673520 shutdown_connections 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 --2- 172.21.15.28:0/1928673520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326dc310 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 -- 172.21.15.28:0/1928673520 >> 172.21.15.28:0/1928673520 conn(0x558e326d89a0 msgr2=0x558e326dbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 -- 172.21.15.28:0/1928673520 shutdown_connections 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.673+0000 7fc603766740 1 -- 172.21.15.28:0/1928673520 wait complete. 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 -- start start 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 Processor -- start 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326f2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1a6db0 con 0x558e326e1840 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1a8220 con 0x558e326e73a0 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5fc1a7480 con 0x558e326e2230 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc603766740 10 client.0 _ll_register_callbacks cb 0x558e325fdb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37724/0 (socket says 172.21.15.28:37724) 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1954541597 learned_addr learned my addr 172.21.15.28:0/1954541597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1954541597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 msgr2=0x558e326f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 msgr2=0x558e326f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 -- 172.21.15.28:0/1954541597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e326f2730 con 0x558e326e1840 2024-03-22T02:25:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fad9d640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5fa59c640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005600 tx=0x7fc5ec00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b853c94c283d1fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.674+0000 7fc5e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec038060 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc5ec009bf0 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5ec004cc0 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc5ec02f070 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 10 client.18822 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x558e3275dfd0 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5ec004cc0 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc5ec033670 con 0x558e326e1840 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 10 client.18822 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 10 client.18822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 20 client.18822 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e3275e730 0x558e32760b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 4 client.18822 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558e3264a150 con 0x558e3275e730 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.675+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e3275e730 0x558e32760b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.676+0000 7fc5f9d9b640 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e3275e730 0x558e32760b50 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65be2c3d399db13d server_cookie=69a23c108da382de in_seq=0 out_seq=0 2024-03-22T02:25:16.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.676+0000 7fc5e1ffb640 10 client.18822 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc5e1ffb640 1 -- 172.21.15.28:0/1954541597 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558e3264a150 con 0x558e3275e730 2024-03-22T02:25:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc5e1ffb640 10 client.18822 handle_command_reply: tid=0 2024-03-22T02:25:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 client.18822 shutdown 2024-03-22T02:25:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 msgr2=0x558e326f0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec005600 tx=0x7fc5ec00f3c0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 shutdown_connections 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558e3275e730 0x558e32760b50 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558e326e73a0 0x558e326f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558e326e2230 0x558e326f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc5e1ffb640 0 client.18822 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 --2- 172.21.15.28:0/1954541597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e326e1840 0x558e326f0bf0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 >> 172.21.15.28:0/1954541597 conn(0x558e326d89a0 msgr2=0x558e326dc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 shutdown_connections 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 1 -- 172.21.15.28:0/1954541597 wait complete. 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.678+0000 7fc603766740 20 client.18822 trim_cache size 0 max 16384 2024-03-22T02:25:16.684 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:25:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:25:16.685 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 msgr2=0x7fc5e40a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 0x7fc5e40a5580 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x558e326e2a90 tx=0x7fc5f4004730 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 msgr2=0x7fc5fc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 0x7fc5fc198440 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc5f002c170 tx=0x7fc5f002c650 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 shutdown_connections 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5e40a30c0 0x7fc5e40a5580 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc5fc107950 0x7fc5fc19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc5fc105020 0x7fc5fc198440 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 --2- 172.21.15.28:0/519867518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5fc1026f0 0x7fc5fc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 >> 172.21.15.28:0/519867518 conn(0x7fc5fc0fc040 msgr2=0x7fc5fc103110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 shutdown_connections 2024-03-22T02:25:16.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.679+0000 7fc600a9d640 1 -- 172.21.15.28:0/519867518 wait complete. 2024-03-22T02:25:16.691 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:25:16.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- 172.21.15.28:0/2848894295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c101ca0 msgr2=0x7fc05c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2848894295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c101ca0 0x7fc05c102120 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009900 tx=0x7fc04c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- 172.21.15.28:0/2848894295 shutdown_connections 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2848894295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c102660 0x7fc05c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2848894295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c101ca0 0x7fc05c102120 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2848894295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c100ab0 0x7fc05c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- 172.21.15.28:0/2848894295 >> 172.21.15.28:0/2848894295 conn(0x7fc05c0fc020 msgr2=0x7fc05c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- 172.21.15.28:0/2848894295 shutdown_connections 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- 172.21.15.28:0/2848894295 wait complete. 2024-03-22T02:25:16.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 Processor -- start 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.799+0000 7fc0639bc640 1 -- start start 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 0x7fc05c0698c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c101ca0 0x7fc05c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10c730 con 0x7fc05c102660 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10c5b0 con 0x7fc05c100ab0 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc05c10c8b0 con 0x7fc05c101ca0 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061758640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 0x7fc05c0698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37748/0 (socket says 172.21.15.28:37748) 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061758640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 0x7fc05c0698c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56718/0 (socket says 172.21.15.28:56718) 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 -- 172.21.15.28:0/2350619559 learned_addr learned my addr 172.21.15.28:0/2350619559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:16.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc060f57640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c101ca0 0x7fc05c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 -- 172.21.15.28:0/2350619559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c101ca0 msgr2=0x7fc05c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c101ca0 0x7fc05c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 -- 172.21.15.28:0/2350619559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 msgr2=0x7fc05c0698c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 0x7fc05c0698c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 -- 172.21.15.28:0/2350619559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc05c19fb30 con 0x7fc05c102660 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc061f59640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc05400eb10 tx=0x7fc05400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054004b20 con 0x7fc05c102660 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc054004cc0 con 0x7fc05c102660 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc05c19fe20 con 0x7fc05c102660 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.800+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc054010760 con 0x7fc05c102660 2024-03-22T02:25:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.801+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc05c1a0290 con 0x7fc05c102660 2024-03-22T02:25:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.802+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc05c107df0 con 0x7fc05c102660 2024-03-22T02:25:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.803+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc054005020 con 0x7fc05c102660 2024-03-22T02:25:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.803+0000 7fc0527fc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 0x7fc0480a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.803+0000 7fc061758640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 0x7fc0480a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.803+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc0540cfd60 con 0x7fc05c102660 2024-03-22T02:25:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.804+0000 7fc061758640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 0x7fc0480a53e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc05c101b00 tx=0x7fc058005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:16.808+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc05409c780 con 0x7fc05c102660 2024-03-22T02:25:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.052+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc05c02d3b0 con 0x7fc05c102660 2024-03-22T02:25:17.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.053+0000 7fc0527fc640 1 -- 172.21.15.28:0/2350619559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7fc05409c080 con 0x7fc05c102660 2024-03-22T02:25:17.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:25:17.053 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:24:45.372919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:25:17.054 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 msgr2=0x7fc0480a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 0x7fc0480a53e0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc05c101b00 tx=0x7fc058005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 msgr2=0x7fc05c19f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fc05400eb10 tx=0x7fc05400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 shutdown_connections 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0480a2f20 0x7fc0480a53e0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc05c102660 0x7fc05c19f270 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc05c101ca0 0x7fc05c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 --2- 172.21.15.28:0/2350619559 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc05c100ab0 0x7fc05c0698c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 >> 172.21.15.28:0/2350619559 conn(0x7fc05c0fc020 msgr2=0x7fc05c104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 shutdown_connections 2024-03-22T02:25:17.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.055+0000 7fc0639bc640 1 -- 172.21.15.28:0/2350619559 wait complete. 2024-03-22T02:25:17.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:25:17.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- 172.21.15.28:0/1447912030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058102630 msgr2=0x7f5058104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 --2- 172.21.15.28:0/1447912030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058102630 0x7f5058104a20 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f5054009900 tx=0x7f505402bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- 172.21.15.28:0/1447912030 shutdown_connections 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 --2- 172.21.15.28:0/1447912030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058107890 0x7f5058109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 --2- 172.21.15.28:0/1447912030 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058104f60 0x7f5058107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 --2- 172.21.15.28:0/1447912030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058102630 0x7f5058104a20 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- 172.21.15.28:0/1447912030 >> 172.21.15.28:0/1447912030 conn(0x7f50580fc040 msgr2=0x7f50580fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- 172.21.15.28:0/1447912030 shutdown_connections 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- 172.21.15.28:0/1447912030 wait complete. 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 Processor -- start 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.200+0000 7f505feb5640 1 -- start start 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058102630 0x7f5058197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 0x7f505819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f505810d830 con 0x7f5058104f60 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f505810d6b0 con 0x7f5058107890 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f505810d9b0 con 0x7f5058102630 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37778/0 (socket says 172.21.15.28:37778) 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 -- 172.21.15.28:0/1297369581 learned_addr learned my addr 172.21.15.28:0/1297369581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505dc51640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058102630 0x7f5058197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505e452640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 0x7f505819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 -- 172.21.15.28:0/1297369581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058102630 msgr2=0x7f5058197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058102630 0x7f5058197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 -- 172.21.15.28:0/1297369581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 msgr2=0x7f505819f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 0x7f505819f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 -- 172.21.15.28:0/1297369581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f505819fb50 con 0x7f5058104f60 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505e452640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 0x7f505819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505d450640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f504c00d770 tx=0x7f504c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f504affd640 1 -- 172.21.15.28:0/1297369581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c017070 con 0x7f5058104f60 2024-03-22T02:25:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f504affd640 1 -- 172.21.15.28:0/1297369581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f504c009c30 con 0x7f5058104f60 2024-03-22T02:25:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f505819fe40 con 0x7f5058104f60 2024-03-22T02:25:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f504affd640 1 -- 172.21.15.28:0/1297369581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c00f620 con 0x7f5058104f60 2024-03-22T02:25:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.201+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50581a0300 con 0x7f5058104f60 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.203+0000 7f504affd640 1 -- 172.21.15.28:0/1297369581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f504c01a070 con 0x7f5058104f60 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 Processor -- start 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 -- start start 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d75820 0x564626d75c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d7d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564626cda8d0 con 0x564626d7b380 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564626c82e00 con 0x564626d75820 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f5062b7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564626cd6880 con 0x564626d76210 2024-03-22T02:25:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d7d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505dc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505e452640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d75820 0x564626d75c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505dc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45912/0 (socket says 172.21.15.28:45912) 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d7d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37786/0 (socket says 172.21.15.28:37786) 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f505dc51640 1 -- 172.21.15.28:0/2386811204 learned_addr learned my addr 172.21.15.28:0/2386811204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.204+0000 7f504affd640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 0x7f50440a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 -- 172.21.15.28:0/2386811204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d75820 msgr2=0x564626d75c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505e452640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 0x7f50440a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d75820 0x564626d75c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 -- 172.21.15.28:0/2386811204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 msgr2=0x564626d7d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d7d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 -- 172.21.15.28:0/2386811204 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646269ee3f0 con 0x564626d76210 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f504affd640 1 -- 172.21.15.28:0/1297369581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f504c0cf2b0 con 0x7f5058104f60 2024-03-22T02:25:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505dc51640 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5054009900 tx=0x7f50540095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5338280bcabb136e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f5043fff640 1 -- 172.21.15.28:0/2386811204 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f505402cb80 con 0x564626d76210 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f505e452640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 0x7f50440a53e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x564626d76a70 tx=0x7f5050004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f5043fff640 1 -- 172.21.15.28:0/2386811204 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5054005040 con 0x564626d76210 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.205+0000 7f5043fff640 1 -- 172.21.15.28:0/2386811204 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50540048e0 con 0x564626d76210 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- 172.21.15.28:0/2386811204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 msgr2=0x564626d7ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5054009900 tx=0x7f50540095e0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- 172.21.15.28:0/2386811204 shutdown_connections 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d7d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d76210 0x564626d7ad10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2386811204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d75820 0x564626d75c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- 172.21.15.28:0/2386811204 >> 172.21.15.28:0/2386811204 conn(0x564626d70d90 msgr2=0x564626d71190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- 172.21.15.28:0/2386811204 shutdown_connections 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- 172.21.15.28:0/2386811204 wait complete. 2024-03-22T02:25:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 -- start start 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.206+0000 7f5062b7a740 1 Processor -- start 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d7fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d7b380 0x564626d872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564626d7e2d0 con 0x564626d76210 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564626c96300 con 0x564626d7b380 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564626d7e450 con 0x564626d75820 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f5062b7a740 10 client.0 _ll_register_callbacks cb 0x564626cd1e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505dc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d7fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37798/0 (socket says 172.21.15.28:37798) 2024-03-22T02:25:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 -- 172.21.15.28:0/608818596 learned_addr learned my addr 172.21.15.28:0/608818596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505e452640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d7b380 0x564626d872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 -- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 msgr2=0x564626d7fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d7fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 -- 172.21.15.28:0/608818596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d7b380 msgr2=0x564626d872b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d7b380 0x564626d872b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 -- 172.21.15.28:0/608818596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564626ab87d0 con 0x564626d76210 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505dc51640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d7fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f505d450640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f504c09b560 tx=0x7f504c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c111eff7fd064db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f50417fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.207+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504c0cce60 con 0x564626d76210 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f504c0a3b50 con 0x564626d76210 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5054005040 con 0x564626d76210 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f504c021070 con 0x564626d76210 2024-03-22T02:25:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 10 client.18840 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564626d8a740 con 0x564626d76210 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5054005040 con 0x564626d76210 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f504c084b60 con 0x564626d76210 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 10 client.18840 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 10 client.18840 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 20 client.18840 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626d8bd60 0x564626d8e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 4 client.18840 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564626ce7e10 con 0x564626d8bd60 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.208+0000 7f505dc51640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626d8bd60 0x564626d8e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f50417fa640 10 client.18840 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f505dc51640 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626d8bd60 0x564626d8e1d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21451c317b68203c server_cookie=a4ee13cd80f69845 in_seq=0 out_seq=0 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f50417fa640 1 -- 172.21.15.28:0/608818596 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564626ce7e10 con 0x564626d8bd60 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f50417fa640 10 client.18840 handle_command_reply: tid=0 2024-03-22T02:25:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f5062b7a740 1 client.18840 shutdown 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 msgr2=0x564626d80270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.209+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f504c09b560 tx=0x7f504c0057f0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 shutdown_connections 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626d8bd60 0x564626d8e1d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d7b380 0x564626d872b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d76210 0x564626d80270 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f50417fa640 0 client.18840 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 --2- 172.21.15.28:0/608818596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d7fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 >> 172.21.15.28:0/608818596 conn(0x564626d70d90 msgr2=0x564626d6ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 shutdown_connections 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 1 -- 172.21.15.28:0/608818596 wait complete. 2024-03-22T02:25:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.210+0000 7f5062b7a740 20 client.18840 trim_cache size 0 max 16384 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 Processor -- start 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 -- start start 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d6f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d87270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50581ad210 con 0x564626d7b380 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50581aaae0 con 0x564626d76210 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f5062b7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f50581a8930 con 0x564626d75820 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f505e452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f505dc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d6f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f505e452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37808/0 (socket says 172.21.15.28:37808) 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f505e452640 1 -- 172.21.15.28:0/3936025179 learned_addr learned my addr 172.21.15.28:0/3936025179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.224+0000 7f505d450640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d87270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 -- 172.21.15.28:0/3936025179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 msgr2=0x564626d6f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d6f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 -- 172.21.15.28:0/3936025179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 msgr2=0x564626d87270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d87270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 -- 172.21.15.28:0/3936025179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564626d8e800 con 0x564626d7b380 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505dc51640 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d6f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f505e452640 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f505000e950 tx=0x7f5050009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbcc6f808631ce0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f50437fe640 1 -- 172.21.15.28:0/3936025179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050010aa0 con 0x564626d7b380 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f50437fe640 1 -- 172.21.15.28:0/3936025179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5050025070 con 0x564626d7b380 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.225+0000 7f50437fe640 1 -- 172.21.15.28:0/3936025179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50500206b0 con 0x564626d7b380 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- 172.21.15.28:0/3936025179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 msgr2=0x564626d6fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f505000e950 tx=0x7f5050009b20 comp rx=0 tx=0).stop 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- 172.21.15.28:0/3936025179 shutdown_connections 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d6fc80 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d87270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- 172.21.15.28:0/3936025179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d6f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- 172.21.15.28:0/3936025179 >> 172.21.15.28:0/3936025179 conn(0x564626d70d90 msgr2=0x564626d7fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- 172.21.15.28:0/3936025179 shutdown_connections 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- 172.21.15.28:0/3936025179 wait complete. 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- start start 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 Processor -- start 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d846f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d84c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50581a8fa0 con 0x564626d7b380 2024-03-22T02:25:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50581aa410 con 0x564626d76210 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.226+0000 7f5062b7a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f50581a9670 con 0x564626d75820 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f5062b7a740 10 client.0 _ll_register_callbacks cb 0x7f504c00de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505dc51640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d846f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37824/0 (socket says 172.21.15.28:37824) 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 -- 172.21.15.28:0/2602704007 learned_addr learned my addr 172.21.15.28:0/2602704007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505d450640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d84c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 -- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 msgr2=0x564626d846f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d846f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 -- 172.21.15.28:0/2602704007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 msgr2=0x564626d84c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d84c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 -- 172.21.15.28:0/2602704007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564626d861e0 con 0x564626d7b380 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f505e452640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5050004a80 tx=0x7f5050021680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7680ef0e3bd01b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f50417fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:17.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5050037060 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5050010aa0 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5050025070 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f505003e050 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f5062b7a740 10 client.18846 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.227+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x564626d874e0 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5050025070 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f505001e040 con 0x564626d7b380 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 10 client.18846 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 10 client.18846 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 20 client.18846 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626df26b0 0x564626df4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 4 client.18846 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:17.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564626ce2160 con 0x564626df26b0 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f505dc51640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626df26b0 0x564626df4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f50417fa640 10 client.18846 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f505dc51640 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626df26b0 0x564626df4ad0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dddd5ff96a1ddeb0 server_cookie=8bb8c0086dae36d2 in_seq=0 out_seq=0 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f50417fa640 1 -- 172.21.15.28:0/2602704007 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564626ce2160 con 0x564626df26b0 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f50417fa640 10 client.18846 handle_command_reply: tid=0 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.228+0000 7f5062b7a740 1 client.18846 shutdown 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 msgr2=0x564626d85bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f5050004a80 tx=0x7f5050021680 comp rx=0 tx=0).stop 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 shutdown_connections 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564626d7b380 0x564626d85bf0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564626df26b0 0x564626df4ad0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564626d76210 0x564626d84c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 --2- 172.21.15.28:0/2602704007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564626d75820 0x564626d846f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f50417fa640 0 client.18846 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 >> 172.21.15.28:0/2602704007 conn(0x564626d70d90 msgr2=0x564626d73ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 shutdown_connections 2024-03-22T02:25:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 1 -- 172.21.15.28:0/2602704007 wait complete. 2024-03-22T02:25:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f5062b7a740 20 client.18846 trim_cache size 0 max 16384 2024-03-22T02:25:17.236 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 msgr2=0x7f50440a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 0x7f50440a53e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x564626d76a70 tx=0x7f5050004ab0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 msgr2=0x7f5058198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f504c00d770 tx=0x7f504c00dc30 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 shutdown_connections 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50440a2f20 0x7f50440a53e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5058107890 0x7f505819f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5058104f60 0x7f5058198470 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 --2- 172.21.15.28:0/1297369581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5058102630 0x7f5058197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 >> 172.21.15.28:0/1297369581 conn(0x7f50580fc040 msgr2=0x7f5058103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 shutdown_connections 2024-03-22T02:25:17.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.229+0000 7f505feb5640 1 -- 172.21.15.28:0/1297369581 wait complete. 2024-03-22T02:25:17.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:25:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 -- 172.21.15.28:0/858263940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 msgr2=0x7f249c10a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/858263940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c10a190 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f2494009ca0 tx=0x7f249402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 -- 172.21.15.28:0/858263940 shutdown_connections 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/858263940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c10a190 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/858263940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c0ffc20 0x7f249c1000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/858263940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c1031b0 0x7f249c1035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.349+0000 7f24a0eeb640 1 -- 172.21.15.28:0/858263940 >> 172.21.15.28:0/858263940 conn(0x7f249c0fb780 msgr2=0x7f249c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 -- 172.21.15.28:0/858263940 shutdown_connections 2024-03-22T02:25:17.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 -- 172.21.15.28:0/858263940 wait complete. 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 Processor -- start 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 -- start start 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 0x7f249c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.350+0000 7f24a0eeb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f24a0eeb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 0x7f249c19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f24a0eeb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f249c10d830 con 0x7f249c1005e0 2024-03-22T02:25:17.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f24a0eeb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f249c10d6b0 con 0x7f249c1031b0 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f24a0eeb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c10d9b0 con 0x7f249c0ffc20 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37854/0 (socket says 172.21.15.28:37854) 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 -- 172.21.15.28:0/2960191833 learned_addr learned my addr 172.21.15.28:0/2960191833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f249a59c640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 0x7f249c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f249ad9d640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 0x7f249c19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 -- 172.21.15.28:0/2960191833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 msgr2=0x7f249c197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 0x7f249c197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 -- 172.21.15.28:0/2960191833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 msgr2=0x7f249c19f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 0x7f249c19f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 -- 172.21.15.28:0/2960191833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249c19fa60 con 0x7f249c1005e0 2024-03-22T02:25:17.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f249a59c640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 0x7f249c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f249ad9d640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 0x7f249c19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f2493fff640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f248400e9f0 tx=0x7f248400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.351+0000 7f24937fe640 1 -- 172.21.15.28:0/2960191833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248400cd80 con 0x7f249c1005e0 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.352+0000 7f24937fe640 1 -- 172.21.15.28:0/2960191833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2484010910 con 0x7f249c1005e0 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.352+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f249c19fd50 con 0x7f249c1005e0 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.352+0000 7f24937fe640 1 -- 172.21.15.28:0/2960191833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484018990 con 0x7f249c1005e0 2024-03-22T02:25:17.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.352+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f249c1a0260 con 0x7f249c1005e0 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.354+0000 7f24937fe640 1 -- 172.21.15.28:0/2960191833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2484010ac0 con 0x7f249c1005e0 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.354+0000 7f24a3bae740 1 Processor -- start 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 -- start start 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 0x556e98b63c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b69380 0x556e98b6b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e98ac88d0 con 0x556e98b69380 2024-03-22T02:25:17.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556e98a70e00 con 0x556e98b63820 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24a3bae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556e98ac4880 con 0x556e98b64210 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f2493fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b69380 0x556e98b6b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 0x556e98b63c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45968/0 (socket says 172.21.15.28:45968) 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 0x556e98b63c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56792/0 (socket says 172.21.15.28:56792) 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 -- 172.21.15.28:0/4020560571 learned_addr learned my addr 172.21.15.28:0/4020560571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24937fe640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 0x7f24880a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 -- 172.21.15.28:0/4020560571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 msgr2=0x556e98b63c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 0x556e98b63c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 -- 172.21.15.28:0/4020560571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b69380 msgr2=0x556e98b6b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b69380 0x556e98b6b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249ad9d640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 0x7f24880a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f249a59c640 1 -- 172.21.15.28:0/4020560571 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e987dc3f0 con 0x556e98b64210 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.355+0000 7f24937fe640 1 -- 172.21.15.28:0/2960191833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f24840cfa30 con 0x7f249c1005e0 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.356+0000 7f249a59c640 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f248c0114f0 tx=0x7f248c0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a43b421d4bf8a908 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.356+0000 7f2490ff9640 1 -- 172.21.15.28:0/4020560571 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c01e070 con 0x556e98b64210 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.356+0000 7f2490ff9640 1 -- 172.21.15.28:0/4020560571 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248c0035f0 con 0x556e98b64210 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.356+0000 7f2490ff9640 1 -- 172.21.15.28:0/4020560571 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c00f800 con 0x556e98b64210 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- 172.21.15.28:0/4020560571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 msgr2=0x556e98b68d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f248c0114f0 tx=0x7f248c0119b0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f249ad9d640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 0x7f24880a5460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2494005e30 tx=0x7f249402c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- 172.21.15.28:0/4020560571 shutdown_connections 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b69380 0x556e98b6b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b64210 0x556e98b68d10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- 172.21.15.28:0/4020560571 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b63820 0x556e98b63c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- 172.21.15.28:0/4020560571 >> 172.21.15.28:0/4020560571 conn(0x556e98b5ed90 msgr2=0x556e98b5f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- 172.21.15.28:0/4020560571 shutdown_connections 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- 172.21.15.28:0/4020560571 wait complete. 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- start start 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 Processor -- start 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b6e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b69380 0x556e98b752e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e98b6c330 con 0x556e98b63820 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556e98a84300 con 0x556e98b64210 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556e98b6c4b0 con 0x556e98b69380 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f24a3bae740 10 client.0 _ll_register_callbacks cb 0x556e98abfe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f249a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f249a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37872/0 (socket says 172.21.15.28:37872) 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f249a59c640 1 -- 172.21.15.28:0/3409855937 learned_addr learned my addr 172.21.15.28:0/3409855937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f2493fff640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b6e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.357+0000 7f249ad9d640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b69380 0x556e98b752e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 -- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b69380 msgr2=0x556e98b752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b69380 0x556e98b752e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 -- 172.21.15.28:0/3409855937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 msgr2=0x556e98b6e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b6e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 -- 172.21.15.28:0/3409855937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e988a67d0 con 0x556e98b63820 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f249a59c640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f248c011eb0 tx=0x7f248c0042b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e080f0d360854e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f2481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c026020 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f248c005040 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f248c0035f0 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f248c031050 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f24a3bae740 10 client.18858 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.358+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556e98b78650 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248c0035f0 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f248c021030 con 0x556e98b63820 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 10 client.18858 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 10 client.18858 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 20 client.18858 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b79b10 0x556e98b7bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 4 client.18858 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e98ad5e10 con 0x556e98b79b10 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f2493fff640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b79b10 0x556e98b7bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f2493fff640 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b79b10 0x556e98b7bf80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd12d7a40bcef64f server_cookie=6ef4254c01aa6606 in_seq=0 out_seq=0 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.359+0000 7f2481ffb640 10 client.18858 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.360+0000 7f2481ffb640 1 -- 172.21.15.28:0/3409855937 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e98ad5e10 con 0x556e98b79b10 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.360+0000 7f2481ffb640 10 client.18858 handle_command_reply: tid=0 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.360+0000 7f24a3bae740 1 client.18858 shutdown 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 msgr2=0x556e98b6dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f248c011eb0 tx=0x7f248c0042b0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 shutdown_connections 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b79b10 0x556e98b7bf80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b69380 0x556e98b752e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b6e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f2481ffb640 0 client.18858 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3409855937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b6dd60 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 >> 172.21.15.28:0/3409855937 conn(0x556e98b5ed90 msgr2=0x556e98b5dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 shutdown_connections 2024-03-22T02:25:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 1 -- 172.21.15.28:0/3409855937 wait complete. 2024-03-22T02:25:17.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.361+0000 7f24a3bae740 20 client.18858 trim_cache size 0 max 16384 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 Processor -- start 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 -- start start 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b5ed90 0x556e98b61ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b81850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1ad280 con 0x556e98b63820 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1aab50 con 0x556e98b64210 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f24a3bae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1a89a0 con 0x556e98b5ed90 2024-03-22T02:25:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f2493fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f2493fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37876/0 (socket says 172.21.15.28:37876) 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f2493fff640 1 -- 172.21.15.28:0/3396142511 learned_addr learned my addr 172.21.15.28:0/3396142511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f249a59c640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b5ed90 0x556e98b61ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.375+0000 7f249ad9d640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b81850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 -- 172.21.15.28:0/3396142511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b5ed90 msgr2=0x556e98b61ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b5ed90 0x556e98b61ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 -- 172.21.15.28:0/3396142511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 msgr2=0x556e98b81850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b81850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 -- 172.21.15.28:0/3396142511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e98b5e6e0 con 0x556e98b63820 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f249ad9d640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b81850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2493fff640 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f24840cd120 tx=0x7f24840ccff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc2500bd5229bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2483fff640 1 -- 172.21.15.28:0/3396142511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248409be00 con 0x556e98b63820 2024-03-22T02:25:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2483fff640 1 -- 172.21.15.28:0/3396142511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248409fea0 con 0x556e98b63820 2024-03-22T02:25:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.376+0000 7f2483fff640 1 -- 172.21.15.28:0/3396142511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24840cc570 con 0x556e98b63820 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- 172.21.15.28:0/3396142511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 msgr2=0x556e98b752a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f24840cd120 tx=0x7f24840ccff0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- 172.21.15.28:0/3396142511 shutdown_connections 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b64210 0x556e98b81850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b63820 0x556e98b752a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 --2- 172.21.15.28:0/3396142511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b5ed90 0x556e98b61ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- 172.21.15.28:0/3396142511 >> 172.21.15.28:0/3396142511 conn(0x556e98b64ff0 msgr2=0x556e98b5dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- 172.21.15.28:0/3396142511 shutdown_connections 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- 172.21.15.28:0/3396142511 wait complete. 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 -- start start 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.377+0000 7f24a3bae740 1 Processor -- start 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b63820 0x556e98b732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b64210 0x556e98b742d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1a9010 con 0x556e98b64210 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1aa480 con 0x556e98b5ed90 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f249c1a96e0 con 0x556e98b63820 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 10 client.0 _ll_register_callbacks cb 0x7f248c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b64210 0x556e98b742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f2493fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b63820 0x556e98b732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56836/0 (socket says 172.21.15.28:56836) 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 -- 172.21.15.28:0/2527889581 learned_addr learned my addr 172.21.15.28:0/2527889581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 -- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b63820 msgr2=0x556e98b732c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b63820 0x556e98b732c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 -- 172.21.15.28:0/2527889581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b64210 msgr2=0x556e98b742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b64210 0x556e98b742d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 -- 172.21.15.28:0/2527889581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e98b748c0 con 0x556e98b5ed90 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f249a59c640 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f248c004c90 tx=0x7f248c0093b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ef971a71944f8f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f2481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248c026050 con 0x556e98b5ed90 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f248409fea0 con 0x556e98b5ed90 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.378+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f248c005100 con 0x556e98b5ed90 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f248c01e040 con 0x556e98b5ed90 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 10 client.28685 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556e98be01c0 con 0x556e98b5ed90 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f248409fea0 con 0x556e98b5ed90 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f248c021040 con 0x556e98b5ed90 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 10 client.28685 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 10 client.28685 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 20 client.28685 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b69380 0x556e98be2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 4 client.28685 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556e98ad0160 con 0x556e98b69380 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2493fff640 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b69380 0x556e98be2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2481ffb640 10 client.28685 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.379+0000 7f2493fff640 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b69380 0x556e98be2b70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=60e5070934cf4f3c server_cookie=c4f83060f082f992 in_seq=0 out_seq=0 2024-03-22T02:25:17.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f2481ffb640 1 -- 172.21.15.28:0/2527889581 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556e98ad0160 con 0x556e98b69380 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f2481ffb640 10 client.28685 handle_command_reply: tid=0 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 client.28685 shutdown 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 msgr2=0x556e98b72d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f248c004c90 tx=0x7f248c0093b0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 shutdown_connections 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x556e98b69380 0x556e98be2b70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e98b64210 0x556e98b742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e98b63820 0x556e98b732c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f2481ffb640 0 client.28685 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 --2- 172.21.15.28:0/2527889581 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e98b5ed90 0x556e98b72d80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 >> 172.21.15.28:0/2527889581 conn(0x556e98b64ff0 msgr2=0x556e98b80840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 shutdown_connections 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 1 -- 172.21.15.28:0/2527889581 wait complete. 2024-03-22T02:25:17.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.380+0000 7f24a3bae740 20 client.28685 trim_cache size 0 max 16384 2024-03-22T02:25:17.388 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 msgr2=0x7f24880a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 0x7f24880a5460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2494005e30 tx=0x7f249402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 msgr2=0x7f249c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f248400e9f0 tx=0x7f248400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 shutdown_connections 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24880a2fa0 0x7f24880a5460 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f249c1031b0 0x7f249c19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f249c1005e0 0x7f249c198380 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 --2- 172.21.15.28:0/2960191833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f249c0ffc20 0x7f249c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 >> 172.21.15.28:0/2960191833 conn(0x7f249c0fb780 msgr2=0x7f249c1061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 shutdown_connections 2024-03-22T02:25:17.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.381+0000 7f24a0eeb640 1 -- 172.21.15.28:0/2960191833 wait complete. 2024-03-22T02:25:17.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:25:17.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1194182 11.30 MB/sec execute 479 sec latency 30.875 ms 2024-03-22T02:25:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/1940981125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8101cc0 msgr2=0x7f4eb8102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/1940981125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8102140 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009860 tx=0x7f4ea002bb70 comp rx=0 tx=0).stop 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/1940981125 shutdown_connections 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/1940981125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8102680 0x7f4eb8108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/1940981125 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8102140 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/1940981125 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/1940981125 >> 172.21.15.28:0/1940981125 conn(0x7f4eb80fc040 msgr2=0x7f4eb80fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/1940981125 shutdown_connections 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.507+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/1940981125 wait complete. 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 Processor -- start 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 -- start start 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102680 0x7f4eb819f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb810c750 con 0x7f4eb8101cc0 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb810c5d0 con 0x7f4eb8100ad0 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4ebd8a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb810c8d0 con 0x7f4eb8102680 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37898/0 (socket says 172.21.15.28:37898) 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 -- 172.21.15.28:0/3813660830 learned_addr learned my addr 172.21.15.28:0/3813660830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102680 0x7f4eb819f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 -- 172.21.15.28:0/3813660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102680 msgr2=0x7f4eb819f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102680 0x7f4eb819f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 -- 172.21.15.28:0/3813660830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 msgr2=0x7f4eb8197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eae7fc640 1 -- 172.21.15.28:0/3813660830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb819fba0 con 0x7f4eb8101cc0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.508+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0034b10 tx=0x7f4ea0032480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4ebc8a2640 1 -- 172.21.15.28:0/3813660830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0041020 con 0x7f4eb8101cc0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4eb819fe30 con 0x7f4eb8101cc0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4ebc8a2640 1 -- 172.21.15.28:0/3813660830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ea002c9f0 con 0x7f4eb8101cc0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4ebc8a2640 1 -- 172.21.15.28:0/3813660830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0039040 con 0x7f4eb8101cc0 2024-03-22T02:25:17.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.509+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4eb81a02a0 con 0x7f4eb8101cc0 2024-03-22T02:25:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ebc8a2640 1 -- 172.21.15.28:0/3813660830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4ea0044020 con 0x7f4eb8101cc0 2024-03-22T02:25:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 Processor -- start 2024-03-22T02:25:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 -- start start 2024-03-22T02:25:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520b6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599520138d0 con 0x5599520ae820 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559951fbbe00 con 0x5599520af210 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4ec056d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55995200f880 con 0x5599520b4380 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4eb6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.511+0000 7f4eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56874/0 (socket says 172.21.15.28:56874) 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2647626063 learned_addr learned my addr 172.21.15.28:0/2647626063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2647626063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 msgr2=0x5599520b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2647626063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 msgr2=0x5599520aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4ebc8a2640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 0x7f4ea40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2647626063 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559951d273f0 con 0x5599520af210 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 0x7f4ea40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80114f0 tx=0x7f4ea80119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2903fe5ce68a35f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4ebc8a2640 1 -- 172.21.15.28:0/3813660830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4ea00f0f80 con 0x7f4eb8101cc0 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eadffb640 1 -- 172.21.15.28:0/2647626063 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea801e070 con 0x5599520af210 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eadffb640 1 -- 172.21.15.28:0/2647626063 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ea80035f0 con 0x5599520af210 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.512+0000 7f4eadffb640 1 -- 172.21.15.28:0/2647626063 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea800f800 con 0x5599520af210 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- 172.21.15.28:0/2647626063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 msgr2=0x5599520b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4ea80114f0 tx=0x7f4ea80119b0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4eae7fc640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 0x7f4ea40a53e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009980 tx=0x7f4ea00f4a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- 172.21.15.28:0/2647626063 shutdown_connections 2024-03-22T02:25:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520af210 0x5599520b3d10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2647626063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- 172.21.15.28:0/2647626063 >> 172.21.15.28:0/2647626063 conn(0x5599520a9d90 msgr2=0x5599520aa190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- 172.21.15.28:0/2647626063 shutdown_connections 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- 172.21.15.28:0/2647626063 wait complete. 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.513+0000 7f4ec056d740 1 -- start start 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 Processor -- start 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599520b7330 con 0x5599520af210 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559951fcf300 con 0x5599520b4380 2024-03-22T02:25:17.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5599520b74b0 con 0x5599520ae820 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4ec056d740 10 client.0 _ll_register_callbacks cb 0x55995200ae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37912/0 (socket says 172.21.15.28:37912) 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/682509988 learned_addr learned my addr 172.21.15.28:0/682509988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/682509988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520ae820 msgr2=0x5599520b8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 msgr2=0x5599520c02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520c02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/682509988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559951df17d0 con 0x5599520af210 2024-03-22T02:25:17.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8011eb0 tx=0x7f4ea8004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ffa75b648b90bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.514+0000 7f4e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8027020 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4ea8005250 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ea80035f0 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4ea8032050 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 10 client.18876 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5599520c3610 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ea80035f0 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f4ea800f5c0 con 0x5599520af210 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 10 client.18876 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 10 client.18876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 20 client.18876 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5599520c4b50 0x5599520c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 4 client.18876 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:17.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559952020e10 con 0x5599520c4b50 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.515+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5599520c4b50 0x5599520c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.516+0000 7f4e96ffd640 10 client.18876 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.516+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5599520c4b50 0x5599520c6f80 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9098226a25989080 server_cookie=ec47f1b99d81909f in_seq=0 out_seq=0 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.516+0000 7f4e96ffd640 1 -- 172.21.15.28:0/682509988 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559952020e10 con 0x5599520c4b50 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.516+0000 7f4e96ffd640 10 client.18876 handle_command_reply: tid=0 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.516+0000 7f4ec056d740 1 client.18876 shutdown 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 msgr2=0x5599520b9260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8011eb0 tx=0x7f4ea8004230 comp rx=0 tx=0).stop 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 shutdown_connections 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5599520c4b50 0x5599520c6f80 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520c02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520af210 0x5599520b9260 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 --2- 172.21.15.28:0/682509988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 >> 172.21.15.28:0/682509988 conn(0x5599520a9d90 msgr2=0x5599520a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4e96ffd640 0 client.18876 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 shutdown_connections 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 1 -- 172.21.15.28:0/682509988 wait complete. 2024-03-22T02:25:17.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.517+0000 7f4ec056d740 20 client.18876 trim_cache size 0 max 16384 2024-03-22T02:25:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 Processor -- start 2024-03-22T02:25:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 -- start start 2024-03-22T02:25:17.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520b9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520c0260 0x5599520a8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81ad300 con 0x5599520ae820 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81aabd0 con 0x5599520b4380 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4ec056d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81a8a20 con 0x5599520c0260 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37916/0 (socket says 172.21.15.28:37916) 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4eb77fe640 1 -- 172.21.15.28:0/3925632407 learned_addr learned my addr 172.21.15.28:0/3925632407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520b9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.533+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520c0260 0x5599520a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 -- 172.21.15.28:0/3925632407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520c0260 msgr2=0x5599520a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520c0260 0x5599520a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 -- 172.21.15.28:0/3925632407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 msgr2=0x5599520b9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520b9260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 -- 172.21.15.28:0/3925632407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599520aab00 con 0x5599520ae820 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009980 tx=0x7f4eb0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4c8a7eb6ff6778 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4ead7fa640 1 -- 172.21.15.28:0/3925632407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0004b20 con 0x5599520ae820 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4ead7fa640 1 -- 172.21.15.28:0/3925632407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb0004cc0 con 0x5599520ae820 2024-03-22T02:25:17.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.534+0000 7f4ead7fa640 1 -- 172.21.15.28:0/3925632407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00055f0 con 0x5599520ae820 2024-03-22T02:25:17.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- 172.21.15.28:0/3925632407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 msgr2=0x5599520b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009980 tx=0x7f4eb0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- 172.21.15.28:0/3925632407 shutdown_connections 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520c0260 0x5599520a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520b4380 0x5599520b9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- 172.21.15.28:0/3925632407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520b8d20 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- 172.21.15.28:0/3925632407 >> 172.21.15.28:0/3925632407 conn(0x5599520a9d90 msgr2=0x5599520afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- 172.21.15.28:0/3925632407 shutdown_connections 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- 172.21.15.28:0/3925632407 wait complete. 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- start start 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 Processor -- start 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520bdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520c0260 0x5599520bef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81a9090 con 0x5599520ae820 2024-03-22T02:25:17.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81aa500 con 0x5599520c0260 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb81a9760 con 0x5599520b4380 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.535+0000 7f4ec056d740 10 client.0 _ll_register_callbacks cb 0x7f4ea801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520bdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37924/0 (socket says 172.21.15.28:37924) 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb77fe640 1 -- 172.21.15.28:0/2093308212 learned_addr learned my addr 172.21.15.28:0/2093308212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520c0260 0x5599520bef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520c0260 msgr2=0x5599520bef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520c0260 0x5599520bef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2093308212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 msgr2=0x5599520bda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 -- 172.21.15.28:0/2093308212 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599520bf570 con 0x5599520b4380 2024-03-22T02:25:17.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4eb6ffd640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520bdf70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8022000 tx=0x7f4ea8003f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b4cc2d84be690be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4e96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea803e060 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f4ea800f5c0 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.536+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4eb0004cc0 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4ea8045050 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 10 client.28699 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55995212b4a0 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4eb0004cc0 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f4ea800fc20 con 0x5599520b4380 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 10 client.28699 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 10 client.28699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 20 client.28699 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:17.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55995212bdc0 0x55995212e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 4 client.28699 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55995201b160 con 0x55995212bdc0 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.537+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55995212bdc0 0x55995212e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4eb77fe640 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55995212bdc0 0x55995212e1e0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599431faba6b460a server_cookie=817a421ffd2fcd9b in_seq=0 out_seq=0 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4e96ffd640 10 client.28699 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4e96ffd640 1 -- 172.21.15.28:0/2093308212 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55995201b160 con 0x55995212bdc0 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4e96ffd640 10 client.28699 handle_command_reply: tid=0 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4ec056d740 1 client.28699 shutdown 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 msgr2=0x5599520bdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520bdf70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8022000 tx=0x7f4ea8003f80 comp rx=0 tx=0).stop 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 shutdown_connections 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.538+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55995212bdc0 0x55995212e1e0 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5599520c0260 0x5599520bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5599520b4380 0x5599520bdf70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 --2- 172.21.15.28:0/2093308212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599520ae820 0x5599520bda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4e96ffd640 0 client.28699 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 >> 172.21.15.28:0/2093308212 conn(0x5599520a9d90 msgr2=0x5599520b8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 shutdown_connections 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 1 -- 172.21.15.28:0/2093308212 wait complete. 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ec056d740 20 client.28699 trim_cache size 0 max 16384 2024-03-22T02:25:17.545 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 msgr2=0x7f4ea40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 0x7f4ea40a53e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009980 tx=0x7f4ea00f4a60 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 msgr2=0x7f4eb8198510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0034b10 tx=0x7f4ea0032480 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 shutdown_connections 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4ea40a2f20 0x7f4ea40a53e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4eb8102680 0x7f4eb819f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb8101cc0 0x7f4eb8198510 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 --2- 172.21.15.28:0/3813660830 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4eb8100ad0 0x7f4eb8197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:17.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 >> 172.21.15.28:0/3813660830 conn(0x7f4eb80fc040 msgr2=0x7f4eb8069ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 shutdown_connections 2024-03-22T02:25:17.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:17.539+0000 7f4ebd8a4640 1 -- 172.21.15.28:0/3813660830 wait complete. 2024-03-22T02:25:18.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:17 smithi120 ceph-mon[36803]: pgmap v711: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:25:18.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:17 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2350619559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:17 smithi083 ceph-mon[36340]: pgmap v711: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:25:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:17 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2350619559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:18.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:17 smithi028 ceph-mon[31916]: pgmap v711: 129 pgs: 129 active+clean; 65 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:25:18.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:17 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2350619559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:18.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1196578 11.31 MB/sec execute 480 sec latency 115.256 ms 2024-03-22T02:25:18.552 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:25:18.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:25:18.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:25:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.674+0000 7f347dcb8640 1 -- 172.21.15.28:0/3821992388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 msgr2=0x7f347810a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.674+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3821992388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347810a130 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f3470009ca0 tx=0x7f347002c140 comp rx=0 tx=0).stop 2024-03-22T02:25:18.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.674+0000 7f347dcb8640 1 -- 172.21.15.28:0/3821992388 shutdown_connections 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.674+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3821992388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347810a130 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.674+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3821992388 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3478102ee0 0x7f3478103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3821992388 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f34780ffe70 0x7f3478100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 -- 172.21.15.28:0/3821992388 >> 172.21.15.28:0/3821992388 conn(0x7f34780fb7a0 msgr2=0x7f34780fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 -- 172.21.15.28:0/3821992388 shutdown_connections 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 -- 172.21.15.28:0/3821992388 wait complete. 2024-03-22T02:25:18.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 Processor -- start 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 -- start start 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 0x7f3478197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3478102ee0 0x7f34781983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.675+0000 7f347dcb8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f347dcb8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f347810d820 con 0x7f34781038a0 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f347dcb8640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f347810d6a0 con 0x7f3478102ee0 2024-03-22T02:25:18.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f347dcb8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f347810d9a0 con 0x7f34780ffe70 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37936/0 (socket says 172.21.15.28:37936) 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f34777fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 0x7f3478197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 -- 172.21.15.28:0/3113949875 learned_addr learned my addr 172.21.15.28:0/3113949875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3476ffd640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3478102ee0 0x7f34781983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 -- 172.21.15.28:0/3113949875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 msgr2=0x7f3478197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 0x7f3478197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 -- 172.21.15.28:0/3113949875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3478102ee0 msgr2=0x7f34781983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3478102ee0 0x7f34781983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 -- 172.21.15.28:0/3113949875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347819fa80 con 0x7f34781038a0 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f34777fe640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 0x7f3478197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3477fff640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3470005cc0 tx=0x7f3470035ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.676+0000 7f3474ff9640 1 -- 172.21.15.28:0/3113949875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470039040 con 0x7f34781038a0 2024-03-22T02:25:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.677+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347819fd10 con 0x7f34781038a0 2024-03-22T02:25:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.677+0000 7f3474ff9640 1 -- 172.21.15.28:0/3113949875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3470044070 con 0x7f34781038a0 2024-03-22T02:25:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.677+0000 7f3474ff9640 1 -- 172.21.15.28:0/3113949875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3470004440 con 0x7f34781038a0 2024-03-22T02:25:18.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.677+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34781a01b0 con 0x7f34781038a0 2024-03-22T02:25:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.679+0000 7f3474ff9640 1 -- 172.21.15.28:0/3113949875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f34700045e0 con 0x7f34781038a0 2024-03-22T02:25:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.679+0000 7f3474ff9640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 0x7f34600a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f34777fe640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 0x7f34600a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3474ff9640 1 -- 172.21.15.28:0/3113949875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f34700f1dc0 con 0x7f34781038a0 2024-03-22T02:25:18.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f34777fe640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 0x7f34600a53e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3478069ee0 tx=0x7f3468005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 Processor -- start 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 -- start start 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053de720 0x5604053e0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56040533d570 con 0x5604053d95b0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5604052e5e10 con 0x5604053de720 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.680+0000 7f3480977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560405339520 con 0x5604053d8bc0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37946/0 (socket says 172.21.15.28:37946) 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f3476ffd640 1 -- 172.21.15.28:0/2801552208 learned_addr learned my addr 172.21.15.28:0/2801552208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f3477fff640 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053de720 0x5604053e0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 -- 172.21.15.28:0/2801552208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053de720 msgr2=0x5604053e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053de720 0x5604053e0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 -- 172.21.15.28:0/2801552208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 msgr2=0x5604053de0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 -- 172.21.15.28:0/2801552208 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604051197e0 con 0x5604053d8bc0 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f3476ffd640 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f34777fe640 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053d8fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f34680041f0 tx=0x7f3468014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c12f06a4e805cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f345dffb640 1 -- 172.21.15.28:0/2801552208 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3468026480 con 0x5604053d8bc0 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f345dffb640 1 -- 172.21.15.28:0/2801552208 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346800f960 con 0x5604053d8bc0 2024-03-22T02:25:18.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.681+0000 7f345dffb640 1 -- 172.21.15.28:0/2801552208 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346801c900 con 0x5604053d8bc0 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 -- 172.21.15.28:0/2801552208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 msgr2=0x5604053d8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053d8fe0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f34680041f0 tx=0x7f3468014000 comp rx=0 tx=0).stop 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 -- 172.21.15.28:0/2801552208 shutdown_connections 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053de720 0x5604053e0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053de0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 --2- 172.21.15.28:0/2801552208 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053d8fe0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.682+0000 7f3480977740 1 -- 172.21.15.28:0/2801552208 >> 172.21.15.28:0/2801552208 conn(0x5604053d4130 msgr2=0x5604053d4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.683+0000 7f3480977740 1 -- 172.21.15.28:0/2801552208 shutdown_connections 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.683+0000 7f3480977740 1 -- 172.21.15.28:0/2801552208 wait complete. 2024-03-22T02:25:18.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.683+0000 7f3480977740 1 -- start start 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.683+0000 7f3480977740 1 Processor -- start 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.683+0000 7f3480977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053e2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053de720 0x5604053ea400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5604053e1720 con 0x5604053d95b0 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5604052f9110 con 0x5604053d8bc0 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5604053e18a0 con 0x5604053de720 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 10 client.0 _ll_register_callbacks cb 0x560405335be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37956/0 (socket says 172.21.15.28:37956) 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053e2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3476ffd640 1 -- 172.21.15.28:0/2659475557 learned_addr learned my addr 172.21.15.28:0/2659475557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3477fff640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053de720 0x5604053ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 -- 172.21.15.28:0/2659475557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053de720 msgr2=0x5604053ea400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053de720 0x5604053ea400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 -- 172.21.15.28:0/2659475557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 msgr2=0x5604053e3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 -- 172.21.15.28:0/2659475557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560405179f70 con 0x5604053d8bc0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3476ffd640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f34777fe640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053e2ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3468011ca0 tx=0x7f3468003f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c2e970d9cc44305 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346801c680 con 0x5604053d8bc0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346800f960 con 0x5604053d8bc0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.684+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f346801c820 con 0x5604053d8bc0 2024-03-22T02:25:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f3468038050 con 0x5604053d8bc0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 10 client.28715 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560405011170 con 0x5604053d8bc0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346800f960 con 0x5604053d8bc0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f346802e030 con 0x5604053d8bc0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 10 client.28715 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 10 client.28715 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 20 client.28715 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053ee8f0 0x5604053f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 4 client.28715 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56040534aab0 con 0x5604053ee8f0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.685+0000 7f3476ffd640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053ee8f0 0x5604053f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.686+0000 7f3476ffd640 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053ee8f0 0x5604053f0ce0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2acd42ea6894067d server_cookie=de47e74e0c297cb0 in_seq=0 out_seq=0 2024-03-22T02:25:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.686+0000 7f3456ffd640 10 client.28715 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.686+0000 7f3456ffd640 1 -- 172.21.15.28:0/2659475557 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56040534aab0 con 0x5604053ee8f0 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.686+0000 7f3456ffd640 10 client.28715 handle_command_reply: tid=0 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 client.28715 shutdown 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 msgr2=0x5604053e2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053e2ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3468011ca0 tx=0x7f3468003f20 comp rx=0 tx=0).stop 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 shutdown_connections 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053ee8f0 0x5604053f0ce0 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053de720 0x5604053ea400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053e3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3456ffd640 0 client.28715 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 --2- 172.21.15.28:0/2659475557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053e2ef0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 >> 172.21.15.28:0/2659475557 conn(0x5604053d4130 msgr2=0x5604053d33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 shutdown_connections 2024-03-22T02:25:18.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 1 -- 172.21.15.28:0/2659475557 wait complete. 2024-03-22T02:25:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.687+0000 7f3480977740 20 client.28715 trim_cache size 0 max 16384 2024-03-22T02:25:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 Processor -- start 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 -- start start 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 0x5604053e3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34781ad180 con 0x5604053d95b0 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f34781aaa50 con 0x5604053d8bc0 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3480977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34781a88a0 con 0x5604053d4130 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f34777fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 0x5604053e3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053ea3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56946/0 (socket says 172.21.15.28:56946) 2024-03-22T02:25:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f34777fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 0x5604053e3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46060/0 (socket says 172.21.15.28:46060) 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3476ffd640 1 -- 172.21.15.28:0/3447977877 learned_addr learned my addr 172.21.15.28:0/3447977877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.725+0000 7f3477fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37958/0 (socket says 172.21.15.28:37958) 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 -- 172.21.15.28:0/3447977877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 msgr2=0x5604053e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 0x5604053e3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 -- 172.21.15.28:0/3447977877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 msgr2=0x5604053ea3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053ea3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 -- 172.21.15.28:0/3447977877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604053d9e10 con 0x5604053d95b0 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f3477fff640 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f34700ef4b0 tx=0x7f34700de040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd40d310e0c415c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f345d7fa640 1 -- 172.21.15.28:0/3447977877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34700be7e0 con 0x5604053d95b0 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f345d7fa640 1 -- 172.21.15.28:0/3447977877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34700dce50 con 0x5604053d95b0 2024-03-22T02:25:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.726+0000 7f345d7fa640 1 -- 172.21.15.28:0/3447977877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34700de0b0 con 0x5604053d95b0 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- 172.21.15.28:0/3447977877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 msgr2=0x5604053da000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f34700ef4b0 tx=0x7f34700de040 comp rx=0 tx=0).stop 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- 172.21.15.28:0/3447977877 shutdown_connections 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d95b0 0x5604053da000 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d8bc0 0x5604053ea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 --2- 172.21.15.28:0/3447977877 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d4130 0x5604053e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- 172.21.15.28:0/3447977877 >> 172.21.15.28:0/3447977877 conn(0x5604053de220 msgr2=0x5604053d7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- 172.21.15.28:0/3447977877 shutdown_connections 2024-03-22T02:25:18.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- 172.21.15.28:0/3447977877 wait complete. 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 -- start start 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.727+0000 7f3480977740 1 Processor -- start 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d95b0 0x5604053e9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34781a8f10 con 0x5604053d4130 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f34781aa380 con 0x5604053d95b0 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f34781a95e0 con 0x5604053d8bc0 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3480977740 10 client.0 _ll_register_callbacks cb 0x7f346c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f34777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46062/0 (socket says 172.21.15.28:46062) 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f34777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37974/0 (socket says 172.21.15.28:37974) 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 -- 172.21.15.28:0/3397848474 learned_addr learned my addr 172.21.15.28:0/3397848474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3477fff640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d95b0 0x5604053e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 -- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d95b0 msgr2=0x5604053e9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d95b0 0x5604053e9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 -- 172.21.15.28:0/3397848474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 msgr2=0x5604053e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 -- 172.21.15.28:0/3397848474 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604053e9820 con 0x5604053d8bc0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f34777fe640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3476ffd640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x5604053e4360 tx=0x7f346c012b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc2b1bcdb3cd077 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.728+0000 7f3456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c004250 con 0x5604053d8bc0 2024-03-22T02:25:18.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34700dce50 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f346c0043f0 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f346c030050 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 10 client.28717 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560405454ed0 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34700dce50 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f346c020510 con 0x5604053d8bc0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 10 client.28717 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 10 client.28717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 20 client.28717 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053de720 0x560405457910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 4 client.28717 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560405344e00 con 0x5604053de720 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.729+0000 7f34777fe640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053de720 0x560405457910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.730+0000 7f3456ffd640 10 client.28717 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.730+0000 7f34777fe640 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053de720 0x560405457910 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7db3f088ea1b6bc9 server_cookie=f04943d2f2e61b0 in_seq=0 out_seq=0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3456ffd640 1 -- 172.21.15.28:0/3397848474 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560405344e00 con 0x5604053de720 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3456ffd640 10 client.28717 handle_command_reply: tid=0 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 client.28717 shutdown 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 msgr2=0x5604053e8220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x5604053e4360 tx=0x7f346c012b40 comp rx=0 tx=0).stop 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 shutdown_connections 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5604053de720 0x560405457910 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5604053d95b0 0x5604053e9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5604053d8bc0 0x5604053e8220 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 --2- 172.21.15.28:0/3397848474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5604053d4130 0x5604053e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3456ffd640 0 client.28717 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 >> 172.21.15.28:0/3397848474 conn(0x5604053de220 msgr2=0x5604053d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 shutdown_connections 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 1 -- 172.21.15.28:0/3397848474 wait complete. 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.732+0000 7f3480977740 20 client.28717 trim_cache size 0 max 16384 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c64a9206-ab77-4459-8083-c68210a68213", 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:25:18.738 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 msgr2=0x7f34600a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 0x7f34600a53e0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3478069ee0 tx=0x7f3468005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 msgr2=0x7f347819f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3470005cc0 tx=0x7f3470035ae0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 shutdown_connections 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34600a2f20 0x7f34600a53e0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34781038a0 0x7f347819f410 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3478102ee0 0x7f34781983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 --2- 172.21.15.28:0/3113949875 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34780ffe70 0x7f3478197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 >> 172.21.15.28:0/3113949875 conn(0x7f34780fb7a0 msgr2=0x7f34781061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 shutdown_connections 2024-03-22T02:25:18.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.733+0000 7f347dcb8640 1 -- 172.21.15.28:0/3113949875 wait complete. 2024-03-22T02:25:18.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:25:18.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 -- 172.21.15.28:0/2159148727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50102660 msgr2=0x7fde50108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 --2- 172.21.15.28:0/2159148727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50102660 0x7fde50108ef0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde48009ca0 tx=0x7fde4802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 -- 172.21.15.28:0/2159148727 shutdown_connections 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 --2- 172.21.15.28:0/2159148727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50102660 0x7fde50108ef0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 --2- 172.21.15.28:0/2159148727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 --2- 172.21.15.28:0/2159148727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50100ab0 0x7fde50100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 -- 172.21.15.28:0/2159148727 >> 172.21.15.28:0/2159148727 conn(0x7fde500fc040 msgr2=0x7fde500fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 -- 172.21.15.28:0/2159148727 shutdown_connections 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 -- 172.21.15.28:0/2159148727 wait complete. 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.855+0000 7fde55fa1640 1 Processor -- start 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 -- start start 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 0x7fde501035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010c730 con 0x7fde50101ca0 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010c5b0 con 0x7fde50100ab0 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde55fa1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fde5010c8b0 con 0x7fde50102660 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37988/0 (socket says 172.21.15.28:37988) 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 0x7fde501035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4effd640 1 -- 172.21.15.28:0/424667056 learned_addr learned my addr 172.21.15.28:0/424667056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:56968/0 (socket says 172.21.15.28:56968) 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 -- 172.21.15.28:0/424667056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 msgr2=0x7fde501035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 0x7fde501035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 -- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 msgr2=0x7fde50103080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4f7fe640 1 -- 172.21.15.28:0/424667056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde50103e50 con 0x7fde50100ab0 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4effd640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.856+0000 7fde4ffff640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 0x7fde501035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fde4400e9c0 tx=0x7fde4400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde4cff9640 1 -- 172.21.15.28:0/424667056 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4400ccc0 con 0x7fde50100ab0 2024-03-22T02:25:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde4cff9640 1 -- 172.21.15.28:0/424667056 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde440035f0 con 0x7fde50100ab0 2024-03-22T02:25:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde4cff9640 1 -- 172.21.15.28:0/424667056 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4400ce60 con 0x7fde50100ab0 2024-03-22T02:25:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde501a1fe0 con 0x7fde50100ab0 2024-03-22T02:25:18.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.857+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde501a23f0 con 0x7fde50100ab0 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.859+0000 7fde4cff9640 1 -- 172.21.15.28:0/424667056 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fde440037a0 con 0x7fde50100ab0 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.859+0000 7fde58b89740 1 Processor -- start 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 -- start start 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a65fbda8c0 con 0x55a65fc75430 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a65fb82c20 con 0x55a65fc75e20 2024-03-22T02:25:18.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde58b89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a65fbd6870 con 0x55a65fc7af90 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4cff9640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 0x7fde380a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37996/0 (socket says 172.21.15.28:37996) 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4ffff640 1 -- 172.21.15.28:0/4104482637 learned_addr learned my addr 172.21.15.28:0/4104482637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4ffff640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 0x7fde380a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4cff9640 1 -- 172.21.15.28:0/424667056 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fde440cfca0 con 0x7fde50100ab0 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 -- 172.21.15.28:0/4104482637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 msgr2=0x55a65fc7a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 -- 172.21.15.28:0/4104482637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 msgr2=0x55a65fc75850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4effd640 1 -- 172.21.15.28:0/4104482637 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a65fba7150 con 0x55a65fc7af90 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.860+0000 7fde4ffff640 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.861+0000 7fde4effd640 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7d420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fde40009fd0 tx=0x7fde400119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1a1eebbd5fc4bce server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.861+0000 7fde4ffff640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 0x7fde380a5460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55a65fc76680 tx=0x7fde48032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.861+0000 7fde35ffb640 1 -- 172.21.15.28:0/4104482637 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40005020 con 0x55a65fc7af90 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.861+0000 7fde35ffb640 1 -- 172.21.15.28:0/4104482637 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde400047a0 con 0x55a65fc7af90 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.861+0000 7fde35ffb640 1 -- 172.21.15.28:0/4104482637 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4000f9d0 con 0x55a65fc7af90 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- 172.21.15.28:0/4104482637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 msgr2=0x55a65fc7d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7d420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fde40009fd0 tx=0x7fde400119a0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- 172.21.15.28:0/4104482637 shutdown_connections 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7d420 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- 172.21.15.28:0/4104482637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc75850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- 172.21.15.28:0/4104482637 >> 172.21.15.28:0/4104482637 conn(0x55a65fc709a0 msgr2=0x55a65fc70da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- 172.21.15.28:0/4104482637 shutdown_connections 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- 172.21.15.28:0/4104482637 wait complete. 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- start start 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 Processor -- start 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc86ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a65fc7df40 con 0x55a65fc75430 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a65fb96130 con 0x55a65fc75e20 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.862+0000 7fde58b89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a65fc7e0c0 con 0x55a65fc7af90 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde58b89740 10 client.0 _ll_register_callbacks cb 0x55a65fb95b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38000/0 (socket says 172.21.15.28:38000) 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 -- 172.21.15.28:0/2330164411 learned_addr learned my addr 172.21.15.28:0/2330164411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4ffff640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc86ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4effd640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 -- 172.21.15.28:0/2330164411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 msgr2=0x55a65fc86ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc86ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 -- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 msgr2=0x55a65fc7fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 -- 172.21.15.28:0/2330164411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a65f9b87d0 con 0x55a65fc75430 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4ffff640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc86ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4effd640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fde4409bea0 tx=0x7fde4406ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1494df22217172d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde2effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4409bf60 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde400047a0 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.863+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fde44003b10 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde4409b3b0 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 10 client.18912 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a65f8ee3f0 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde400047a0 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fde44097cb0 con 0x55a65fc75430 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 10 client.18912 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 10 client.18912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 20 client.18912 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fc8b7a0 0x55a65fc8dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 4 client.18912 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:18.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a65fbe7e00 con 0x55a65fc8b7a0 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.864+0000 7fde4effd640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fc8b7a0 0x55a65fc8dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.865+0000 7fde2effd640 10 client.18912 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.865+0000 7fde4effd640 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fc8b7a0 0x55a65fc8dc30 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cbb3e285ef07965 server_cookie=e1ca761c5efa31d5 in_seq=0 out_seq=0 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde2effd640 1 -- 172.21.15.28:0/2330164411 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a65fbe7e00 con 0x55a65fc8b7a0 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde2effd640 10 client.18912 handle_command_reply: tid=0 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 client.18912 shutdown 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 msgr2=0x55a65fc7f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fde4409bea0 tx=0x7fde4406ab50 comp rx=0 tx=0).stop 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 shutdown_connections 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fc8b7a0 0x55a65fc8dc30 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc86ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc7fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde2effd640 0 client.18912 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 --2- 172.21.15.28:0/2330164411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75430 0x55a65fc7f950 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.866+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 >> 172.21.15.28:0/2330164411 conn(0x55a65fc709a0 msgr2=0x55a65fc6fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.867+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 shutdown_connections 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.867+0000 7fde58b89740 1 -- 172.21.15.28:0/2330164411 wait complete. 2024-03-22T02:25:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.867+0000 7fde58b89740 20 client.18912 trim_cache size 0 max 16384 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:25:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:18.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:18 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 Processor -- start 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 -- start start 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75430 0x55a65fc6ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde501af530 con 0x55a65fc75e20 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fde501ace00 con 0x55a65fc75430 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde58b89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fde501aac50 con 0x55a65fc7af90 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38012/0 (socket says 172.21.15.28:38012) 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4effd640 1 -- 172.21.15.28:0/1086375763 learned_addr learned my addr 172.21.15.28:0/1086375763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75430 0x55a65fc6ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.904+0000 7fde4effd640 1 -- 172.21.15.28:0/1086375763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 msgr2=0x55a65fc7fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4effd640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4effd640 1 -- 172.21.15.28:0/1086375763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75430 msgr2=0x55a65fc6ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4effd640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75430 0x55a65fc6ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4effd640 1 -- 172.21.15.28:0/1086375763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a65f8ae0f0 con 0x55a65fc75e20 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde4effd640 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fde40011fd0 tx=0x7fde40004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298ad3290c2fb2c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde357fa640 1 -- 172.21.15.28:0/1086375763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde40005200 con 0x55a65fc75e20 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde357fa640 1 -- 172.21.15.28:0/1086375763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde400053a0 con 0x55a65fc75e20 2024-03-22T02:25:18.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.905+0000 7fde357fa640 1 -- 172.21.15.28:0/1086375763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde400035f0 con 0x55a65fc75e20 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- 172.21.15.28:0/1086375763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 msgr2=0x55a65fc86e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fde40011fd0 tx=0x7fde40004f70 comp rx=0 tx=0).stop 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- 172.21.15.28:0/1086375763 shutdown_connections 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc7af90 0x55a65fc7fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc75e20 0x55a65fc86e90 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 --2- 172.21.15.28:0/1086375763 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75430 0x55a65fc6ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- 172.21.15.28:0/1086375763 >> 172.21.15.28:0/1086375763 conn(0x55a65fc709a0 msgr2=0x55a65fc7f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- 172.21.15.28:0/1086375763 shutdown_connections 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- 172.21.15.28:0/1086375763 wait complete. 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.906+0000 7fde58b89740 1 -- start start 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 Processor -- start 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc75430 0x55a65fc83ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc84420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde501ab2c0 con 0x55a65fc7af90 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fde501ac730 con 0x55a65fc75e20 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fde501ab990 con 0x55a65fc75430 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde58b89740 10 client.0 _ll_register_callbacks cb 0x7fde4409b970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc75430 0x55a65fc83ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde4effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc84420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde4ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38016/0 (socket says 172.21.15.28:38016) 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.907+0000 7fde4ffff640 1 -- 172.21.15.28:0/1344415298 learned_addr learned my addr 172.21.15.28:0/1344415298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 -- 172.21.15.28:0/1344415298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc75430 msgr2=0x55a65fc83ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc75430 0x55a65fc83ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 -- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 msgr2=0x55a65fc84420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc84420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 -- 172.21.15.28:0/1344415298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a65fc85a20 con 0x55a65fc7af90 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4effd640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc84420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde4ffff640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fde480324c0 tx=0x7fde4802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f2ccc72efac3a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde2effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde48039040 con 0x55a65fc7af90 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fde48004040 con 0x55a65fc7af90 2024-03-22T02:25:18.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde400053a0 con 0x55a65fc7af90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fde4805e050 con 0x55a65fc7af90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde58b89740 10 client.18918 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a65fc86da0 con 0x55a65fc7af90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.908+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde400053a0 con 0x55a65fc7af90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fde480508c0 con 0x55a65fc7af90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 10 client.18918 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 10 client.18918 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 20 client.18918 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fcf1f90 0x55a65fcf43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 4 client.18918 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a65fbe2150 con 0x55a65fcf1f90 2024-03-22T02:25:18.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fcf1f90 0x55a65fcf43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde2effd640 10 client.18918 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.909+0000 7fde4f7fe640 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fcf1f90 0x55a65fcf43b0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24b00ed2b27aa947 server_cookie=988afd56b7d3282a in_seq=0 out_seq=0 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde2effd640 1 -- 172.21.15.28:0/1344415298 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a65fbe2150 con 0x55a65fcf1f90 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde2effd640 10 client.18918 handle_command_reply: tid=0 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 client.18918 shutdown 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 msgr2=0x55a65fc85430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fde480324c0 tx=0x7fde4802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 shutdown_connections 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a65fcf1f90 0x55a65fcf43b0 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a65fc7af90 0x55a65fc85430 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde2effd640 0 client.18918 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a65fc75e20 0x55a65fc84420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 --2- 172.21.15.28:0/1344415298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a65fc75430 0x55a65fc83ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 >> 172.21.15.28:0/1344415298 conn(0x55a65fc709a0 msgr2=0x55a65fc6f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 shutdown_connections 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 1 -- 172.21.15.28:0/1344415298 wait complete. 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.912+0000 7fde58b89740 20 client.18918 trim_cache size 0 max 16384 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:25:18.917 INFO:teuthology.orchestra.run.smithi028.stdout: "c64a9206-ab77-4459-8083-c68210a68213": { 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "c64a9206-ab77-4459-8083-c68210a68213", 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:25:18.918 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 msgr2=0x7fde380a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 0x7fde380a5460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x55a65fc76680 tx=0x7fde48032590 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 msgr2=0x7fde50104a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fde4400e9c0 tx=0x7fde4400ee80 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 shutdown_connections 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fde380a2fa0 0x7fde380a5460 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fde50102660 0x7fde501035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde50101ca0 0x7fde50103080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 --2- 172.21.15.28:0/424667056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fde50100ab0 0x7fde50104a40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:18.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 >> 172.21.15.28:0/424667056 conn(0x7fde500fc040 msgr2=0x7fde500fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 shutdown_connections 2024-03-22T02:25:18.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:18.913+0000 7fde55fa1640 1 -- 172.21.15.28:0/424667056 wait complete. 2024-03-22T02:25:18.924 INFO:tasks.cephfs.filesystem:scrub status for tag:c64a9206-ab77-4459-8083-c68210a68213 - {'path': '/', 'tag': 'c64a9206-ab77-4459-8083-c68210a68213', 'options': 'recursive,force'} 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:19.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:18 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:19.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:25:19.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:19.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:18 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:25:19.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1197517 11.29 MB/sec execute 481 sec latency 71.021 ms 2024-03-22T02:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 -- 172.21.15.28:0/2200002753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 msgr2=0x7f25681086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 --2- 172.21.15.28:0/2200002753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f25681086b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f2560009ca0 tx=0x7f256002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 -- 172.21.15.28:0/2200002753 shutdown_connections 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 --2- 172.21.15.28:0/2200002753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f25681086b0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 --2- 172.21.15.28:0/2200002753 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568101460 0x7f25681018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 --2- 172.21.15.28:0/2200002753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568100270 0x7f2568100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.811+0000 7f256f80e640 1 -- 172.21.15.28:0/2200002753 >> 172.21.15.28:0/2200002753 conn(0x7f25680fb7e0 msgr2=0x7f25680fdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- 172.21.15.28:0/2200002753 shutdown_connections 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- 172.21.15.28:0/2200002753 wait complete. 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 Processor -- start 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- start start 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568100270 0x7f2568109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101460 0x7f256819cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f256810c7a0 con 0x7f2568101e20 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f256810c620 con 0x7f2568100270 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256f80e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f256810c920 con 0x7f2568101460 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.812+0000 7f256ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38034/0 (socket says 172.21.15.28:38034) 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568100270 0x7f2568109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 -- 172.21.15.28:0/1500804511 learned_addr learned my addr 172.21.15.28:0/1500804511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256cda9640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101460 0x7f256819cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 -- 172.21.15.28:0/1500804511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101460 msgr2=0x7f256819cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101460 0x7f256819cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 -- 172.21.15.28:0/1500804511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568100270 msgr2=0x7f2568109a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568100270 0x7f2568109a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 -- 172.21.15.28:0/1500804511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256819fa00 con 0x7f2568101e20 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256ddab640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2560037040 tx=0x7f2560032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f255e7fc640 1 -- 172.21.15.28:0/1500804511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560041020 con 0x7f2568101e20 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f255e7fc640 1 -- 172.21.15.28:0/1500804511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2560034cd0 con 0x7f2568101e20 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f256819fc90 con 0x7f2568101e20 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f255e7fc640 1 -- 172.21.15.28:0/1500804511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560039040 con 0x7f2568101e20 2024-03-22T02:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.813+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25681a0180 con 0x7f2568101e20 2024-03-22T02:25:19.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f255e7fc640 1 -- 172.21.15.28:0/1500804511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2560044020 con 0x7f2568101e20 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 Processor -- start 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 -- start start 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f255e7fc640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 0x7f25540a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c5c016040 con 0x561c5c0b14e0 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c5bfbe9e0 con 0x561c5c0b7040 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.816+0000 7f25723f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c5c011ff0 con 0x561c5c0b1ed0 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256ddab640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46136/0 (socket says 172.21.15.28:46136) 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 -- 172.21.15.28:0/259604425 learned_addr learned my addr 172.21.15.28:0/259604425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256cda9640 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256cda9640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 0x7f25540a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f255e7fc640 1 -- 172.21.15.28:0/1500804511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f25600f1630 con 0x7f2568101e20 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 -- 172.21.15.28:0/259604425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 msgr2=0x561c5c0b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 -- 172.21.15.28:0/259604425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 msgr2=0x561c5c0b1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 -- 172.21.15.28:0/259604425 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c5bd2a210 con 0x561c5c0b1ed0 2024-03-22T02:25:19.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256d5aa640 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2564009ca0 tx=0x7f2564011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b856364d66c1a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f256cda9640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 0x7f25540a5480 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x561c5c0b2730 tx=0x7f2558005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f25537fe640 1 -- 172.21.15.28:0/259604425 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564004220 con 0x561c5c0b1ed0 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f25537fe640 1 -- 172.21.15.28:0/259604425 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25640043c0 con 0x561c5c0b1ed0 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.817+0000 7f25537fe640 1 -- 172.21.15.28:0/259604425 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640198b0 con 0x561c5c0b1ed0 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- 172.21.15.28:0/259604425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 msgr2=0x561c5c0b69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2564009ca0 tx=0x7f2564011810 comp rx=0 tx=0).stop 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- 172.21.15.28:0/259604425 shutdown_connections 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0b69d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 --2- 172.21.15.28:0/259604425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- 172.21.15.28:0/259604425 >> 172.21.15.28:0/259604425 conn(0x561c5c0aca50 msgr2=0x561c5c0ace50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- 172.21.15.28:0/259604425 shutdown_connections 2024-03-22T02:25:19.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- 172.21.15.28:0/259604425 wait complete. 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.818+0000 7f25723f8740 1 -- start start 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 Processor -- start 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 0x561c5c0bf890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0bfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561c5c0be400 con 0x561c5c0b7040 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561c5bfd1db0 con 0x561c5c0b14e0 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561c5c0be580 con 0x561c5c0b1ed0 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f25723f8740 10 client.0 _ll_register_callbacks cb 0x561c5bd29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256cda9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0bfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 0x561c5c0bf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38058/0 (socket says 172.21.15.28:38058) 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 -- 172.21.15.28:0/1324797326 learned_addr learned my addr 172.21.15.28:0/1324797326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 -- 172.21.15.28:0/1324797326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 msgr2=0x561c5c0bfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0bfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 -- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 msgr2=0x561c5c0bf890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 0x561c5c0bf890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256ddab640 1 -- 172.21.15.28:0/1324797326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c5bdf6620 con 0x561c5c0b7040 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.819+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 0x561c5c0bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f256ddab640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2560033000 tx=0x7f25600099f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f6a4fcbdb98b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f2550ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25600bd340 con 0x561c5c0b7040 2024-03-22T02:25:19.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f25600f12d0 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25640043c0 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f25600c4070 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f25723f8740 10 client.18930 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561c5c0ca180 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.820+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25640043c0 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f256003c030 con 0x561c5c0b7040 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 10 client.18930 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 10 client.18930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 20 client.18930 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c0cb960 0x561c5c0cde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 4 client.18930 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:19.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561c5c023580 con 0x561c5c0cb960 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.821+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c0cb960 0x561c5c0cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.822+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c0cb960 0x561c5c0cde20 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcc6060b265914c6 server_cookie=aefe3269a9494faf in_seq=0 out_seq=0 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.822+0000 7f2550ff9640 10 client.18930 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.822+0000 7f2550ff9640 1 -- 172.21.15.28:0/1324797326 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561c5c023580 con 0x561c5c0cb960 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.822+0000 7f2550ff9640 10 client.18930 handle_command_reply: tid=0 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.822+0000 7f25723f8740 1 client.18930 shutdown 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 msgr2=0x561c5c0c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2560033000 tx=0x7f25600099f0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 shutdown_connections 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c0cb960 0x561c5c0cde20 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.831 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b7040 0x561c5c0c6e10 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0bfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 --2- 172.21.15.28:0/1324797326 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b14e0 0x561c5c0bf890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f2550ff9640 0 client.18930 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 >> 172.21.15.28:0/1324797326 conn(0x561c5c0aca50 msgr2=0x561c5c0abc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 shutdown_connections 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 1 -- 172.21.15.28:0/1324797326 wait complete. 2024-03-22T02:25:19.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.823+0000 7f25723f8740 20 client.18930 trim_cache size 0 max 16384 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 Processor -- start 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 -- start start 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 0x561c5c0ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25681aae50 con 0x561c5c0b14e0 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25681a8720 con 0x561c5c0b1ed0 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f25723f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25681a6570 con 0x561c5c0b7040 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38064/0 (socket says 172.21.15.28:38064) 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 -- 172.21.15.28:0/1287938616 learned_addr learned my addr 172.21.15.28:0/1287938616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256ddab640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 0x561c5c0ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256cda9640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 -- 172.21.15.28:0/1287938616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 msgr2=0x561c5c0ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 0x561c5c0ab3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 -- 172.21.15.28:0/1287938616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b1ed0 msgr2=0x561c5c0c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.845 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256d5aa640 1 -- 172.21.15.28:0/1287938616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c5c0ad7c0 con 0x561c5c0b14e0 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.839+0000 7f256ddab640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 0x561c5c0ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f256d5aa640 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2564009bc0 tx=0x7f2564005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe98edd2d9eb8f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f2552ffd640 1 -- 172.21.15.28:0/1287938616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2564031070 con 0x561c5c0b14e0 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f2552ffd640 1 -- 172.21.15.28:0/1287938616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2564003bd0 con 0x561c5c0b14e0 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f2552ffd640 1 -- 172.21.15.28:0/1287938616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25640198c0 con 0x561c5c0b14e0 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 -- 172.21.15.28:0/1287938616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 msgr2=0x561c5c0b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f2564009bc0 tx=0x7f2564005d50 comp rx=0 tx=0).stop 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 -- 172.21.15.28:0/1287938616 shutdown_connections 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b7040 0x561c5c0ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 --2- 172.21.15.28:0/1287938616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0b3080 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.846 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 -- 172.21.15.28:0/1287938616 >> 172.21.15.28:0/1287938616 conn(0x561c5c0aca50 msgr2=0x561c5c0b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 -- 172.21.15.28:0/1287938616 shutdown_connections 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.840+0000 7f25723f8740 1 -- 172.21.15.28:0/1287938616 wait complete. 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 -- start start 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 Processor -- start 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0c5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25681a6be0 con 0x561c5c0b14e0 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f25681a8050 con 0x561c5c0b7040 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f25681a72b0 con 0x561c5c0b1ed0 2024-03-22T02:25:19.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f25723f8740 10 client.0 _ll_register_callbacks cb 0x7f25600095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38074/0 (socket says 172.21.15.28:38074) 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 -- 172.21.15.28:0/2859655093 learned_addr learned my addr 172.21.15.28:0/2859655093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256cda9640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256ddab640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 -- 172.21.15.28:0/2859655093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 msgr2=0x561c5c0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 -- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 msgr2=0x561c5c0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.841+0000 7f256d5aa640 1 -- 172.21.15.28:0/2859655093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561c5c0c5c40 con 0x561c5c0b14e0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f256d5aa640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2564009380 tx=0x7f2564009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad6447a66fdebdf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:19.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256403a060 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f25640036f0 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2564003bd0 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2564031050 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 10 client.18936 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x561c5c0c6f50 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2564003bd0 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f256402c040 con 0x561c5c0b14e0 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 10 client.18936 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 10 client.18936 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 20 client.18936 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:19.849 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c132100 0x561c5c134520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 4 client.18936 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.842+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561c5c01d8d0 con 0x561c5c132100 2024-03-22T02:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.843+0000 7f256cda9640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c132100 0x561c5c134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.843+0000 7f256cda9640 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c132100 0x561c5c134520 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c920c38fe7b1f9f server_cookie=ffda92574e14ec5f in_seq=0 out_seq=0 2024-03-22T02:25:19.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:19.843+0000 7f2550ff9640 10 client.18936 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f2550ff9640 1 -- 172.21.15.28:0/2859655093 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561c5c01d8d0 con 0x561c5c132100 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f2550ff9640 10 client.18936 handle_command_reply: tid=0 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 client.18936 shutdown 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 msgr2=0x561c5c0c4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f2564009380 tx=0x7f2564009f10 comp rx=0 tx=0).stop 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 shutdown_connections 2024-03-22T02:25:20.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561c5c132100 0x561c5c134520 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561c5c0b7040 0x561c5c0c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561c5c0b1ed0 0x561c5c0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 --2- 172.21.15.28:0/2859655093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561c5c0b14e0 0x561c5c0c4170 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 >> 172.21.15.28:0/2859655093 conn(0x561c5c0aca50 msgr2=0x561c5c0b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.156+0000 7f2550ff9640 0 client.18936 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 shutdown_connections 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f25723f8740 1 -- 172.21.15.28:0/2859655093 wait complete. 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f25723f8740 20 client.18936 trim_cache size 0 max 16384 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:25:20.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:25:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:25:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:25:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 msgr2=0x7f25540a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:20.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 0x7f25540a5480 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x561c5c0b2730 tx=0x7f2558005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 msgr2=0x7f256819f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2560037040 tx=0x7f2560032f40 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 shutdown_connections 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f25540a2fc0 0x7f25540a5480 secure :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0x561c5c0b2730 tx=0x7f2558005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2568101e20 0x7f256819f140 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2568101460 0x7f256819cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 --2- 172.21.15.28:0/1500804511 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2568100270 0x7f2568109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.157+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 >> 172.21.15.28:0/1500804511 conn(0x7f25680fb7e0 msgr2=0x7f2568104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.158+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 shutdown_connections 2024-03-22T02:25:20.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:20.158+0000 7f256f80e640 1 -- 172.21.15.28:0/1500804511 wait complete. 2024-03-22T02:25:20.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: pgmap v712: 129 pgs: 129 active+clean; 61 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 393 op/s 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:20.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:19 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: pgmap v712: 129 pgs: 129 active+clean; 61 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 393 op/s 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:19 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: pgmap v712: 129 pgs: 129 active+clean; 61 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.8 MiB/s wr, 393 op/s 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:25:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:19 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:25:20.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1198634 11.28 MB/sec execute 482 sec latency 226.178 ms 2024-03-22T02:25:21.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1200921 11.29 MB/sec execute 483 sec latency 9.174 ms 2024-03-22T02:25:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:21 smithi120 ceph-mon[36803]: pgmap v713: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-03-22T02:25:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:21 smithi083 ceph-mon[36340]: pgmap v713: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-03-22T02:25:22.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:21 smithi028 ceph-mon[31916]: pgmap v713: 129 pgs: 129 active+clean; 59 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-03-22T02:25:22.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1203032 11.28 MB/sec execute 484 sec latency 33.567 ms 2024-03-22T02:25:23.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1204687 11.28 MB/sec execute 485 sec latency 37.846 ms 2024-03-22T02:25:24.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:23 smithi120 ceph-mon[36803]: pgmap v714: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:25:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:23 smithi083 ceph-mon[36340]: pgmap v714: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:25:24.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:23 smithi028 ceph-mon[31916]: pgmap v714: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 321 op/s 2024-03-22T02:25:24.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1206521 11.27 MB/sec execute 486 sec latency 93.301 ms 2024-03-22T02:25:25.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:24 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:25:25.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:24 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:25:25.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:24 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:25:25.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1208242 11.27 MB/sec execute 487 sec latency 40.942 ms 2024-03-22T02:25:26.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:25 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:25:26.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:25 smithi120 ceph-mon[36803]: pgmap v715: 129 pgs: 129 active+clean; 54 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 375 op/s 2024-03-22T02:25:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:25 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:25:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:25 smithi083 ceph-mon[36340]: pgmap v715: 129 pgs: 129 active+clean; 54 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 375 op/s 2024-03-22T02:25:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:25 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:25:26.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:25 smithi028 ceph-mon[31916]: pgmap v715: 129 pgs: 129 active+clean; 54 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.6 MiB/s wr, 375 op/s 2024-03-22T02:25:26.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1210769 11.28 MB/sec execute 488 sec latency 13.120 ms 2024-03-22T02:25:27.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1212056 11.27 MB/sec execute 489 sec latency 32.021 ms 2024-03-22T02:25:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:27 smithi120 ceph-mon[36803]: pgmap v716: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 341 op/s 2024-03-22T02:25:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:27 smithi083 ceph-mon[36340]: pgmap v716: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 341 op/s 2024-03-22T02:25:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:27 smithi028 ceph-mon[31916]: pgmap v716: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.0 MiB/s wr, 341 op/s 2024-03-22T02:25:28.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1214661 11.28 MB/sec execute 490 sec latency 12.111 ms 2024-03-22T02:25:29.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1215548 11.26 MB/sec execute 491 sec latency 95.459 ms 2024-03-22T02:25:30.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1218358 11.27 MB/sec execute 492 sec latency 23.270 ms 2024-03-22T02:25:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:30 smithi120 ceph-mon[36803]: pgmap v717: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:30 smithi083 ceph-mon[36340]: pgmap v717: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:30 smithi028 ceph-mon[31916]: pgmap v717: 129 pgs: 129 active+clean; 55 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:31.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1219916 11.26 MB/sec execute 493 sec latency 28.849 ms 2024-03-22T02:25:32.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1222224 11.27 MB/sec execute 494 sec latency 13.057 ms 2024-03-22T02:25:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:32 smithi120 ceph-mon[36803]: pgmap v718: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 364 op/s 2024-03-22T02:25:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:32 smithi083 ceph-mon[36340]: pgmap v718: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 364 op/s 2024-03-22T02:25:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:32 smithi028 ceph-mon[31916]: pgmap v718: 129 pgs: 129 active+clean; 58 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.4 MiB/s wr, 364 op/s 2024-03-22T02:25:33.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- 172.21.15.28:0/1714778164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 msgr2=0x7fd730109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 --2- 172.21.15.28:0/1714778164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730109ce0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fd728009ca0 tx=0x7fd72802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- 172.21.15.28:0/1714778164 shutdown_connections 2024-03-22T02:25:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 --2- 172.21.15.28:0/1714778164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730109ce0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 --2- 172.21.15.28:0/1714778164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730104f60 0x7fd730107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 --2- 172.21.15.28:0/1714778164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730102630 0x7fd730104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- 172.21.15.28:0/1714778164 >> 172.21.15.28:0/1714778164 conn(0x7fd7300fc020 msgr2=0x7fd7300fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- 172.21.15.28:0/1714778164 shutdown_connections 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- 172.21.15.28:0/1714778164 wait complete. 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 Processor -- start 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.252+0000 7fd736132640 1 -- start start 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730102630 0x7fd730101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730104f60 0x7fd7301005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd73010d850 con 0x7fd730107890 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd73010d6d0 con 0x7fd730104f60 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd73010d9d0 con 0x7fd730102630 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60764/0 (socket says 172.21.15.28:60764) 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 -- 172.21.15.28:0/3368533399 learned_addr learned my addr 172.21.15.28:0/3368533399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730102630 0x7fd730101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72effd640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730104f60 0x7fd7301005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 -- 172.21.15.28:0/3368533399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730102630 msgr2=0x7fd730101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730102630 0x7fd730101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 -- 172.21.15.28:0/3368533399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730104f60 msgr2=0x7fd7301005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730104f60 0x7fd7301005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 -- 172.21.15.28:0/3368533399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7301013b0 con 0x7fd730107890 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd72802c630 tx=0x7fd728032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72cff9640 1 -- 172.21.15.28:0/3368533399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728041020 con 0x7fd730107890 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72cff9640 1 -- 172.21.15.28:0/3368533399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd728034cd0 con 0x7fd730107890 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7301a1ed0 con 0x7fd730107890 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd72cff9640 1 -- 172.21.15.28:0/3368533399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd728039040 con 0x7fd730107890 2024-03-22T02:25:33.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.253+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7301a2330 con 0x7fd730107890 2024-03-22T02:25:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.256+0000 7fd72cff9640 1 -- 172.21.15.28:0/3368533399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd728044020 con 0x7fd730107890 2024-03-22T02:25:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 Processor -- start 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 -- start start 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72cff9640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 0x7fd7180a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x5575186889d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 0x55751868b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5575185e8040 con 0x557518683ed0 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5575185909e0 con 0x5575186834e0 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd738df1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5575185e3ff0 con 0x557518689040 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 0x55751868b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41224/0 (socket says 172.21.15.28:41224) 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 -- 172.21.15.28:0/932120156 learned_addr learned my addr 172.21.15.28:0/932120156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72effd640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 0x7fd7180a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72effd640 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x5575186889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72cff9640 1 -- 172.21.15.28:0/3368533399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd7280f15e0 con 0x7fd730107890 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 -- 172.21.15.28:0/932120156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 msgr2=0x55751868b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 0x55751868b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 -- 172.21.15.28:0/932120156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 msgr2=0x5575186889d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x5575186889d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72f7fe640 1 -- 172.21.15.28:0/932120156 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575182fc210 con 0x5575186834e0 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72ffff640 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 0x55751868b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.257+0000 7fd72effd640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 0x7fd7180a5480 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd724009fd0 tx=0x7fd724009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd720009cf0 tx=0x7fd720011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd0a3eb1630dc2e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd715ffb640 1 -- 172.21.15.28:0/932120156 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720004230 con 0x5575186834e0 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd715ffb640 1 -- 172.21.15.28:0/932120156 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7200043d0 con 0x5575186834e0 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd715ffb640 1 -- 172.21.15.28:0/932120156 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720019800 con 0x5575186834e0 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 -- 172.21.15.28:0/932120156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 msgr2=0x557518683900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd720009cf0 tx=0x7fd720011810 comp rx=0 tx=0).stop 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 -- 172.21.15.28:0/932120156 shutdown_connections 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557518689040 0x55751868b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x5575186889d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 --2- 172.21.15.28:0/932120156 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518683900 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.258+0000 7fd738df1740 1 -- 172.21.15.28:0/932120156 >> 172.21.15.28:0/932120156 conn(0x55751867ea50 msgr2=0x55751867ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- 172.21.15.28:0/932120156 shutdown_connections 2024-03-22T02:25:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- 172.21.15.28:0/932120156 wait complete. 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- start start 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 Processor -- start 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x55751868d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x55751868d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55751868bff0 con 0x557518689040 2024-03-22T02:25:33.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5575185a3db0 con 0x557518683ed0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55751868c170 con 0x5575186834e0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd738df1740 10 client.0 _ll_register_callbacks cb 0x5575182fba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x55751868d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x55751868d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60788/0 (socket says 172.21.15.28:60788) 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x55751868d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56886/0 (socket says 172.21.15.28:56886) 2024-03-22T02:25:33.274 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.259+0000 7fd72ffff640 1 -- 172.21.15.28:0/2192232963 learned_addr learned my addr 172.21.15.28:0/2192232963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 -- 172.21.15.28:0/2192232963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 msgr2=0x55751868d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x55751868d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 -- 172.21.15.28:0/2192232963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 msgr2=0x5575186949d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 -- 172.21.15.28:0/2192232963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575183c8620 con 0x557518683ed0 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72ffff640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd72effd640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x55751868d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5575186842d0 tx=0x7fd72401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7137ad1637ac06bc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd70effd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72400fe60 con 0x557518683ed0 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7200043d0 con 0x557518683ed0 2024-03-22T02:25:33.275 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd724003800 con 0x557518683ed0 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd7240044c0 con 0x557518683ed0 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd738df1740 10 client.28769 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557518697e00 con 0x557518683ed0 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.260+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7200043d0 con 0x557518683ed0 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fd72401fb90 con 0x557518683ed0 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 10 client.28769 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 10 client.28769 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 20 client.28769 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5575186994b0 0x55751869b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 4 client.28769 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:33.276 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575185f5580 con 0x5575186994b0 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5575186994b0 0x55751869b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd70effd640 10 client.28769 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.261+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5575186994b0 0x55751869b8e0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e50cea14d187e482 server_cookie=fc6a9e9185fa1f9c in_seq=0 out_seq=0 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.262+0000 7fd70effd640 1 -- 172.21.15.28:0/2192232963 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575185f5580 con 0x5575186994b0 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.262+0000 7fd70effd640 10 client.28769 handle_command_reply: tid=0 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.262+0000 7fd738df1740 1 client.28769 shutdown 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.262+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 msgr2=0x55751868d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.262+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x55751868d990 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5575186842d0 tx=0x7fd72401d000 comp rx=0 tx=0).stop 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 shutdown_connections 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5575186994b0 0x55751869b8e0 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518689040 0x5575186949d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.277 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x55751868d990 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd70effd640 0 client.28769 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 --2- 172.21.15.28:0/2192232963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x55751868d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 >> 172.21.15.28:0/2192232963 conn(0x55751867ea50 msgr2=0x55751867dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 shutdown_connections 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 1 -- 172.21.15.28:0/2192232963 wait complete. 2024-03-22T02:25:33.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.263+0000 7fd738df1740 20 client.28769 trim_cache size 0 max 16384 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 Processor -- start 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 -- start start 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55751867ea50 0x557518684760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518694990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301af440 con 0x557518683ed0 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301acd10 con 0x5575186834e0 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd738df1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301aab60 con 0x55751867ea50 2024-03-22T02:25:33.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60790/0 (socket says 172.21.15.28:60790) 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 -- 172.21.15.28:0/3914812691 learned_addr learned my addr 172.21.15.28:0/3914812691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72effd640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518694990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55751867ea50 0x557518684760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 -- 172.21.15.28:0/3914812691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55751867ea50 msgr2=0x557518684760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55751867ea50 0x557518684760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 -- 172.21.15.28:0/3914812691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 msgr2=0x557518694990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518694990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 -- 172.21.15.28:0/3914812691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55751868c510 con 0x557518683ed0 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72effd640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518694990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd72ffff640 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd7280bd8c0 tx=0x7fd7280c4040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd42d5c2e8b762b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.279+0000 7fd7157fa640 1 -- 172.21.15.28:0/3914812691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280d6db0 con 0x557518683ed0 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd7157fa640 1 -- 172.21.15.28:0/3914812691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7280f1070 con 0x557518683ed0 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd7157fa640 1 -- 172.21.15.28:0/3914812691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7280d6bd0 con 0x557518683ed0 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 -- 172.21.15.28:0/3914812691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 msgr2=0x557518684ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7fd7280bd8c0 tx=0x7fd7280c4040 comp rx=0 tx=0).stop 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 -- 172.21.15.28:0/3914812691 shutdown_connections 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557518683ed0 0x557518684ca0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5575186834e0 0x557518694990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 --2- 172.21.15.28:0/3914812691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55751867ea50 0x557518684760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 -- 172.21.15.28:0/3914812691 >> 172.21.15.28:0/3914812691 conn(0x55751868d010 msgr2=0x557518682540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 -- 172.21.15.28:0/3914812691 shutdown_connections 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.280+0000 7fd738df1740 1 -- 172.21.15.28:0/3914812691 wait complete. 2024-03-22T02:25:33.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 -- start start 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 Processor -- start 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x5575186925a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x557518693540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301ab1d0 con 0x55751867ea50 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301ac640 con 0x557518683ed0 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7301ab8a0 con 0x5575186834e0 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd738df1740 10 client.0 _ll_register_callbacks cb 0x7fd724009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60796/0 (socket says 172.21.15.28:60796) 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x5575186925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 -- 172.21.15.28:0/2968209439 learned_addr learned my addr 172.21.15.28:0/2968209439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72ffff640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x557518693540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 -- 172.21.15.28:0/2968209439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 msgr2=0x5575186925a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x5575186925a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 -- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 msgr2=0x557518693540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x557518693540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72f7fe640 1 -- 172.21.15.28:0/2968209439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557518693b30 con 0x55751867ea50 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.281+0000 7fd72ffff640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x557518693540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd72f7fe640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fd720009730 tx=0x7fd720011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94d18a403e7d6ab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:33.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720009ea0 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fd72002b350 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7280f1070 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fd72003b050 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 10 client.18954 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557518708030 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7280f1070 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fd720003680 con 0x55751867ea50 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 10 client.18954 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 10 client.18954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 20 client.18954 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557518689040 0x55751870a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 4 client.18954 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:25:33.290 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.282+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575185ef8d0 con 0x557518689040 2024-03-22T02:25:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.283+0000 7fd72effd640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557518689040 0x55751870a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.283+0000 7fd70effd640 10 client.18954 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:33.291 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.283+0000 7fd72effd640 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557518689040 0x55751870a7f0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83db9f5580072203 server_cookie=d5990e351086a5e8 in_seq=0 out_seq=0 2024-03-22T02:25:33.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1224244 11.26 MB/sec execute 495 sec latency 114.565 ms 2024-03-22T02:25:33.633 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.632+0000 7fd70effd640 1 -- 172.21.15.28:0/2968209439 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575185ef8d0 con 0x557518689040 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.632+0000 7fd70effd640 10 client.18954 handle_command_reply: tid=0 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.632+0000 7fd738df1740 1 client.18954 shutdown 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 msgr2=0x557518692060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fd720009730 tx=0x7fd720011dc0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 shutdown_connections 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557518689040 0x55751870a7f0 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd70effd640 0 client.18954 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557518683ed0 0x557518693540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5575186834e0 0x5575186925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.634 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 --2- 172.21.15.28:0/2968209439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55751867ea50 0x557518692060 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 >> 172.21.15.28:0/2968209439 conn(0x55751868d010 msgr2=0x55751867d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 shutdown_connections 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 1 -- 172.21.15.28:0/2968209439 wait complete. 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd738df1740 20 client.18954 trim_cache size 0 max 16384 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:25:33.635 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:25:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 msgr2=0x7fd7180a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.633+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 0x7fd7180a5480 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd724009fd0 tx=0x7fd724009b80 comp rx=0 tx=0).stop 2024-03-22T02:25:33.636 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 msgr2=0x7fd730100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fd72802c630 tx=0x7fd728032ea0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 shutdown_connections 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7180a2fc0 0x7fd7180a5480 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd730107890 0x7fd730100b20 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd730104f60 0x7fd7301005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 --2- 172.21.15.28:0/3368533399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd730102630 0x7fd730101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 >> 172.21.15.28:0/3368533399 conn(0x7fd7300fc020 msgr2=0x7fd730103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 shutdown_connections 2024-03-22T02:25:33.637 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:33.634+0000 7fd736132640 1 -- 172.21.15.28:0/3368533399 wait complete. 2024-03-22T02:25:34.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1225533 11.26 MB/sec execute 496 sec latency 216.272 ms 2024-03-22T02:25:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:34 smithi120 ceph-mon[36803]: pgmap v719: 129 pgs: 129 active+clean; 59 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:34 smithi083 ceph-mon[36340]: pgmap v719: 129 pgs: 129 active+clean; 59 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:34 smithi028 ceph-mon[31916]: pgmap v719: 129 pgs: 129 active+clean; 59 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.5 MiB/s wr, 355 op/s 2024-03-22T02:25:35.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1227442 11.25 MB/sec execute 497 sec latency 34.361 ms 2024-03-22T02:25:36.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1229396 11.25 MB/sec execute 498 sec latency 13.006 ms 2024-03-22T02:25:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:36 smithi120 ceph-mon[36803]: pgmap v720: 129 pgs: 129 active+clean; 57 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 383 op/s 2024-03-22T02:25:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:36 smithi083 ceph-mon[36340]: pgmap v720: 129 pgs: 129 active+clean; 57 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 383 op/s 2024-03-22T02:25:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:36 smithi028 ceph-mon[31916]: pgmap v720: 129 pgs: 129 active+clean; 57 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 383 op/s 2024-03-22T02:25:37.435 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1231853 11.26 MB/sec execute 499 sec latency 30.405 ms 2024-03-22T02:25:38.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1233426 11.25 MB/sec execute 500 sec latency 41.817 ms 2024-03-22T02:25:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:38 smithi120 ceph-mon[36803]: pgmap v721: 129 pgs: 129 active+clean; 53 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 314 op/s 2024-03-22T02:25:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:38 smithi083 ceph-mon[36340]: pgmap v721: 129 pgs: 129 active+clean; 53 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 314 op/s 2024-03-22T02:25:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:38 smithi028 ceph-mon[31916]: pgmap v721: 129 pgs: 129 active+clean; 53 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.7 MiB/s wr, 314 op/s 2024-03-22T02:25:39.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1235741 11.25 MB/sec execute 501 sec latency 55.568 ms 2024-03-22T02:25:40.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1237122 11.25 MB/sec execute 502 sec latency 40.863 ms 2024-03-22T02:25:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:40 smithi120 ceph-mon[36803]: pgmap v722: 129 pgs: 129 active+clean; 56 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 402 op/s 2024-03-22T02:25:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:40 smithi083 ceph-mon[36340]: pgmap v722: 129 pgs: 129 active+clean; 56 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 402 op/s 2024-03-22T02:25:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:40 smithi028 ceph-mon[31916]: pgmap v722: 129 pgs: 129 active+clean; 56 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.4 MiB/s wr, 402 op/s 2024-03-22T02:25:41.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1239783 11.26 MB/sec execute 503 sec latency 18.691 ms 2024-03-22T02:25:42.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1241818 11.25 MB/sec execute 504 sec latency 31.887 ms 2024-03-22T02:25:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:42 smithi120 ceph-mon[36803]: pgmap v723: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:25:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:42 smithi083 ceph-mon[36340]: pgmap v723: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:25:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:42 smithi028 ceph-mon[31916]: pgmap v723: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.8 MiB/s wr, 385 op/s 2024-03-22T02:25:43.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1243623 11.25 MB/sec execute 505 sec latency 9.314 ms 2024-03-22T02:25:44.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1244982 11.24 MB/sec execute 506 sec latency 47.693 ms 2024-03-22T02:25:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:44 smithi120 ceph-mon[36803]: pgmap v724: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 343 op/s 2024-03-22T02:25:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:44 smithi083 ceph-mon[36340]: pgmap v724: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 343 op/s 2024-03-22T02:25:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:44 smithi028 ceph-mon[31916]: pgmap v724: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 343 op/s 2024-03-22T02:25:45.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1247051 11.24 MB/sec execute 507 sec latency 10.495 ms 2024-03-22T02:25:46.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1249467 11.24 MB/sec execute 508 sec latency 27.240 ms 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.724+0000 7f14ddf79640 1 -- 172.21.15.28:0/2150758926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 msgr2=0x7f14d8107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.724+0000 7f14ddf79640 1 --2- 172.21.15.28:0/2150758926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d8107350 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f14cc009900 tx=0x7f14cc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 -- 172.21.15.28:0/2150758926 shutdown_connections 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 --2- 172.21.15.28:0/2150758926 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8107890 0x7f14d8109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 --2- 172.21.15.28:0/2150758926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d8107350 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 --2- 172.21.15.28:0/2150758926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8102630 0x7f14d8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 -- 172.21.15.28:0/2150758926 >> 172.21.15.28:0/2150758926 conn(0x7f14d80fc020 msgr2=0x7f14d80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 -- 172.21.15.28:0/2150758926 shutdown_connections 2024-03-22T02:25:46.729 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 -- 172.21.15.28:0/2150758926 wait complete. 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 Processor -- start 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.725+0000 7f14ddf79640 1 -- start start 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8102630 0x7f14d8101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8107890 0x7f14d8100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14d810d840 con 0x7f14d8104f60 2024-03-22T02:25:46.730 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14d810d6c0 con 0x7f14d8102630 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14d810d9c0 con 0x7f14d8107890 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34968/0 (socket says 172.21.15.28:34968) 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 -- 172.21.15.28:0/3721294928 learned_addr learned my addr 172.21.15.28:0/3721294928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d77fe640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8102630 0x7f14d8101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d7fff640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8107890 0x7f14d8100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 -- 172.21.15.28:0/3721294928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8107890 msgr2=0x7f14d8100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8107890 0x7f14d8100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 -- 172.21.15.28:0/3721294928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8102630 msgr2=0x7f14d8101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8102630 0x7f14d8101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 -- 172.21.15.28:0/3721294928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d81013b0 con 0x7f14d8104f60 2024-03-22T02:25:46.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f14cc000c00 tx=0x7f14cc005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d4ff9640 1 -- 172.21.15.28:0/3721294928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc041070 con 0x7f14d8104f60 2024-03-22T02:25:46.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d4ff9640 1 -- 172.21.15.28:0/3721294928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14cc0044d0 con 0x7f14d8104f60 2024-03-22T02:25:46.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d81a1f10 con 0x7f14d8104f60 2024-03-22T02:25:46.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.726+0000 7f14d4ff9640 1 -- 172.21.15.28:0/3721294928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14cc034440 con 0x7f14d8104f60 2024-03-22T02:25:46.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.727+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d81a2370 con 0x7f14d8104f60 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.728+0000 7f14d4ff9640 1 -- 172.21.15.28:0/3721294928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f14cc044070 con 0x7f14d8104f60 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d4ff9640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 0x7f14c00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 Processor -- start 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 -- start start 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d77fe640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 0x7f14c00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d4ff9640 1 -- 172.21.15.28:0/3721294928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f14cc0f1240 con 0x7f14d8104f60 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d79a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d79f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55666d6ff040 con 0x55666d7a0040 2024-03-22T02:25:46.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55666d6a79e0 con 0x55666d79aed0 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14e0c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55666d6faff0 con 0x55666d79a4e0 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d79f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34970/0 (socket says 172.21.15.28:34970) 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d7fff640 1 -- 172.21.15.28:0/2575849076 learned_addr learned my addr 172.21.15.28:0/2575849076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.729+0000 7f14d77fe640 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d79a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 -- 172.21.15.28:0/2575849076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 msgr2=0x55666d79a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d79a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 -- 172.21.15.28:0/2575849076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 msgr2=0x55666d79f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d79f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 -- 172.21.15.28:0/2575849076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55666d413210 con 0x55666d7a0040 2024-03-22T02:25:46.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d77fe640 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d79a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d77fe640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 0x7f14c00a5480 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f14c8011550 tx=0x7f14c8011950 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14d7fff640 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f14d0010370 tx=0x7f14d0010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0698d07a1d495b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14bdffb640 1 -- 172.21.15.28:0/2575849076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0016070 con 0x55666d7a0040 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14bdffb640 1 -- 172.21.15.28:0/2575849076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d0004d20 con 0x55666d7a0040 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.730+0000 7f14bdffb640 1 -- 172.21.15.28:0/2575849076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0005420 con 0x55666d7a0040 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- 172.21.15.28:0/2575849076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 msgr2=0x55666d7a24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f14d0010370 tx=0x7f14d0010830 comp rx=0 tx=0).stop 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- 172.21.15.28:0/2575849076 shutdown_connections 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7a24d0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d79f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 --2- 172.21.15.28:0/2575849076 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d79a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- 172.21.15.28:0/2575849076 >> 172.21.15.28:0/2575849076 conn(0x55666d795a50 msgr2=0x55666d795e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- 172.21.15.28:0/2575849076 shutdown_connections 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- 172.21.15.28:0/2575849076 wait complete. 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 -- start start 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 Processor -- start 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.731+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d7a4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55666d7a2ff0 con 0x55666d7a0040 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55666d6badb0 con 0x55666d79aed0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55666d7a3170 con 0x55666d79a4e0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 10 client.0 _ll_register_callbacks cb 0x55666d412a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d7a4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d7a4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34176/0 (socket says 172.21.15.28:34176) 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34972/0 (socket says 172.21.15.28:34972) 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d77fe640 1 -- 172.21.15.28:0/3045741964 learned_addr learned my addr 172.21.15.28:0/3045741964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 -- 172.21.15.28:0/3045741964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 msgr2=0x55666d7a4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d7a4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 -- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 msgr2=0x55666d7a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 -- 172.21.15.28:0/3045741964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55666d4df620 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14d7fff640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x55666d7969c0 tx=0x7f14d0004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217ab501f818bc92 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0029060 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f14d0003c00 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14d0004d20 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f14d0030050 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 10 client.18966 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.732+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55666d7aed70 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d0004d20 con 0x55666d7a0040 2024-03-22T02:25:46.748 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f14d000f040 con 0x55666d7a0040 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 10 client.18966 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 10 client.18966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 20 client.18966 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d7b0170 0x55666d7b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 4 client.18966 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55666d70c580 con 0x55666d7b0170 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14d77fe640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d7b0170 0x55666d7b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14d77fe640 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d7b0170 0x55666d7b2560 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b56564acd1d66844 server_cookie=b127531354e3c10b in_seq=0 out_seq=0 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.733+0000 7f14b6ffd640 10 client.18966 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14b6ffd640 1 -- 172.21.15.28:0/3045741964 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55666d70c580 con 0x55666d7b0170 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14b6ffd640 10 client.18966 handle_command_reply: tid=0 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 client.18966 shutdown 2024-03-22T02:25:46.749 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 msgr2=0x55666d7aba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x55666d7969c0 tx=0x7f14d0004c50 comp rx=0 tx=0).stop 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 shutdown_connections 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d7b0170 0x55666d7b2560 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d7a0040 0x55666d7aba00 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 --2- 172.21.15.28:0/3045741964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d79a4e0 0x55666d7a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14b6ffd640 0 client.18966 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 >> 172.21.15.28:0/3045741964 conn(0x55666d795a50 msgr2=0x55666d794c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 shutdown_connections 2024-03-22T02:25:46.750 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.734+0000 7f14e0c34740 1 -- 172.21.15.28:0/3045741964 wait complete. 2024-03-22T02:25:46.751 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.735+0000 7f14e0c34740 20 client.18966 trim_cache size 0 max 16384 2024-03-22T02:25:46.751 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:46 smithi120 ceph-mon[36803]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-03-22T02:25:46.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 Processor -- start 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 -- start start 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79a4e0 0x55666d794ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 0x55666d795010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81af450 con 0x55666d79aed0 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81acd20 con 0x55666d79a4e0 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.749+0000 7f14e0c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81aab70 con 0x55666d7a0040 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 0x55666d795010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34180/0 (socket says 172.21.15.28:34180) 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 -- 172.21.15.28:0/1545103207 learned_addr learned my addr 172.21.15.28:0/1545103207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:46.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 0x55666d795010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34982/0 (socket says 172.21.15.28:34982) 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d77fe640 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79a4e0 0x55666d794ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 -- 172.21.15.28:0/1545103207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79a4e0 msgr2=0x55666d794ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79a4e0 0x55666d794ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 -- 172.21.15.28:0/1545103207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 msgr2=0x55666d795010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 0x55666d795010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 -- 172.21.15.28:0/1545103207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55666d7967c0 con 0x55666d7a0040 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14d7fff640 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009640 tx=0x7f14d0010e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71652d830d8aed6d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14bd7fa640 1 -- 172.21.15.28:0/1545103207 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0029030 con 0x55666d7a0040 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14bd7fa640 1 -- 172.21.15.28:0/1545103207 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d00040e0 con 0x55666d7a0040 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.750+0000 7f14bd7fa640 1 -- 172.21.15.28:0/1545103207 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0016070 con 0x55666d7a0040 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- 172.21.15.28:0/1545103207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 msgr2=0x55666d7ab9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f14d0009640 tx=0x7f14d0010e60 comp rx=0 tx=0).stop 2024-03-22T02:25:46.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- 172.21.15.28:0/1545103207 shutdown_connections 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7ab9c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79aed0 0x55666d795010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1545103207 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79a4e0 0x55666d794ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- 172.21.15.28:0/1545103207 >> 172.21.15.28:0/1545103207 conn(0x55666d795a50 msgr2=0x55666d7943c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- 172.21.15.28:0/1545103207 shutdown_connections 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- 172.21.15.28:0/1545103207 wait complete. 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.751+0000 7f14e0c34740 1 -- start start 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 Processor -- start 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7aa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81ab1e0 con 0x55666d79a4e0 2024-03-22T02:25:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81ac650 con 0x55666d79aed0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f14d81ab8b0 con 0x55666d7a0040 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14e0c34740 10 client.0 _ll_register_callbacks cb 0x7f14cc005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34992/0 (socket says 172.21.15.28:34992) 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 -- 172.21.15.28:0/1705550405 learned_addr learned my addr 172.21.15.28:0/1705550405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d7fff640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7aa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 -- 172.21.15.28:0/1705550405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 msgr2=0x55666d7aa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7aa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 -- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 msgr2=0x55666d7a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 -- 172.21.15.28:0/1705550405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55666d7aa910 con 0x55666d79a4e0 2024-03-22T02:25:46.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d7fff640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14d77fe640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f14c8013010 tx=0x7f14c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed3491aa77c647d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.752+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c801a040 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f14c8009040 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14d00040e0 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f14c803a050 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 10 client.18972 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55666d7abc90 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d00040e0 con 0x55666d79a4e0 2024-03-22T02:25:46.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f14c800f560 con 0x55666d79a4e0 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 10 client.18972 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 10 client.18972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 20 client.18972 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d816b40 0x55666d818f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 4 client.18972 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55666d7a4270 con 0x55666d816b40 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.753+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d816b40 0x55666d818f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.754+0000 7f14d6ffd640 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d816b40 0x55666d818f60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12b37871be837bca server_cookie=163f319b44a741ec in_seq=0 out_seq=0 2024-03-22T02:25:46.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:46.754+0000 7f14b6ffd640 10 client.18972 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:46 smithi083 ceph-mon[36340]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-03-22T02:25:46.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:46 smithi028 ceph-mon[31916]: pgmap v725: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 415 op/s 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14b6ffd640 1 -- 172.21.15.28:0/1705550405 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55666d7a4270 con 0x55666d816b40 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14b6ffd640 10 client.18972 handle_command_reply: tid=0 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 client.18972 shutdown 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 msgr2=0x55666d7a8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f14c8013010 tx=0x7f14c8004730 comp rx=0 tx=0).stop 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 shutdown_connections 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55666d816b40 0x55666d818f60 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55666d7a0040 0x55666d7aa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14b6ffd640 0 client.18972 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55666d79aed0 0x55666d7a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 --2- 172.21.15.28:0/1705550405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55666d79a4e0 0x55666d7a8e40 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 >> 172.21.15.28:0/1705550405 conn(0x55666d795a50 msgr2=0x55666d7943c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 shutdown_connections 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 1 -- 172.21.15.28:0/1705550405 wait complete. 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.053+0000 7f14e0c34740 20 client.18972 trim_cache size 0 max 16384 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:25:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:25:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 msgr2=0x7f14c00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 0x7f14c00a5480 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f14c8011550 tx=0x7f14c8011950 comp rx=0 tx=0).stop 2024-03-22T02:25:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 msgr2=0x7f14d81005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f14cc000c00 tx=0x7f14cc005d50 comp rx=0 tx=0).stop 2024-03-22T02:25:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 shutdown_connections 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f14c00a2fc0 0x7f14c00a5480 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f14d8107890 0x7f14d8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14d8104f60 0x7f14d81005e0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 --2- 172.21.15.28:0/3721294928 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f14d8102630 0x7f14d8101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 >> 172.21.15.28:0/3721294928 conn(0x7f14d80fc020 msgr2=0x7f14d8102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 shutdown_connections 2024-03-22T02:25:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:25:47.054+0000 7f14ddf79640 1 -- 172.21.15.28:0/3721294928 wait complete. 2024-03-22T02:25:47.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1250578 11.24 MB/sec execute 509 sec latency 167.675 ms 2024-03-22T02:25:48.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1252722 11.23 MB/sec execute 510 sec latency 41.533 ms 2024-03-22T02:25:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:48 smithi120 ceph-mon[36803]: pgmap v726: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 327 op/s 2024-03-22T02:25:48.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:48 smithi083 ceph-mon[36340]: pgmap v726: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 327 op/s 2024-03-22T02:25:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:48 smithi028 ceph-mon[31916]: pgmap v726: 129 pgs: 129 active+clean; 60 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 327 op/s 2024-03-22T02:25:48.925 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:25:48.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3688214651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 msgr2=0x7f0db4102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3688214651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db4102100 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009900 tx=0x7f0da402bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3688214651 shutdown_connections 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3688214651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4102640 0x7f0db4108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3688214651 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db4102100 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3688214651 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4100a90 0x7f0db4100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3688214651 >> 172.21.15.28:0/3688214651 conn(0x7f0db40fc020 msgr2=0x7f0db40fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3688214651 shutdown_connections 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3688214651 wait complete. 2024-03-22T02:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 Processor -- start 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.034+0000 7f0dbc6af640 1 -- start start 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4102640 0x7f0db419f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410c710 con 0x7f0db4100a90 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410c590 con 0x7f0db4102640 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0db410c890 con 0x7f0db4101c80 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35016/0 (socket says 172.21.15.28:35016) 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 -- 172.21.15.28:0/3015195412 learned_addr learned my addr 172.21.15.28:0/3015195412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbac4c640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4102640 0x7f0db419f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 -- 172.21.15.28:0/3015195412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 msgr2=0x7f0db419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 -- 172.21.15.28:0/3015195412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4102640 msgr2=0x7f0db419f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4102640 0x7f0db419f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 -- 172.21.15.28:0/3015195412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db419fb50 con 0x7f0db4100a90 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dba44b640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f0db000e9c0 tx=0x7f0db000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dab7fe640 1 -- 172.21.15.28:0/3015195412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db000cd80 con 0x7f0db4100a90 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db419fde0 con 0x7f0db4100a90 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dab7fe640 1 -- 172.21.15.28:0/3015195412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db00035f0 con 0x7f0db4100a90 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dab7fe640 1 -- 172.21.15.28:0/3015195412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0010690 con 0x7f0db4100a90 2024-03-22T02:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.035+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db41a01f0 con 0x7f0db4100a90 2024-03-22T02:25:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.037+0000 7f0dab7fe640 1 -- 172.21.15.28:0/3015195412 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0db00037a0 con 0x7f0db4100a90 2024-03-22T02:25:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.038+0000 7f0dbf373740 1 Processor -- start 2024-03-22T02:25:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.038+0000 7f0dbf373740 1 -- start start 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.038+0000 7f0dab7fe640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 0x7f0da00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd767850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd76cf90 0x55bdbd76f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd6cc8c0 con 0x55bdbd76cf90 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd674c20 con 0x55bdbd767430 2024-03-22T02:25:49.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dbf373740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd6c8870 con 0x55bdbd767e20 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.039+0000 7f0dab7fe640 1 -- 172.21.15.28:0/3015195412 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0db00cfb60 con 0x7f0db4100a90 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.040+0000 7f0db9c4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.040+0000 7f0db9c4a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34224/0 (socket says 172.21.15.28:34224) 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.040+0000 7f0db9c4a640 1 -- 172.21.15.28:0/3749500591 learned_addr learned my addr 172.21.15.28:0/3749500591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.040+0000 7f0dba44b640 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd767850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.041+0000 7f0db9c4a640 1 -- 172.21.15.28:0/3749500591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 msgr2=0x55bdbd767850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.041+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd767850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.041+0000 7f0db9c4a640 1 -- 172.21.15.28:0/3749500591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd76cf90 msgr2=0x55bdbd76f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.041+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd76cf90 0x55bdbd76f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.041+0000 7f0db9c4a640 1 -- 172.21.15.28:0/3749500591 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbd699150 con 0x55bdbd767e20 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.042+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009900 tx=0x7f0da4009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab309940da7b4c94 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.042+0000 7f0dbac4c640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 0x7f0da00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.043+0000 7f0da8ff9640 1 -- 172.21.15.28:0/3749500591 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da402cb80 con 0x55bdbd767e20 2024-03-22T02:25:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.043+0000 7f0da8ff9640 1 -- 172.21.15.28:0/3749500591 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da4005040 con 0x55bdbd767e20 2024-03-22T02:25:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0da8ff9640 1 -- 172.21.15.28:0/3749500591 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da4004920 con 0x55bdbd767e20 2024-03-22T02:25:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 -- 172.21.15.28:0/3749500591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 msgr2=0x55bdbd76c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0da4009900 tx=0x7f0da4009560 comp rx=0 tx=0).stop 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbac4c640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 0x7f0da00a53e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005fd0 tx=0x7f0dac009c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 -- 172.21.15.28:0/3749500591 shutdown_connections 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd76cf90 0x55bdbd76f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767e20 0x55bdbd76c920 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 --2- 172.21.15.28:0/3749500591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd767850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.044+0000 7f0dbf373740 1 -- 172.21.15.28:0/3749500591 >> 172.21.15.28:0/3749500591 conn(0x55bdbd7629a0 msgr2=0x55bdbd762da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.052+0000 7f0dbf373740 1 -- 172.21.15.28:0/3749500591 shutdown_connections 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.052+0000 7f0dbf373740 1 -- 172.21.15.28:0/3749500591 wait complete. 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.052+0000 7f0dbf373740 1 -- start start 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.052+0000 7f0dbf373740 1 Processor -- start 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd774d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd76cf90 0x55bdbd771830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd76ff80 con 0x55bdbd767e20 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd688130 con 0x55bdbd767430 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bdbd770100 con 0x55bdbd76cf90 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.053+0000 7f0dbf373740 10 client.0 _ll_register_callbacks cb 0x55bdbd687b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.054+0000 7f0db9c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.054+0000 7f0db9c4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35036/0 (socket says 172.21.15.28:35036) 2024-03-22T02:25:49.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.054+0000 7f0db9c4a640 1 -- 172.21.15.28:0/1860164374 learned_addr learned my addr 172.21.15.28:0/1860164374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.055+0000 7f0dbac4c640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd76cf90 0x55bdbd771830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.056+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd774d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.058+0000 7f0db9c4a640 1 -- 172.21.15.28:0/1860164374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd76cf90 msgr2=0x55bdbd771830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.058+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd76cf90 0x55bdbd771830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.058+0000 7f0db9c4a640 1 -- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 msgr2=0x55bdbd774d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.058+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd774d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.058+0000 7f0db9c4a640 1 -- 172.21.15.28:0/1860164374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbd4aa7d0 con 0x55bdbd767e20 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.059+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0da4036040 tx=0x7f0da4005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9964c496e7f6b7c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.059+0000 7f0d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.067+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da40048d0 con 0x55bdbd767e20 2024-03-22T02:25:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.067+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0da4004a70 con 0x55bdbd767e20 2024-03-22T02:25:49.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.071+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0da4005040 con 0x55bdbd767e20 2024-03-22T02:25:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.074+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0da4004c70 con 0x55bdbd767e20 2024-03-22T02:25:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.076+0000 7f0dbf373740 10 client.18981 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.076+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bdbd3e03f0 con 0x55bdbd767e20 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.081+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0da4005040 con 0x55bdbd767e20 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.081+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f0da4056420 con 0x55bdbd767e20 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 10 client.18981 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 10 client.18981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 20 client.18981 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd779300 0x55bdbd77b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 4 client.18981 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.082+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdbd6d9e00 con 0x55bdbd779300 2024-03-22T02:25:49.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.086+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd779300 0x55bdbd77b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.096+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd779300 0x55bdbd77b6f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6dbfafb4d44ffe server_cookie=7343eba593398cd1 in_seq=0 out_seq=0 2024-03-22T02:25:49.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.096+0000 7f0d9dffb640 10 client.18981 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.100+0000 7f0d9dffb640 1 -- 172.21.15.28:0/1860164374 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdbd6d9e00 con 0x55bdbd779300 2024-03-22T02:25:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.100+0000 7f0d9dffb640 10 client.18981 handle_command_reply: tid=0 2024-03-22T02:25:49.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.106+0000 7f0dbf373740 1 client.18981 shutdown 2024-03-22T02:25:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.108+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 msgr2=0x55bdbd7752d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.108+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0da4036040 tx=0x7f0da4005420 comp rx=0 tx=0).stop 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 shutdown_connections 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd779300 0x55bdbd77b6f0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd76cf90 0x55bdbd771830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd767e20 0x55bdbd7752d0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1860164374 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767430 0x55bdbd774d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.111+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 >> 172.21.15.28:0/1860164374 conn(0x55bdbd7629a0 msgr2=0x55bdbd761c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.112+0000 7f0d9dffb640 0 client.18981 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.112+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 shutdown_connections 2024-03-22T02:25:49.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.113+0000 7f0dbf373740 1 -- 172.21.15.28:0/1860164374 wait complete. 2024-03-22T02:25:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.114+0000 7f0dbf373740 20 client.18981 trim_cache size 0 max 16384 2024-03-22T02:25:49.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 Processor -- start 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 -- start start 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd761370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd7618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0db41a28d0 con 0x55bdbd7629a0 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4073ae0 con 0x55bdbd767e20 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.145+0000 7f0dbf373740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4071930 con 0x55bdbd767430 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35048/0 (socket says 172.21.15.28:35048) 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 -- 172.21.15.28:0/1900901679 learned_addr learned my addr 172.21.15.28:0/1900901679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dbac4c640 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd7618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd761370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 -- 172.21.15.28:0/1900901679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 msgr2=0x55bdbd761370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd761370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 -- 172.21.15.28:0/1900901679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 msgr2=0x55bdbd7618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd7618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 -- 172.21.15.28:0/1900901679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbd771ba0 con 0x55bdbd7629a0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0dba44b640 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0db0098850 tx=0x7f0db0098120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af4f6bf051463b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0d9ffff640 1 -- 172.21.15.28:0/1900901679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db009be80 con 0x55bdbd7629a0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0d9ffff640 1 -- 172.21.15.28:0/1900901679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db00bae50 con 0x55bdbd7629a0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.146+0000 7f0d9ffff640 1 -- 172.21.15.28:0/1900901679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0003b10 con 0x55bdbd7629a0 2024-03-22T02:25:49.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- 172.21.15.28:0/1900901679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 msgr2=0x55bdbd7704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0db0098850 tx=0x7f0db0098120 comp rx=0 tx=0).stop 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- 172.21.15.28:0/1900901679 shutdown_connections 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd7618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd761370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 --2- 172.21.15.28:0/1900901679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd7704f0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- 172.21.15.28:0/1900901679 >> 172.21.15.28:0/1900901679 conn(0x55bdbd77b9f0 msgr2=0x55bdbd774d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- 172.21.15.28:0/1900901679 shutdown_connections 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- 172.21.15.28:0/1900901679 wait complete. 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.147+0000 7f0dbf373740 1 -- start start 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 Processor -- start 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd77fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd780d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4071fa0 con 0x55bdbd7629a0 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4073410 con 0x55bdbd767e20 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0db4072670 con 0x55bdbd767430 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbf373740 10 client.0 _ll_register_callbacks cb 0x7f0da402c330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35050/0 (socket says 172.21.15.28:35050) 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dbac4c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd780d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd77fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 -- 172.21.15.28:0/2385132747 learned_addr learned my addr 172.21.15.28:0/2385132747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 -- 172.21.15.28:0/2385132747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 msgr2=0x55bdbd77fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd77fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 -- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 msgr2=0x55bdbd780d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd780d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 -- 172.21.15.28:0/2385132747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbd3a00f0 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0dba44b640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0db009be10 tx=0x7f0db000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e70ba597bf311d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.148+0000 7f0d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db009ffd0 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f0db00984f0 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db00bae50 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0db00a4890 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 10 client.18987 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bdbd7f4b70 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0db00bae50 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f0db009b3f0 con 0x55bdbd7629a0 2024-03-22T02:25:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 10 client.18987 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 10 client.18987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 20 client.18987 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd76cf90 0x55bdbd7f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 4 client.18987 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bdbd6d4150 con 0x55bdbd76cf90 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.149+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd76cf90 0x55bdbd7f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.150+0000 7f0db9c4a640 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd76cf90 0x55bdbd7f74d0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75eafc68551830e8 server_cookie=bdba360f21805ec9 in_seq=0 out_seq=0 2024-03-22T02:25:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.150+0000 7f0d9dffb640 10 client.18987 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:49.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.151+0000 7f0d9dffb640 1 -- 172.21.15.28:0/2385132747 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bdbd6d4150 con 0x55bdbd76cf90 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.151+0000 7f0d9dffb640 10 client.18987 handle_command_reply: tid=0 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.151+0000 7f0dbf373740 1 client.18987 shutdown 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.151+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 msgr2=0x55bdbd77f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.151+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0db009be10 tx=0x7f0db000c7f0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 shutdown_connections 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bdbd76cf90 0x55bdbd7f74d0 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bdbd767e20 0x55bdbd780d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bdbd767430 0x55bdbd77fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0d9dffb640 0 client.18987 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 --2- 172.21.15.28:0/2385132747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bdbd7629a0 0x55bdbd77f820 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 >> 172.21.15.28:0/2385132747 conn(0x55bdbd77b9f0 msgr2=0x55bdbd765c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 shutdown_connections 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 1 -- 172.21.15.28:0/2385132747 wait complete. 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbf373740 20 client.18987 trim_cache size 0 max 16384 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:25:49.160 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 msgr2=0x7f0da00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 0x7f0da00a53e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0dac005fd0 tx=0x7f0dac009c90 comp rx=0 tx=0).stop 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 msgr2=0x7f0db4069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f0db000e9c0 tx=0x7f0db000ee80 comp rx=0 tx=0).stop 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 shutdown_connections 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0da00a2f20 0x7f0da00a53e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0db4102640 0x7f0db419f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0db4101c80 0x7f0db419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 --2- 172.21.15.28:0/3015195412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0db4100a90 0x7f0db4069860 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 >> 172.21.15.28:0/3015195412 conn(0x7f0db40fc020 msgr2=0x7f0db4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 shutdown_connections 2024-03-22T02:25:49.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.152+0000 7f0dbc6af640 1 -- 172.21.15.28:0/3015195412 wait complete. 2024-03-22T02:25:49.164 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:25:49.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:25:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 -- 172.21.15.28:0/4272334771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410ded0 msgr2=0x7f511410e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 --2- 172.21.15.28:0/4272334771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410ded0 0x7f511410e380 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f510c009ca0 tx=0x7f510c02c100 comp rx=0 tx=0).stop 2024-03-22T02:25:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 -- 172.21.15.28:0/4272334771 shutdown_connections 2024-03-22T02:25:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 --2- 172.21.15.28:0/4272334771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410ded0 0x7f511410e380 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 --2- 172.21.15.28:0/4272334771 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f511410b500 0x7f511410d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 --2- 172.21.15.28:0/4272334771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5114108b90 0x7f511410afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.311+0000 7f5119c88640 1 -- 172.21.15.28:0/4272334771 >> 172.21.15.28:0/4272334771 conn(0x7f511406ea00 msgr2=0x7f5114070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.312+0000 7f5119c88640 1 -- 172.21.15.28:0/4272334771 shutdown_connections 2024-03-22T02:25:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.313+0000 7f5119c88640 1 -- 172.21.15.28:0/4272334771 wait complete. 2024-03-22T02:25:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.314+0000 7f5119c88640 1 Processor -- start 2024-03-22T02:25:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 -- start start 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5114108b90 0x7f51141a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 0x7f51141a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 0x7f51141ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f511411c640 con 0x7f511410ded0 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f511411c4c0 con 0x7f5114108b90 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.315+0000 7f5119c88640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f511411c7c0 con 0x7f511410b500 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5113fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 0x7f51141ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5113fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 0x7f51141ae090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35070/0 (socket says 172.21.15.28:35070) 2024-03-22T02:25:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5113fff640 1 -- 172.21.15.28:0/2360588410 learned_addr learned my addr 172.21.15.28:0/2360588410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 0x7f51141a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f51137fe640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5114108b90 0x7f51141a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 -- 172.21.15.28:0/2360588410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5114108b90 msgr2=0x7f51141a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5114108b90 0x7f51141a6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 -- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 msgr2=0x7f51141ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 0x7f51141ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.316+0000 7f5112ffd640 1 -- 172.21.15.28:0/2360588410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51141ae700 con 0x7f511410b500 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5112ffd640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 0x7f51141a70c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f510800e9f0 tx=0x7f510800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510800ccc0 con 0x7f511410b500 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5108010910 con 0x7f511410b500 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080188e0 con 0x7f511410b500 2024-03-22T02:25:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51141ae9f0 con 0x7f511410b500 2024-03-22T02:25:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.317+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51141aee30 con 0x7f511410b500 2024-03-22T02:25:49.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.319+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5114071500 con 0x7f511410b500 2024-03-22T02:25:49.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.321+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5108010ac0 con 0x7f511410b500 2024-03-22T02:25:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.322+0000 7f5110ff9640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 0x7f50fc0a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.322+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f51080d0270 con 0x7f511410b500 2024-03-22T02:25:49.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.322+0000 7f51137fe640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 0x7f50fc0a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.323+0000 7f51137fe640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 0x7f50fc0a55e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5114072c80 tx=0x7f5104005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.328+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f510809ccb0 con 0x7f511410b500 2024-03-22T02:25:49.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1254135 11.23 MB/sec execute 511 sec latency 75.632 ms 2024-03-22T02:25:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.577+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f511402d3b0 con 0x7f511410b500 2024-03-22T02:25:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.578+0000 7f5110ff9640 1 -- 172.21.15.28:0/2360588410 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7f510809c5b0 con 0x7f511410b500 2024-03-22T02:25:49.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:25:49.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:24:45.372919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:25:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 msgr2=0x7f50fc0a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 0x7f50fc0a55e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f5114072c80 tx=0x7f5104005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 msgr2=0x7f51141a70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 0x7f51141a70c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f510800e9f0 tx=0x7f510800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 shutdown_connections 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50fc0a3120 0x7f50fc0a55e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f511410ded0 0x7f51141ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f511410b500 0x7f51141a70c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 --2- 172.21.15.28:0/2360588410 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5114108b90 0x7f51141a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 >> 172.21.15.28:0/2360588410 conn(0x7f511406ea00 msgr2=0x7f511410c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 shutdown_connections 2024-03-22T02:25:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.580+0000 7f5119c88640 1 -- 172.21.15.28:0/2360588410 wait complete. 2024-03-22T02:25:49.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:25:49.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- 172.21.15.28:0/3852944032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec101c80 msgr2=0x7fecec102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 --2- 172.21.15.28:0/3852944032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec101c80 0x7fecec102100 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fece0009900 tx=0x7fece002bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- 172.21.15.28:0/3852944032 shutdown_connections 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 --2- 172.21.15.28:0/3852944032 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec102640 0x7fecec108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 --2- 172.21.15.28:0/3852944032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec101c80 0x7fecec102100 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 --2- 172.21.15.28:0/3852944032 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 0x7fecec100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- 172.21.15.28:0/3852944032 >> 172.21.15.28:0/3852944032 conn(0x7fecec0fc020 msgr2=0x7fecec0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- 172.21.15.28:0/3852944032 shutdown_connections 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- 172.21.15.28:0/3852944032 wait complete. 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 Processor -- start 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.706+0000 7fecf253a640 1 -- start start 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 0x7fecec197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 0x7fecec198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecec069080 con 0x7fecec102640 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecec068f00 con 0x7fecec101c80 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecec069200 con 0x7fecec100a90 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35088/0 (socket says 172.21.15.28:35088) 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7feceb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 0x7fecec198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 -- 172.21.15.28:0/2368618332 learned_addr learned my addr 172.21.15.28:0/2368618332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecebfff640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 0x7fecec197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 -- 172.21.15.28:0/2368618332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 msgr2=0x7fecec197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 0x7fecec197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 -- 172.21.15.28:0/2368618332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 msgr2=0x7fecec198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 0x7fecec198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 -- 172.21.15.28:0/2368618332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecec19fb10 con 0x7fecec102640 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7feceb7fe640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 0x7fecec198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fece400eb10 tx=0x7fece400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fece97fa640 1 -- 172.21.15.28:0/2368618332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4004b20 con 0x7fecec102640 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecec19fe00 con 0x7fecec102640 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fece97fa640 1 -- 172.21.15.28:0/2368618332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece4004cc0 con 0x7fecec102640 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fece97fa640 1 -- 172.21.15.28:0/2368618332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4010760 con 0x7fecec102640 2024-03-22T02:25:49.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.707+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecec1a0270 con 0x7fecec102640 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fece97fa640 1 -- 172.21.15.28:0/2368618332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fece4005020 con 0x7fecec102640 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 Processor -- start 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 -- start start 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fece97fa640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 0x7fecd40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9391b0 0x563cfe93bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe949140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cfe89d8d0 con 0x563cfe93c310 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563cfe845e00 con 0x563cfe9387c0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fece97fa640 1 -- 172.21.15.28:0/2368618332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fece40cfd50 con 0x7fecec102640 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7fecf51fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563cfe899880 con 0x563cfe9391b0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.710+0000 7feceb7fe640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 0x7fecd40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52352/0 (socket says 172.21.15.28:52352) 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 -- 172.21.15.28:0/475466153 learned_addr learned my addr 172.21.15.28:0/475466153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe949140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7feceb7fe640 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9391b0 0x563cfe93bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 -- 172.21.15.28:0/475466153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9391b0 msgr2=0x563cfe93bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9391b0 0x563cfe93bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 -- 172.21.15.28:0/475466153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 msgr2=0x563cfe949140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe949140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 -- 172.21.15.28:0/475466153 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfe5b13f0 con 0x563cfe9387c0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7feceb7fe640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 0x7fecd40a53e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fece002c7b0 tx=0x7fece0005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecebfff640 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009ca0 tx=0x7fecdc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b875ccc019d639e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.711+0000 7fecd27fc640 1 -- 172.21.15.28:0/475466153 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc004220 con 0x563cfe9387c0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.712+0000 7fecd27fc640 1 -- 172.21.15.28:0/475466153 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecdc0043c0 con 0x563cfe9387c0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.712+0000 7fecd27fc640 1 -- 172.21.15.28:0/475466153 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc018870 con 0x563cfe9387c0 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.712+0000 7fecf51fa740 1 -- 172.21.15.28:0/475466153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 msgr2=0x563cfe938be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.712+0000 7fecf51fa740 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009ca0 tx=0x7fecdc011810 comp rx=0 tx=0).stop 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 -- 172.21.15.28:0/475466153 shutdown_connections 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe949140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9391b0 0x563cfe93bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 --2- 172.21.15.28:0/475466153 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9387c0 0x563cfe938be0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 -- 172.21.15.28:0/475466153 >> 172.21.15.28:0/475466153 conn(0x563cfe933d90 msgr2=0x563cfe934190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 -- 172.21.15.28:0/475466153 shutdown_connections 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 -- 172.21.15.28:0/475466153 wait complete. 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.713+0000 7fecf51fa740 1 -- start start 2024-03-22T02:25:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 Processor -- start 2024-03-22T02:25:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 0x563cfe94ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe93c310 0x563cfe952a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cfe949b40 con 0x563cfe9391b0 2024-03-22T02:25:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563cfe859300 con 0x563cfe93c310 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563cfe949cc0 con 0x563cfe9387c0 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf51fa740 10 client.0 _ll_register_callbacks cb 0x563cfe894e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7feceb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 0x563cfe94ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7feceb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 0x563cfe94ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35100/0 (socket says 172.21.15.28:35100) 2024-03-22T02:25:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7feceb7fe640 1 -- 172.21.15.28:0/2521886132 learned_addr learned my addr 172.21.15.28:0/2521886132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe93c310 0x563cfe952a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 -- 172.21.15.28:0/2521886132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe93c310 msgr2=0x563cfe952a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe93c310 0x563cfe952a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 -- 172.21.15.28:0/2521886132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 msgr2=0x563cfe94ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 0x563cfe94ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.714+0000 7fecebfff640 1 -- 172.21.15.28:0/2521886132 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfe67b7d0 con 0x563cfe9387c0 2024-03-22T02:25:49.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7fecebfff640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94b550 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=116286b8f16beff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:49.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc027020 con 0x563cfe9387c0 2024-03-22T02:25:49.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fecdc0035f0 con 0x563cfe9387c0 2024-03-22T02:25:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecdc0043c0 con 0x563cfe9387c0 2024-03-22T02:25:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fecdc032050 con 0x563cfe9387c0 2024-03-22T02:25:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7fecf51fa740 10 client.28825 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563cfe955dd0 con 0x563cfe9387c0 2024-03-22T02:25:49.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecdc003cf0 con 0x563cfe9387c0 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.715+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fecdc018570 con 0x563cfe9387c0 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 10 client.28825 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 10 client.28825 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 20 client.28825 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9572e0 0x563cfe959710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 4 client.28825 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fecdc0043c0 con 0x563cfe9572e0 2024-03-22T02:25:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7feceb7fe640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9572e0 0x563cfe959710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7feceb7fe640 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9572e0 0x563cfe959710 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5a262bd797799782 server_cookie=2c9e905209b2ef5 in_seq=0 out_seq=0 2024-03-22T02:25:49.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.716+0000 7feccb7fe640 10 client.28825 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:49.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.717+0000 7feccb7fe640 1 -- 172.21.15.28:0/2521886132 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fecdc0043c0 con 0x563cfe9572e0 2024-03-22T02:25:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.717+0000 7feccb7fe640 10 client.28825 handle_command_reply: tid=0 2024-03-22T02:25:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.717+0000 7fecf51fa740 1 client.28825 shutdown 2024-03-22T02:25:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 msgr2=0x563cfe94b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94b550 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fecdc009870 tx=0x7fecdc0052f0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 shutdown_connections 2024-03-22T02:25:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9572e0 0x563cfe959710 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe93c310 0x563cfe952a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7feccb7fe640 0 client.28825 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9391b0 0x563cfe94ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 --2- 172.21.15.28:0/2521886132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94b550 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 >> 172.21.15.28:0/2521886132 conn(0x563cfe933d90 msgr2=0x563cfe932fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 shutdown_connections 2024-03-22T02:25:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 1 -- 172.21.15.28:0/2521886132 wait complete. 2024-03-22T02:25:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.718+0000 7fecf51fa740 20 client.28825 trim_cache size 0 max 16384 2024-03-22T02:25:49.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 Processor -- start 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 -- start start 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe952980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1ab090 con 0x563cfe93c310 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1a8960 con 0x563cfe9391b0 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf51fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1a67b0 con 0x563cfe9387c0 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf0ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7feceb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe952980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf0ad7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35116/0 (socket says 172.21.15.28:35116) 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecf0ad7640 1 -- 172.21.15.28:0/3860822246 learned_addr learned my addr 172.21.15.28:0/3860822246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.737+0000 7fecebfff640 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 -- 172.21.15.28:0/3860822246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 msgr2=0x563cfe94ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 -- 172.21.15.28:0/3860822246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 msgr2=0x563cfe952980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe952980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 -- 172.21.15.28:0/3860822246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cfe92e3e0 con 0x563cfe93c310 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7feceb7fe640 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe952980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fece40cd7b0 tx=0x7fece40bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bad44780a218d81 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecd1ffb640 1 -- 172.21.15.28:0/3860822246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40b5170 con 0x563cfe93c310 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecd1ffb640 1 -- 172.21.15.28:0/3860822246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece40b4070 con 0x563cfe93c310 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.738+0000 7fecd1ffb640 1 -- 172.21.15.28:0/3860822246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4098870 con 0x563cfe93c310 2024-03-22T02:25:49.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- 172.21.15.28:0/3860822246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 msgr2=0x563cfe94b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fece40cd7b0 tx=0x7fece40bc040 comp rx=0 tx=0).stop 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- 172.21.15.28:0/3860822246 shutdown_connections 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe93c310 0x563cfe94b0b0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe952980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 --2- 172.21.15.28:0/3860822246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe9387c0 0x563cfe94ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- 172.21.15.28:0/3860822246 >> 172.21.15.28:0/3860822246 conn(0x563cfe933d90 msgr2=0x563cfe937880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- 172.21.15.28:0/3860822246 shutdown_connections 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- 172.21.15.28:0/3860822246 wait complete. 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 -- start start 2024-03-22T02:25:49.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.739+0000 7fecf51fa740 1 Processor -- start 2024-03-22T02:25:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe950510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 0x563cfe951520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1a6e20 con 0x563cfe9387c0 2024-03-22T02:25:49.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1a8290 con 0x563cfe9391b0 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecec1a74f0 con 0x563cfe93c310 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 10 client.0 _ll_register_callbacks cb 0x7fece40057d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35122/0 (socket says 172.21.15.28:35122) 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 -- 172.21.15.28:0/79235017 learned_addr learned my addr 172.21.15.28:0/79235017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 0x563cfe951520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7feceb7fe640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe950510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 -- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 msgr2=0x563cfe951520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 0x563cfe951520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 -- 172.21.15.28:0/79235017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 msgr2=0x563cfe950510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe950510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 -- 172.21.15.28:0/79235017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cfe951b10 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf0ad7640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 0x563cfe951520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecebfff640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fecdc018bd0 tx=0x7fecdc018e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62690b0533026ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7feccb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc03b060 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fecdc0183d0 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.740+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece40b4070 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fecdc042050 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 10 client.19011 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x563cfe9bd610 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece40b4070 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fecdc019430 con 0x563cfe9387c0 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 10 client.19011 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 10 client.19011 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:25:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 20 client.19011 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9bdeb0 0x563cfe9c02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 4 client.19011 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cfe8aae10 con 0x563cfe9bdeb0 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feceb7fe640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9bdeb0 0x563cfe9c02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feccb7fe640 10 client.19011 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.741+0000 7feceb7fe640 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9bdeb0 0x563cfe9c02d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=391078536c564078 server_cookie=2ac026dbaf6e3d19 in_seq=0 out_seq=0 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7feccb7fe640 1 -- 172.21.15.28:0/79235017 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563cfe8aae10 con 0x563cfe9bdeb0 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7feccb7fe640 10 client.19011 handle_command_reply: tid=0 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 client.19011 shutdown 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 msgr2=0x563cfe94ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fecdc018bd0 tx=0x7fecdc018e80 comp rx=0 tx=0).stop 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 shutdown_connections 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cfe93c310 0x563cfe951520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x563cfe9bdeb0 0x563cfe9c02d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cfe9391b0 0x563cfe950510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 --2- 172.21.15.28:0/79235017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cfe9387c0 0x563cfe94ffd0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7feccb7fe640 0 client.19011 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 >> 172.21.15.28:0/79235017 conn(0x563cfe933d90 msgr2=0x563cfe932740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 shutdown_connections 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 1 -- 172.21.15.28:0/79235017 wait complete. 2024-03-22T02:25:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.742+0000 7fecf51fa740 20 client.19011 trim_cache size 0 max 16384 2024-03-22T02:25:49.752 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 msgr2=0x7fecd40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 0x7fecd40a53e0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fece002c7b0 tx=0x7fece0005e80 comp rx=0 tx=0).stop 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 msgr2=0x7fecec19f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fece400eb10 tx=0x7fece400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 shutdown_connections 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fecd40a2f20 0x7fecd40a53e0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecec102640 0x7fecec19f4a0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecec101c80 0x7fecec198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 --2- 172.21.15.28:0/2368618332 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecec100a90 0x7fecec197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 >> 172.21.15.28:0/2368618332 conn(0x7fecec0fc020 msgr2=0x7fecec0fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 shutdown_connections 2024-03-22T02:25:49.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.743+0000 7fecf253a640 1 -- 172.21.15.28:0/2368618332 wait complete. 2024-03-22T02:25:49.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/1591357253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8101c80 msgr2=0x7f2fd8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/1591357253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8101c80 0x7f2fd8102100 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009900 tx=0x7f2fcc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/1591357253 shutdown_connections 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/1591357253 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 0x7f2fd8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/1591357253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8101c80 0x7f2fd8102100 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/1591357253 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8100a90 0x7f2fd8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/1591357253 >> 172.21.15.28:0/1591357253 conn(0x7f2fd80fc020 msgr2=0x7f2fd80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/1591357253 shutdown_connections 2024-03-22T02:25:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/1591357253 wait complete. 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.870+0000 7f2fdd4eb640 1 Processor -- start 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 -- start start 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8101c80 0x7f2fd8103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 0x7f2fd81035a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd810c710 con 0x7f2fd8100a90 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd810c590 con 0x7f2fd8101c80 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fdd4eb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd810c890 con 0x7f2fd8102640 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35132/0 (socket says 172.21.15.28:35132) 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2844159099 learned_addr learned my addr 172.21.15.28:0/2844159099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd77fe640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 0x7f2fd81035a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8101c80 0x7f2fd8103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2844159099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 msgr2=0x7f2fd81035a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 0x7f2fd81035a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2844159099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8101c80 msgr2=0x7f2fd8103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8101c80 0x7f2fd8103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2844159099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd8103dc0 con 0x7f2fd8100a90 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800e9c0 tx=0x7f2fc800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.871+0000 7f2fbffff640 1 -- 172.21.15.28:0/2844159099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc800cde0 con 0x7f2fd8100a90 2024-03-22T02:25:49.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.872+0000 7f2fbffff640 1 -- 172.21.15.28:0/2844159099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc8010910 con 0x7f2fd8100a90 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.872+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd81a1fd0 con 0x7f2fd8100a90 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.872+0000 7f2fbffff640 1 -- 172.21.15.28:0/2844159099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8018a10 con 0x7f2fd8100a90 2024-03-22T02:25:49.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.872+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd81a2430 con 0x7f2fd8100a90 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fbffff640 1 -- 172.21.15.28:0/2844159099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2fc801f020 con 0x7f2fd8100a90 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 Processor -- start 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 -- start start 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x56170994ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709951810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617098ae8d0 con 0x56170994f380 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561709856e00 con 0x561709949820 2024-03-22T02:25:49.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.874+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617098aa880 con 0x56170994a210 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fbffff640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 0x7f2fc00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x56170994ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52408/0 (socket says 172.21.15.28:52408) 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709951810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 -- 172.21.15.28:0/4057669616 learned_addr learned my addr 172.21.15.28:0/4057669616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 0x7f2fc00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/4057669616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 msgr2=0x561709949c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/4057669616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 msgr2=0x561709951810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709951810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fbffff640 1 -- 172.21.15.28:0/2844159099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2fc80cf9a0 con 0x7f2fd8100a90 2024-03-22T02:25:49.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/4057669616 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617095c23f0 con 0x56170994a210 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x56170994ed10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8098690 tx=0x7f2fc801af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f64e5b7b4b7a72fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fd77fe640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 0x7f2fc00a53e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x56170994aa70 tx=0x7f2fd0004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.875+0000 7f2fbd7fa640 1 -- 172.21.15.28:0/4057669616 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc809b620 con 0x56170994a210 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fbd7fa640 1 -- 172.21.15.28:0/4057669616 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc80cfe50 con 0x56170994a210 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fbd7fa640 1 -- 172.21.15.28:0/4057669616 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a46d0 con 0x56170994a210 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 -- 172.21.15.28:0/4057669616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 msgr2=0x56170994ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x56170994ed10 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8098690 tx=0x7f2fc801af90 comp rx=0 tx=0).stop 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 -- 172.21.15.28:0/4057669616 shutdown_connections 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709951810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x56170994ed10 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/4057669616 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709949c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 -- 172.21.15.28:0/4057669616 >> 172.21.15.28:0/4057669616 conn(0x561709944d90 msgr2=0x561709945190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 -- 172.21.15.28:0/4057669616 shutdown_connections 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.876+0000 7f2fe01ab740 1 -- 172.21.15.28:0/4057669616 wait complete. 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 -- start start 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 Processor -- start 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x5617099540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994f380 0x56170995b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561709952270 con 0x561709949820 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56170986a300 con 0x56170994a210 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5617099523f0 con 0x56170994f380 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fe01ab740 10 client.0 _ll_register_callbacks cb 0x5617098a5e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35138/0 (socket says 172.21.15.28:35138) 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994f380 0x56170995b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2629801868 learned_addr learned my addr 172.21.15.28:0/2629801868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.877+0000 7f2fd67fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x5617099540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994f380 msgr2=0x56170995b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994f380 0x56170995b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2629801868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 msgr2=0x5617099540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x5617099540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/2629801868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56170968c7d0 con 0x561709949820 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80cd400 tx=0x7f2fc809bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e17ab724f95944e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc809b620 con 0x561709949820 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2fc80046c0 con 0x561709949820 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fc80cfe50 con 0x561709949820 2024-03-22T02:25:49.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2fc809c3c0 con 0x561709949820 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fe01ab740 10 client.19023 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56170995e470 con 0x561709949820 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.878+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fc80cfe50 con 0x561709949820 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f2fc809b110 con 0x561709949820 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 10 client.19023 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 10 client.19023 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 20 client.19023 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x56170995f7c0 0x561709961bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 4 client.19023 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617098bbe10 con 0x56170995f7c0 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x56170995f7c0 0x561709961bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fbaff5640 10 client.19023 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.879+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x56170995f7c0 0x561709961bb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=807e772fdb8e1193 server_cookie=64a4579130523264 in_seq=0 out_seq=0 2024-03-22T02:25:49.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fbaff5640 1 -- 172.21.15.28:0/2629801868 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617098bbe10 con 0x56170995f7c0 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fbaff5640 10 client.19023 handle_command_reply: tid=0 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 client.19023 shutdown 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 msgr2=0x561709953b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2fc80cd400 tx=0x7f2fc809bd30 comp rx=0 tx=0).stop 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 shutdown_connections 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x56170995f7c0 0x561709961bb0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994f380 0x56170995b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x5617099540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/2629801868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561709949820 0x561709953b80 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 >> 172.21.15.28:0/2629801868 conn(0x561709944d90 msgr2=0x561709943fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fbaff5640 0 client.19023 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 shutdown_connections 2024-03-22T02:25:49.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 1 -- 172.21.15.28:0/2629801868 wait complete. 2024-03-22T02:25:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.880+0000 7f2fe01ab740 20 client.19023 trim_cache size 0 max 16384 2024-03-22T02:25:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.894+0000 7f2fe01ab740 1 Processor -- start 2024-03-22T02:25:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 -- start start 2024-03-22T02:25:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709943740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x561709943c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x5617099441c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81af420 con 0x56170994f380 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81accf0 con 0x561709949820 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81aab20 con 0x56170994a210 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x5617099441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x561709943c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x5617099441c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35144/0 (socket says 172.21.15.28:35144) 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd77fe640 1 -- 172.21.15.28:0/479290488 learned_addr learned my addr 172.21.15.28:0/479290488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709943740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 -- 172.21.15.28:0/479290488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 msgr2=0x561709943740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709943740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 -- 172.21.15.28:0/479290488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 msgr2=0x5617099441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x5617099441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 -- 172.21.15.28:0/479290488 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561709961f20 con 0x56170994a210 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.895+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x561709943c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009230 tx=0x7f2fcc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa319787b2e5806 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fbcff9640 1 -- 172.21.15.28:0/479290488 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc02cb20 con 0x56170994a210 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fbcff9640 1 -- 172.21.15.28:0/479290488 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fcc02ccc0 con 0x56170994a210 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fbcff9640 1 -- 172.21.15.28:0/479290488 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc005040 con 0x56170994a210 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 -- 172.21.15.28:0/479290488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 msgr2=0x561709943c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x561709943c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009230 tx=0x7f2fcc005fd0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 -- 172.21.15.28:0/479290488 shutdown_connections 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x5617099441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56170994a210 0x561709943c80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/479290488 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561709949820 0x561709943740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 -- 172.21.15.28:0/479290488 >> 172.21.15.28:0/479290488 conn(0x561709944d90 msgr2=0x561709953050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 -- 172.21.15.28:0/479290488 shutdown_connections 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.896+0000 7f2fe01ab740 1 -- 172.21.15.28:0/479290488 wait complete. 2024-03-22T02:25:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 -- start start 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 Processor -- start 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 0x56170995a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x56170995aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709958bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81ab1b0 con 0x56170994f380 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81ac620 con 0x56170994a210 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd81ab880 con 0x561709949820 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fe01ab740 10 client.0 _ll_register_callbacks cb 0x7f2fc809b890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709958bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 0x56170995a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709958bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35156/0 (socket says 172.21.15.28:35156) 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd77fe640 1 -- 172.21.15.28:0/68247452 learned_addr learned my addr 172.21.15.28:0/68247452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x56170995aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/68247452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 msgr2=0x56170995aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x56170995aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/68247452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 msgr2=0x561709958bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:49.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709958bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.897+0000 7f2fd6ffd640 1 -- 172.21.15.28:0/68247452 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617099591c0 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fd6ffd640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 0x56170995a5a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fc809b9c0 tx=0x7f2fc809fe90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faa0d635401f1f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80b9020 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fcc02ccc0 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2fc80a4450 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2fc809b620 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 10 client.28849 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5617099c62c0 con 0x561709949820 2024-03-22T02:25:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fcc02ccc0 con 0x561709949820 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f2fc807abc0 con 0x561709949820 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 10 client.28849 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 10 client.28849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 20 client.28849 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5617099c6ab0 0x5617099c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 4 client.28849 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.898+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617098b6160 con 0x5617099c6ab0 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5617099c6ab0 0x5617099c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fd67fc640 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5617099c6ab0 0x5617099c8ed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=735f7476b2320c57 server_cookie=42db4848096107fd in_seq=0 out_seq=0 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fbaff5640 10 client.28849 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fbaff5640 1 -- 172.21.15.28:0/68247452 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617098b6160 con 0x5617099c6ab0 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fbaff5640 10 client.28849 handle_command_reply: tid=0 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 client.28849 shutdown 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 msgr2=0x56170995a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 0x56170995a5a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fc809b9c0 tx=0x7f2fc809fe90 comp rx=0 tx=0).stop 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 shutdown_connections 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5617099c6ab0 0x5617099c8ed0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56170994f380 0x561709958bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fbaff5640 0 client.28849 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56170994a210 0x56170995aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 --2- 172.21.15.28:0/68247452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561709949820 0x56170995a5a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 >> 172.21.15.28:0/68247452 conn(0x561709944d90 msgr2=0x561709947ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 shutdown_connections 2024-03-22T02:25:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 1 -- 172.21.15.28:0/68247452 wait complete. 2024-03-22T02:25:49.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.899+0000 7f2fe01ab740 20 client.28849 trim_cache size 0 max 16384 2024-03-22T02:25:49.911 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 msgr2=0x7f2fc00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 0x7f2fc00a53e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x56170994aa70 tx=0x7f2fd0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 msgr2=0x7f2fd81049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800e9c0 tx=0x7f2fc800ee80 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 shutdown_connections 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2fc00a2f20 0x7f2fc00a53e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2fd8102640 0x7f2fd81035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2fd8101c80 0x7f2fd8103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 --2- 172.21.15.28:0/2844159099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fd8100a90 0x7f2fd81049b0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 >> 172.21.15.28:0/2844159099 conn(0x7f2fd80fc020 msgr2=0x7f2fd80fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 shutdown_connections 2024-03-22T02:25:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:49.900+0000 7f2fdd4eb640 1 -- 172.21.15.28:0/2844159099 wait complete. 2024-03-22T02:25:49.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:25:50.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 -- 172.21.15.28:0/1721863331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4340ffc50 msgr2=0x7ff4341000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 --2- 172.21.15.28:0/1721863331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4340ffc50 0x7ff4341000d0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff424009900 tx=0x7ff42402bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 -- 172.21.15.28:0/1721863331 shutdown_connections 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 --2- 172.21.15.28:0/1721863331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 0x7ff43410a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 --2- 172.21.15.28:0/1721863331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4340ffc50 0x7ff4341000d0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 --2- 172.21.15.28:0/1721863331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4341031e0 0x7ff434103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 -- 172.21.15.28:0/1721863331 >> 172.21.15.28:0/1721863331 conn(0x7ff4340fb790 msgr2=0x7ff4340fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 -- 172.21.15.28:0/1721863331 shutdown_connections 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.025+0000 7ff43b720640 1 -- 172.21.15.28:0/1721863331 wait complete. 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 Processor -- start 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 -- start start 2024-03-22T02:25:50.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4340ffc50 0x7ff4341028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 0x7ff434100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410d840 con 0x7ff4341031e0 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410d6c0 con 0x7ff4340ffc50 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff43b720640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff43410d9c0 con 0x7ff434100610 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35174/0 (socket says 172.21.15.28:35174) 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 -- 172.21.15.28:0/1566057129 learned_addr learned my addr 172.21.15.28:0/1566057129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4340ffc50 0x7ff4341028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 0x7ff434100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 -- 172.21.15.28:0/1566057129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 msgr2=0x7ff434100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 0x7ff434100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 -- 172.21.15.28:0/1566057129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4340ffc50 msgr2=0x7ff4341028e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4340ffc50 0x7ff4341028e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 -- 172.21.15.28:0/1566057129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff434101cf0 con 0x7ff4341031e0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.026+0000 7ff439cbd640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00eb10 tx=0x7ff42c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.027+0000 7ff42a7fc640 1 -- 172.21.15.28:0/1566057129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c004b20 con 0x7ff4341031e0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.027+0000 7ff42a7fc640 1 -- 172.21.15.28:0/1566057129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff42c004cc0 con 0x7ff4341031e0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.027+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4341a1fe0 con 0x7ff4341031e0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.027+0000 7ff42a7fc640 1 -- 172.21.15.28:0/1566057129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c010760 con 0x7ff4341031e0 2024-03-22T02:25:50.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.027+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a2440 con 0x7ff4341031e0 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.029+0000 7ff43e3df740 1 Processor -- start 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.029+0000 7ff42a7fc640 1 -- 172.21.15.28:0/1566057129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff42c005020 con 0x7ff4341031e0 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 -- start start 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 0x5601217d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56012172d8d0 con 0x5601217ce380 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5601216d5e00 con 0x5601217c8820 2024-03-22T02:25:50.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff43e3df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560121729880 con 0x5601217c9210 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff438cbb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34404/0 (socket says 172.21.15.28:34404) 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 -- 172.21.15.28:0/462315847 learned_addr learned my addr 172.21.15.28:0/462315847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff438cbb640 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 0x5601217d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff42a7fc640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 0x7ff4200a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 -- 172.21.15.28:0/462315847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 msgr2=0x5601217c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 -- 172.21.15.28:0/462315847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 msgr2=0x5601217d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 0x5601217d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff4394bc640 1 -- 172.21.15.28:0/462315847 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601214413f0 con 0x5601217c9210 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff438cbb640 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 0x5601217d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.030+0000 7ff439cbd640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 0x7ff4200a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.031+0000 7ff42a7fc640 1 -- 172.21.15.28:0/1566057129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff42c0cfeb0 con 0x7ff4341031e0 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.031+0000 7ff4394bc640 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff430009ca0 tx=0x7ff430011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68bef17f7cc7c10b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.031+0000 7ff41f7fe640 1 -- 172.21.15.28:0/462315847 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430004220 con 0x5601217c9210 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.031+0000 7ff41f7fe640 1 -- 172.21.15.28:0/462315847 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4300043c0 con 0x5601217c9210 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.031+0000 7ff41f7fe640 1 -- 172.21.15.28:0/462315847 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430019950 con 0x5601217c9210 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- 172.21.15.28:0/462315847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 msgr2=0x5601217cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff439cbd640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 0x7ff4200a53e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff42c098ba0 tx=0x7ff42c06bcf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff430009ca0 tx=0x7ff430011810 comp rx=0 tx=0).stop 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- 172.21.15.28:0/462315847 shutdown_connections 2024-03-22T02:25:50.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217ce380 0x5601217d0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217c9210 0x5601217cdd10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- 172.21.15.28:0/462315847 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- 172.21.15.28:0/462315847 >> 172.21.15.28:0/462315847 conn(0x5601217c3d90 msgr2=0x5601217c4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- 172.21.15.28:0/462315847 shutdown_connections 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- 172.21.15.28:0/462315847 wait complete. 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- start start 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 Processor -- start 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217da260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601217d1330 con 0x5601217c8820 2024-03-22T02:25:50.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5601216e9300 con 0x5601217c9210 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.032+0000 7ff43e3df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5601217d14b0 con 0x5601217ce380 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff43e3df740 10 client.0 _ll_register_callbacks cb 0x560121724e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35196/0 (socket says 172.21.15.28:35196) 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 -- 172.21.15.28:0/1563060556 learned_addr learned my addr 172.21.15.28:0/1563060556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217da260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 -- 172.21.15.28:0/1563060556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 msgr2=0x5601217da260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217da260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 -- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 msgr2=0x5601217d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 -- 172.21.15.28:0/1563060556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56012150b7d0 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff424036040 tx=0x7ff42402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5915a912d5c8fa78 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff41cff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4240038b0 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff424003a50 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4300043c0 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff424003c50 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff43e3df740 10 client.19041 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5601217dd640 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.033+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4300043c0 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7ff42403f410 con 0x5601217c8820 2024-03-22T02:25:50.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 10 client.19041 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 10 client.19041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 20 client.19041 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5601217deba0 0x5601217e1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 4 client.19041 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56012173ae10 con 0x5601217deba0 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5601217deba0 0x5601217e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff41cff9640 10 client.19041 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.034+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5601217deba0 0x5601217e1010 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6d0c2598bcfb944 server_cookie=76795980d7e2a40f in_seq=0 out_seq=0 2024-03-22T02:25:50.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff41cff9640 1 -- 172.21.15.28:0/1563060556 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56012173ae10 con 0x5601217deba0 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff41cff9640 10 client.19041 handle_command_reply: tid=0 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 client.19041 shutdown 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 msgr2=0x5601217d2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7ff424036040 tx=0x7ff42402bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 shutdown_connections 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5601217deba0 0x5601217e1010 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217da260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1563060556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d2ce0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff41cff9640 0 client.19041 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 >> 172.21.15.28:0/1563060556 conn(0x5601217c3d90 msgr2=0x5601217c2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 shutdown_connections 2024-03-22T02:25:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 1 -- 172.21.15.28:0/1563060556 wait complete. 2024-03-22T02:25:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.036+0000 7ff43e3df740 20 client.19041 trim_cache size 0 max 16384 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.050+0000 7ff43e3df740 1 Processor -- start 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.050+0000 7ff43e3df740 1 -- start start 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217d3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217c7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341af4c0 con 0x5601217c9210 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341acd90 con 0x5601217c8820 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff43e3df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341aabe0 con 0x5601217ce380 2024-03-22T02:25:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35200/0 (socket says 172.21.15.28:35200) 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 -- 172.21.15.28:0/1535599489 learned_addr learned my addr 172.21.15.28:0/1535599489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff438cbb640 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 -- 172.21.15.28:0/1535599489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 msgr2=0x5601217c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 -- 172.21.15.28:0/1535599489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 msgr2=0x5601217d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 -- 172.21.15.28:0/1535599489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601217c4b00 con 0x5601217c9210 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff4394bc640 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ff430014040 tx=0x7ff430009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd60c704c3d3e5e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff41effd640 1 -- 172.21.15.28:0/1535599489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430009e50 con 0x5601217c9210 2024-03-22T02:25:50.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff41effd640 1 -- 172.21.15.28:0/1535599489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff430031070 con 0x5601217c9210 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.051+0000 7ff41effd640 1 -- 172.21.15.28:0/1535599489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff430005040 con 0x5601217c9210 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 -- 172.21.15.28:0/1535599489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 msgr2=0x5601217da220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ff430014040 tx=0x7ff430009310 comp rx=0 tx=0).stop 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 -- 172.21.15.28:0/1535599489 shutdown_connections 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c9210 0x5601217da220 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 --2- 172.21.15.28:0/1535599489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c8820 0x5601217d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 -- 172.21.15.28:0/1535599489 >> 172.21.15.28:0/1535599489 conn(0x5601217c3d90 msgr2=0x5601217c6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 -- 172.21.15.28:0/1535599489 shutdown_connections 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.052+0000 7ff43e3df740 1 -- 172.21.15.28:0/1535599489 wait complete. 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 -- start start 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 Processor -- start 2024-03-22T02:25:50.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217d95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ab250 con 0x5601217c8820 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ac6c0 con 0x5601217c9210 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4341ab920 con 0x5601217ce380 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff43e3df740 10 client.0 _ll_register_callbacks cb 0x7ff424009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff438cbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff438cbb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35202/0 (socket says 172.21.15.28:35202) 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff4394bc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff438cbb640 1 -- 172.21.15.28:0/2855005025 learned_addr learned my addr 172.21.15.28:0/2855005025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.053+0000 7ff438cbb640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217d95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 -- 172.21.15.28:0/2855005025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 msgr2=0x5601217d95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217d95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 -- 172.21.15.28:0/2855005025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 msgr2=0x5601217d8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 -- 172.21.15.28:0/2855005025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601217d9b90 con 0x5601217c9210 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff438cbb640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff4394bc640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d8590 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff430003fd0 tx=0x7ff430009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1afea51b9c196b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff41cff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43003b020 con 0x5601217c9210 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff430031070 con 0x5601217c9210 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff430005040 con 0x5601217c9210 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff43002b390 con 0x5601217c9210 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff43e3df740 10 client.28871 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:50.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x560121845550 con 0x5601217c9210 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.054+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff430031070 con 0x5601217c9210 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7ff4300035f0 con 0x5601217c9210 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 10 client.28871 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 10 client.28871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 20 client.28871 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560121845db0 0x5601218481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 4 client.28871 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560121735160 con 0x560121845db0 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff438cbb640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560121845db0 0x5601218481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff41cff9640 10 client.28871 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.055+0000 7ff438cbb640 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560121845db0 0x5601218481d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6bcdbf0f726af4c server_cookie=9b86bf0fe8bad5b3 in_seq=0 out_seq=0 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.056+0000 7ff41cff9640 1 -- 172.21.15.28:0/2855005025 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560121735160 con 0x560121845db0 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.056+0000 7ff41cff9640 10 client.28871 handle_command_reply: tid=0 2024-03-22T02:25:50.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.056+0000 7ff43e3df740 1 client.28871 shutdown 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.056+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 msgr2=0x5601217d8590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.056+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d8590 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff430003fd0 tx=0x7ff430009c30 comp rx=0 tx=0).stop 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 shutdown_connections 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560121845db0 0x5601218481d0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5601217ce380 0x5601217d95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5601217c9210 0x5601217d8590 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 --2- 172.21.15.28:0/2855005025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601217c8820 0x5601217d8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff41cff9640 0 client.28871 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 >> 172.21.15.28:0/2855005025 conn(0x5601217c3d90 msgr2=0x5601217d2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 shutdown_connections 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 1 -- 172.21.15.28:0/2855005025 wait complete. 2024-03-22T02:25:50.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43e3df740 20 client.28871 trim_cache size 0 max 16384 2024-03-22T02:25:50.067 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 msgr2=0x7ff4200a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 0x7ff4200a53e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff42c098ba0 tx=0x7ff42c06bcf0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 msgr2=0x7ff434101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7ff42c00eb10 tx=0x7ff42c00c2d0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 shutdown_connections 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4200a2f20 0x7ff4200a53e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4341031e0 0x7ff434101460 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff434100610 0x7ff434100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 --2- 172.21.15.28:0/1566057129 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4340ffc50 0x7ff4341028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 >> 172.21.15.28:0/1566057129 conn(0x7ff4340fb790 msgr2=0x7ff434106270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 shutdown_connections 2024-03-22T02:25:50.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:50.057+0000 7ff43b720640 1 -- 172.21.15.28:0/1566057129 wait complete. 2024-03-22T02:25:50.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1256587 11.23 MB/sec execute 512 sec latency 46.171 ms 2024-03-22T02:25:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:50 smithi120 ceph-mon[36803]: pgmap v727: 129 pgs: 129 active+clean; 52 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-22T02:25:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:50 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2360588410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:50 smithi083 ceph-mon[36340]: pgmap v727: 129 pgs: 129 active+clean; 52 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-22T02:25:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:50 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2360588410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:50 smithi028 ceph-mon[31916]: pgmap v727: 129 pgs: 129 active+clean; 52 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 375 op/s 2024-03-22T02:25:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:50 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2360588410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:25:51.071 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:25:51.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:25:51.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- 172.21.15.28:0/331160603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 msgr2=0x7f2690100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 --2- 172.21.15.28:0/331160603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690100dc0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2684009900 tx=0x7f268402bc30 comp rx=0 tx=0).stop 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- 172.21.15.28:0/331160603 shutdown_connections 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 --2- 172.21.15.28:0/331160603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 0x7f2690108de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 --2- 172.21.15.28:0/331160603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f2690102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 --2- 172.21.15.28:0/331160603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690100dc0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- 172.21.15.28:0/331160603 >> 172.21.15.28:0/331160603 conn(0x7f26900fbf10 msgr2=0x7f26900fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- 172.21.15.28:0/331160603 shutdown_connections 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- 172.21.15.28:0/331160603 wait complete. 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 Processor -- start 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.181+0000 7f2695e8b640 1 -- start start 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f269019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 0x7f26901a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f269010c640 con 0x7f26901009a0 2024-03-22T02:25:51.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f269010c4c0 con 0x7f2690101b90 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2695e8b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f269010c7c0 con 0x7f2690102550 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2694e89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2694e89640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35220/0 (socket says 172.21.15.28:35220) 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f2694e89640 1 -- 172.21.15.28:0/1228496508 learned_addr learned my addr 172.21.15.28:0/1228496508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f269019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f269568a640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 0x7f26901a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 -- 172.21.15.28:0/1228496508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 msgr2=0x7f26901a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 0x7f26901a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 -- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 msgr2=0x7f2690199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.182+0000 7f268ffff640 1 -- 172.21.15.28:0/1228496508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26901a1c00 con 0x7f2690101b90 2024-03-22T02:25:51.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.186+0000 7f268ffff640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f269019a520 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f267c00d6d0 tx=0x7f267c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.187+0000 7f268dffb640 1 -- 172.21.15.28:0/1228496508 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c017070 con 0x7f2690101b90 2024-03-22T02:25:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.187+0000 7f268dffb640 1 -- 172.21.15.28:0/1228496508 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f267c009820 con 0x7f2690101b90 2024-03-22T02:25:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.187+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26901a1ef0 con 0x7f2690101b90 2024-03-22T02:25:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.188+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26901a23b0 con 0x7f2690101b90 2024-03-22T02:25:51.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.188+0000 7f268dffb640 1 -- 172.21.15.28:0/1228496508 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c004d00 con 0x7f2690101b90 2024-03-22T02:25:51.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.191+0000 7f2699db3740 1 Processor -- start 2024-03-22T02:25:51.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.192+0000 7f268dffb640 1 -- 172.21.15.28:0/1228496508 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f267c010670 con 0x7f2690101b90 2024-03-22T02:25:51.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f268dffb640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 0x7f26800a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f268dffb640 1 -- 172.21.15.28:0/1228496508 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f267c0cf290 con 0x7f2690101b90 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 -- start start 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc956b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f269568a640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 0x7f26800a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc95710f0 0x55cbc9573580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbc94c7570 con 0x55cbc956bf80 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cbc946fe10 con 0x55cbc95710f0 2024-03-22T02:25:51.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f2699db3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbc94c3520 con 0x55cbc956b590 2024-03-22T02:25:51.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f268ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.193+0000 7f268ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35230/0 (socket says 172.21.15.28:35230) 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 -- 172.21.15.28:0/2914782537 learned_addr learned my addr 172.21.15.28:0/2914782537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f2694e89640 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc956b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 -- 172.21.15.28:0/2914782537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 msgr2=0x55cbc956b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc956b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 -- 172.21.15.28:0/2914782537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc95710f0 msgr2=0x55cbc9573580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc95710f0 0x55cbc9573580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 -- 172.21.15.28:0/2914782537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc92a37e0 con 0x55cbc956bf80 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f268ffff640 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f267c097230 tx=0x7f267c09b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8af5d921f96d513 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f269568a640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 0x7f26800a55b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2688005fd0 tx=0x7f2688009af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f267affd640 1 -- 172.21.15.28:0/2914782537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c020040 con 0x55cbc956bf80 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f267affd640 1 -- 172.21.15.28:0/2914782537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f267c09f6a0 con 0x55cbc956bf80 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.194+0000 7f267affd640 1 -- 172.21.15.28:0/2914782537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c09ac00 con 0x55cbc956bf80 2024-03-22T02:25:51.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 -- 172.21.15.28:0/2914782537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 msgr2=0x55cbc9570a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f267c097230 tx=0x7f267c09b1b0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 -- 172.21.15.28:0/2914782537 shutdown_connections 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc95710f0 0x55cbc9573580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc956bf80 0x55cbc9570a80 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 --2- 172.21.15.28:0/2914782537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc956b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.195+0000 7f2699db3740 1 -- 172.21.15.28:0/2914782537 >> 172.21.15.28:0/2914782537 conn(0x55cbc955e2f0 msgr2=0x55cbc955e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.196+0000 7f2699db3740 1 -- 172.21.15.28:0/2914782537 shutdown_connections 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.196+0000 7f2699db3740 1 -- 172.21.15.28:0/2914782537 wait complete. 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.196+0000 7f2699db3740 1 -- start start 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.196+0000 7f2699db3740 1 Processor -- start 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9586750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc9586c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cbc9574040 con 0x55cbc95710f0 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cbc9483110 con 0x55cbc956bf80 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cbc95741c0 con 0x55cbc956b590 2024-03-22T02:25:51.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f2699db3740 10 client.0 _ll_register_callbacks cb 0x55cbc94bfbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f269568a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f269568a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35246/0 (socket says 172.21.15.28:35246) 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.197+0000 7f269568a640 1 -- 172.21.15.28:0/1499202340 learned_addr learned my addr 172.21.15.28:0/1499202340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f2694e89640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9586750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 -- 172.21.15.28:0/1499202340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 msgr2=0x55cbc9586750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9586750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 -- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 msgr2=0x55cbc9586c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc9586c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 -- 172.21.15.28:0/1499202340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc9303f70 con 0x55cbc95710f0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f269568a640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f2688003f60 tx=0x7f2688004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7456dc3e13f7dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f266ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:25:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688022070 con 0x55cbc95710f0 2024-03-22T02:25:51.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.198+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f268800ea20 con 0x55cbc95710f0 2024-03-22T02:25:51.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.199+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f267c09f6a0 con 0x55cbc95710f0 2024-03-22T02:25:51.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.199+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f268800ec20 con 0x55cbc95710f0 2024-03-22T02:25:51.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.199+0000 7f2699db3740 10 client.19059 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.199+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cbc919b170 con 0x55cbc95710f0 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f267c09f6a0 con 0x55cbc95710f0 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f268801e3f0 con 0x55cbc95710f0 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 10 client.19059 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 10 client.19059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 20 client.19059 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9578740 0x55cbc957ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 4 client.19059 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbc94d4ab0 con 0x55cbc9578740 2024-03-22T02:25:51.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.200+0000 7f2694e89640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9578740 0x55cbc957ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.204+0000 7f2694e89640 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9578740 0x55cbc957ac00 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc497efc093d7dc3 server_cookie=a09b7672fa8f8687 in_seq=0 out_seq=0 2024-03-22T02:25:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.204+0000 7f266ffff640 10 client.19059 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.208+0000 7f266ffff640 1 -- 172.21.15.28:0/1499202340 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cbc94d4ab0 con 0x55cbc9578740 2024-03-22T02:25:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.208+0000 7f266ffff640 10 client.19059 handle_command_reply: tid=0 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.208+0000 7f2699db3740 1 client.19059 shutdown 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 msgr2=0x55cbc958dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f2688003f60 tx=0x7f2688004ab0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 shutdown_connections 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9578740 0x55cbc957ac00 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958dcd0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc9586c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 --2- 172.21.15.28:0/1499202340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9586750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 >> 172.21.15.28:0/1499202340 conn(0x55cbc955e2f0 msgr2=0x55cbc955d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f266ffff640 0 client.19059 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 shutdown_connections 2024-03-22T02:25:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.209+0000 7f2699db3740 1 -- 172.21.15.28:0/1499202340 wait complete. 2024-03-22T02:25:51.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.210+0000 7f2699db3740 20 client.19059 trim_cache size 0 max 16384 2024-03-22T02:25:51.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 Processor -- start 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 -- start start 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9561c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc958dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26740126e0 con 0x55cbc95710f0 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f267400ffb0 con 0x55cbc956bf80 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2699db3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f267400de00 con 0x55cbc956b590 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f269568a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f2694e89640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9561c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f269568a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35260/0 (socket says 172.21.15.28:35260) 2024-03-22T02:25:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f268ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc958dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.239+0000 7f269568a640 1 -- 172.21.15.28:0/4162092106 learned_addr learned my addr 172.21.15.28:0/4162092106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 -- 172.21.15.28:0/4162092106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 msgr2=0x55cbc9561c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9561c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 -- 172.21.15.28:0/4162092106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 msgr2=0x55cbc958dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc958dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 -- 172.21.15.28:0/4162092106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbc9559bb0 con 0x55cbc95710f0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f268ffff640 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc958dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f269568a640 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f268801e8e0 tx=0x7f26880099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d706cc02225f6ba1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f267a7fc640 1 -- 172.21.15.28:0/4162092106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268803e070 con 0x55cbc95710f0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f267a7fc640 1 -- 172.21.15.28:0/4162092106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2688009df0 con 0x55cbc95710f0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.240+0000 7f267a7fc640 1 -- 172.21.15.28:0/4162092106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2688022040 con 0x55cbc95710f0 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- 172.21.15.28:0/4162092106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 msgr2=0x55cbc95854f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f268801e8e0 tx=0x7f26880099a0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- 172.21.15.28:0/4162092106 shutdown_connections 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc95854f0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956bf80 0x55cbc958dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- 172.21.15.28:0/4162092106 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956b590 0x55cbc9561c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- 172.21.15.28:0/4162092106 >> 172.21.15.28:0/4162092106 conn(0x55cbc955e2f0 msgr2=0x55cbc9561550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- 172.21.15.28:0/4162092106 shutdown_connections 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- 172.21.15.28:0/4162092106 wait complete. 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 -- start start 2024-03-22T02:25:51.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 Processor -- start 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.241+0000 7f2699db3740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 0x55cbc958b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f267400e470 con 0x55cbc95710f0 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f267400f8e0 con 0x55cbc956b590 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f267400eb40 con 0x55cbc956bf80 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 10 client.0 _ll_register_callbacks cb 0x7f2688009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f268ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 0x55cbc958b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52536/0 (socket says 172.21.15.28:52536) 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 -- 172.21.15.28:0/1534788988 learned_addr learned my addr 172.21.15.28:0/1534788988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f269568a640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 -- 172.21.15.28:0/1534788988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 msgr2=0x55cbc958b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 0x55cbc958b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 -- 172.21.15.28:0/1534788988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 msgr2=0x55cbc958c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 -- 172.21.15.28:0/1534788988 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbc958ce20 con 0x55cbc956b590 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f268ffff640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 0x55cbc958b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f269568a640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2694e89640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2684036040 tx=0x7f268402cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05ab5e2854ed0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f266ffff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:25:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684046e40 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2684004040 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.242+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2688009df0 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f2684057050 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 10 client.28883 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55cbc9609450 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2688009df0 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f26840052b0 con 0x55cbc956b590 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 10 client.28883 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 10 client.28883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 20 client.28883 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9609e70 0x55cbc960c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 4 client.28883 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:25:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cbc94cee00 con 0x55cbc9609e70 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f268ffff640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9609e70 0x55cbc960c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f268ffff640 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9609e70 0x55cbc960c290 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a20daae3d6792ee server_cookie=ef6f98695143d2bc in_seq=0 out_seq=0 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.243+0000 7f266ffff640 10 client.28883 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.244+0000 7f266ffff640 1 -- 172.21.15.28:0/1534788988 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cbc94cee00 con 0x55cbc9609e70 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.244+0000 7f266ffff640 10 client.28883 handle_command_reply: tid=0 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.244+0000 7f2699db3740 1 client.28883 shutdown 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.244+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 msgr2=0x55cbc958b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.244+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f2684036040 tx=0x7f268402cc70 comp rx=0 tx=0).stop 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 shutdown_connections 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cbc9609e70 0x55cbc960c290 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cbc95710f0 0x55cbc958c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f266ffff640 0 client.28883 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cbc956bf80 0x55cbc958b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 --2- 172.21.15.28:0/1534788988 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cbc956b590 0x55cbc958b330 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 >> 172.21.15.28:0/1534788988 conn(0x55cbc955e2f0 msgr2=0x55cbc955ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 shutdown_connections 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 1 -- 172.21.15.28:0/1534788988 wait complete. 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.245+0000 7f2699db3740 20 client.28883 trim_cache size 0 max 16384 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:25:51.252 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "638a2e64-6fa8-4eb2-a430-224c3346a04e", 2024-03-22T02:25:51.253 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:25:51.253 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 msgr2=0x7f26800a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 0x7f26800a55b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2688005fd0 tx=0x7f2688009af0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 msgr2=0x7f269019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f269019a520 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f267c00d6d0 tx=0x7f267c00db90 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 shutdown_connections 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f26800a30f0 0x7f26800a55b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2690102550 0x7f26901a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2690101b90 0x7f269019a520 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 --2- 172.21.15.28:0/1228496508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26901009a0 0x7f2690199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 >> 172.21.15.28:0/1228496508 conn(0x7f26900fbf10 msgr2=0x7f2690069c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 shutdown_connections 2024-03-22T02:25:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.246+0000 7f2695e8b640 1 -- 172.21.15.28:0/1228496508 wait complete. 2024-03-22T02:25:51.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:25:51.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:25:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- 172.21.15.28:0/1894610215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a501078d0 msgr2=0x7f8a50109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1894610215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a501078d0 0x7f8a50109cc0 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f8a48009ca0 tx=0x7f8a4802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- 172.21.15.28:0/1894610215 shutdown_connections 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1894610215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a501078d0 0x7f8a50109cc0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1894610215 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 0x7f8a50107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1894610215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a50102670 0x7f8a50104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- 172.21.15.28:0/1894610215 >> 172.21.15.28:0/1894610215 conn(0x7f8a500fc020 msgr2=0x7f8a500fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- 172.21.15.28:0/1894610215 shutdown_connections 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- 172.21.15.28:0/1894610215 wait complete. 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 Processor -- start 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.377+0000 7f8a57a02640 1 -- start start 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 0x7f8a50100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a501078d0 0x7f8a50100b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5010d830 con 0x7f8a50102670 2024-03-22T02:25:51.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5010d6b0 con 0x7f8a50104fa0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5010d9b0 con 0x7f8a501078d0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35290/0 (socket says 172.21.15.28:35290) 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 -- 172.21.15.28:0/1631410357 learned_addr learned my addr 172.21.15.28:0/1631410357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 0x7f8a50100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a501078d0 0x7f8a50100b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 -- 172.21.15.28:0/1631410357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a501078d0 msgr2=0x7f8a50100b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a501078d0 0x7f8a50100b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 -- 172.21.15.28:0/1631410357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 msgr2=0x7f8a50100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 0x7f8a50100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 -- 172.21.15.28:0/1631410357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a501013f0 con 0x7f8a50102670 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9f0 tx=0x7f8a4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a467fc640 1 -- 172.21.15.28:0/1631410357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c00cd80 con 0x7f8a50102670 2024-03-22T02:25:51.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a467fc640 1 -- 172.21.15.28:0/1631410357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c005500 con 0x7f8a50102670 2024-03-22T02:25:51.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a501a1fd0 con 0x7f8a50102670 2024-03-22T02:25:51.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a467fc640 1 -- 172.21.15.28:0/1631410357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c010690 con 0x7f8a50102670 2024-03-22T02:25:51.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.378+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a501a23e0 con 0x7f8a50102670 2024-03-22T02:25:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.381+0000 7f8a467fc640 1 -- 172.21.15.28:0/1631410357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8a4c0046c0 con 0x7f8a50102670 2024-03-22T02:25:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.381+0000 7f8a5a5ed740 1 Processor -- start 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.381+0000 7f8a5a5ed740 1 -- start start 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 0x55bb824e3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a467fc640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 0x7f8a3c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824e3f90 0x55bb824e6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb824438c0 con 0x55bb824de430 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb823ebc20 con 0x55bb824dee20 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb8243f870 con 0x55bb824e3f90 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a55f9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 0x55bb824e3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 0x7f8a3c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a55f9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35306/0 (socket says 172.21.15.28:35306) 2024-03-22T02:25:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a55f9f640 1 -- 172.21.15.28:0/2570607538 learned_addr learned my addr 172.21.15.28:0/2570607538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824e3f90 0x55bb824e6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a467fc640 1 -- 172.21.15.28:0/1631410357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8a4c0cfbb0 con 0x7f8a50102670 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 -- 172.21.15.28:0/2570607538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824e3f90 msgr2=0x55bb824e6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824e3f90 0x55bb824e6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 -- 172.21.15.28:0/2570607538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 msgr2=0x55bb824de850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 -- 172.21.15.28:0/2570607538 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb82410150 con 0x55bb824dee20 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 0x7f8a3c0a53e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8a50101d50 tx=0x7f8a40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.382+0000 7f8a5579e640 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 0x55bb824e3920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0cd2f0 tx=0x7f8a4c097b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de09fc4bcb014637 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a3b7fe640 1 -- 172.21.15.28:0/2570607538 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c09b830 con 0x55bb824dee20 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a3b7fe640 1 -- 172.21.15.28:0/2570607538 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c09c5d0 con 0x55bb824dee20 2024-03-22T02:25:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a3b7fe640 1 -- 172.21.15.28:0/2570607538 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0b4530 con 0x55bb824dee20 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/2570607538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 msgr2=0x55bb824e3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 0x55bb824e3920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0cd2f0 tx=0x7f8a4c097b10 comp rx=0 tx=0).stop 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/2570607538 shutdown_connections 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824e3f90 0x55bb824e6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824dee20 0x55bb824e3920 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/2570607538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824de850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/2570607538 >> 172.21.15.28:0/2570607538 conn(0x55bb824d99a0 msgr2=0x55bb824d9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/2570607538 shutdown_connections 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.383+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/2570607538 wait complete. 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 -- start start 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 Processor -- start 2024-03-22T02:25:51.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824e8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824e3f90 0x55bb824efea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb824e6f40 con 0x55bb824de430 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb823ff130 con 0x55bb824e3f90 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb824e70c0 con 0x55bb824dee20 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5a5ed740 10 client.0 _ll_register_callbacks cb 0x55bb823feb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5579e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a54f9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824e8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5579e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35310/0 (socket says 172.21.15.28:35310) 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a5579e640 1 -- 172.21.15.28:0/1072988604 learned_addr learned my addr 172.21.15.28:0/1072988604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.384+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824e3f90 0x55bb824efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 -- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824e3f90 msgr2=0x55bb824efea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824e3f90 0x55bb824efea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 -- 172.21.15.28:0/1072988604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 msgr2=0x55bb824e8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 -- 172.21.15.28:0/1072988604 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb822217d0 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824e8e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a500697f0 tx=0x7f8a40004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dd9f44c89051a11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4001caa0 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8a4001cc40 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c09c5d0 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a4001ce40 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a5a5ed740 10 client.28894 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.385+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb821573f0 con 0x55bb824dee20 2024-03-22T02:25:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c09c5d0 con 0x55bb824dee20 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f8a4001d670 con 0x55bb824dee20 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 10 client.28894 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 10 client.28894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 20 client.28894 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824f47f0 0x55bb824f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 4 client.28894 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb82450e00 con 0x55bb824f47f0 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824f47f0 0x55bb824f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a38ff9640 10 client.28894 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.386+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824f47f0 0x55bb824f6c50 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef2e2c6e44192aeb server_cookie=fc471143ca649e91 in_seq=0 out_seq=0 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1072988604 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb82450e00 con 0x55bb824f47f0 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a38ff9640 10 client.28894 handle_command_reply: tid=0 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 client.28894 shutdown 2024-03-22T02:25:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 msgr2=0x55bb824e8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824e8e60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a500697f0 tx=0x7f8a40004730 comp rx=0 tx=0).stop 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 shutdown_connections 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824f47f0 0x55bb824f6c50 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824e3f90 0x55bb824efea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824e8e60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a38ff9640 0 client.28894 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1072988604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824e8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.387+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 >> 172.21.15.28:0/1072988604 conn(0x55bb824d99a0 msgr2=0x55bb824d8c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.388+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 shutdown_connections 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.388+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1072988604 wait complete. 2024-03-22T02:25:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.388+0000 7f8a5a5ed740 20 client.28894 trim_cache size 0 max 16384 2024-03-22T02:25:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 Processor -- start 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 -- start start 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 0x55bb824dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824fb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501af4e0 con 0x55bb824de430 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501acdb0 con 0x55bb824d99a0 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501aac00 con 0x55bb824dee20 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35322/0 (socket says 172.21.15.28:35322) 2024-03-22T02:25:51.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 -- 172.21.15.28:0/603369240 learned_addr learned my addr 172.21.15.28:0/603369240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a5579e640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 0x55bb824dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824fb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 -- 172.21.15.28:0/603369240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 msgr2=0x55bb824fb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824fb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 -- 172.21.15.28:0/603369240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 msgr2=0x55bb824dd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 0x55bb824dd310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.408+0000 7f8a54f9d640 1 -- 172.21.15.28:0/603369240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb821170f0 con 0x55bb824de430 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.409+0000 7f8a5579e640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 0x55bb824dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.409+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004700 tx=0x7f8a40005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d348f0936654ffc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.409+0000 7f8a3affd640 1 -- 172.21.15.28:0/603369240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40004030 con 0x55bb824de430 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.409+0000 7f8a3affd640 1 -- 172.21.15.28:0/603369240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a400041d0 con 0x55bb824de430 2024-03-22T02:25:51.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.409+0000 7f8a3affd640 1 -- 172.21.15.28:0/603369240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400035f0 con 0x55bb824de430 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/603369240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 msgr2=0x55bb824efe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004700 tx=0x7f8a40005c90 comp rx=0 tx=0).stop 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/603369240 shutdown_connections 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824dee20 0x55bb824fb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824de430 0x55bb824efe60 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/603369240 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824d99a0 0x55bb824dd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/603369240 >> 172.21.15.28:0/603369240 conn(0x55bb824dfc00 msgr2=0x55bb824dcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/603369240 shutdown_connections 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/603369240 wait complete. 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 -- start start 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 Processor -- start 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 0x55bb824edb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824de430 0x55bb824ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.410+0000 7f8a5a5ed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501ab270 con 0x55bb824dee20 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501ac6e0 con 0x55bb824de430 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5a5ed740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501ab940 con 0x55bb824d99a0 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5a5ed740 10 client.0 _ll_register_callbacks cb 0x7f8a4000e8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a55f9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a55f9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35328/0 (socket says 172.21.15.28:35328) 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a55f9f640 1 -- 172.21.15.28:0/1034145859 learned_addr learned my addr 172.21.15.28:0/1034145859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:25:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 0x55bb824edb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824de430 0x55bb824ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 -- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824de430 msgr2=0x55bb824ee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824de430 0x55bb824ee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 -- 172.21.15.28:0/1034145859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 msgr2=0x55bb824ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 -- 172.21.15.28:0/1034145859 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb824ef630 con 0x55bb824d99a0 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a55f9f640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5579e640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 0x55bb824edb60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c09bbd0 tx=0x7f8a4c004b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fd45e30df97b60f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a38ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:25:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c09b830 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f8a4c0a4450 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a400041d0 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.411+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8a4c09bca0 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 10 client.28900 fetch_fsmap learned FSMap version 37 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55bb8255b1a0 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a400041d0 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f8a4c0a4b30 con 0x55bb824d99a0 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 10 client.28900 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 10 client.28900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 20 client.28900 populate_metadata read hostname 'smithi028' 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824e3f90 0x55bb8255d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:25:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 4 client.28900 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bb8244b150 con 0x55bb824e3f90 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824e3f90 0x55bb8255d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a38ff9640 10 client.28900 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.412+0000 7f8a54f9d640 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824e3f90 0x55bb8255d780 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a1675dea8cf9c49 server_cookie=204d88b90363e8c in_seq=0 out_seq=0 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a38ff9640 1 -- 172.21.15.28:0/1034145859 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bb8244b150 con 0x55bb824e3f90 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a38ff9640 10 client.28900 handle_command_reply: tid=0 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 client.28900 shutdown 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 msgr2=0x55bb824edb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 0x55bb824edb60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c09bbd0 tx=0x7f8a4c004b70 comp rx=0 tx=0).stop 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 shutdown_connections 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb824e3f90 0x55bb8255d780 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb824dee20 0x55bb824ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb824de430 0x55bb824ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 --2- 172.21.15.28:0/1034145859 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb824d99a0 0x55bb824edb60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a38ff9640 0 client.28900 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.413+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 >> 172.21.15.28:0/1034145859 conn(0x55bb824dfc00 msgr2=0x55bb824dd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 shutdown_connections 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a5a5ed740 1 -- 172.21.15.28:0/1034145859 wait complete. 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a5a5ed740 20 client.28900 trim_cache size 0 max 16384 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:25:51.421 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "638a2e64-6fa8-4eb2-a430-224c3346a04e": { 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "638a2e64-6fa8-4eb2-a430-224c3346a04e", 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:25:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:25:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 msgr2=0x7f8a3c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 0x7f8a3c0a53e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f8a50101d50 tx=0x7f8a40005f90 comp rx=0 tx=0).stop 2024-03-22T02:25:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 msgr2=0x7f8a50101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:25:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e9f0 tx=0x7f8a4c00edf0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 shutdown_connections 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8a3c0a2f20 0x7f8a3c0a53e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a501078d0 0x7f8a50100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8a50104fa0 0x7f8a50100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 --2- 172.21.15.28:0/1631410357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a50102670 0x7f8a50101fe0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 >> 172.21.15.28:0/1631410357 conn(0x7f8a500fc020 msgr2=0x7f8a50102fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 shutdown_connections 2024-03-22T02:25:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:25:51.414+0000 7f8a57a02640 1 -- 172.21.15.28:0/1631410357 wait complete. 2024-03-22T02:25:51.426 INFO:tasks.cephfs.filesystem:scrub status for tag:638a2e64-6fa8-4eb2-a430-224c3346a04e - {'path': '/', 'tag': '638a2e64-6fa8-4eb2-a430-224c3346a04e', 'options': 'recursive,force'} 2024-03-22T02:25:51.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1258199 11.23 MB/sec execute 513 sec latency 29.741 ms 2024-03-22T02:25:52.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1261049 11.24 MB/sec execute 514 sec latency 11.349 ms 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: pgmap v728: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:52 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: pgmap v728: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:25:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:25:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:25:52.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:25:52.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:52 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: pgmap v728: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:52 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:25:53.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1263233 11.23 MB/sec execute 515 sec latency 36.465 ms 2024-03-22T02:25:54.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1264896 11.23 MB/sec execute 516 sec latency 63.715 ms 2024-03-22T02:25:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:54 smithi120 ceph-mon[36803]: pgmap v729: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:25:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:54 smithi083 ceph-mon[36340]: pgmap v729: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:25:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:54 smithi028 ceph-mon[31916]: pgmap v729: 129 pgs: 129 active+clean; 55 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:25:55.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1267288 11.24 MB/sec execute 517 sec latency 45.648 ms 2024-03-22T02:25:56.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1268883 11.23 MB/sec execute 518 sec latency 30.830 ms 2024-03-22T02:25:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:56 smithi120 ceph-mon[36803]: pgmap v730: 129 pgs: 129 active+clean; 55 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-03-22T02:25:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:56 smithi083 ceph-mon[36340]: pgmap v730: 129 pgs: 129 active+clean; 55 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-03-22T02:25:56.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:56 smithi028 ceph-mon[31916]: pgmap v730: 129 pgs: 129 active+clean; 55 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 371 op/s 2024-03-22T02:25:57.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1271684 11.24 MB/sec execute 519 sec latency 11.747 ms 2024-03-22T02:25:58.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1273387 11.23 MB/sec execute 520 sec latency 49.466 ms 2024-03-22T02:25:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:25:58 smithi120 ceph-mon[36803]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 352 op/s 2024-03-22T02:25:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:25:58 smithi083 ceph-mon[36340]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 352 op/s 2024-03-22T02:25:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:25:58 smithi028 ceph-mon[31916]: pgmap v731: 129 pgs: 129 active+clean; 60 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.6 MiB/s wr, 352 op/s 2024-03-22T02:25:59.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1275340 11.23 MB/sec execute 521 sec latency 103.850 ms 2024-03-22T02:26:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 -- 172.21.15.28:0/3347229561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0101cb0 msgr2=0x7fc0c0102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 --2- 172.21.15.28:0/3347229561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0101cb0 0x7fc0c0102130 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009900 tx=0x7fc0b402bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 -- 172.21.15.28:0/3347229561 shutdown_connections 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 --2- 172.21.15.28:0/3347229561 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0102670 0x7fc0c0108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 --2- 172.21.15.28:0/3347229561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0101cb0 0x7fc0c0102130 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 --2- 172.21.15.28:0/3347229561 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0100ac0 0x7fc0c0100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 -- 172.21.15.28:0/3347229561 >> 172.21.15.28:0/3347229561 conn(0x7fc0c00fc050 msgr2=0x7fc0c00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 -- 172.21.15.28:0/3347229561 shutdown_connections 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.151+0000 7fc0c6439640 1 -- 172.21.15.28:0/3347229561 wait complete. 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 Processor -- start 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 -- start start 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0100ac0 0x7fc0c006aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0101cb0 0x7fc0c019cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010c740 con 0x7fc0c0102670 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010c5c0 con 0x7fc0c0101cb0 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c6439640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c010c8c0 con 0x7fc0c0100ac0 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c49d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c49d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40146/0 (socket says 172.21.15.28:40146) 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0c49d6640 1 -- 172.21.15.28:0/1788597825 learned_addr learned my addr 172.21.15.28:0/1788597825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0101cb0 0x7fc0c019cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.152+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0100ac0 0x7fc0c006aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 -- 172.21.15.28:0/1788597825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0100ac0 msgr2=0x7fc0c006aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0100ac0 0x7fc0c006aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 -- 172.21.15.28:0/1788597825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0101cb0 msgr2=0x7fc0c019cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0101cb0 0x7fc0c019cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 -- 172.21.15.28:0/1788597825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c019fa30 con 0x7fc0c0102670 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800eb10 tx=0x7fc0b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0bd7fa640 1 -- 172.21.15.28:0/1788597825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8004b20 con 0x7fc0c0102670 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c019fd20 con 0x7fc0c0102670 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0bd7fa640 1 -- 172.21.15.28:0/1788597825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b8004cc0 con 0x7fc0c0102670 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0bd7fa640 1 -- 172.21.15.28:0/1788597825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8010760 con 0x7fc0c0102670 2024-03-22T02:26:00.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.153+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c01a0290 con 0x7fc0c0102670 2024-03-22T02:26:00.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0bd7fa640 1 -- 172.21.15.28:0/1788597825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc0b8005020 con 0x7fc0c0102670 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 Processor -- start 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 -- start start 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 0x55a354b84900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0bd7fa640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 0x7fc0a80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b84ed0 0x55a354b899d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b8c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a354ae9040 con 0x55a354b844e0 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a354a919e0 con 0x55a354b84ed0 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.156+0000 7fc0c9028740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a354ae4ff0 con 0x55a354b8a040 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 0x55a354b84900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 0x7fc0a80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b8c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 0x55a354b84900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40148/0 (socket says 172.21.15.28:40148) 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/4057083325 learned_addr learned my addr 172.21.15.28:0/4057083325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b84ed0 0x55a354b899d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bd7fa640 1 -- 172.21.15.28:0/1788597825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc0b80cfea0 con 0x7fc0c0102670 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 -- 172.21.15.28:0/4057083325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b84ed0 msgr2=0x55a354b899d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b84ed0 0x55a354b899d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 -- 172.21.15.28:0/4057083325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 msgr2=0x55a354b84900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 0x55a354b84900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 -- 172.21.15.28:0/4057083325 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3547fd210 con 0x55a354b8a040 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 0x7fc0a80a5480 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0101b10 tx=0x7fc0b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b8c4d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc0b809bec0 tx=0x7fc0b800efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feece322e8b868be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0a67fc640 1 -- 172.21.15.28:0/4057083325 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b809bd00 con 0x55a354b8a040 2024-03-22T02:26:00.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0a67fc640 1 -- 172.21.15.28:0/4057083325 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b809c8c0 con 0x55a354b8a040 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.157+0000 7fc0a67fc640 1 -- 172.21.15.28:0/4057083325 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b80aee30 con 0x55a354b8a040 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- 172.21.15.28:0/4057083325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 msgr2=0x55a354b8c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b8c4d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc0b809bec0 tx=0x7fc0b800efd0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- 172.21.15.28:0/4057083325 shutdown_connections 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b8c4d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b84ed0 0x55a354b899d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 --2- 172.21.15.28:0/4057083325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b844e0 0x55a354b84900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- 172.21.15.28:0/4057083325 >> 172.21.15.28:0/4057083325 conn(0x55a354b7fa50 msgr2=0x55a354b7fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- 172.21.15.28:0/4057083325 shutdown_connections 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- 172.21.15.28:0/4057083325 wait complete. 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.158+0000 7fc0c9028740 1 -- start start 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 Processor -- start 2024-03-22T02:26:00.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b8e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b8a040 0x55a354b959b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a354b8cf90 con 0x55a354b8a040 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a354aa4db0 con 0x55a354b844e0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a354b8d110 con 0x55a354b84ed0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c9028740 10 client.0 _ll_register_callbacks cb 0x55a3547fca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0c49d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b8a040 0x55a354b959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b8e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40634/0 (socket says 172.21.15.28:40634) 2024-03-22T02:26:00.166 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1914391130 learned_addr learned my addr 172.21.15.28:0/1914391130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b8e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:36612/0 (socket says 172.21.15.28:36612) 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 msgr2=0x55a354b8e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b8e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1914391130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b8a040 msgr2=0x55a354b959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b8a040 0x55a354b959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1914391130 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3548c9620 con 0x55a354b84ed0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009230 tx=0x7fc0b4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c019ff80d54d620 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.159+0000 7fc09f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4005250 con 0x55a354b84ed0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc0b40053f0 con 0x55a354b84ed0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0b809c8c0 con 0x55a354b84ed0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc0b4051050 con 0x55a354b84ed0 2024-03-22T02:26:00.167 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 10 client.28912 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a354b98d90 con 0x55a354b84ed0 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b809c8c0 con 0x55a354b84ed0 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc0b40036f0 con 0x55a354b84ed0 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 10 client.28912 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 10 client.28912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 20 client.28912 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354b9a480 0x55a354b9c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 4 client.28912 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a354af6580 con 0x55a354b9a480 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.160+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354b9a480 0x55a354b9c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.161+0000 7fc09f7fe640 10 client.28912 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:00.168 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.161+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354b9a480 0x55a354b9c940 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0316b05381046f5 server_cookie=53426bc30e5ad55 in_seq=0 out_seq=0 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1914391130 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a354af6580 con 0x55a354b9a480 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc09f7fe640 10 client.28912 handle_command_reply: tid=0 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 client.28912 shutdown 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 msgr2=0x55a354b8e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009230 tx=0x7fc0b4004270 comp rx=0 tx=0).stop 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 shutdown_connections 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354b9a480 0x55a354b9c940 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b8a040 0x55a354b959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b84ed0 0x55a354b8e970 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1914391130 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b8e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.170 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc09f7fe640 0 client.28912 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:00.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 >> 172.21.15.28:0/1914391130 conn(0x55a354b7fa50 msgr2=0x55a354b7ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 shutdown_connections 2024-03-22T02:26:00.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 1 -- 172.21.15.28:0/1914391130 wait complete. 2024-03-22T02:26:00.171 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.163+0000 7fc0c9028740 20 client.28912 trim_cache size 0 max 16384 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.180+0000 7fc0c9028740 1 Processor -- start 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.180+0000 7fc0c9028740 1 -- start start 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b85b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.182 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01ad220 con 0x55a354b84ed0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01aaaf0 con 0x55a354b844e0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c9028740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01a8940 con 0x55a354b8a040 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40164/0 (socket says 172.21.15.28:40164) 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/171268474 learned_addr learned my addr 172.21.15.28:0/171268474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bffff640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b855c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b85b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/171268474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 msgr2=0x55a354b85b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b85b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/171268474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 msgr2=0x55a354b855c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b855c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/171268474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a354b9cfc0 con 0x55a354b84ed0 2024-03-22T02:26:00.183 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bffff640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b855c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009850 tx=0x7fc0b4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f7baccdc353e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.181+0000 7fc0a5ffb640 1 -- 172.21.15.28:0/171268474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b405a070 con 0x55a354b84ed0 2024-03-22T02:26:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0a5ffb640 1 -- 172.21.15.28:0/171268474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b4048070 con 0x55a354b84ed0 2024-03-22T02:26:00.184 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0a5ffb640 1 -- 172.21.15.28:0/171268474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b40046c0 con 0x55a354b84ed0 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 -- 172.21.15.28:0/171268474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 msgr2=0x55a354b95970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009850 tx=0x7fc0b4001e70 comp rx=0 tx=0).stop 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 -- 172.21.15.28:0/171268474 shutdown_connections 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b8a040 0x55a354b85b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b95970 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 --2- 172.21.15.28:0/171268474 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b844e0 0x55a354b855c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 -- 172.21.15.28:0/171268474 >> 172.21.15.28:0/171268474 conn(0x55a354b7fa50 msgr2=0x55a354b82cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.182+0000 7fc0c9028740 1 -- 172.21.15.28:0/171268474 shutdown_connections 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 -- 172.21.15.28:0/171268474 wait complete. 2024-03-22T02:26:00.188 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 -- start start 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 Processor -- start 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 0x55a354b93520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b8a040 0x55a354b94a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01a8fb0 con 0x55a354b84ed0 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01aa420 con 0x55a354b8a040 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01a9680 con 0x55a354b844e0 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c9028740 10 client.0 _ll_register_callbacks cb 0x7fc0b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40166/0 (socket says 172.21.15.28:40166) 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1504516446 learned_addr learned my addr 172.21.15.28:0/1504516446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0bffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 0x55a354b93520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.183+0000 7fc0c49d6640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b8a040 0x55a354b94a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1504516446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 msgr2=0x55a354b93520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 0x55a354b93520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b8a040 msgr2=0x55a354b94a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b8a040 0x55a354b94a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 -- 172.21.15.28:0/1504516446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a354b807c0 con 0x55a354b84ed0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 0x55a354b93520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0bf7fe640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009230 tx=0x7fc0b4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4380067806529286 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc09f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b405a070 con 0x55a354b84ed0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc0b4005450 con 0x55a354b84ed0 2024-03-22T02:26:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0b4048070 con 0x55a354b84ed0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc0b4005650 con 0x55a354b84ed0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0c9028740 10 client.19101 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55a354c00ac0 con 0x55a354b84ed0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.184+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0b4048070 con 0x55a354b84ed0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc0b4034730 con 0x55a354b84ed0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 10 client.19101 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 10 client.19101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 20 client.19101 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354c013b0 0x55a354c037d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 4 client.19101 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a354af08d0 con 0x55a354c013b0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354c013b0 0x55a354c037d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc0bffff640 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354c013b0 0x55a354c037d0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=466a1c0aa706c95c server_cookie=97e990676e6b1bc9 in_seq=0 out_seq=0 2024-03-22T02:26:00.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.185+0000 7fc09f7fe640 10 client.19101 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:00.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1277284 11.23 MB/sec execute 522 sec latency 202.110 ms 2024-03-22T02:26:00.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc09f7fe640 1 -- 172.21.15.28:0/1504516446 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a354af08d0 con 0x55a354c013b0 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc09f7fe640 10 client.19101 handle_command_reply: tid=0 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc0c9028740 1 client.19101 shutdown 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 msgr2=0x55a354b93a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009230 tx=0x7fc0b4000bc0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 shutdown_connections 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.590+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a354c013b0 0x55a354c037d0 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a354b8a040 0x55a354b94a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc09f7fe640 0 client.19101 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a354b84ed0 0x55a354b93a60 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 --2- 172.21.15.28:0/1504516446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a354b844e0 0x55a354b93520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 >> 172.21.15.28:0/1504516446 conn(0x55a354b7fa50 msgr2=0x55a354b7e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 shutdown_connections 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 1 -- 172.21.15.28:0/1504516446 wait complete. 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.591+0000 7fc0c9028740 20 client.19101 trim_cache size 0 max 16384 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:26:00.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:26:00.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 msgr2=0x7fc0a80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 0x7fc0a80a5480 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0101b10 tx=0x7fc0b0005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:00.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 msgr2=0x7fc0c019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800eb10 tx=0x7fc0b800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 shutdown_connections 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0a80a2fc0 0x7fc0a80a5480 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc0c0102670 0x7fc0c019f170 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc0c0101cb0 0x7fc0c019cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 --2- 172.21.15.28:0/1788597825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0c0100ac0 0x7fc0c006aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 >> 172.21.15.28:0/1788597825 conn(0x7fc0c00fc050 msgr2=0x7fc0c0104f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 shutdown_connections 2024-03-22T02:26:00.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:00.592+0000 7fc0c6439640 1 -- 172.21.15.28:0/1788597825 wait complete. 2024-03-22T02:26:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:00 smithi120 ceph-mon[36803]: pgmap v732: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:26:00.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:00 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:26:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:00 smithi083 ceph-mon[36340]: pgmap v732: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:26:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:00 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:26:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:00 smithi028 ceph-mon[31916]: pgmap v732: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.1 MiB/s wr, 399 op/s 2024-03-22T02:26:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:00 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:26:01.436 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1278905 11.23 MB/sec execute 523 sec latency 237.771 ms 2024-03-22T02:26:01.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:01 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:26:01.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:01 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:26:01.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:01 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:26:02.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1281262 11.23 MB/sec execute 524 sec latency 30.401 ms 2024-03-22T02:26:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:02 smithi120 ceph-mon[36803]: pgmap v733: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-03-22T02:26:02.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:02 smithi083 ceph-mon[36340]: pgmap v733: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-03-22T02:26:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:02 smithi028 ceph-mon[31916]: pgmap v733: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-03-22T02:26:03.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1282782 11.22 MB/sec execute 525 sec latency 61.128 ms 2024-03-22T02:26:04.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1285018 11.22 MB/sec execute 526 sec latency 76.200 ms 2024-03-22T02:26:04.751 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:04 smithi120 ceph-mon[36803]: pgmap v734: 129 pgs: 129 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-22T02:26:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:04 smithi083 ceph-mon[36340]: pgmap v734: 129 pgs: 129 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-22T02:26:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:04 smithi028 ceph-mon[31916]: pgmap v734: 129 pgs: 129 active+clean; 63 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-03-22T02:26:05.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1286233 11.22 MB/sec execute 527 sec latency 89.542 ms 2024-03-22T02:26:06.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1288491 11.22 MB/sec execute 528 sec latency 24.616 ms 2024-03-22T02:26:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:06 smithi120 ceph-mon[36803]: pgmap v735: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-03-22T02:26:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:06 smithi083 ceph-mon[36340]: pgmap v735: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-03-22T02:26:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:06 smithi028 ceph-mon[31916]: pgmap v735: 129 pgs: 129 active+clean; 62 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.0 MiB/s wr, 394 op/s 2024-03-22T02:26:07.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1289909 11.21 MB/sec execute 529 sec latency 51.669 ms 2024-03-22T02:26:08.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1292672 11.22 MB/sec execute 530 sec latency 16.891 ms 2024-03-22T02:26:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:08 smithi120 ceph-mon[36803]: pgmap v736: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 390 op/s 2024-03-22T02:26:08.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:08 smithi083 ceph-mon[36340]: pgmap v736: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 390 op/s 2024-03-22T02:26:08.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:08 smithi028 ceph-mon[31916]: pgmap v736: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 390 op/s 2024-03-22T02:26:09.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1294522 11.21 MB/sec execute 531 sec latency 29.361 ms 2024-03-22T02:26:10.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1296549 11.22 MB/sec execute 532 sec latency 47.168 ms 2024-03-22T02:26:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:10 smithi120 ceph-mon[36803]: pgmap v737: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:26:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:10 smithi083 ceph-mon[36340]: pgmap v737: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:26:10.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:10 smithi028 ceph-mon[31916]: pgmap v737: 129 pgs: 129 active+clean; 58 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.8 MiB/s wr, 360 op/s 2024-03-22T02:26:11.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1298847 11.21 MB/sec execute 533 sec latency 25.488 ms 2024-03-22T02:26:12.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1300605 11.21 MB/sec execute 534 sec latency 31.875 ms 2024-03-22T02:26:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:12 smithi120 ceph-mon[36803]: pgmap v738: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 349 op/s 2024-03-22T02:26:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:12 smithi083 ceph-mon[36340]: pgmap v738: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 349 op/s 2024-03-22T02:26:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:12 smithi028 ceph-mon[31916]: pgmap v738: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 349 op/s 2024-03-22T02:26:13.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1303480 11.22 MB/sec execute 535 sec latency 11.554 ms 2024-03-22T02:26:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- 172.21.15.28:0/2308497080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c100ab0 msgr2=0x7f5d6c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/2308497080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c100ed0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c009900 tx=0x7f5d5c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:13.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- 172.21.15.28:0/2308497080 shutdown_connections 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/2308497080 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c102660 0x7f5d6c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/2308497080 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/2308497080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c100ed0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- 172.21.15.28:0/2308497080 >> 172.21.15.28:0/2308497080 conn(0x7f5d6c0fc020 msgr2=0x7f5d6c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- 172.21.15.28:0/2308497080 shutdown_connections 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- 172.21.15.28:0/2308497080 wait complete. 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 Processor -- start 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.689+0000 7f5d72d0d640 1 -- start start 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c10b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c102660 0x7f5d6c109fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10d850 con 0x7f5d6c101ca0 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10d6d0 con 0x7f5d6c102660 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c10d9d0 con 0x7f5d6c100ab0 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54296/0 (socket says 172.21.15.28:54296) 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d70aa9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c10b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 -- 172.21.15.28:0/3549527260 learned_addr learned my addr 172.21.15.28:0/3549527260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d712aa640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c102660 0x7f5d6c109fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 -- 172.21.15.28:0/3549527260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c100ab0 msgr2=0x7f5d6c10b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c10b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 -- 172.21.15.28:0/3549527260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c102660 msgr2=0x7f5d6c109fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c102660 0x7f5d6c109fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 -- 172.21.15.28:0/3549527260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6c10a840 con 0x7f5d6c101ca0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d6bfff640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000e9f0 tx=0x7f5d6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d69ffb640 1 -- 172.21.15.28:0/3549527260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6000cd80 con 0x7f5d6c101ca0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d69ffb640 1 -- 172.21.15.28:0/3549527260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d60010910 con 0x7f5d6c101ca0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d6c1a1f60 con 0x7f5d6c101ca0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.690+0000 7f5d69ffb640 1 -- 172.21.15.28:0/3549527260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60018990 con 0x7f5d6c101ca0 2024-03-22T02:26:13.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.691+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6c1a23d0 con 0x7f5d6c101ca0 2024-03-22T02:26:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d69ffb640 1 -- 172.21.15.28:0/3549527260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5d60010ac0 con 0x7f5d6c101ca0 2024-03-22T02:26:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 Processor -- start 2024-03-22T02:26:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 -- start start 2024-03-22T02:26:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a63f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d69ffb640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 0x7f5d540a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a640040 0x55e55a6424d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e55a59f040 con 0x55e55a63a4e0 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e55a5479e0 con 0x55e55a63aed0 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d759cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e55a59aff0 con 0x55e55a640040 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54306/0 (socket says 172.21.15.28:54306) 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d6bfff640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 0x7f5d540a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d712aa640 1 -- 172.21.15.28:0/2706084214 learned_addr learned my addr 172.21.15.28:0/2706084214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a63f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.701 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.693+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a640040 0x55e55a6424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d69ffb640 1 -- 172.21.15.28:0/3549527260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5d600cfac0 con 0x7f5d6c101ca0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 -- 172.21.15.28:0/2706084214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a640040 msgr2=0x55e55a6424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a640040 0x55e55a6424d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 -- 172.21.15.28:0/2706084214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 msgr2=0x55e55a63f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a63f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 -- 172.21.15.28:0/2706084214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e55a2b3210 con 0x55e55a63a4e0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a63f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f5d64012ac0 tx=0x7f5d64009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ec832f7b82051b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d6bfff640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 0x7f5d540a5500 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5d60098240 tx=0x7f5d60066e90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d52ffd640 1 -- 172.21.15.28:0/2706084214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64012110 con 0x55e55a63a4e0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d52ffd640 1 -- 172.21.15.28:0/2706084214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d640122b0 con 0x55e55a63a4e0 2024-03-22T02:26:13.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.694+0000 7f5d52ffd640 1 -- 172.21.15.28:0/2706084214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400eb30 con 0x55e55a63a4e0 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- 172.21.15.28:0/2706084214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 msgr2=0x55e55a63a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f5d64012ac0 tx=0x7f5d64009aa0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- 172.21.15.28:0/2706084214 shutdown_connections 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a640040 0x55e55a6424d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a63f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2706084214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63a4e0 0x55e55a63a900 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- 172.21.15.28:0/2706084214 >> 172.21.15.28:0/2706084214 conn(0x55e55a635a50 msgr2=0x55e55a635e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- 172.21.15.28:0/2706084214 shutdown_connections 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- 172.21.15.28:0/2706084214 wait complete. 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 -- start start 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.695+0000 7f5d759cd740 1 Processor -- start 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a644510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a640040 0x55e55a64ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e55a642ff0 con 0x55e55a63aed0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e55a55adb0 con 0x55e55a640040 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e55a643170 con 0x55e55a63a4e0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d759cd740 10 client.0 _ll_register_callbacks cb 0x55e55a2b2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54308/0 (socket says 172.21.15.28:54308) 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 -- 172.21.15.28:0/1945518790 learned_addr learned my addr 172.21.15.28:0/1945518790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d712aa640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a640040 0x55e55a64ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d712aa640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a644510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 -- 172.21.15.28:0/1945518790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 msgr2=0x55e55a644510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a644510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 -- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a640040 msgr2=0x55e55a64ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a640040 0x55e55a64ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 -- 172.21.15.28:0/1945518790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e55a37f620 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02cfd0 tx=0x7f5d5c004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c90b5154e95cc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.696+0000 7f5d4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c005d50 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5d5c005230 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e55a5ac580 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5d5c04e050 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 10 client.19113 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e55a64ede0 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e55a5ac580 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f5d5c0035f0 con 0x55e55a63aed0 2024-03-22T02:26:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 10 client.19113 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 10 client.19113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 20 client.19113 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a650520 0x55e55a652950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 4 client.19113 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e55a5a68d0 con 0x55e55a650520 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.697+0000 7f5d712aa640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a650520 0x55e55a652950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.698+0000 7f5d4bfff640 10 client.19113 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.698+0000 7f5d712aa640 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a650520 0x55e55a652950 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e67d8863dd1815d2 server_cookie=c149780df7aef327 in_seq=0 out_seq=0 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d4bfff640 1 -- 172.21.15.28:0/1945518790 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e55a5a68d0 con 0x55e55a650520 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d4bfff640 10 client.19113 handle_command_reply: tid=0 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 client.19113 shutdown 2024-03-22T02:26:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 msgr2=0x55e55a644a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c02cfd0 tx=0x7f5d5c004250 comp rx=0 tx=0).stop 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 shutdown_connections 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a650520 0x55e55a652950 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a640040 0x55e55a64ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a63aed0 0x55e55a644a50 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d4bfff640 0 client.19113 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 --2- 172.21.15.28:0/1945518790 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a644510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 >> 172.21.15.28:0/1945518790 conn(0x55e55a635a50 msgr2=0x55e55a634c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 shutdown_connections 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 1 -- 172.21.15.28:0/1945518790 wait complete. 2024-03-22T02:26:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.699+0000 7f5d759cd740 20 client.19113 trim_cache size 0 max 16384 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 Processor -- start 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 -- start start 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a64b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1af450 con 0x55e55a640040 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1acd20 con 0x55e55a63aed0 2024-03-22T02:26:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d759cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1aab70 con 0x55e55a63a4e0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55124/0 (socket says 172.21.15.28:55124) 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d712aa640 1 -- 172.21.15.28:0/3384362679 learned_addr learned my addr 172.21.15.28:0/3384362679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d70aa9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a64b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.715+0000 7f5d712aa640 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 -- 172.21.15.28:0/3384362679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 msgr2=0x55e55a64b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a64b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 -- 172.21.15.28:0/3384362679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 msgr2=0x55e55a6343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 -- 172.21.15.28:0/3384362679 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e55a63a8c0 con 0x55e55a63a4e0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d712aa640 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006010 tx=0x7f5d64009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d23398cb2a28bc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d527fc640 1 -- 172.21.15.28:0/3384362679 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d640313d0 con 0x55e55a63a4e0 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d527fc640 1 -- 172.21.15.28:0/3384362679 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d6400ecf0 con 0x55e55a63a4e0 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.716+0000 7f5d527fc640 1 -- 172.21.15.28:0/3384362679 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d64005250 con 0x55e55a63a4e0 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- 172.21.15.28:0/3384362679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 msgr2=0x55e55a63f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5d64006010 tx=0x7f5d64009f00 comp rx=0 tx=0).stop 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- 172.21.15.28:0/3384362679 shutdown_connections 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63aed0 0x55e55a64b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- 172.21.15.28:0/3384362679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63a4e0 0x55e55a63f540 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- 172.21.15.28:0/3384362679 >> 172.21.15.28:0/3384362679 conn(0x55e55a635a50 msgr2=0x55e55a639540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- 172.21.15.28:0/3384362679 shutdown_connections 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- 172.21.15.28:0/3384362679 wait complete. 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- start start 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 Processor -- start 2024-03-22T02:26:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63a4e0 0x55e55a64b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 0x55e55a64b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1ab1e0 con 0x55e55a640040 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1ac650 con 0x55e55a63a4e0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6c1ab8b0 con 0x55e55a63aed0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.717+0000 7f5d759cd740 10 client.0 _ll_register_callbacks cb 0x7f5d5c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54336/0 (socket says 172.21.15.28:54336) 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 -- 172.21.15.28:0/2531845146 learned_addr learned my addr 172.21.15.28:0/2531845146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 0x55e55a64b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63a4e0 0x55e55a64b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 -- 172.21.15.28:0/2531845146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 msgr2=0x55e55a64b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 0x55e55a64b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 -- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63a4e0 msgr2=0x55e55a64b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63a4e0 0x55e55a64b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 -- 172.21.15.28:0/2531845146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e55a649e30 con 0x55e55a640040 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 0x55e55a64b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d712aa640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f5d640128d0 tx=0x7f5d64012470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebcc3479a7ebc68 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d4bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d640315f0 con 0x55e55a640040 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d6400ecf0 con 0x55e55a640040 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f5d6400e7e0 con 0x55e55a640040 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5d64041050 con 0x55e55a640040 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d759cd740 10 client.19119 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:13.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e55a6b71e0 con 0x55e55a640040 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.718+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d6400ecf0 con 0x55e55a640040 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f5d6401f460 con 0x55e55a640040 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 10 client.19119 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 10 client.19119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 20 client.19119 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a6b7900 0x55e55a6b9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 4 client.19119 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e55a524770 con 0x55e55a6b7900 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a6b7900 0x55e55a6b9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:13.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d70aa9640 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a6b7900 0x55e55a6b9cf0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72f273c5cb558763 server_cookie=78a8cde7422abf6e in_seq=0 out_seq=0 2024-03-22T02:26:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:13.719+0000 7f5d4bfff640 10 client.19119 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d4bfff640 1 -- 172.21.15.28:0/2531845146 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e55a524770 con 0x55e55a6b7900 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d4bfff640 10 client.19119 handle_command_reply: tid=0 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 client.19119 shutdown 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 msgr2=0x55e55a6497f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f5d640128d0 tx=0x7f5d64012470 comp rx=0 tx=0).stop 2024-03-22T02:26:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 shutdown_connections 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e55a6b7900 0x55e55a6b9cf0 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e55a640040 0x55e55a6497f0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d4bfff640 0 client.19119 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e55a63aed0 0x55e55a64b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 --2- 172.21.15.28:0/2531845146 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e55a63a4e0 0x55e55a64b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 >> 172.21.15.28:0/2531845146 conn(0x55e55a635a50 msgr2=0x55e55a639540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 shutdown_connections 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 1 -- 172.21.15.28:0/2531845146 wait complete. 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.160+0000 7f5d759cd740 20 client.19119 trim_cache size 0 max 16384 2024-03-22T02:26:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:26:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:26:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:26:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:26:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 msgr2=0x7f5d540a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 0x7f5d540a5500 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5d60098240 tx=0x7f5d60066e90 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 msgr2=0x7f5d6c109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000e9f0 tx=0x7f5d6000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 shutdown_connections 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5d540a3040 0x7f5d540a5500 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5d6c102660 0x7f5d6c109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d6c101ca0 0x7f5d6c109a70 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 --2- 172.21.15.28:0/3549527260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5d6c100ab0 0x7f5d6c10b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.161+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 >> 172.21.15.28:0/3549527260 conn(0x7f5d6c0fc020 msgr2=0x7f5d6c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.162+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 shutdown_connections 2024-03-22T02:26:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:14.162+0000 7f5d72d0d640 1 -- 172.21.15.28:0/3549527260 wait complete. 2024-03-22T02:26:14.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1304048 11.21 MB/sec execute 536 sec latency 334.087 ms 2024-03-22T02:26:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:14 smithi120 ceph-mon[36803]: pgmap v739: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-03-22T02:26:14.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:14 smithi083 ceph-mon[36340]: pgmap v739: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-03-22T02:26:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:14 smithi028 ceph-mon[31916]: pgmap v739: 129 pgs: 129 active+clean; 61 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-03-22T02:26:15.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1306123 11.21 MB/sec execute 537 sec latency 46.512 ms 2024-03-22T02:26:16.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1307774 11.20 MB/sec execute 538 sec latency 28.893 ms 2024-03-22T02:26:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:16 smithi120 ceph-mon[36803]: pgmap v740: 129 pgs: 129 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-03-22T02:26:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:16 smithi083 ceph-mon[36340]: pgmap v740: 129 pgs: 129 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-03-22T02:26:16.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:16 smithi028 ceph-mon[31916]: pgmap v740: 129 pgs: 129 active+clean; 64 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 386 op/s 2024-03-22T02:26:17.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1310597 11.21 MB/sec execute 539 sec latency 9.456 ms 2024-03-22T02:26:18.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1312655 11.20 MB/sec execute 540 sec latency 28.840 ms 2024-03-22T02:26:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:18 smithi120 ceph-mon[36803]: pgmap v741: 129 pgs: 129 active+clean; 65 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 305 op/s 2024-03-22T02:26:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:18 smithi083 ceph-mon[36340]: pgmap v741: 129 pgs: 129 active+clean; 65 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 305 op/s 2024-03-22T02:26:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:18 smithi028 ceph-mon[31916]: pgmap v741: 129 pgs: 129 active+clean; 65 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 305 op/s 2024-03-22T02:26:19.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1314599 11.21 MB/sec execute 541 sec latency 14.844 ms 2024-03-22T02:26:20.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1316405 11.20 MB/sec execute 542 sec latency 55.510 ms 2024-03-22T02:26:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:20 smithi120 ceph-mon[36803]: pgmap v742: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 391 op/s 2024-03-22T02:26:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:20 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:26:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:20 smithi083 ceph-mon[36340]: pgmap v742: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 391 op/s 2024-03-22T02:26:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:20 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:26:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:20 smithi028 ceph-mon[31916]: pgmap v742: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 391 op/s 2024-03-22T02:26:20.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:20 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:26:21.427 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:26:21.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:26:21.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1318149 11.20 MB/sec execute 543 sec latency 25.578 ms 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- 172.21.15.28:0/2408191961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00100a90 msgr2=0x7f9f00100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 --2- 172.21.15.28:0/2408191961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00100a90 0x7f9f00100eb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0009900 tx=0x7f9ef002bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- 172.21.15.28:0/2408191961 shutdown_connections 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 --2- 172.21.15.28:0/2408191961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00102640 0x7f9f00108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 --2- 172.21.15.28:0/2408191961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00101c80 0x7f9f00102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 --2- 172.21.15.28:0/2408191961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00100a90 0x7f9f00100eb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- 172.21.15.28:0/2408191961 >> 172.21.15.28:0/2408191961 conn(0x7f9f000fc020 msgr2=0x7f9f000fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- 172.21.15.28:0/2408191961 shutdown_connections 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- 172.21.15.28:0/2408191961 wait complete. 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 Processor -- start 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.538+0000 7f9f04e39640 1 -- start start 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00101c80 0x7f9f00198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 0x7f9f0019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0010d830 con 0x7f9f00100a90 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0010d6b0 con 0x7f9f00102640 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9f04e39640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0010d9b0 con 0x7f9f00101c80 2024-03-22T02:26:21.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54354/0 (socket says 172.21.15.28:54354) 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 -- 172.21.15.28:0/568683867 learned_addr learned my addr 172.21.15.28:0/568683867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00101c80 0x7f9f00198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efed9d640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 0x7f9f0019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 -- 172.21.15.28:0/568683867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00101c80 msgr2=0x7f9f00198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00101c80 0x7f9f00198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 -- 172.21.15.28:0/568683867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 msgr2=0x7f9f0019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 0x7f9f0019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 -- 172.21.15.28:0/568683867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0019fb30 con 0x7f9f00100a90 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efed9d640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 0x7f9f0019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:21.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9efe59c640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0000c00 tx=0x7f9ef002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.539+0000 7f9eef7fe640 1 -- 172.21.15.28:0/568683867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0041070 con 0x7f9f00100a90 2024-03-22T02:26:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.540+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0019fdc0 con 0x7f9f00100a90 2024-03-22T02:26:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.540+0000 7f9eef7fe640 1 -- 172.21.15.28:0/568683867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef0004450 con 0x7f9f00100a90 2024-03-22T02:26:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.540+0000 7f9eef7fe640 1 -- 172.21.15.28:0/568683867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0034440 con 0x7f9f00100a90 2024-03-22T02:26:21.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.540+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f001a01f0 con 0x7f9f00100a90 2024-03-22T02:26:21.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.542+0000 7f9eef7fe640 1 -- 172.21.15.28:0/568683867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9ef0044070 con 0x7f9f00100a90 2024-03-22T02:26:21.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.542+0000 7f9eef7fe640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 0x7f9ee80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.542+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 0x7f9ee80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.542+0000 7f9eef7fe640 1 -- 172.21.15.28:0/568683867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f9ef00f12a0 con 0x7f9f00100a90 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.542+0000 7f9f07a24740 1 Processor -- start 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 -- start start 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 0x7f9ee80a53e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40095f0 tx=0x7f9ef4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582c3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5571582208c0 con 0x5571582bb430 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5571581c8c20 con 0x5571582bbe20 2024-03-22T02:26:21.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9f07a24740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55715821c870 con 0x5571582c0f90 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582c3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54362/0 (socket says 172.21.15.28:54362) 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efe59c640 1 -- 172.21.15.28:0/3657147894 learned_addr learned my addr 172.21.15.28:0/3657147894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 -- 172.21.15.28:0/3657147894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 msgr2=0x5571582c0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 -- 172.21.15.28:0/3657147894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 msgr2=0x5571582bb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efed9d640 1 -- 172.21.15.28:0/3657147894 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571581ed150 con 0x5571582c0f90 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.543+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9efed9d640 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582c3420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8005fd0 tx=0x7f9ef8011bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da1e57181359e529 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9eecff9640 1 -- 172.21.15.28:0/3657147894 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8004d20 con 0x5571582c0f90 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9eecff9640 1 -- 172.21.15.28:0/3657147894 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef800f9a0 con 0x5571582c0f90 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9eecff9640 1 -- 172.21.15.28:0/3657147894 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef80044a0 con 0x5571582c0f90 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9f07a24740 1 -- 172.21.15.28:0/3657147894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 msgr2=0x5571582c3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.544+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582c3420 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8005fd0 tx=0x7f9ef8011bf0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- 172.21.15.28:0/3657147894 shutdown_connections 2024-03-22T02:26:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582c3420 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3657147894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582bb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- 172.21.15.28:0/3657147894 >> 172.21.15.28:0/3657147894 conn(0x5571582b69a0 msgr2=0x5571582b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- 172.21.15.28:0/3657147894 shutdown_connections 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- 172.21.15.28:0/3657147894 wait complete. 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- start start 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 Processor -- start 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5571582c3f40 con 0x5571582bb430 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5571581dc130 con 0x5571582bbe20 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5571582c40c0 con 0x5571582c0f90 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9f07a24740 10 client.0 _ll_register_callbacks cb 0x5571581dbb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9efe59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9efed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9efed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55162/0 (socket says 172.21.15.28:55162) 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9efed9d640 1 -- 172.21.15.28:0/3208214426 learned_addr learned my addr 172.21.15.28:0/3208214426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.545+0000 7f9efdd9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 -- 172.21.15.28:0/3208214426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 msgr2=0x5571582cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 -- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 msgr2=0x5571582c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 -- 172.21.15.28:0/3208214426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557157ffe7d0 con 0x5571582bb430 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efed9d640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582c58f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00bd400 tx=0x7f9ef00bd590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2b75b24dcab998 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9ee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef004b040 con 0x5571582bb430 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f9ef00035f0 con 0x5571582bb430 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef800f9a0 con 0x5571582bb430 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9ef0004c80 con 0x5571582bb430 2024-03-22T02:26:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9f07a24740 10 client.19131 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557157f343f0 con 0x5571582bb430 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.546+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef800f9a0 con 0x5571582bb430 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f9ef00c5bd0 con 0x5571582bb430 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 10 client.19131 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 10 client.19131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 20 client.19131 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5571582d16a0 0x5571582d3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 4 client.19131 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55715822de00 con 0x5571582d16a0 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5571582d16a0 0x5571582d3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5571582d16a0 0x5571582d3b30 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d74c5cb0511aff server_cookie=fee1820812bdc78a in_seq=0 out_seq=0 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.547+0000 7f9ee5ffb640 10 client.19131 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3208214426 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55715822de00 con 0x5571582d16a0 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9ee5ffb640 10 client.19131 handle_command_reply: tid=0 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 client.19131 shutdown 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 msgr2=0x5571582c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582c58f0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f9ef00bd400 tx=0x7f9ef00bd590 comp rx=0 tx=0).stop 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 shutdown_connections 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5571582d16a0 0x5571582d3b30 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9ee5ffb640 0 client.19131 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bbe20 0x5571582c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3208214426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bb430 0x5571582c58f0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 >> 172.21.15.28:0/3208214426 conn(0x5571582b69a0 msgr2=0x5571582b5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 shutdown_connections 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 1 -- 172.21.15.28:0/3208214426 wait complete. 2024-03-22T02:26:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.548+0000 7f9f07a24740 20 client.19131 trim_cache size 0 max 16384 2024-03-22T02:26:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 Processor -- start 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 -- start start 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582b5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001ad300 con 0x5571582bbe20 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001aabd0 con 0x5571582bb430 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9f07a24740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001a8a20 con 0x5571582c0f90 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efed9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582b58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55166/0 (socket says 172.21.15.28:55166) 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efed9d640 1 -- 172.21.15.28:0/3563086294 learned_addr learned my addr 172.21.15.28:0/3563086294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582b5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3563086294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 msgr2=0x5571582b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3563086294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 msgr2=0x5571582b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582b5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.568+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3563086294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557157ef40f0 con 0x5571582bbe20 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cce00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x5571582d43e0 tx=0x7f9ef4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac693db5b106d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9ee7fff640 1 -- 172.21.15.28:0/3563086294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef4031030 con 0x5571582bbe20 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9ee7fff640 1 -- 172.21.15.28:0/3563086294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef400edf0 con 0x5571582bbe20 2024-03-22T02:26:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9ee7fff640 1 -- 172.21.15.28:0/3563086294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef402c340 con 0x5571582bbe20 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9f07a24740 1 -- 172.21.15.28:0/3563086294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 msgr2=0x5571582cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.569+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cce00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x5571582d43e0 tx=0x7f9ef4014000 comp rx=0 tx=0).stop 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- 172.21.15.28:0/3563086294 shutdown_connections 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cce00 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3563086294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582b5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- 172.21.15.28:0/3563086294 >> 172.21.15.28:0/3563086294 conn(0x5571582b69a0 msgr2=0x5571582c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- 172.21.15.28:0/3563086294 shutdown_connections 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- 172.21.15.28:0/3563086294 wait complete. 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- start start 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 Processor -- start 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582cad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cc250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001a9090 con 0x5571582bbe20 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001aa500 con 0x5571582bb430 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9f001a9760 con 0x5571582c0f90 2024-03-22T02:26:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9f07a24740 10 client.0 _ll_register_callbacks cb 0x7f9ef00095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9efdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9efdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54392/0 (socket says 172.21.15.28:54392) 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3978207132 learned_addr learned my addr 172.21.15.28:0/3978207132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9efed9d640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.570+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3978207132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 msgr2=0x5571582cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 msgr2=0x5571582cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582cad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 -- 172.21.15.28:0/3978207132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571582cc840 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9efdd9b640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f9ef402f040 tx=0x7f9ef401d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5211bb484bf7973d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef403b060 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f9ef4005040 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ef400edf0 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9ef4042050 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 10 client.19137 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x557158338320 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ef400edf0 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f9ef40046c0 con 0x5571582bbe20 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 10 client.19137 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 10 client.19137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 20 client.19137 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557158338c90 0x55715833b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 4 client.19137 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.571+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557158228150 con 0x557158338c90 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.572+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557158338c90 0x55715833b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.572+0000 7f9efe59c640 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557158338c90 0x55715833b0b0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21b6c012431ff1a server_cookie=9218600cb4348a9 in_seq=0 out_seq=0 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.572+0000 7f9ee5ffb640 10 client.19137 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9ee5ffb640 1 -- 172.21.15.28:0/3978207132 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557158228150 con 0x557158338c90 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9ee5ffb640 10 client.19137 handle_command_reply: tid=0 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 client.19137 shutdown 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 msgr2=0x5571582cb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f9ef402f040 tx=0x7f9ef401d8f0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 shutdown_connections 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557158338c90 0x55715833b0b0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5571582c0f90 0x5571582cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5571582bbe20 0x5571582cb290 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9ee5ffb640 0 client.19137 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 --2- 172.21.15.28:0/3978207132 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5571582bb430 0x5571582cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 >> 172.21.15.28:0/3978207132 conn(0x5571582b69a0 msgr2=0x5571582b9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 shutdown_connections 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.575+0000 7f9f07a24740 1 -- 172.21.15.28:0/3978207132 wait complete. 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f07a24740 20 client.19137 trim_cache size 0 max 16384 2024-03-22T02:26:21.583 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:26:21.584 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:26:21.584 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 msgr2=0x7f9ee80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 0x7f9ee80a53e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40095f0 tx=0x7f9ef4005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 msgr2=0x7f9f00197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f9ef0000c00 tx=0x7f9ef002c640 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 shutdown_connections 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9ee80a2f20 0x7f9ee80a53e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9f00102640 0x7f9f0019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f00101c80 0x7f9f00198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 --2- 172.21.15.28:0/568683867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f00100a90 0x7f9f00197f10 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 >> 172.21.15.28:0/568683867 conn(0x7f9f000fc020 msgr2=0x7f9f00105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 shutdown_connections 2024-03-22T02:26:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.576+0000 7f9f04e39640 1 -- 172.21.15.28:0/568683867 wait complete. 2024-03-22T02:26:21.588 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:26:21.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:26:21.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:21.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:21.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:26:21.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- 172.21.15.28:0/2077534356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688107890 msgr2=0x7f7688109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 --2- 172.21.15.28:0/2077534356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688107890 0x7f7688109ce0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f7678009900 tx=0x7f767802bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- 172.21.15.28:0/2077534356 shutdown_connections 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 --2- 172.21.15.28:0/2077534356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688107890 0x7f7688109ce0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 --2- 172.21.15.28:0/2077534356 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688104f60 0x7f7688107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 --2- 172.21.15.28:0/2077534356 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688102630 0x7f7688104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- 172.21.15.28:0/2077534356 >> 172.21.15.28:0/2077534356 conn(0x7f76880fc020 msgr2=0x7f76880fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- 172.21.15.28:0/2077534356 shutdown_connections 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- 172.21.15.28:0/2077534356 wait complete. 2024-03-22T02:26:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 Processor -- start 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.738+0000 7f768ec2a640 1 -- start start 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688102630 0x7f7688197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688107890 0x7f768819f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f768810d850 con 0x7f7688104f60 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f768810d6d0 con 0x7f7688102630 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f768810d9d0 con 0x7f7688107890 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41816/0 (socket says 172.21.15.28:41816) 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 -- 172.21.15.28:0/1319737109 learned_addr learned my addr 172.21.15.28:0/1319737109 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768d1c7640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688107890 0x7f768819f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768c9c6640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688102630 0x7f7688197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 -- 172.21.15.28:0/1319737109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688107890 msgr2=0x7f768819f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688107890 0x7f768819f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 -- 172.21.15.28:0/1319737109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688102630 msgr2=0x7f7688197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688102630 0x7f7688197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 -- 172.21.15.28:0/1319737109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768819fac0 con 0x7f7688104f60 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7687fff640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f767c00e9c0 tx=0x7f767c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c00cd80 con 0x7f7688104f60 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767c005500 con 0x7f7688104f60 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768819fdb0 con 0x7f7688104f60 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.739+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c010690 con 0x7f7688104f60 2024-03-22T02:26:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.740+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76881a0270 con 0x7f7688104f60 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.741+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76880fdb10 con 0x7f7688104f60 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.742+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f767c0046c0 con 0x7f7688104f60 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.742+0000 7f7685ffb640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 0x7f76700a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.743+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f767c0cfd30 con 0x7f7688104f60 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.743+0000 7f768c9c6640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 0x7f76700a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:21.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.743+0000 7f768c9c6640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 0x7f76700a53e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7680005fd0 tx=0x7f7680009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:21.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.746+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f767c053740 con 0x7f7688104f60 2024-03-22T02:26:21.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:21.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:21.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:26:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:26:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:26:21.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:26:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.983+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f768802d3b0 con 0x7f7688104f60 2024-03-22T02:26:21.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.984+0000 7f7685ffb640 1 -- 172.21.15.28:0/1319737109 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+11866 (secure 0 0 0) 0x7f767c09c750 con 0x7f7688104f60 2024-03-22T02:26:21.985 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:26:21.986 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":37,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:24:45.372919+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_27700":{"gid":27700,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/3084572880","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":3084572880},{"type":"v1","addr":"172.21.15.83:6837","nonce":3084572880}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24509,24478,24508]},"id":1}]} 2024-03-22T02:26:21.986 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 msgr2=0x7f76700a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 0x7f76700a53e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7680005fd0 tx=0x7f7680009500 comp rx=0 tx=0).stop 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 msgr2=0x7f76881983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f767c00e9c0 tx=0x7f767c00ee80 comp rx=0 tx=0).stop 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 shutdown_connections 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f76700a2f20 0x7f76700a53e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7688107890 0x7f768819f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7688104f60 0x7f76881983e0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 --2- 172.21.15.28:0/1319737109 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7688102630 0x7f7688197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 >> 172.21.15.28:0/1319737109 conn(0x7f76880fc020 msgr2=0x7f7688102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 shutdown_connections 2024-03-22T02:26:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:21.986+0000 7f768ec2a640 1 -- 172.21.15.28:0/1319737109 wait complete. 2024-03-22T02:26:21.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- 172.21.15.28:0/2612822514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8101c80 msgr2=0x7f13b8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 --2- 172.21.15.28:0/2612822514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8101c80 0x7f13b8102100 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f13ac009900 tx=0x7f13ac02bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- 172.21.15.28:0/2612822514 shutdown_connections 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 --2- 172.21.15.28:0/2612822514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8102640 0x7f13b8108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 --2- 172.21.15.28:0/2612822514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8101c80 0x7f13b8102100 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 --2- 172.21.15.28:0/2612822514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8100a90 0x7f13b8100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- 172.21.15.28:0/2612822514 >> 172.21.15.28:0/2612822514 conn(0x7f13b80fc020 msgr2=0x7f13b80fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- 172.21.15.28:0/2612822514 shutdown_connections 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- 172.21.15.28:0/2612822514 wait complete. 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 Processor -- start 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.150+0000 7f13be58c640 1 -- start start 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 0x7f13b8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8102640 0x7f13b819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13b810d830 con 0x7f13b8100a90 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b810d6b0 con 0x7f13b8101c80 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13be58c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f13b810d9b0 con 0x7f13b8102640 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41836/0 (socket says 172.21.15.28:41836) 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 -- 172.21.15.28:0/2090260720 learned_addr learned my addr 172.21.15.28:0/2090260720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8102640 0x7f13b819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 0x7f13b8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 -- 172.21.15.28:0/2090260720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8102640 msgr2=0x7f13b819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8102640 0x7f13b819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 -- 172.21.15.28:0/2090260720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 msgr2=0x7f13b8198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 0x7f13b8198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 -- 172.21.15.28:0/2090260720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b819fb00 con 0x7f13b8100a90 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 0x7f13b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.151+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f13a800e9c0 tx=0x7f13a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.152+0000 7f13b57fa640 1 -- 172.21.15.28:0/2090260720 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a800cd80 con 0x7f13b8100a90 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.152+0000 7f13b57fa640 1 -- 172.21.15.28:0/2090260720 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a80035f0 con 0x7f13b8100a90 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.152+0000 7f13b57fa640 1 -- 172.21.15.28:0/2090260720 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a8010690 con 0x7f13b8100a90 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.152+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b819fdf0 con 0x7f13b8100a90 2024-03-22T02:26:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.152+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b81a0200 con 0x7f13b8100a90 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.154+0000 7f13b57fa640 1 -- 172.21.15.28:0/2090260720 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f13a80037a0 con 0x7f13b8100a90 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.154+0000 7f13c1179740 1 Processor -- start 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.154+0000 7f13c1179740 1 -- start start 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ac820 0x5557327acc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ad210 0x5557327b1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5557327118d0 con 0x5557327b2380 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5557326b9e00 con 0x5557327ac820 2024-03-22T02:26:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13c1179740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55573270d880 con 0x5557327ad210 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13bcb29640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ac820 0x5557327acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41844/0 (socket says 172.21.15.28:41844) 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 -- 172.21.15.28:0/2697980039 learned_addr learned my addr 172.21.15.28:0/2697980039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ad210 0x5557327b1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b57fa640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 0x7f13a00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 -- 172.21.15.28:0/2697980039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ad210 msgr2=0x5557327b1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ad210 0x5557327b1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 -- 172.21.15.28:0/2697980039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ac820 msgr2=0x5557327acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ac820 0x5557327acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 -- 172.21.15.28:0/2697980039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557324253f0 con 0x5557327b2380 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 0x7f13a00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b57fa640 1 -- 172.21.15.28:0/2090260720 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f13a80cfc10 con 0x7f13b8100a90 2024-03-22T02:26:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f13ac036040 tx=0x7f13ac009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1214a093c860cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f139e7fc640 1 -- 172.21.15.28:0/2697980039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac02cb80 con 0x5557327b2380 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f139e7fc640 1 -- 172.21.15.28:0/2697980039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13ac005040 con 0x5557327b2380 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.155+0000 7f139e7fc640 1 -- 172.21.15.28:0/2697980039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ac004920 con 0x5557327b2380 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- 172.21.15.28:0/2697980039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 msgr2=0x5557327b4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 0x7f13a00a53e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009fd0 tx=0x7f13b0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f13ac036040 tx=0x7f13ac009560 comp rx=0 tx=0).stop 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- 172.21.15.28:0/2697980039 shutdown_connections 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327b4810 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ad210 0x5557327b1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 --2- 172.21.15.28:0/2697980039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ac820 0x5557327acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- 172.21.15.28:0/2697980039 >> 172.21.15.28:0/2697980039 conn(0x5557327a7d90 msgr2=0x5557327a8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- 172.21.15.28:0/2697980039 shutdown_connections 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- 172.21.15.28:0/2697980039 wait complete. 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.156+0000 7f13c1179740 1 -- start start 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 Processor -- start 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327b6c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5557327b52d0 con 0x5557327b2380 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5557326cd300 con 0x5557327ad210 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5557327b5450 con 0x5557327ac820 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13c1179740 10 client.0 _ll_register_callbacks cb 0x555732708e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327b6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41858/0 (socket says 172.21.15.28:41858) 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 -- 172.21.15.28:0/2910111061 learned_addr learned my addr 172.21.15.28:0/2910111061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 -- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 msgr2=0x5557327b6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327b6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 -- 172.21.15.28:0/2910111061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 msgr2=0x5557327b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13bcb29640 1 -- 172.21.15.28:0/2910111061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557324ef7d0 con 0x5557327b2380 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.157+0000 7f13b77fe640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13bcb29640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f13b000fb90 tx=0x7f13b0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8c718c88b67752 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b001ea00 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f13b001eba0 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13ac005040 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f13b001eda0 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 10 client.19155 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5557327c15c0 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13ac005040 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f13b0028440 con 0x5557327b2380 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 10 client.19155 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 10 client.19155 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 20 client.19155 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5557327c2d30 0x5557327c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 4 client.19155 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.158+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55573271ee10 con 0x5557327c2d30 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5557327c2d30 0x5557327c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13977fe640 10 client.19155 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13b7fff640 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5557327c2d30 0x5557327c5190 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=14a2fe6e5f506e43 server_cookie=6eea0c7cd40d4d26 in_seq=0 out_seq=0 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13977fe640 1 -- 172.21.15.28:0/2910111061 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55573271ee10 con 0x5557327c2d30 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13977fe640 10 client.19155 handle_command_reply: tid=0 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.159+0000 7f13c1179740 1 client.19155 shutdown 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 msgr2=0x5557327be1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f13b000fb90 tx=0x7f13b0004ab0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 shutdown_connections 2024-03-22T02:26:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327be1e0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5557327c2d30 0x5557327c5190 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 --2- 172.21.15.28:0/2910111061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327b6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13977fe640 0 client.19155 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 >> 172.21.15.28:0/2910111061 conn(0x5557327a7d90 msgr2=0x5557327a6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 shutdown_connections 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 1 -- 172.21.15.28:0/2910111061 wait complete. 2024-03-22T02:26:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.160+0000 7f13c1179740 20 client.19155 trim_cache size 0 max 16384 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 Processor -- start 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 -- start start 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327be1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 0x5557327a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81ad250 con 0x5557327ac820 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81aab20 con 0x5557327ad210 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13c1179740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81a8970 con 0x5557327b2380 2024-03-22T02:26:22.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.173+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41870/0 (socket says 172.21.15.28:41870) 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 -- 172.21.15.28:0/3298868102 learned_addr learned my addr 172.21.15.28:0/3298868102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13bcb29640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 0x5557327a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b77fe640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327be1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 -- 172.21.15.28:0/3298868102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 msgr2=0x5557327a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 0x5557327a6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 -- 172.21.15.28:0/3298868102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 msgr2=0x5557327be1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327be1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 -- 172.21.15.28:0/3298868102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557327a23e0 con 0x5557327ac820 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13bcb29640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 0x5557327a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f13b7fff640 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c6a0 tx=0x7f13a80cd1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5f7c94745a23af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f139dffb640 1 -- 172.21.15.28:0/3298868102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a809ba70 con 0x5557327ac820 2024-03-22T02:26:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f139dffb640 1 -- 172.21.15.28:0/3298868102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a809c630 con 0x5557327ac820 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.174+0000 7f139dffb640 1 -- 172.21.15.28:0/3298868102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a809b890 con 0x5557327ac820 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- 172.21.15.28:0/3298868102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 msgr2=0x5557327ad900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f13a800c6a0 tx=0x7f13a80cd1d0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- 172.21.15.28:0/3298868102 shutdown_connections 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327b2380 0x5557327a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327be1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 --2- 172.21.15.28:0/3298868102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327ac820 0x5557327ad900 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- 172.21.15.28:0/3298868102 >> 172.21.15.28:0/3298868102 conn(0x5557327a7d90 msgr2=0x5557327a6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- 172.21.15.28:0/3298868102 shutdown_connections 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- 172.21.15.28:0/3298868102 wait complete. 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 -- start start 2024-03-22T02:26:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 Processor -- start 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.175+0000 7f13c1179740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327bc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327bc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81a8fe0 con 0x5557327b2380 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81aa450 con 0x5557327ad210 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f13b81a96b0 con 0x5557327ac820 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 10 client.0 _ll_register_callbacks cb 0x7f13b0009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13b77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327bc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13b7fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327bc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41878/0 (socket says 172.21.15.28:41878) 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 -- 172.21.15.28:0/3772157911 learned_addr learned my addr 172.21.15.28:0/3772157911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 -- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 msgr2=0x5557327bc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327bc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 -- 172.21.15.28:0/3772157911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 msgr2=0x5557327bc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327bc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 -- 172.21.15.28:0/3772157911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557327bdb50 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13bcb29640 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f13b0028900 tx=0x7f13b0028e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7f433b5a34a980 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13977fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13b003a070 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f13b0028630 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13a809c630 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f13b000f950 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 10 client.19161 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.176+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x555732829810 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13a809c630 con 0x5557327b2380 2024-03-22T02:26:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f13b0036420 con 0x5557327b2380 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 10 client.19161 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 10 client.19161 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 20 client.19161 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55573282a270 0x55573282c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 4 client.19161 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555732719160 con 0x55573282a270 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13b7fff640 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55573282a270 0x55573282c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13b7fff640 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55573282a270 0x55573282c690 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4afd0e572bb3d7e server_cookie=d7bd73d14f0b1cd8 in_seq=0 out_seq=0 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.177+0000 7f13977fe640 10 client.19161 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13977fe640 1 -- 172.21.15.28:0/3772157911 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555732719160 con 0x55573282a270 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13977fe640 10 client.19161 handle_command_reply: tid=0 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 client.19161 shutdown 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 msgr2=0x5557327bd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f13b0028900 tx=0x7f13b0028e20 comp rx=0 tx=0).stop 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 shutdown_connections 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5557327b2380 0x5557327bd560 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x55573282a270 0x55573282c690 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5557327ad210 0x5557327bc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 --2- 172.21.15.28:0/3772157911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5557327ac820 0x5557327bc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13977fe640 0 client.19161 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 >> 172.21.15.28:0/3772157911 conn(0x5557327a7d90 msgr2=0x5557327ab880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 shutdown_connections 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 1 -- 172.21.15.28:0/3772157911 wait complete. 2024-03-22T02:26:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13c1179740 20 client.19161 trim_cache size 0 max 16384 2024-03-22T02:26:22.187 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 msgr2=0x7f13a00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 0x7f13a00a53e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f13b0009fd0 tx=0x7f13b0004a40 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 msgr2=0x7f13b8197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.178+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f13a800e9c0 tx=0x7f13a800ee80 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 shutdown_connections 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f13a00a2f20 0x7f13a00a53e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f13b8102640 0x7f13b819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f13b8101c80 0x7f13b8198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 --2- 172.21.15.28:0/2090260720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13b8100a90 0x7f13b8197ee0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 >> 172.21.15.28:0/2090260720 conn(0x7f13b80fc020 msgr2=0x7f13b8104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 shutdown_connections 2024-03-22T02:26:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.179+0000 7f13be58c640 1 -- 172.21.15.28:0/2090260720 wait complete. 2024-03-22T02:26:22.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 -- 172.21.15.28:0/3231426642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2110104fc0 msgr2=0x7f21101073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 --2- 172.21.15.28:0/3231426642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2110104fc0 0x7f21101073b0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f20f8009900 tx=0x7f20f802bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 -- 172.21.15.28:0/3231426642 shutdown_connections 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 --2- 172.21.15.28:0/3231426642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f21101078f0 0x7f2110109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 --2- 172.21.15.28:0/3231426642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2110104fc0 0x7f21101073b0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 --2- 172.21.15.28:0/3231426642 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110102690 0x7f2110104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 -- 172.21.15.28:0/3231426642 >> 172.21.15.28:0/3231426642 conn(0x7f21100fc020 msgr2=0x7f21100fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 -- 172.21.15.28:0/3231426642 shutdown_connections 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.295+0000 7f2114fe4640 1 -- 172.21.15.28:0/3231426642 wait complete. 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 Processor -- start 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 -- start start 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 0x7f2110102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110104fc0 0x7f2110100640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f211010d7e0 con 0x7f21101078f0 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f211010d660 con 0x7f2110104fc0 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2114fe4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f211010d960 con 0x7f2110102690 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 0x7f2110102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41900/0 (socket says 172.21.15.28:41900) 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210ed9d640 1 -- 172.21.15.28:0/1527308255 learned_addr learned my addr 172.21.15.28:0/1527308255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f2107fff640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110104fc0 0x7f2110100640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 -- 172.21.15.28:0/1527308255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110104fc0 msgr2=0x7f2110100640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110104fc0 0x7f2110100640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 -- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 msgr2=0x7f2110100b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.296+0000 7f210e59c640 1 -- 172.21.15.28:0/1527308255 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2110101410 con 0x7f2110102690 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f210e59c640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 0x7f2110102000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f210000e9f0 tx=0x7f210000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f21077fe640 1 -- 172.21.15.28:0/1527308255 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210000ccc0 con 0x7f2110102690 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f21077fe640 1 -- 172.21.15.28:0/1527308255 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2100010910 con 0x7f2110102690 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21101a1fd0 con 0x7f2110102690 2024-03-22T02:26:22.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f21077fe640 1 -- 172.21.15.28:0/1527308255 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100018960 con 0x7f2110102690 2024-03-22T02:26:22.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.297+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21101a2430 con 0x7f2110102690 2024-03-22T02:26:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.299+0000 7f21077fe640 1 -- 172.21.15.28:0/1527308255 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2100010ac0 con 0x7f2110102690 2024-03-22T02:26:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.299+0000 7f2117bc5740 1 Processor -- start 2024-03-22T02:26:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.299+0000 7f2117bc5740 1 -- start start 2024-03-22T02:26:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f87cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f87f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f11f7dc8d0 con 0x55f11f877820 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f11f784e00 con 0x55f11f87d380 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2117bc5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f11f7d8880 con 0x55f11f878210 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f21077fe640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 0x7f20fc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41914/0 (socket says 172.21.15.28:41914) 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 -- 172.21.15.28:0/1265044583 learned_addr learned my addr 172.21.15.28:0/1265044583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2107fff640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f87f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210e59c640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f87cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f21077fe640 1 -- 172.21.15.28:0/1527308255 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f21000cfb60 con 0x7f2110102690 2024-03-22T02:26:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 0x7f20fc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 -- 172.21.15.28:0/1265044583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 msgr2=0x55f11f87cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f87cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 -- 172.21.15.28:0/1265044583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 msgr2=0x55f11f87f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f87f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210ed9d640 1 -- 172.21.15.28:0/1265044583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11f4f03f0 con 0x55f11f877820 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f2107fff640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f87f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.300+0000 7f210e59c640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f87cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f210800ff20 tx=0x7f21080138f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f331e64ad548be46 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f210ed9d640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 0x7f20fc0a53e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f210801e000 tx=0x7f2108020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2104ff9640 1 -- 172.21.15.28:0/1265044583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210800f470 con 0x55f11f877820 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2104ff9640 1 -- 172.21.15.28:0/1265044583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2108023030 con 0x55f11f877820 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2104ff9640 1 -- 172.21.15.28:0/1265044583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108010690 con 0x55f11f877820 2024-03-22T02:26:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 -- 172.21.15.28:0/1265044583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 msgr2=0x55f11f877c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f210800ff20 tx=0x7f21080138f0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 -- 172.21.15.28:0/1265044583 shutdown_connections 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f87f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f87cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1265044583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f877c40 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.301+0000 7f2117bc5740 1 -- 172.21.15.28:0/1265044583 >> 172.21.15.28:0/1265044583 conn(0x55f11f872d90 msgr2=0x55f11f873190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- 172.21.15.28:0/1265044583 shutdown_connections 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- 172.21.15.28:0/1265044583 wait complete. 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- start start 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 Processor -- start 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f882280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f8892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f11f8802d0 con 0x55f11f877820 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f11f798300 con 0x55f11f87d380 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f11f880450 con 0x55f11f878210 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f210e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2117bc5740 10 client.0 _ll_register_callbacks cb 0x55f11f7d3e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f210e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41926/0 (socket says 172.21.15.28:41926) 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f2107fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f882280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f210e59c640 1 -- 172.21.15.28:0/4189246348 learned_addr learned my addr 172.21.15.28:0/4189246348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.302+0000 7f210ed9d640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f8892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 -- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 msgr2=0x55f11f882280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f882280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 -- 172.21.15.28:0/4189246348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 msgr2=0x55f11f8892c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f8892c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 -- 172.21.15.28:0/4189246348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f11f5ba7d0 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2107fff640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f882280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210ed9d640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f8892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f210e59c640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f2100098900 tx=0x7f210009b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ef08a172837c5c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210009b580 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f2100010430 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2108023030 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f210009ffd0 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 10 client.19173 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f11f88c6f0 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2108023030 con 0x55f11f877820 2024-03-22T02:26:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f2100097da0 con 0x55f11f877820 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 10 client.19173 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 10 client.19173 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.303+0000 7f2117bc5740 20 client.19173 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f88ddc0 0x55f11f890210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f2117bc5740 4 client.19173 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f11f7e9e10 con 0x55f11f88ddc0 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f2107fff640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f88ddc0 0x55f11f890210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f2107fff640 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f88ddc0 0x55f11f890210 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b2ea4d899d8ca64 server_cookie=73b538fd2e8daf10 in_seq=0 out_seq=0 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.304+0000 7f20f5ffb640 10 client.19173 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f20f5ffb640 1 -- 172.21.15.28:0/4189246348 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f11f7e9e10 con 0x55f11f88ddc0 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f20f5ffb640 10 client.19173 handle_command_reply: tid=0 2024-03-22T02:26:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 client.19173 shutdown 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 msgr2=0x55f11f881d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f2100098900 tx=0x7f210009b950 comp rx=0 tx=0).stop 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 shutdown_connections 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f88ddc0 0x55f11f890210 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f8892c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f20f5ffb640 0 client.19173 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f878210 0x55f11f882280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 --2- 172.21.15.28:0/4189246348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f881d40 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 >> 172.21.15.28:0/4189246348 conn(0x55f11f872d90 msgr2=0x55f11f871fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 shutdown_connections 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 1 -- 172.21.15.28:0/4189246348 wait complete. 2024-03-22T02:26:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.305+0000 7f2117bc5740 20 client.19173 trim_cache size 0 max 16384 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.320+0000 7f2117bc5740 1 Processor -- start 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.320+0000 7f2117bc5740 1 -- start start 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f877820 0x55f11f86de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f882060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21101af430 con 0x55f11f878210 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21101acd00 con 0x55f11f87d380 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2117bc5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f21101aab50 con 0x55f11f877820 2024-03-22T02:26:22.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f210e59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f877820 0x55f11f86de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f210ed9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f882060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41932/0 (socket says 172.21.15.28:41932) 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 -- 172.21.15.28:0/1004254460 learned_addr learned my addr 172.21.15.28:0/1004254460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 -- 172.21.15.28:0/1004254460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f877820 msgr2=0x55f11f86de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f877820 0x55f11f86de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 -- 172.21.15.28:0/1004254460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 msgr2=0x55f11f882060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f882060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 -- 172.21.15.28:0/1004254460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f11f877460 con 0x55f11f878210 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f2107fff640 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f20f802cfd0 tx=0x7f20f802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa2bbe0392eefd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f20f7fff640 1 -- 172.21.15.28:0/1004254460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f8004180 con 0x55f11f878210 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f20f7fff640 1 -- 172.21.15.28:0/1004254460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20f8004320 con 0x55f11f878210 2024-03-22T02:26:22.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.321+0000 7f20f7fff640 1 -- 172.21.15.28:0/1004254460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f80052a0 con 0x55f11f878210 2024-03-22T02:26:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- 172.21.15.28:0/1004254460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 msgr2=0x55f11f889280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f20f802cfd0 tx=0x7f20f802e040 comp rx=0 tx=0).stop 2024-03-22T02:26:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- 172.21.15.28:0/1004254460 shutdown_connections 2024-03-22T02:26:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f87d380 0x55f11f882060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f878210 0x55f11f889280 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 --2- 172.21.15.28:0/1004254460 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f877820 0x55f11f86de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- 172.21.15.28:0/1004254460 >> 172.21.15.28:0/1004254460 conn(0x55f11f872d90 msgr2=0x55f11f881950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- 172.21.15.28:0/1004254460 shutdown_connections 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- 172.21.15.28:0/1004254460 wait complete. 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 -- start start 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.322+0000 7f2117bc5740 1 Processor -- start 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f878210 0x55f11f886df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f87d380 0x55f11f887d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f21101ab1c0 con 0x55f11f877820 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f21101ac630 con 0x55f11f878210 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f21101ab890 con 0x55f11f87d380 2024-03-22T02:26:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2117bc5740 10 client.0 _ll_register_callbacks cb 0x7f210000ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41944/0 (socket says 172.21.15.28:41944) 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 -- 172.21.15.28:0/2273098099 learned_addr learned my addr 172.21.15.28:0/2273098099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f2107fff640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f878210 0x55f11f886df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210ed9d640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f87d380 0x55f11f887d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 -- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f87d380 msgr2=0x55f11f887d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f87d380 0x55f11f887d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 -- 172.21.15.28:0/2273098099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f878210 msgr2=0x55f11f886df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f878210 0x55f11f886df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 -- 172.21.15.28:0/2273098099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f11f888310 con 0x55f11f877820 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f210e59c640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f2100098390 tx=0x7f2100009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c4dde7f4773996 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f20f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.323+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210009adf0 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20f8004320 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f210009af90 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f21000a4890 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 10 client.19179 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f11f8f40b0 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20f8004320 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f2100096aa0 con 0x55f11f877820 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 10 client.19179 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 10 client.19179 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 20 client.19179 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f8f48f0 0x55f11f8f6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 4 client.19179 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f11f7e4160 con 0x55f11f8f48f0 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.324+0000 7f2107fff640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f8f48f0 0x55f11f8f6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f20f5ffb640 10 client.19179 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2107fff640 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f8f48f0 0x55f11f8f6d10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5ec00e062ab2c92b server_cookie=fcfb6ea34a4f7b65 in_seq=0 out_seq=0 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f20f5ffb640 1 -- 172.21.15.28:0/2273098099 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f11f7e4160 con 0x55f11f8f48f0 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f20f5ffb640 10 client.19179 handle_command_reply: tid=0 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 client.19179 shutdown 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 msgr2=0x55f11f8868b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f2100098390 tx=0x7f2100009700 comp rx=0 tx=0).stop 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 shutdown_connections 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55f11f8f48f0 0x55f11f8f6d10 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f11f87d380 0x55f11f887d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f11f878210 0x55f11f886df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 --2- 172.21.15.28:0/2273098099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f11f877820 0x55f11f8868b0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f20f5ffb640 0 client.19179 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 >> 172.21.15.28:0/2273098099 conn(0x55f11f872d90 msgr2=0x55f11f875ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 shutdown_connections 2024-03-22T02:26:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 1 -- 172.21.15.28:0/2273098099 wait complete. 2024-03-22T02:26:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2117bc5740 20 client.19179 trim_cache size 0 max 16384 2024-03-22T02:26:22.334 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 msgr2=0x7f20fc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 0x7f20fc0a53e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f210801e000 tx=0x7f2108020000 comp rx=0 tx=0).stop 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 msgr2=0x7f2110102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.325+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 0x7f2110102000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f210000e9f0 tx=0x7f210000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 shutdown_connections 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20fc0a2f20 0x7f20fc0a53e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21101078f0 0x7f2110100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2110104fc0 0x7f2110100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 --2- 172.21.15.28:0/1527308255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2110102690 0x7f2110102000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 >> 172.21.15.28:0/1527308255 conn(0x7f21100fc020 msgr2=0x7f2110103130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 shutdown_connections 2024-03-22T02:26:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.326+0000 7f2114fe4640 1 -- 172.21.15.28:0/1527308255 wait complete. 2024-03-22T02:26:22.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:26:22.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1320702 11.21 MB/sec execute 544 sec latency 32.443 ms 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 -- 172.21.15.28:0/235760752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 msgr2=0x7f99cc105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 --2- 172.21.15.28:0/235760752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc105440 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009900 tx=0x7f99b402bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 -- 172.21.15.28:0/235760752 shutdown_connections 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 --2- 172.21.15.28:0/235760752 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 0x7f99cc10c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 --2- 172.21.15.28:0/235760752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc105440 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 --2- 172.21.15.28:0/235760752 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 0x7f99cc1041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 -- 172.21.15.28:0/235760752 >> 172.21.15.28:0/235760752 conn(0x7f99cc0ff340 msgr2=0x7f99cc101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 -- 172.21.15.28:0/235760752 shutdown_connections 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.446+0000 7f99d1756640 1 -- 172.21.15.28:0/235760752 wait complete. 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 Processor -- start 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 -- start start 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 0x7f99cc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 0x7f99cc1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10fa30 con 0x7f99cc104fc0 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10f8b0 con 0x7f99cc105980 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99d1756640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc10fbb0 con 0x7f99cc103dd0 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41962/0 (socket says 172.21.15.28:41962) 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99c27fc640 1 -- 172.21.15.28:0/949770594 learned_addr learned my addr 172.21.15.28:0/949770594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99caffd640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 0x7f99cc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.447+0000 7f99cb7fe640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 0x7f99cc1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 -- 172.21.15.28:0/949770594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 msgr2=0x7f99cc19a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 0x7f99cc19a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 -- 172.21.15.28:0/949770594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 msgr2=0x7f99cc1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 0x7f99cc1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 -- 172.21.15.28:0/949770594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99cc1a1cf0 con 0x7f99cc104fc0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c27fc640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009230 tx=0x7f99b4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c8ff9640 1 -- 172.21.15.28:0/949770594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4041070 con 0x7f99cc104fc0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99cc1a1f80 con 0x7f99cc104fc0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c8ff9640 1 -- 172.21.15.28:0/949770594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b40044d0 con 0x7f99cc104fc0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99c8ff9640 1 -- 172.21.15.28:0/949770594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b4034440 con 0x7f99cc104fc0 2024-03-22T02:26:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.448+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99cc1a23f0 con 0x7f99cc104fc0 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.450+0000 7f99c8ff9640 1 -- 172.21.15.28:0/949770594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f99b4044070 con 0x7f99cc104fc0 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 Processor -- start 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 -- start start 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c932c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b08c8978d0 con 0x55b08c932820 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b08c83fe00 con 0x55b08c938380 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99d441a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b08c893880 con 0x55b08c933210 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99c8ff9640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 0x7f99b80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c932c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99caffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33158/0 (socket says 172.21.15.28:33158) 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99caffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43396/0 (socket says 172.21.15.28:43396) 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99c27fc640 1 -- 172.21.15.28:0/2961390520 learned_addr learned my addr 172.21.15.28:0/2961390520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.451+0000 7f99cb7fe640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 0x7f99b80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c8ff9640 1 -- 172.21.15.28:0/949770594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f99b40f1370 con 0x7f99cc104fc0 2024-03-22T02:26:22.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 -- 172.21.15.28:0/2961390520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 msgr2=0x55b08c937d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 -- 172.21.15.28:0/2961390520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 msgr2=0x55b08c932c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c932c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 -- 172.21.15.28:0/2961390520 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08c5ab3f0 con 0x55b08c938380 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99caffd640 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c27fc640 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99b40eea00 tx=0x7f99b4049000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b32942bae715c005 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99cb7fe640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 0x7f99b80a5460 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f99b4005820 tx=0x7f99c4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c17fa640 1 -- 172.21.15.28:0/2961390520 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b404b040 con 0x55b08c938380 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c17fa640 1 -- 172.21.15.28:0/2961390520 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b40bd6e0 con 0x55b08c938380 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.452+0000 7f99c17fa640 1 -- 172.21.15.28:0/2961390520 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b40bd940 con 0x55b08c938380 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- 172.21.15.28:0/2961390520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 msgr2=0x55b08c93a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f99b40eea00 tx=0x7f99b4049000 comp rx=0 tx=0).stop 2024-03-22T02:26:22.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- 172.21.15.28:0/2961390520 shutdown_connections 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c93a810 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c933210 0x55b08c937d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- 172.21.15.28:0/2961390520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c932c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- 172.21.15.28:0/2961390520 >> 172.21.15.28:0/2961390520 conn(0x55b08c92dd90 msgr2=0x55b08c92e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- 172.21.15.28:0/2961390520 shutdown_connections 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- 172.21.15.28:0/2961390520 wait complete. 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- start start 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 Processor -- start 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c93cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c944010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b08c93b0f0 con 0x55b08c933210 2024-03-22T02:26:22.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b08c853300 con 0x55b08c938380 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b08c93b270 con 0x55b08c932820 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.453+0000 7f99d441a740 10 client.0 _ll_register_callbacks cb 0x55b08c88ee30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99cb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c944010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99caffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c93cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41982/0 (socket says 172.21.15.28:41982) 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 -- 172.21.15.28:0/1675565409 learned_addr learned my addr 172.21.15.28:0/1675565409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 -- 172.21.15.28:0/1675565409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 msgr2=0x55b08c93cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c93cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 -- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 msgr2=0x55b08c944010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c944010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 -- 172.21.15.28:0/1675565409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08c6757d0 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f99b40bd3e0 tx=0x7f99b402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c769f5e0e04bc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b40d5060 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f99b40c5bb0 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99b40bd6e0 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f99b404b070 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99d441a740 10 client.19191 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.454+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b08c93b0f0 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b40bd6e0 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f99b40bcb30 con 0x55b08c933210 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 10 client.19191 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 10 client.19191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 20 client.19191 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c948680 0x55b08c94aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 4 client.19191 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b08c8a4e10 con 0x55b08c948680 2024-03-22T02:26:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99caffd640 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c948680 0x55b08c94aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99caffd640 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c948680 0x55b08c94aa70 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af64d1624a4caf13 server_cookie=3510819a49183fe4 in_seq=0 out_seq=0 2024-03-22T02:26:22.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.455+0000 7f99b27fc640 10 client.19191 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.457+0000 7f99b27fc640 1 -- 172.21.15.28:0/1675565409 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b08c8a4e10 con 0x55b08c948680 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.457+0000 7f99b27fc640 10 client.19191 handle_command_reply: tid=0 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.457+0000 7f99d441a740 1 client.19191 shutdown 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 msgr2=0x55b08c93d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f99b40bd3e0 tx=0x7f99b402bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 shutdown_connections 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c948680 0x55b08c94aa70 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c944010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c93d040 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 --2- 172.21.15.28:0/1675565409 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c93cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99b27fc640 0 client.19191 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 >> 172.21.15.28:0/1675565409 conn(0x55b08c92dd90 msgr2=0x55b08c92cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 shutdown_connections 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 1 -- 172.21.15.28:0/1675565409 wait complete. 2024-03-22T02:26:22.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.458+0000 7f99d441a740 20 client.19191 trim_cache size 0 max 16384 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 Processor -- start 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 -- start start 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c931700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c94f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1af460 con 0x55b08c933210 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1acd30 con 0x55b08c938380 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.473+0000 7f99d441a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1aab80 con 0x55b08c932820 2024-03-22T02:26:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c931700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41988/0 (socket says 172.21.15.28:41988) 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c27fc640 1 -- 172.21.15.28:0/2286668219 learned_addr learned my addr 172.21.15.28:0/2286668219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99cb7fe640 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c94f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 -- 172.21.15.28:0/2286668219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 msgr2=0x55b08c94f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c94f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 -- 172.21.15.28:0/2286668219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 msgr2=0x55b08c943fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 -- 172.21.15.28:0/2286668219 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b08c94b0f0 con 0x55b08c932820 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c27fc640 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99caffd640 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c931700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55b08c94b320 tx=0x7f99bc00b410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e353634763745f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c0ff9640 1 -- 172.21.15.28:0/2286668219 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc026070 con 0x55b08c932820 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c0ff9640 1 -- 172.21.15.28:0/2286668219 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99bc020690 con 0x55b08c932820 2024-03-22T02:26:22.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.474+0000 7f99c0ff9640 1 -- 172.21.15.28:0/2286668219 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc011b60 con 0x55b08c932820 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- 172.21.15.28:0/2286668219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 msgr2=0x55b08c931700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c931700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55b08c94b320 tx=0x7f99bc00b410 comp rx=0 tx=0).stop 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- 172.21.15.28:0/2286668219 shutdown_connections 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c938380 0x55b08c94f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c933210 0x55b08c943fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 --2- 172.21.15.28:0/2286668219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c932820 0x55b08c931700 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- 172.21.15.28:0/2286668219 >> 172.21.15.28:0/2286668219 conn(0x55b08c92dd90 msgr2=0x55b08c930ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- 172.21.15.28:0/2286668219 shutdown_connections 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- 172.21.15.28:0/2286668219 wait complete. 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 -- start start 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.475+0000 7f99d441a740 1 Processor -- start 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c933210 0x55b08c941bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c938380 0x55b08c942bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ab1f0 con 0x55b08c932820 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ac660 con 0x55b08c933210 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f99cc1ab8c0 con 0x55b08c938380 2024-03-22T02:26:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 10 client.0 _ll_register_callbacks cb 0x7f99c4009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41990/0 (socket says 172.21.15.28:41990) 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 -- 172.21.15.28:0/1701157717 learned_addr learned my addr 172.21.15.28:0/1701157717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99cb7fe640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c938380 0x55b08c942bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c933210 0x55b08c941bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 -- 172.21.15.28:0/1701157717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c938380 msgr2=0x55b08c942bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c938380 0x55b08c942bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 -- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c933210 msgr2=0x55b08c941bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c933210 0x55b08c941bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 -- 172.21.15.28:0/1701157717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b08c9431c0 con 0x55b08c932820 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99caffd640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000c00 tx=0x7f99bc0208d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934dc553e07768ec server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc026040 con 0x55b08c932820 2024-03-22T02:26:22.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f99bc00bc80 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.476+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99bc020690 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f99bc037050 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 10 client.19197 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55b08c9ae9b0 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99bc020690 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f99bc032030 con 0x55b08c932820 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 10 client.19197 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 10 client.19197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 20 client.19197 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c9af320 0x55b08c9b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 4 client.19197 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b08c89f160 con 0x55b08c9af320 2024-03-22T02:26:22.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c9af320 0x55b08c9b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99c27fc640 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c9af320 0x55b08c9b1740 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=831cdfaeacd145f9 server_cookie=30535d44a0d3b82b in_seq=0 out_seq=0 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.477+0000 7f99b27fc640 10 client.19197 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99b27fc640 1 -- 172.21.15.28:0/1701157717 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b08c89f160 con 0x55b08c9af320 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99b27fc640 10 client.19197 handle_command_reply: tid=0 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 client.19197 shutdown 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 msgr2=0x55b08c941680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f99bc000c00 tx=0x7f99bc0208d0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 shutdown_connections 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b08c9af320 0x55b08c9b1740 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b08c938380 0x55b08c942bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99b27fc640 0 client.19197 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b08c933210 0x55b08c941bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 --2- 172.21.15.28:0/1701157717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b08c932820 0x55b08c941680 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 >> 172.21.15.28:0/1701157717 conn(0x55b08c92dd90 msgr2=0x55b08c931880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 shutdown_connections 2024-03-22T02:26:22.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 1 -- 172.21.15.28:0/1701157717 wait complete. 2024-03-22T02:26:22.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d441a740 20 client.19197 trim_cache size 0 max 16384 2024-03-22T02:26:22.487 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 msgr2=0x7f99b80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 0x7f99b80a5460 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f99b4005820 tx=0x7f99c4004ab0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 msgr2=0x7f99cc19a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.480+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f99b4009230 tx=0x7f99b4005d50 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 shutdown_connections 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f99b80a2fa0 0x7f99b80a5460 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f99cc105980 0x7f99cc1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99cc104fc0 0x7f99cc19a610 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 --2- 172.21.15.28:0/949770594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f99cc103dd0 0x7f99cc19a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:22.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 >> 172.21.15.28:0/949770594 conn(0x7f99cc0ff340 msgr2=0x7f99cc108390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 shutdown_connections 2024-03-22T02:26:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:22.481+0000 7f99d1756640 1 -- 172.21.15.28:0/949770594 wait complete. 2024-03-22T02:26:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:22 smithi120 ceph-mon[36803]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 424 op/s 2024-03-22T02:26:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:22 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1319737109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:22.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:22 smithi083 ceph-mon[36340]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 424 op/s 2024-03-22T02:26:22.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:22 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1319737109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:22 smithi028 ceph-mon[31916]: pgmap v743: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 424 op/s 2024-03-22T02:26:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:22 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1319737109' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:23.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1322069 11.20 MB/sec execute 545 sec latency 32.389 ms 2024-03-22T02:26:23.493 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:26:23.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:23.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:26:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 -- 172.21.15.28:0/526277726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c101ba0 msgr2=0x7f087c101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 --2- 172.21.15.28:0/526277726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c101ba0 0x7f087c101fc0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f0864009900 tx=0x7f086402bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 -- 172.21.15.28:0/526277726 shutdown_connections 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 --2- 172.21.15.28:0/526277726 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c103750 0x7f087c109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 --2- 172.21.15.28:0/526277726 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c102d90 0x7f087c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 --2- 172.21.15.28:0/526277726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c101ba0 0x7f087c101fc0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 -- 172.21.15.28:0/526277726 >> 172.21.15.28:0/526277726 conn(0x7f087c0fd130 msgr2=0x7f087c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 -- 172.21.15.28:0/526277726 shutdown_connections 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 -- 172.21.15.28:0/526277726 wait complete. 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.600+0000 7f0882266640 1 Processor -- start 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 -- start start 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101ba0 0x7f087c197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 0x7f087c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10d820 con 0x7f087c102d90 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10d6a0 con 0x7f087c103750 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0882266640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c10d9a0 con 0x7f087c101ba0 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42006/0 (socket says 172.21.15.28:42006) 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 -- 172.21.15.28:0/2067369836 learned_addr learned my addr 172.21.15.28:0/2067369836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0880803640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 0x7f087c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087bfff640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101ba0 0x7f087c197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 -- 172.21.15.28:0/2067369836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101ba0 msgr2=0x7f087c197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101ba0 0x7f087c197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 -- 172.21.15.28:0/2067369836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 msgr2=0x7f087c19f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 0x7f087c19f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f087b7fe640 1 -- 172.21.15.28:0/2067369836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087c19fb20 con 0x7f087c102d90 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.601+0000 7f0880803640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 0x7f087c19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f087b7fe640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e9f0 tx=0x7f086c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f08797fa640 1 -- 172.21.15.28:0/2067369836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c00cd80 con 0x7f087c102d90 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f08797fa640 1 -- 172.21.15.28:0/2067369836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086c010910 con 0x7f087c102d90 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f08797fa640 1 -- 172.21.15.28:0/2067369836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c018990 con 0x7f087c102d90 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f087c19fe10 con 0x7f087c102d90 2024-03-22T02:26:23.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.602+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f087c1a02d0 con 0x7f087c102d90 2024-03-22T02:26:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.604+0000 7f08797fa640 1 -- 172.21.15.28:0/2067369836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f086c010ac0 con 0x7f087c102d90 2024-03-22T02:26:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.604+0000 7f0884e47740 1 Processor -- start 2024-03-22T02:26:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.604+0000 7f0884e47740 1 -- start start 2024-03-22T02:26:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.604+0000 7f0884e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0884e47740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 0x55c54fe5e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0884e47740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0884e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c54fdbf570 con 0x55c54fe5ab40 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0884e47740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c54fd67e10 con 0x55c54fe5e690 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0884e47740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c54fdbb520 con 0x55c54fe5b530 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0880803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0880803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42020/0 (socket says 172.21.15.28:42020) 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 0x55c54fe5e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0880803640 1 -- 172.21.15.28:0/2739852373 learned_addr learned my addr 172.21.15.28:0/2739852373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087b7fe640 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f08797fa640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 0x7f08680a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 -- 172.21.15.28:0/2739852373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 msgr2=0x55c54fe6b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 -- 172.21.15.28:0/2739852373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 msgr2=0x55c54fe5af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 -- 172.21.15.28:0/2739852373 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54fb9b7e0 con 0x55c54fe5b530 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0880803640 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f0880803640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 0x7f08680a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f087bfff640 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 0x55c54fe5e020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f086402c170 tx=0x7f0864009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb196df898a9eea server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.605+0000 7f08797fa640 1 -- 172.21.15.28:0/2067369836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f086c0cfae0 con 0x7f087c102d90 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.606+0000 7f08727fc640 1 -- 172.21.15.28:0/2739852373 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086402cb80 con 0x55c54fe5b530 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.606+0000 7f08727fc640 1 -- 172.21.15.28:0/2739852373 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864005040 con 0x55c54fe5b530 2024-03-22T02:26:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.606+0000 7f08727fc640 1 -- 172.21.15.28:0/2739852373 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864004920 con 0x55c54fe5b530 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- 172.21.15.28:0/2739852373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 msgr2=0x55c54fe5e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0880803640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 0x7f08680a53e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55c54fe5bd90 tx=0x7f0874004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 0x55c54fe5e020 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f086402c170 tx=0x7f0864009560 comp rx=0 tx=0).stop 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- 172.21.15.28:0/2739852373 shutdown_connections 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5b530 0x55c54fe5e020 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 --2- 172.21.15.28:0/2739852373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe5af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- 172.21.15.28:0/2739852373 >> 172.21.15.28:0/2739852373 conn(0x55c54fe56130 msgr2=0x55c54fe56530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- 172.21.15.28:0/2739852373 shutdown_connections 2024-03-22T02:26:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- 172.21.15.28:0/2739852373 wait complete. 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 -- start start 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.607+0000 7f0884e47740 1 Processor -- start 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe651f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c54fe5f070 con 0x55c54fe5b530 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c54fd7b110 con 0x55c54fe5e690 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c54fe5f1f0 con 0x55c54fe5ab40 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0884e47740 10 client.0 _ll_register_callbacks cb 0x55c54fdb7be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0880803640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43484/0 (socket says 172.21.15.28:43484) 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f0880803640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33232/0 (socket says 172.21.15.28:33232) 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 -- 172.21.15.28:0/3866920957 learned_addr learned my addr 172.21.15.28:0/3866920957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087b7fe640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe651f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 -- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 msgr2=0x55c54fe6a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 -- 172.21.15.28:0/3866920957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 msgr2=0x55c54fe651f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe651f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 -- 172.21.15.28:0/3866920957 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54fbfbf70 con 0x55c54fe5ab40 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f087bfff640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0864009900 tx=0x7f0864004260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afb79a603f14812e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.608+0000 7f085b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864004450 con 0x55c54fe5ab40 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f08640046c0 con 0x55c54fe5ab40 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0864005040 con 0x55c54fe5ab40 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f0864005600 con 0x55c54fe5ab40 2024-03-22T02:26:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 10 client.29026 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c54fa93170 con 0x55c54fe5ab40 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0864005040 con 0x55c54fe5ab40 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f0864003ca0 con 0x55c54fe5ab40 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 10 client.29026 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 10 client.29026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 20 client.29026 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fe817f0 0x55c54fe83be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 4 client.29026 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.609+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c54fdccab0 con 0x55c54fe817f0 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.610+0000 7f087b7fe640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fe817f0 0x55c54fe83be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.610+0000 7f085b7fe640 10 client.29026 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.610+0000 7f087b7fe640 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fe817f0 0x55c54fe83be0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74e86aefb9e9798a server_cookie=8c0e9a85754d564c in_seq=0 out_seq=0 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.611+0000 7f085b7fe640 1 -- 172.21.15.28:0/3866920957 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c54fdccab0 con 0x55c54fe817f0 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.611+0000 7f085b7fe640 10 client.29026 handle_command_reply: tid=0 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 client.29026 shutdown 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 msgr2=0x55c54fe64cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0864009900 tx=0x7f0864004260 comp rx=0 tx=0).stop 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 shutdown_connections 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fe817f0 0x55c54fe83be0 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5e690 0x55c54fe6a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f085b7fe640 0 client.29026 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe651f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 --2- 172.21.15.28:0/3866920957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5ab40 0x55c54fe64cb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 >> 172.21.15.28:0/3866920957 conn(0x55c54fe56130 msgr2=0x55c54fe553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 shutdown_connections 2024-03-22T02:26:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 1 -- 172.21.15.28:0/3866920957 wait complete. 2024-03-22T02:26:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.612+0000 7f0884e47740 20 client.29026 trim_cache size 0 max 16384 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 Processor -- start 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 -- start start 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 0x55c54fe644e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe64a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1aaf80 con 0x55c54fe5ab40 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a8850 con 0x55c54fe5b530 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0884e47740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a66a0 con 0x55c54fe5e690 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42036/0 (socket says 172.21.15.28:42036) 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 -- 172.21.15.28:0/3423711574 learned_addr learned my addr 172.21.15.28:0/3423711574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0880803640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe64a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087b7fe640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 0x55c54fe644e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 -- 172.21.15.28:0/3423711574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 msgr2=0x55c54fe64a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe64a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 -- 172.21.15.28:0/3423711574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 msgr2=0x55c54fe644e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 0x55c54fe644e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087bfff640 1 -- 172.21.15.28:0/3423711574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c54fe5a780 con 0x55c54fe5ab40 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f0880803640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe64a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.630+0000 7f087b7fe640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 0x55c54fe644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f087bfff640 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f0864005bc0 tx=0x7f0864005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f0dac105914800 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0871ffb640 1 -- 172.21.15.28:0/3423711574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0864005d50 con 0x55c54fe5ab40 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0871ffb640 1 -- 172.21.15.28:0/3423711574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086402cc90 con 0x55c54fe5ab40 2024-03-22T02:26:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0871ffb640 1 -- 172.21.15.28:0/3423711574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08640039b0 con 0x55c54fe5ab40 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 -- 172.21.15.28:0/3423711574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 msgr2=0x55c54fe63fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f0864005bc0 tx=0x7f0864005820 comp rx=0 tx=0).stop 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 -- 172.21.15.28:0/3423711574 shutdown_connections 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe64a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5b530 0x55c54fe644e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 --2- 172.21.15.28:0/3423711574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5ab40 0x55c54fe63fa0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 -- 172.21.15.28:0/3423711574 >> 172.21.15.28:0/3423711574 conn(0x55c54fe56130 msgr2=0x55c54fe553f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.631+0000 7f0884e47740 1 -- 172.21.15.28:0/3423711574 shutdown_connections 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 -- 172.21.15.28:0/3423711574 wait complete. 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 -- start start 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 Processor -- start 2024-03-22T02:26:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5ab40 0x55c54fe94600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe92c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a6d10 con 0x55c54fe5b530 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a8180 con 0x55c54fe5ab40 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f087c1a73e0 con 0x55c54fe5e690 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0884e47740 10 client.0 _ll_register_callbacks cb 0x7f0874009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f087b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42048/0 (socket says 172.21.15.28:42048) 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f087b7fe640 1 -- 172.21.15.28:0/996268568 learned_addr learned my addr 172.21.15.28:0/996268568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f087bfff640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5ab40 0x55c54fe94600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.632+0000 7f0880803640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe92c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 -- 172.21.15.28:0/996268568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 msgr2=0x55c54fe92c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe92c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 -- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5ab40 msgr2=0x55c54fe94600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5ab40 0x55c54fe94600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 -- 172.21.15.28:0/996268568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c54fe5a1d0 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f087b7fe640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f086c098310 tx=0x7f086c0bc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ee288bad99ce1d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086c09ff50 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7f086c098520 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f086402cc90 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f086c0a4810 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 10 client.19215 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55c54fe6a640 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f086402cc90 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f086c09b300 con 0x55c54fe5b530 2024-03-22T02:26:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 10 client.19215 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 10 client.19215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 20 client.19215 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fee8540 0x55c54feea960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 4 client.19215 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.633+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c54fdc6e00 con 0x55c54fee8540 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.634+0000 7f087bfff640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fee8540 0x55c54feea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.634+0000 7f087bfff640 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fee8540 0x55c54feea960 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=127bfacec92ec173 server_cookie=474e25b299e28cd6 in_seq=0 out_seq=0 2024-03-22T02:26:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.634+0000 7f085b7fe640 10 client.19215 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.635+0000 7f085b7fe640 1 -- 172.21.15.28:0/996268568 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c54fdc6e00 con 0x55c54fee8540 2024-03-22T02:26:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.635+0000 7f085b7fe640 10 client.19215 handle_command_reply: tid=0 2024-03-22T02:26:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.635+0000 7f0884e47740 1 client.19215 shutdown 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 msgr2=0x55c54fe94b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f086c098310 tx=0x7f086c0bc040 comp rx=0 tx=0).stop 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 shutdown_connections 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c54fee8540 0x55c54feea960 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c54fe5e690 0x55c54fe92c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c54fe5b530 0x55c54fe94b40 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f085b7fe640 0 client.19215 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 --2- 172.21.15.28:0/996268568 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c54fe5ab40 0x55c54fe94600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 >> 172.21.15.28:0/996268568 conn(0x55c54fe56130 msgr2=0x55c54fe5f5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 shutdown_connections 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 1 -- 172.21.15.28:0/996268568 wait complete. 2024-03-22T02:26:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0884e47740 20 client.19215 trim_cache size 0 max 16384 2024-03-22T02:26:23.644 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:23.644 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:26:23.644 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d624ef7e-9322-43c7-9e93-4b790d48cb42", 2024-03-22T02:26:23.644 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:26:23.644 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 msgr2=0x7f08680a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 0x7f08680a53e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x55c54fe5bd90 tx=0x7f0874004a40 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 msgr2=0x7f087c198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f086c00e9f0 tx=0x7f086c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 shutdown_connections 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f08680a2f20 0x7f08680a53e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f087c103750 0x7f087c19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f087c102d90 0x7f087c198440 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 --2- 172.21.15.28:0/2067369836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f087c101ba0 0x7f087c197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.636+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 >> 172.21.15.28:0/2067369836 conn(0x7f087c0fd130 msgr2=0x7f087c1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.637+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 shutdown_connections 2024-03-22T02:26:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.637+0000 7f0882266640 1 -- 172.21.15.28:0/2067369836 wait complete. 2024-03-22T02:26:23.649 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:26:23.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:26:23.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/1269865210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 msgr2=0x7fc2b4108ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/1269865210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b4108ee0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009ca0 tx=0x7fc2ac02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/1269865210 shutdown_connections 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/1269865210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b4108ee0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/1269865210 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 0x7fc2b4102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/1269865210 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 0x7fc2b4100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/1269865210 >> 172.21.15.28:0/1269865210 conn(0x7fc2b40fc030 msgr2=0x7fc2b40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/1269865210 shutdown_connections 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/1269865210 wait complete. 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 Processor -- start 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.757+0000 7fc2b9f5e640 1 -- start start 2024-03-22T02:26:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 0x7fc2b406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 0x7fc2b419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410c720 con 0x7fc2b4102650 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410c5a0 con 0x7fc2b4100aa0 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b410c8a0 con 0x7fc2b4101c90 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 0x7fc2b406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42064/0 (socket says 172.21.15.28:42064) 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 -- 172.21.15.28:0/3415058021 learned_addr learned my addr 172.21.15.28:0/3415058021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 0x7fc2b419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 -- 172.21.15.28:0/3415058021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 msgr2=0x7fc2b419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 0x7fc2b419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 -- 172.21.15.28:0/3415058021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 msgr2=0x7fc2b406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 0x7fc2b406ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 -- 172.21.15.28:0/3415058021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b419fac0 con 0x7fc2b4102650 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac037040 tx=0x7fc2ac032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b0ff9640 1 -- 172.21.15.28:0/3415058021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac041020 con 0x7fc2b4102650 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b0ff9640 1 -- 172.21.15.28:0/3415058021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2ac034cd0 con 0x7fc2b4102650 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b0ff9640 1 -- 172.21.15.28:0/3415058021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac039040 con 0x7fc2b4102650 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b419fd50 con 0x7fc2b4102650 2024-03-22T02:26:23.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.758+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b41a01f0 con 0x7fc2b4102650 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b0ff9640 1 -- 172.21.15.28:0/3415058021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc2ac044020 con 0x7fc2b4102650 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 Processor -- start 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 -- start start 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b0ff9640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 0x7fc29c0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703e20 0x55f15a708920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a70b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f15a6688c0 con 0x55f15a708f90 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f15a610c20 con 0x55f15a703430 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f15a664870 con 0x55f15a703e20 2024-03-22T02:26:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 0x7fc29c0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b0ff9640 1 -- 172.21.15.28:0/3415058021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc2ac0f1580 con 0x7fc2b4102650 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a70b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33264/0 (socket says 172.21.15.28:33264) 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b37fe640 1 -- 172.21.15.28:0/2376708862 learned_addr learned my addr 172.21.15.28:0/2376708862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.761+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703e20 0x55f15a708920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 -- 172.21.15.28:0/2376708862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703e20 msgr2=0x55f15a708920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703e20 0x55f15a708920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 -- 172.21.15.28:0/2376708862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 msgr2=0x55f15a70b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a70b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 -- 172.21.15.28:0/2376708862 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15a635150 con 0x55f15a703430 2024-03-22T02:26:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a70b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 0x7fc29c0a5460 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009710 tx=0x7fc2a80098b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009ca0 tx=0x7fc2a4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7dc3ff13a658809 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc299ffb640 1 -- 172.21.15.28:0/2376708862 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4009d90 con 0x55f15a703430 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc299ffb640 1 -- 172.21.15.28:0/2376708862 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a40035f0 con 0x55f15a703430 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.762+0000 7fc299ffb640 1 -- 172.21.15.28:0/2376708862 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4019670 con 0x55f15a703430 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2376708862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 msgr2=0x55f15a703850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009ca0 tx=0x7fc2a4011810 comp rx=0 tx=0).stop 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2376708862 shutdown_connections 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a70b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703e20 0x55f15a708920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2376708862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a703850 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2376708862 >> 172.21.15.28:0/2376708862 conn(0x55f15a6fe9a0 msgr2=0x55f15a6feda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2376708862 shutdown_connections 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2376708862 wait complete. 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 -- start start 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 Processor -- start 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.763+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a70d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a70de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f15a70bf40 con 0x55f15a708f90 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f15a624130 con 0x55f15a703e20 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f15a70c0c0 con 0x55f15a703430 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 10 client.0 _ll_register_callbacks cb 0x55f15a623b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42086/0 (socket says 172.21.15.28:42086) 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 -- 172.21.15.28:0/1984468434 learned_addr learned my addr 172.21.15.28:0/1984468434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a70de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a70d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 -- 172.21.15.28:0/1984468434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 msgr2=0x55f15a70d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a70d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 -- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 msgr2=0x55f15a70de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a70de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 -- 172.21.15.28:0/1984468434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15a4467d0 con 0x55f15a708f90 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0eec70 tx=0x7fc2ac0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901a09b07a7e7de0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc292ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac09adc0 con 0x55f15a708f90 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc2ac0b9fa0 con 0x55f15a708f90 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a40035f0 con 0x55f15a708f90 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc2ac0be030 con 0x55f15a708f90 2024-03-22T02:26:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.764+0000 7fc2bcc28740 10 client.19227 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f15a37c3f0 con 0x55f15a708f90 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a40035f0 con 0x55f15a708f90 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc2ac03c030 con 0x55f15a708f90 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 10 client.19227 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 10 client.19227 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 20 client.19227 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a7196d0 0x55f15a71bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 4 client.19227 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f15a675e00 con 0x55f15a7196d0 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a7196d0 0x55f15a71bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc292ffd640 10 client.19227 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.765+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a7196d0 0x55f15a71bb50 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a307c49d19dde0cd server_cookie=cc7baa4563e42ad4 in_seq=0 out_seq=0 2024-03-22T02:26:23.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc292ffd640 1 -- 172.21.15.28:0/1984468434 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f15a675e00 con 0x55f15a7196d0 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc292ffd640 10 client.19227 handle_command_reply: tid=0 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 client.19227 shutdown 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 msgr2=0x55f15a714ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0eec70 tx=0x7fc2ac0095c0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 shutdown_connections 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a7196d0 0x55f15a71bb50 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a714ea0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a70de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/1984468434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a70d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 >> 172.21.15.28:0/1984468434 conn(0x55f15a6fe9a0 msgr2=0x55f15a6fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc292ffd640 0 client.19227 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 shutdown_connections 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 1 -- 172.21.15.28:0/1984468434 wait complete. 2024-03-22T02:26:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.767+0000 7fc2bcc28740 20 client.19227 trim_cache size 0 max 16384 2024-03-22T02:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 Processor -- start 2024-03-22T02:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 -- start start 2024-03-22T02:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a705280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a708f90 0x55f15a7057c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41ad2d0 con 0x55f15a703e20 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41aaba0 con 0x55f15a703430 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41a89f0 con 0x55f15a708f90 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42102/0 (socket says 172.21.15.28:42102) 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 -- 172.21.15.28:0/2414551587 learned_addr learned my addr 172.21.15.28:0/2414551587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a708f90 0x55f15a7057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a705280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 -- 172.21.15.28:0/2414551587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a708f90 msgr2=0x55f15a7057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a708f90 0x55f15a7057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 -- 172.21.15.28:0/2414551587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 msgr2=0x55f15a705280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a705280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.785+0000 7fc2b2ffd640 1 -- 172.21.15.28:0/2414551587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15a33c0f0 con 0x55f15a703e20 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2b2ffd640 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80133e0 tx=0x7fc2a8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=636aa0844cf8656 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2997fa640 1 -- 172.21.15.28:0/2414551587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8003af0 con 0x55f15a703e20 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2997fa640 1 -- 172.21.15.28:0/2414551587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a8005500 con 0x55f15a703e20 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2997fa640 1 -- 172.21.15.28:0/2414551587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a800f5c0 con 0x55f15a703e20 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2414551587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 msgr2=0x55f15a714e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80133e0 tx=0x7fc2a8005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2414551587 shutdown_connections 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a708f90 0x55f15a7057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a703e20 0x55f15a714e60 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/2414551587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703430 0x55f15a705280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2414551587 >> 172.21.15.28:0/2414551587 conn(0x55f15a6fe9a0 msgr2=0x55f15a6fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.786+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2414551587 shutdown_connections 2024-03-22T02:26:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 -- 172.21.15.28:0/2414551587 wait complete. 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 -- start start 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 Processor -- start 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a712e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41a9060 con 0x55f15a708f90 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41aa4d0 con 0x55f15a703e20 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41a9730 con 0x55f15a703430 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2bcc28740 10 client.0 _ll_register_callbacks cb 0x7fc2ac0b9cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a712e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43578/0 (socket says 172.21.15.28:43578) 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42106/0 (socket says 172.21.15.28:42106) 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.787+0000 7fc2b37fe640 1 -- 172.21.15.28:0/682968245 learned_addr learned my addr 172.21.15.28:0/682968245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 -- 172.21.15.28:0/682968245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 msgr2=0x55f15a712950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 -- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 msgr2=0x55f15a712e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a712e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 -- 172.21.15.28:0/682968245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15a714490 con 0x55f15a708f90 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2b3fff640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0bd630 tx=0x7fc2ac0bbe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690e4e4330309c5f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0c1a80 con 0x55f15a708f90 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7fc2ac0a8110 con 0x55f15a708f90 2024-03-22T02:26:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2a8005500 con 0x55f15a708f90 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fc2ac0bd930 con 0x55f15a708f90 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2bcc28740 10 client.19233 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f15a77fe40 con 0x55f15a708f90 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2a8005500 con 0x55f15a708f90 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fc2ac0c5e20 con 0x55f15a708f90 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2bcc28740 10 client.19233 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.788+0000 7fc2bcc28740 10 client.19233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2bcc28740 20 client.19233 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a780580 0x55f15a7829a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2bcc28740 4 client.19233 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f15a670150 con 0x55f15a780580 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a780580 0x55f15a7829a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc292ffd640 10 client.19233 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.789+0000 7fc2b37fe640 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a780580 0x55f15a7829a0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b3fa2b170da89c server_cookie=b10113c8b7721435 in_seq=0 out_seq=0 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc292ffd640 1 -- 172.21.15.28:0/682968245 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f15a670150 con 0x55f15a780580 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc292ffd640 10 client.19233 handle_command_reply: tid=0 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 client.19233 shutdown 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 msgr2=0x55f15a713ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac0bd630 tx=0x7fc2ac0bbe40 comp rx=0 tx=0).stop 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 shutdown_connections 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f15a780580 0x55f15a7829a0 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f15a708f90 0x55f15a713ea0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f15a703e20 0x55f15a712e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc292ffd640 0 client.19233 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 --2- 172.21.15.28:0/682968245 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f15a703430 0x55f15a712950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 >> 172.21.15.28:0/682968245 conn(0x55f15a6fe9a0 msgr2=0x55f15a705280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 shutdown_connections 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 1 -- 172.21.15.28:0/682968245 wait complete. 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.791+0000 7fc2bcc28740 20 client.19233 trim_cache size 0 max 16384 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "d624ef7e-9322-43c7-9e93-4b790d48cb42": { 2024-03-22T02:26:23.799 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:26:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d624ef7e-9322-43c7-9e93-4b790d48cb42", 2024-03-22T02:26:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:26:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:26:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:26:23.800 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 msgr2=0x7fc29c0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 0x7fc29c0a5460 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8009710 tx=0x7fc2a80098b0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 msgr2=0x7fc2b419f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac037040 tx=0x7fc2ac032f40 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 shutdown_connections 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc29c0a2fa0 0x7fc29c0a5460 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2b4102650 0x7fc2b419f200 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc2b4101c90 0x7fc2b419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 --2- 172.21.15.28:0/3415058021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc2b4100aa0 0x7fc2b406ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 >> 172.21.15.28:0/3415058021 conn(0x7fc2b40fc030 msgr2=0x7fc2b4105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 shutdown_connections 2024-03-22T02:26:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:23.792+0000 7fc2b9f5e640 1 -- 172.21.15.28:0/3415058021 wait complete. 2024-03-22T02:26:23.804 INFO:tasks.cephfs.filesystem:scrub status for tag:d624ef7e-9322-43c7-9e93-4b790d48cb42 - {'path': '/', 'tag': 'd624ef7e-9322-43c7-9e93-4b790d48cb42', 'options': 'recursive,force'} 2024-03-22T02:26:24.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1324752 11.21 MB/sec execute 546 sec latency 43.094 ms 2024-03-22T02:26:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:24 smithi120 ceph-mon[36803]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:26:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:24 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:26:24.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:24 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:24.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:24 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:26:24.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:24 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:24 smithi083 ceph-mon[36340]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:26:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:24 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:26:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:24 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:24 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:26:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:24 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:24 smithi028 ceph-mon[31916]: pgmap v744: 129 pgs: 129 active+clean; 60 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.6 MiB/s wr, 385 op/s 2024-03-22T02:26:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:24 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:26:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:24 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:24 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:26:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:24 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:25.437 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1325792 11.19 MB/sec execute 547 sec latency 69.859 ms 2024-03-22T02:26:25.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:25 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:26:25.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:25 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:25 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:26:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:25 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:25.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:25 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:26:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:25 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:26.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1328242 11.20 MB/sec execute 548 sec latency 27.971 ms 2024-03-22T02:26:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:26 smithi120 ceph-mon[36803]: pgmap v745: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-03-22T02:26:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:26 smithi083 ceph-mon[36340]: pgmap v745: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-03-22T02:26:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:26 smithi028 ceph-mon[31916]: pgmap v745: 129 pgs: 129 active+clean; 61 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 427 op/s 2024-03-22T02:26:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- 172.21.15.28:0/1521406589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260100ab0 msgr2=0x7ff260100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 --2- 172.21.15.28:0/1521406589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260100ab0 0x7ff260100ed0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009ca0 tx=0x7ff25c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- 172.21.15.28:0/1521406589 shutdown_connections 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 --2- 172.21.15.28:0/1521406589 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260102660 0x7ff260108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 --2- 172.21.15.28:0/1521406589 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260101ca0 0x7ff260102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 --2- 172.21.15.28:0/1521406589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260100ab0 0x7ff260100ed0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- 172.21.15.28:0/1521406589 >> 172.21.15.28:0/1521406589 conn(0x7ff2600fc020 msgr2=0x7ff2600fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- 172.21.15.28:0/1521406589 shutdown_connections 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- 172.21.15.28:0/1521406589 wait complete. 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 Processor -- start 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.253+0000 7ff268113640 1 -- start start 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260101ca0 0x7ff2601983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260102660 0x7ff26019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff26010c700 con 0x7ff260102660 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff26010c580 con 0x7ff260100ab0 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff26010c880 con 0x7ff260101ca0 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff2666b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260102660 0x7ff26019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33300/0 (socket says 172.21.15.28:33300) 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 -- 172.21.15.28:0/3468721401 learned_addr learned my addr 172.21.15.28:0/3468721401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff2656ae640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260101ca0 0x7ff2601983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 -- 172.21.15.28:0/3468721401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260101ca0 msgr2=0x7ff2601983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260101ca0 0x7ff2601983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 -- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260102660 msgr2=0x7ff26019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260102660 0x7ff26019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 -- 172.21.15.28:0/3468721401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26019fa90 con 0x7ff260100ab0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f80 tx=0x7ff25c032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff256ffd640 1 -- 172.21.15.28:0/3468721401 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c041020 con 0x7ff260100ab0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.254+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26019fd20 con 0x7ff260100ab0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.255+0000 7ff256ffd640 1 -- 172.21.15.28:0/3468721401 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c034cd0 con 0x7ff260100ab0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.255+0000 7ff256ffd640 1 -- 172.21.15.28:0/3468721401 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c039040 con 0x7ff260100ab0 2024-03-22T02:26:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.255+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2601a0180 con 0x7ff260100ab0 2024-03-22T02:26:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff256ffd640 1 -- 172.21.15.28:0/3468721401 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff25c044020 con 0x7ff260100ab0 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 Processor -- start 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 -- start start 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x55698212b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 0x5569821309d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x5569821334d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556982090040 con 0x55698212b4e0 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5569820389e0 con 0x556982131040 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.257+0000 7ff26add9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55698208bff0 con 0x55698212bed0 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2666b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x55698212b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 0x5569821309d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2666b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x55698212b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42130/0 (socket says 172.21.15.28:42130) 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2666b0640 1 -- 172.21.15.28:0/2882613671 learned_addr learned my addr 172.21.15.28:0/2882613671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2656ae640 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x5569821334d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff256ffd640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 0x7ff24c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2666b0640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 0x7ff24c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 -- 172.21.15.28:0/2882613671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 msgr2=0x5569821334d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x5569821334d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 -- 172.21.15.28:0/2882613671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 msgr2=0x55698212b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x55698212b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 -- 172.21.15.28:0/2882613671 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556981da4210 con 0x55698212bed0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2656ae640 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x5569821334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff256ffd640 1 -- 172.21.15.28:0/3468721401 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff25c0f15e0 con 0x7ff260100ab0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 0x5569821309d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0bd630 tx=0x7ff25c032590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c311565842355604 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff2666b0640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 0x7ff24c0a5480 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff258009fd0 tx=0x7ff258004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.258+0000 7ff24bfff640 1 -- 172.21.15.28:0/2882613671 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0bd470 con 0x55698212bed0 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff24bfff640 1 -- 172.21.15.28:0/2882613671 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c0be030 con 0x55698212bed0 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff24bfff640 1 -- 172.21.15.28:0/2882613671 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0bd290 con 0x55698212bed0 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 -- 172.21.15.28:0/2882613671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 msgr2=0x5569821309d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 0x5569821309d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0bd630 tx=0x7ff25c032590 comp rx=0 tx=0).stop 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 -- 172.21.15.28:0/2882613671 shutdown_connections 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x5569821334d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212bed0 0x5569821309d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 --2- 172.21.15.28:0/2882613671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x55698212b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 -- 172.21.15.28:0/2882613671 >> 172.21.15.28:0/2882613671 conn(0x556982126a50 msgr2=0x556982126e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 -- 172.21.15.28:0/2882613671 shutdown_connections 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.259+0000 7ff26add9740 1 -- 172.21.15.28:0/2882613671 wait complete. 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 -- start start 2024-03-22T02:26:27.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 Processor -- start 2024-03-22T02:26:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 0x556982135130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982135670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556982133db0 con 0x55698212bed0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55698204bdb0 con 0x55698212b4e0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556982133f30 con 0x556982131040 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff26add9740 10 client.0 _ll_register_callbacks cb 0x556981da3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff2656ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff2666b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 0x556982135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff2666b0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43608/0 (socket says 172.21.15.28:43608) 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff2656ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982135670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42146/0 (socket says 172.21.15.28:42146) 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.260+0000 7ff2666b0640 1 -- 172.21.15.28:0/2279098627 learned_addr learned my addr 172.21.15.28:0/2279098627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 -- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 msgr2=0x556982135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 0x556982135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 -- 172.21.15.28:0/2279098627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 msgr2=0x556982135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982135670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 -- 172.21.15.28:0/2279098627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556981e70620 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 0x556982135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2666b0640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2580042e0 tx=0x7ff2580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4d792adf6622693 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2497fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258024070 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff258012970 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff25c0be030 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff258004490 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff26add9740 10 client.29062 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55698213fa90 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.261+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff25c0be030 con 0x556982131040 2024-03-22T02:26:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7ff25801f040 con 0x556982131040 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 10 client.29062 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 10 client.29062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 20 client.29062 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556982141180 0x5569821435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 4 client.29062 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55698209d580 con 0x556982141180 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556982141180 0x5569821435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff2497fa640 10 client.29062 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.262+0000 7ff265eaf640 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556982141180 0x5569821435d0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45fae7c49f21daae server_cookie=20257bc1f92a6eb4 in_seq=0 out_seq=0 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff2497fa640 1 -- 172.21.15.28:0/2279098627 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55698209d580 con 0x556982141180 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff2497fa640 10 client.29062 handle_command_reply: tid=0 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 client.29062 shutdown 2024-03-22T02:26:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 msgr2=0x55698213c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff2580042e0 tx=0x7ff2580046c0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 shutdown_connections 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556982141180 0x5569821435d0 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x55698213c6b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982135670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 --2- 172.21.15.28:0/2279098627 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212b4e0 0x556982135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff2497fa640 0 client.29062 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 >> 172.21.15.28:0/2279098627 conn(0x556982126a50 msgr2=0x556982125c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 shutdown_connections 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 1 -- 172.21.15.28:0/2279098627 wait complete. 2024-03-22T02:26:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.263+0000 7ff26add9740 20 client.29062 trim_cache size 0 max 16384 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 Processor -- start 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 -- start start 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212b4e0 0x5569821253c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x556982125e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601ad240 con 0x55698212bed0 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601aab10 con 0x556982131040 2024-03-22T02:26:27.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff26add9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a8960 con 0x55698212b4e0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff2656ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff2656ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42148/0 (socket says 172.21.15.28:42148) 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff2656ae640 1 -- 172.21.15.28:0/1793064939 learned_addr learned my addr 172.21.15.28:0/1793064939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff265eaf640 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212b4e0 0x5569821253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.279+0000 7ff2666b0640 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x556982125e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 -- 172.21.15.28:0/1793064939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212b4e0 msgr2=0x5569821253c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212b4e0 0x5569821253c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 -- 172.21.15.28:0/1793064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 msgr2=0x556982125e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x556982125e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 -- 172.21.15.28:0/1793064939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55698212c2b0 con 0x55698212bed0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff2656ae640 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff250009e20 tx=0x7ff250005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6419ce21a3f2691b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff24b7fe640 1 -- 172.21.15.28:0/1793064939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff250010430 con 0x55698212bed0 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff24b7fe640 1 -- 172.21.15.28:0/1793064939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff250021070 con 0x55698212bed0 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.280+0000 7ff24b7fe640 1 -- 172.21.15.28:0/1793064939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25001c780 con 0x55698212bed0 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- 172.21.15.28:0/1793064939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 msgr2=0x556982125900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7ff250009e20 tx=0x7ff250005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- 172.21.15.28:0/1793064939 shutdown_connections 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556982131040 0x556982125e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212bed0 0x556982125900 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 --2- 172.21.15.28:0/1793064939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55698212b4e0 0x5569821253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- 172.21.15.28:0/1793064939 >> 172.21.15.28:0/1793064939 conn(0x556982126a50 msgr2=0x55698212a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- 172.21.15.28:0/1793064939 shutdown_connections 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- 172.21.15.28:0/1793064939 wait complete. 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 -- start start 2024-03-22T02:26:27.286 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.281+0000 7ff26add9740 1 Processor -- start 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212bed0 0x5569821a2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x5569821a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a8fd0 con 0x55698212b4e0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601aa440 con 0x55698212bed0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff2601a96a0 con 0x556982131040 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 10 client.0 _ll_register_callbacks cb 0x7ff258012360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42154/0 (socket says 172.21.15.28:42154) 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2656ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212bed0 0x5569821a2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 -- 172.21.15.28:0/3469544958 learned_addr learned my addr 172.21.15.28:0/3469544958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2666b0640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x5569821a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 -- 172.21.15.28:0/3469544958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 msgr2=0x5569821a9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x5569821a9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 -- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212bed0 msgr2=0x5569821a2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212bed0 0x5569821a2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 -- 172.21.15.28:0/3469544958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569821aa470 con 0x55698212b4e0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff265eaf640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0bd8c0 tx=0x7ff25c034f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c5dfea80ca1019 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2497fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0bd290 con 0x55698212b4e0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9274+0+0 (secure 0 0 0) 0x7ff25c02f160 con 0x55698212b4e0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff250021070 con 0x55698212b4e0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7ff25c0bd470 con 0x55698212b4e0 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 10 client.19251 fetch_fsmap learned FSMap version 37 2024-03-22T02:26:27.288 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.282+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5569821ab7f0 con 0x55698212b4e0 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff250021070 con 0x55698212b4e0 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7ff25c03c030 con 0x55698212b4e0 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 10 client.19251 fetch_fsmap finished waiting for FSMap version 37 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 10 client.19251 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 20 client.19251 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569821abef0 0x5569821ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 4 client.19251 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569820978d0 con 0x5569821abef0 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff2656ae640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569821abef0 0x5569821ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff2497fa640 10 client.19251 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:27.289 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.283+0000 7ff2656ae640 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569821abef0 0x5569821ae310 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e50586c6079759c server_cookie=af4053ab136b7162 in_seq=0 out_seq=0 2024-03-22T02:26:27.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1329430 11.19 MB/sec execute 549 sec latency 108.397 ms 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.663+0000 7ff2497fa640 1 -- 172.21.15.28:0/3469544958 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569820978d0 con 0x5569821abef0 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.663+0000 7ff2497fa640 10 client.19251 handle_command_reply: tid=0 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 client.19251 shutdown 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 msgr2=0x5569821a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0bd8c0 tx=0x7ff25c034f40 comp rx=0 tx=0).stop 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 shutdown_connections 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569821abef0 0x5569821ae310 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556982131040 0x5569821a9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55698212bed0 0x5569821a2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 --2- 172.21.15.28:0/3469544958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698212b4e0 0x5569821a2900 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 >> 172.21.15.28:0/3469544958 conn(0x556982126a50 msgr2=0x55698212c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff2497fa640 0 client.19251 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 shutdown_connections 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 1 -- 172.21.15.28:0/3469544958 wait complete. 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.664+0000 7ff26add9740 20 client.19251 trim_cache size 0 max 16384 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:26:27.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 msgr2=0x7ff24c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 0x7ff24c0a5480 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7ff258009fd0 tx=0x7ff258004730 comp rx=0 tx=0).stop 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 msgr2=0x7ff260197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f80 tx=0x7ff25c032ec0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 shutdown_connections 2024-03-22T02:26:27.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff24c0a2fc0 0x7ff24c0a5480 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff260102660 0x7ff26019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff260101ca0 0x7ff2601983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 --2- 172.21.15.28:0/3468721401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff260100ab0 0x7ff260197e70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 >> 172.21.15.28:0/3468721401 conn(0x7ff2600fc020 msgr2=0x7ff2600fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 shutdown_connections 2024-03-22T02:26:27.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:27.665+0000 7ff268113640 1 -- 172.21.15.28:0/3468721401 wait complete. 2024-03-22T02:26:28.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:26:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setuser ceph since I am not root 2024-03-22T02:26:28.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:26:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setgroup ceph since I am not root 2024-03-22T02:26:28.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:26:27 smithi083 ceph-mds[66500]: -- 172.21.15.83:0/961176537 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f1db253a40 con 0x55f1db227000 2024-03-22T02:26:28.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:26:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: starting mds.h at 2024-03-22T02:26:28.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1331884 11.20 MB/sec execute 550 sec latency 192.782 ms 2024-03-22T02:26:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: pgmap v746: 129 pgs: 129 active+clean; 63 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:26:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: daemon mds.h restarted 2024-03-22T02:26:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T02:26:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/366330553,v1:172.21.15.83:6837/366330553] up:boot 2024-03-22T02:26:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:28.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: pgmap v746: 129 pgs: 129 active+clean; 63 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: daemon mds.h restarted 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/366330553,v1:172.21.15.83:6837/366330553] up:boot 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: pgmap v746: 129 pgs: 129 active+clean; 63 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: daemon mds.h restarted 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: osdmap e97: 12 total, 12 up, 12 in 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/366330553,v1:172.21.15.83:6837/366330553] up:boot 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:26:29.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1333602 11.19 MB/sec execute 551 sec latency 53.125 ms 2024-03-22T02:26:30.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1335599 11.19 MB/sec execute 552 sec latency 51.945 ms 2024-03-22T02:26:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:30 smithi120 ceph-mon[36803]: pgmap v748: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-03-22T02:26:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:30 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:26:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:30 smithi083 ceph-mon[36340]: pgmap v748: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-03-22T02:26:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:30 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:26:30.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:30 smithi028 ceph-mon[31916]: pgmap v748: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-03-22T02:26:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:30 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:26:31.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1337719 11.19 MB/sec execute 553 sec latency 30.755 ms 2024-03-22T02:26:31.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:31 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:26:31.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:31 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:26:31.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:31 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:26:32.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1339564 11.19 MB/sec execute 554 sec latency 29.479 ms 2024-03-22T02:26:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:32 smithi120 ceph-mon[36803]: pgmap v749: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-03-22T02:26:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:32 smithi083 ceph-mon[36340]: pgmap v749: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-03-22T02:26:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:32 smithi028 ceph-mon[31916]: pgmap v749: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-03-22T02:26:33.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1342446 11.20 MB/sec execute 555 sec latency 14.760 ms 2024-03-22T02:26:34.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1343891 11.19 MB/sec execute 556 sec latency 33.821 ms 2024-03-22T02:26:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:34 smithi120 ceph-mon[36803]: pgmap v750: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:26:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:34 smithi083 ceph-mon[36340]: pgmap v750: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:26:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:34 smithi028 ceph-mon[31916]: pgmap v750: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 3.0 MiB/s wr, 417 op/s 2024-03-22T02:26:35.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1346124 11.19 MB/sec execute 557 sec latency 64.002 ms 2024-03-22T02:26:36.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1347559 11.18 MB/sec execute 558 sec latency 28.031 ms 2024-03-22T02:26:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:36 smithi120 ceph-mon[36803]: pgmap v751: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 399 op/s 2024-03-22T02:26:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:36 smithi083 ceph-mon[36340]: pgmap v751: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 399 op/s 2024-03-22T02:26:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:36 smithi028 ceph-mon[31916]: pgmap v751: 129 pgs: 129 active+clean; 53 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 399 op/s 2024-03-22T02:26:37.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1349871 11.19 MB/sec execute 559 sec latency 14.435 ms 2024-03-22T02:26:38.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1352226 11.19 MB/sec execute 560 sec latency 30.700 ms 2024-03-22T02:26:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:38 smithi120 ceph-mon[36803]: pgmap v752: 129 pgs: 129 active+clean; 56 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-03-22T02:26:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:38 smithi083 ceph-mon[36340]: pgmap v752: 129 pgs: 129 active+clean; 56 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-03-22T02:26:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:38 smithi028 ceph-mon[31916]: pgmap v752: 129 pgs: 129 active+clean; 56 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-03-22T02:26:39.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1353781 11.18 MB/sec execute 561 sec latency 29.313 ms 2024-03-22T02:26:40.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1356323 11.19 MB/sec execute 562 sec latency 91.320 ms 2024-03-22T02:26:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:40 smithi120 ceph-mon[36803]: pgmap v753: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 349 op/s 2024-03-22T02:26:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- 172.21.15.28:0/1706325383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c107990 msgr2=0x7f839c109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 --2- 172.21.15.28:0/1706325383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c107990 0x7f839c109d80 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f838c009ca0 tx=0x7f838c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- 172.21.15.28:0/1706325383 shutdown_connections 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 --2- 172.21.15.28:0/1706325383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c107990 0x7f839c109d80 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 --2- 172.21.15.28:0/1706325383 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c105060 0x7f839c107450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 --2- 172.21.15.28:0/1706325383 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c102730 0x7f839c104b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- 172.21.15.28:0/1706325383 >> 172.21.15.28:0/1706325383 conn(0x7f839c0fc0c0 msgr2=0x7f839c0fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- 172.21.15.28:0/1706325383 shutdown_connections 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- 172.21.15.28:0/1706325383 wait complete. 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 Processor -- start 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.754+0000 7f83a155c640 1 -- start start 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 0x7f839c1006e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c107990 0x7f839c100c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10d860 con 0x7f839c102730 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10d6e0 con 0x7f839c105060 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f839c10d9e0 con 0x7f839c107990 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41184/0 (socket says 172.21.15.28:41184) 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 -- 172.21.15.28:0/2127792458 learned_addr learned my addr 172.21.15.28:0/2127792458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 0x7f839c1006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c107990 0x7f839c100c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 -- 172.21.15.28:0/2127792458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c107990 msgr2=0x7f839c100c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c107990 0x7f839c100c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 -- 172.21.15.28:0/2127792458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 msgr2=0x7f839c1006e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 0x7f839c1006e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 -- 172.21.15.28:0/2127792458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c1014b0 con 0x7f839c102730 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 0x7f839c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f839affd640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f839400e9f0 tx=0x7f839400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f8393fff640 1 -- 172.21.15.28:0/2127792458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839400cd80 con 0x7f839c102730 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f8393fff640 1 -- 172.21.15.28:0/2127792458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8394010910 con 0x7f839c102730 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c1a1ee0 con 0x7f839c102730 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.755+0000 7f8393fff640 1 -- 172.21.15.28:0/2127792458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394018990 con 0x7f839c102730 2024-03-22T02:26:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.756+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c1a22f0 con 0x7f839c102730 2024-03-22T02:26:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:40 smithi083 ceph-mon[36340]: pgmap v753: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 349 op/s 2024-03-22T02:26:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.757+0000 7f8393fff640 1 -- 172.21.15.28:0/2127792458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8394010ac0 con 0x7f839c102730 2024-03-22T02:26:40.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 Processor -- start 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 -- start start 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f8393fff640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 0x7f83880a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 0x7f83880a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f8393fff640 1 -- 172.21.15.28:0/2127792458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f83940cfa80 con 0x7f839c102730 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd5d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56435dcba040 con 0x56435dd5b040 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56435dc629e0 con 0x56435dd554e0 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f83a4216740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56435dcb5ff0 con 0x56435dd55ed0 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.758+0000 7f839affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45574/0 (socket says 172.21.15.28:45574) 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 -- 172.21.15.28:0/2607302202 learned_addr learned my addr 172.21.15.28:0/2607302202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd5d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 -- 172.21.15.28:0/2607302202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 msgr2=0x56435dd5a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 0x7f83880a5480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f838c005e30 tx=0x7f838c009840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 -- 172.21.15.28:0/2607302202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 msgr2=0x56435dd5d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd5d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 -- 172.21.15.28:0/2607302202 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56435d9ce210 con 0x56435dd554e0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f839affd640 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83940a4280 tx=0x7f839409b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3de86db3d57c195 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f83917fa640 1 -- 172.21.15.28:0/2607302202 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839409a150 con 0x56435dd554e0 2024-03-22T02:26:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f83917fa640 1 -- 172.21.15.28:0/2607302202 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83940984c0 con 0x56435dd554e0 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.759+0000 7f83917fa640 1 -- 172.21.15.28:0/2607302202 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839409b590 con 0x56435dd554e0 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- 172.21.15.28:0/2607302202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 msgr2=0x56435dd55900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f83940a4280 tx=0x7f839409b8b0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- 172.21.15.28:0/2607302202 shutdown_connections 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd5d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 --2- 172.21.15.28:0/2607302202 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd55900 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- 172.21.15.28:0/2607302202 >> 172.21.15.28:0/2607302202 conn(0x56435dd50a50 msgr2=0x56435dd50e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- 172.21.15.28:0/2607302202 shutdown_connections 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- 172.21.15.28:0/2607302202 wait complete. 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 -- start start 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.760+0000 7f83a4216740 1 Processor -- start 2024-03-22T02:26:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56435dd5df90 con 0x56435dd5b040 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56435dc75db0 con 0x56435dd554e0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56435dd5e110 con 0x56435dd55ed0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83a4216740 10 client.0 _ll_register_callbacks cb 0x56435d9cda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41198/0 (socket says 172.21.15.28:41198) 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 -- 172.21.15.28:0/2389703391 learned_addr learned my addr 172.21.15.28:0/2389703391 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 -- 172.21.15.28:0/2389703391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 msgr2=0x56435dd5fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 -- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 msgr2=0x56435dd5f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 -- 172.21.15.28:0/2389703391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56435da9a620 con 0x56435dd5b040 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839affd640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f838c009ca0 tx=0x7f838c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879c43bea9d86736 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.761+0000 7f83827fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c039040 con 0x56435dd5b040 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f838c003d80 con 0x56435dd5b040 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83940984c0 con 0x56435dd5b040 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f838c05a050 con 0x56435dd5b040 2024-03-22T02:26:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 10 client.19269 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56435dd69e40 con 0x56435dd5b040 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83940984c0 con 0x56435dd5b040 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f838c03f040 con 0x56435dd5b040 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 10 client.19269 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 10 client.19269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 20 client.19269 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435dd6b580 0x56435dd6d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 4 client.19269 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56435dcc7580 con 0x56435dd6b580 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.762+0000 7f839affd640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435dd6b580 0x56435dd6d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.763+0000 7f83827fc640 10 client.19269 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.763+0000 7f839affd640 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435dd6b580 0x56435dd6d9d0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ceb836d609332c server_cookie=3c453cb8da16f999 in_seq=0 out_seq=0 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.764+0000 7f83827fc640 1 -- 172.21.15.28:0/2389703391 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56435dcc7580 con 0x56435dd6b580 2024-03-22T02:26:40.772 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.764+0000 7f83827fc640 10 client.19269 handle_command_reply: tid=0 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.764+0000 7f83a4216740 1 client.19269 shutdown 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 msgr2=0x56435dd66a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f838c009ca0 tx=0x7f838c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 shutdown_connections 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435dd6b580 0x56435dd6d9d0 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd66a60 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd55ed0 0x56435dd5fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 --2- 172.21.15.28:0/2389703391 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83827fc640 0 client.19269 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 >> 172.21.15.28:0/2389703391 conn(0x56435dd50a50 msgr2=0x56435dd4fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 shutdown_connections 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 1 -- 172.21.15.28:0/2389703391 wait complete. 2024-03-22T02:26:40.773 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.765+0000 7f83a4216740 20 client.19269 trim_cache size 0 max 16384 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 Processor -- start 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 -- start start 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd5b040 0x56435dd4f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1af490 con 0x56435dd55ed0 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1acd60 con 0x56435dd554e0 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f83a4216740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1aabb0 con 0x56435dd5b040 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.782 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41210/0 (socket says 172.21.15.28:41210) 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f839a7fc640 1 -- 172.21.15.28:0/2423744444 learned_addr learned my addr 172.21.15.28:0/2423744444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.780+0000 7f839affd640 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839b7fe640 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd5b040 0x56435dd4f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 -- 172.21.15.28:0/2423744444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd5b040 msgr2=0x56435dd4f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd5b040 0x56435dd4f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 -- 172.21.15.28:0/2423744444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 msgr2=0x56435dd5a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 -- 172.21.15.28:0/2423744444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56435dd55120 con 0x56435dd55ed0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f839a7fc640 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f83840125e0 tx=0x7f8384005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a87df5132b8b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f8390ff9640 1 -- 172.21.15.28:0/2423744444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384022070 con 0x56435dd55ed0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f8390ff9640 1 -- 172.21.15.28:0/2423744444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83840035f0 con 0x56435dd55ed0 2024-03-22T02:26:40.783 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.781+0000 7f8390ff9640 1 -- 172.21.15.28:0/2423744444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838400f800 con 0x56435dd55ed0 2024-03-22T02:26:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- 172.21.15.28:0/2423744444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 msgr2=0x56435dd66a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f83840125e0 tx=0x7f8384005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- 172.21.15.28:0/2423744444 shutdown_connections 2024-03-22T02:26:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd5b040 0x56435dd4f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd55ed0 0x56435dd66a20 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 --2- 172.21.15.28:0/2423744444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd554e0 0x56435dd5a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- 172.21.15.28:0/2423744444 >> 172.21.15.28:0/2423744444 conn(0x56435dd50a50 msgr2=0x56435dd54540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- 172.21.15.28:0/2423744444 shutdown_connections 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- 172.21.15.28:0/2423744444 wait complete. 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 -- start start 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.782+0000 7f83a4216740 1 Processor -- start 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd554e0 0x56435dd64480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd55ed0 0x56435dd649c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ab220 con 0x56435dd5b040 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ac690 con 0x56435dd55ed0 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f839c1ab8f0 con 0x56435dd554e0 2024-03-22T02:26:40.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 10 client.0 _ll_register_callbacks cb 0x7f8394005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41214/0 (socket says 172.21.15.28:41214) 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 -- 172.21.15.28:0/4280814820 learned_addr learned my addr 172.21.15.28:0/4280814820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839affd640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd554e0 0x56435dd64480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839a7fc640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd55ed0 0x56435dd649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 -- 172.21.15.28:0/4280814820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd554e0 msgr2=0x56435dd64480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd554e0 0x56435dd64480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 -- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd55ed0 msgr2=0x56435dd649c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd55ed0 0x56435dd649c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 -- 172.21.15.28:0/4280814820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56435dd65f50 con 0x56435dd5b040 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f839b7fe640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f838c02c700 tx=0x7f838c032d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79afc0181c4fa2b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:40.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83827fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c039040 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f838c05e840 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.783+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83840035f0 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f838c06c050 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 10 client.19275 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56435ddd1c80 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83840035f0 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f838c0546a0 con 0x56435dd5b040 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 10 client.19275 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 10 client.19275 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 20 client.19275 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:40.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435ddd26a0 0x56435ddd4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 4 client.19275 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:26:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56435dcc18d0 con 0x56435ddd26a0 2024-03-22T02:26:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f839affd640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435ddd26a0 0x56435ddd4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f839affd640 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435ddd26a0 0x56435ddd4ac0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e454b84cdad027f5 server_cookie=71fddb925abf2d05 in_seq=0 out_seq=0 2024-03-22T02:26:40.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:40.784+0000 7f83827fc640 10 client.19275 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:40 smithi028 ceph-mon[31916]: pgmap v753: 129 pgs: 129 active+clean; 57 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 2.8 MiB/s wr, 349 op/s 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.132+0000 7f83827fc640 1 -- 172.21.15.28:0/4280814820 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56435dcc18d0 con 0x56435ddd26a0 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.132+0000 7f83827fc640 10 client.19275 handle_command_reply: tid=0 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.132+0000 7f83a4216740 1 client.19275 shutdown 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 msgr2=0x56435dd65960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f838c02c700 tx=0x7f838c032d00 comp rx=0 tx=0).stop 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 shutdown_connections 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56435ddd26a0 0x56435ddd4ac0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83827fc640 0 client.19275 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56435dd5b040 0x56435dd65960 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56435dd55ed0 0x56435dd649c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 --2- 172.21.15.28:0/4280814820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56435dd554e0 0x56435dd64480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 >> 172.21.15.28:0/4280814820 conn(0x56435dd50a50 msgr2=0x56435dd54540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 shutdown_connections 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 1 -- 172.21.15.28:0/4280814820 wait complete. 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a4216740 20 client.19275 trim_cache size 0 max 16384 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:26:41.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:26:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:26:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 msgr2=0x7f83880a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:41.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.133+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 0x7f83880a5480 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f838c005e30 tx=0x7f838c009840 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 msgr2=0x7f839c1020a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f839400e9f0 tx=0x7f839400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 shutdown_connections 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83880a2fc0 0x7f83880a5480 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f839c107990 0x7f839c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f839c105060 0x7f839c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 --2- 172.21.15.28:0/2127792458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839c102730 0x7f839c1020a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 >> 172.21.15.28:0/2127792458 conn(0x7f839c0fc0c0 msgr2=0x7f839c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 shutdown_connections 2024-03-22T02:26:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:41.134+0000 7f83a155c640 1 -- 172.21.15.28:0/2127792458 wait complete. 2024-03-22T02:26:41.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1357260 11.18 MB/sec execute 563 sec latency 164.243 ms 2024-03-22T02:26:42.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1359900 11.18 MB/sec execute 564 sec latency 17.249 ms 2024-03-22T02:26:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:42 smithi120 ceph-mon[36803]: pgmap v754: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 320 op/s 2024-03-22T02:26:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:42 smithi083 ceph-mon[36340]: pgmap v754: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 320 op/s 2024-03-22T02:26:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:42 smithi028 ceph-mon[31916]: pgmap v754: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 320 op/s 2024-03-22T02:26:43.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1361598 11.18 MB/sec execute 565 sec latency 29.956 ms 2024-03-22T02:26:44.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1364106 11.19 MB/sec execute 566 sec latency 14.087 ms 2024-03-22T02:26:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:44 smithi120 ceph-mon[36803]: pgmap v755: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-22T02:26:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:44 smithi083 ceph-mon[36340]: pgmap v755: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-22T02:26:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:44 smithi028 ceph-mon[31916]: pgmap v755: 129 pgs: 129 active+clean; 59 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 289 op/s 2024-03-22T02:26:45.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1366610 11.19 MB/sec execute 567 sec latency 68.450 ms 2024-03-22T02:26:46.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1367995 11.18 MB/sec execute 568 sec latency 57.179 ms 2024-03-22T02:26:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:46 smithi120 ceph-mon[36803]: pgmap v756: 129 pgs: 129 active+clean; 58 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 371 op/s 2024-03-22T02:26:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:46 smithi083 ceph-mon[36340]: pgmap v756: 129 pgs: 129 active+clean; 58 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 371 op/s 2024-03-22T02:26:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:46 smithi028 ceph-mon[31916]: pgmap v756: 129 pgs: 129 active+clean; 58 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.1 MiB/s wr, 371 op/s 2024-03-22T02:26:47.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1370370 11.18 MB/sec execute 569 sec latency 20.751 ms 2024-03-22T02:26:48.438 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1371649 11.18 MB/sec execute 570 sec latency 44.480 ms 2024-03-22T02:26:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:48 smithi120 ceph-mon[36803]: pgmap v757: 129 pgs: 129 active+clean; 54 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 368 op/s 2024-03-22T02:26:48.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:48 smithi083 ceph-mon[36340]: pgmap v757: 129 pgs: 129 active+clean; 54 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 368 op/s 2024-03-22T02:26:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:48 smithi028 ceph-mon[31916]: pgmap v757: 129 pgs: 129 active+clean; 54 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 368 op/s 2024-03-22T02:26:49.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1374271 11.19 MB/sec execute 571 sec latency 9.458 ms 2024-03-22T02:26:50.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1375295 11.19 MB/sec execute 572 sec latency 62.137 ms 2024-03-22T02:26:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:50 smithi120 ceph-mon[36803]: pgmap v758: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 397 op/s 2024-03-22T02:26:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:50 smithi083 ceph-mon[36340]: pgmap v758: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 397 op/s 2024-03-22T02:26:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:50 smithi028 ceph-mon[31916]: pgmap v758: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.2 MiB/s wr, 397 op/s 2024-03-22T02:26:51.442 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1377170 11.20 MB/sec execute 573 sec latency 98.513 ms 2024-03-22T02:26:52.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1378991 11.20 MB/sec execute 574 sec latency 33.432 ms 2024-03-22T02:26:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:52 smithi120 ceph-mon[36803]: pgmap v759: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-03-22T02:26:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:52 smithi083 ceph-mon[36340]: pgmap v759: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-03-22T02:26:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:52 smithi028 ceph-mon[31916]: pgmap v759: 129 pgs: 129 active+clean; 58 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.7 MiB/s wr, 380 op/s 2024-03-22T02:26:53.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1381149 11.19 MB/sec execute 575 sec latency 38.807 ms 2024-03-22T02:26:53.805 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:26:53.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 -- 172.21.15.28:0/200163312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf8102e40 msgr2=0x7f8bf81032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 --2- 172.21.15.28:0/200163312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf8102e40 0x7f8bf81032c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009930 tx=0x7f8be802bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 -- 172.21.15.28:0/200163312 shutdown_connections 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 --2- 172.21.15.28:0/200163312 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8103800 0x7f8bf810a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 --2- 172.21.15.28:0/200163312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf8102e40 0x7f8bf81032c0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 --2- 172.21.15.28:0/200163312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 -- 172.21.15.28:0/200163312 >> 172.21.15.28:0/200163312 conn(0x7f8bf80fb7a0 msgr2=0x7f8bf80fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 -- 172.21.15.28:0/200163312 shutdown_connections 2024-03-22T02:26:53.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 -- 172.21.15.28:0/200163312 wait complete. 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.916+0000 7f8bffb32640 1 Processor -- start 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 -- start start 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d850 con 0x7f8bf80ffe70 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d6d0 con 0x7f8bf8102e40 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bffb32640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d9d0 con 0x7f8bf8103800 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52408/0 (socket says 172.21.15.28:52408) 2024-03-22T02:26:53.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39352/0 (socket says 172.21.15.28:39352) 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/214275144 learned_addr learned my addr 172.21.15.28:0/214275144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40036/0 (socket says 172.21.15.28:40036) 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 -- 172.21.15.28:0/214275144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 msgr2=0x7f8bf8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 -- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 msgr2=0x7f8bf8197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 -- 172.21.15.28:0/214275144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf819fb10 con 0x7f8bf8103800 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.917+0000 7f8bfe0cf640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009fd0 tx=0x7f8bf000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.918+0000 7f8beeffd640 1 -- 172.21.15.28:0/214275144 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000ce20 con 0x7f8bf8103800 2024-03-22T02:26:53.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.918+0000 7f8beeffd640 1 -- 172.21.15.28:0/214275144 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0005020 con 0x7f8bf8103800 2024-03-22T02:26:53.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.918+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf819fe00 con 0x7f8bf8103800 2024-03-22T02:26:53.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.918+0000 7f8beeffd640 1 -- 172.21.15.28:0/214275144 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0010720 con 0x7f8bf8103800 2024-03-22T02:26:53.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.918+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf81a02c0 con 0x7f8bf8103800 2024-03-22T02:26:53.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.920+0000 7f8beeffd640 1 -- 172.21.15.28:0/214275144 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8bf0005540 con 0x7f8bf8103800 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.920+0000 7f8c027f1740 1 Processor -- start 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 -- start start 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd5f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db7fcbc8c0 con 0x55db7fd57e20 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55db7fc64c20 con 0x55db7fd5cf90 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8c027f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55db7fcb8870 con 0x55db7fd57430 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8beeffd640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 0x7f8be40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40044/0 (socket says 172.21.15.28:40044) 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39364/0 (socket says 172.21.15.28:39364) 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd8ce640 1 -- 172.21.15.28:0/1824112081 learned_addr learned my addr 172.21.15.28:0/1824112081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd5f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8bfe0cf640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 0x7f8be40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.921+0000 7f8beeffd640 1 -- 172.21.15.28:0/214275144 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8bf00cfe90 con 0x7f8bf8103800 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1824112081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 msgr2=0x55db7fd5f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd5f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1824112081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 msgr2=0x55db7fd5c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1824112081 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7fc89150 con 0x55db7fd57430 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:53.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009900 tx=0x7f8be802cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efa55299df127678 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8bfe0cf640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 0x7f8be40a5460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000c760 tx=0x7f8bf00054b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8be3fff640 1 -- 172.21.15.28:0/1824112081 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8004170 con 0x55db7fd57430 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8be3fff640 1 -- 172.21.15.28:0/1824112081 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be8004310 con 0x55db7fd57430 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.922+0000 7f8be3fff640 1 -- 172.21.15.28:0/1824112081 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8003880 con 0x55db7fd57430 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- 172.21.15.28:0/1824112081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 msgr2=0x55db7fd57850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8be8009900 tx=0x7f8be802cf90 comp rx=0 tx=0).stop 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- 172.21.15.28:0/1824112081 shutdown_connections 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd5f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd5c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1824112081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57430 0x55db7fd57850 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- 172.21.15.28:0/1824112081 >> 172.21.15.28:0/1824112081 conn(0x55db7fd529a0 msgr2=0x55db7fd52da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- 172.21.15.28:0/1824112081 shutdown_connections 2024-03-22T02:26:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- 172.21.15.28:0/1824112081 wait complete. 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 -- start start 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.923+0000 7f8c027f1740 1 Processor -- start 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57430 0x55db7fd618f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd68e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55db7fd5fee0 con 0x55db7fd57e20 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55db7fc78130 con 0x55db7fd57430 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55db7fd60060 con 0x55db7fd5cf90 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 10 client.0 _ll_register_callbacks cb 0x55db7fc77b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40060/0 (socket says 172.21.15.28:40060) 2024-03-22T02:26:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 -- 172.21.15.28:0/2518240056 learned_addr learned my addr 172.21.15.28:0/2518240056 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd68e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57430 0x55db7fd618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 -- 172.21.15.28:0/2518240056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 msgr2=0x55db7fd68e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd68e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 -- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57430 msgr2=0x55db7fd618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57430 0x55db7fd618f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 -- 172.21.15.28:0/2518240056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7fa9a7d0 con 0x55db7fd57e20 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4012d20 tx=0x7f8bf400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339e0916ca561de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8be17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf40120d0 con 0x55db7fd57e20 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8bf4010430 con 0x55db7fd57e20 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.924+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be8004310 con 0x55db7fd57e20 2024-03-22T02:26:53.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8bf4012270 con 0x55db7fd57e20 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 10 client.19287 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55db7f9d03f0 con 0x55db7fd57e20 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be8004310 con 0x55db7fd57e20 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f8bf401c680 con 0x55db7fd57e20 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 10 client.19287 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 10 client.19287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 20 client.19287 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fd6d830 0x55db7fd6fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 4 client.19287 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db7fcc9e00 con 0x55db7fd6d830 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fd6d830 0x55db7fd6fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8be17fa640 10 client.19287 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:53.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.925+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fd6d830 0x55db7fd6fcd0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0545a6ba2a9e7a2 server_cookie=1d5c34daabb3b813 in_seq=0 out_seq=0 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8be17fa640 1 -- 172.21.15.28:0/2518240056 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55db7fcc9e00 con 0x55db7fd6d830 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8be17fa640 10 client.19287 handle_command_reply: tid=0 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 client.19287 shutdown 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 msgr2=0x55db7fd61e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4012d20 tx=0x7f8bf400f790 comp rx=0 tx=0).stop 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 shutdown_connections 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fd6d830 0x55db7fd6fcd0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd68e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57e20 0x55db7fd61e30 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8be17fa640 0 client.19287 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 --2- 172.21.15.28:0/2518240056 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57430 0x55db7fd618f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 >> 172.21.15.28:0/2518240056 conn(0x55db7fd529a0 msgr2=0x55db7fd51c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 shutdown_connections 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 1 -- 172.21.15.28:0/2518240056 wait complete. 2024-03-22T02:26:53.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.928+0000 7f8c027f1740 20 client.19287 trim_cache size 0 max 16384 2024-03-22T02:26:53.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 Processor -- start 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 -- start start 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 0x55db7fd68e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81ab010 con 0x55db7fd57430 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81a88e0 con 0x55db7fd57e20 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8c027f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81a6730 con 0x55db7fd5cf90 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40072/0 (socket says 172.21.15.28:40072) 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1265042387 learned_addr learned my addr 172.21.15.28:0/1265042387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 0x55db7fd68e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1265042387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 msgr2=0x55db7fd518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1265042387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 msgr2=0x55db7fd68e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 0x55db7fd68e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/1265042387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db7f9900f0 con 0x55db7fd57430 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 0x55db7fd68e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.948+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8be802c7b0 tx=0x7f8be802cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e227f37bd1489e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.949+0000 7f8be37fe640 1 -- 172.21.15.28:0/1265042387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be80041f0 con 0x55db7fd57430 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.949+0000 7f8be37fe640 1 -- 172.21.15.28:0/1265042387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be8042070 con 0x55db7fd57430 2024-03-22T02:26:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.949+0000 7f8be37fe640 1 -- 172.21.15.28:0/1265042387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be803ee30 con 0x55db7fd57430 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.949+0000 7f8c027f1740 1 -- 172.21.15.28:0/1265042387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 msgr2=0x55db7fd51370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.949+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f8be802c7b0 tx=0x7f8be802cd70 comp rx=0 tx=0).stop 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- 172.21.15.28:0/1265042387 shutdown_connections 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd5cf90 0x55db7fd518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd57e20 0x55db7fd68e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- 172.21.15.28:0/1265042387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd51370 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- 172.21.15.28:0/1265042387 >> 172.21.15.28:0/1265042387 conn(0x55db7fd529a0 msgr2=0x55db7fd59380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- 172.21.15.28:0/1265042387 shutdown_connections 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- 172.21.15.28:0/1265042387 wait complete. 2024-03-22T02:26:53.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- start start 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 Processor -- start 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57e20 0x55db7fd67100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd68110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81a6da0 con 0x55db7fd57430 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81a8210 con 0x55db7fd5cf90 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf81a7470 con 0x55db7fd57e20 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8c027f1740 10 client.0 _ll_register_callbacks cb 0x7f8bf4012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.950+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40076/0 (socket says 172.21.15.28:40076) 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/231606913 learned_addr learned my addr 172.21.15.28:0/231606913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57e20 0x55db7fd67100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd68110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/231606913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57e20 msgr2=0x55db7fd67100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57e20 0x55db7fd67100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 msgr2=0x55db7fd68110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd68110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 -- 172.21.15.28:0/231606913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db7fd68700 con 0x55db7fd57430 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8bfd0cd640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8be8036040 tx=0x7f8be802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67add09869fe082d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8be17fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be8051030 con 0x55db7fd57430 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8be8047040 con 0x55db7fd57430 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8be8042070 con 0x55db7fd57430 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f8be805f030 con 0x55db7fd57430 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8c027f1740 10 client.19293 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:53.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55db7fdd4260 con 0x55db7fd57430 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.951+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be8042070 con 0x55db7fd57430 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f8be8064030 con 0x55db7fd57430 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 10 client.19293 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 10 client.19293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 20 client.19293 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fdd47b0 0x55db7fdd6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 4 client.19293 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db7fcc4150 con 0x55db7fdd47b0 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fdd47b0 0x55db7fdd6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8be17fa640 10 client.19293 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.952+0000 7f8bfd8ce640 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fdd47b0 0x55db7fdd6ba0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ae00873e91837f0 server_cookie=c5c41126ff77459f in_seq=0 out_seq=0 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8be17fa640 1 -- 172.21.15.28:0/231606913 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55db7fcc4150 con 0x55db7fdd47b0 2024-03-22T02:26:53.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8be17fa640 10 client.19293 handle_command_reply: tid=0 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 client.19293 shutdown 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 msgr2=0x55db7fd66bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f8be8036040 tx=0x7f8be802c5d0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 shutdown_connections 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55db7fdd47b0 0x55db7fdd6ba0 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55db7fd5cf90 0x55db7fd68110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55db7fd57e20 0x55db7fd67100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8be17fa640 0 client.19293 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 --2- 172.21.15.28:0/231606913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55db7fd57430 0x55db7fd66bc0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 >> 172.21.15.28:0/231606913 conn(0x55db7fd529a0 msgr2=0x55db7fd55c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 shutdown_connections 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 1 -- 172.21.15.28:0/231606913 wait complete. 2024-03-22T02:26:53.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.953+0000 7f8c027f1740 20 client.19293 trim_cache size 0 max 16384 2024-03-22T02:26:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:53.962 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:26:53.962 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:26:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 msgr2=0x7f8be40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 0x7f8be40a5460 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8bf000c760 tx=0x7f8bf00054b0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 msgr2=0x7f8bf819f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009fd0 tx=0x7f8bf000ef10 comp rx=0 tx=0).stop 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 shutdown_connections 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8be40a2fa0 0x7f8be40a5460 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8bf8103800 0x7f8bf819f4a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8bf8102e40 0x7f8bf8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 --2- 172.21.15.28:0/214275144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf80ffe70 0x7f8bf8197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 >> 172.21.15.28:0/214275144 conn(0x7f8bf80fb7a0 msgr2=0x7f8bf8106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 shutdown_connections 2024-03-22T02:26:53.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:53.954+0000 7f8bffb32640 1 -- 172.21.15.28:0/214275144 wait complete. 2024-03-22T02:26:53.966 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:26:53.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 -- 172.21.15.28:0/3439094344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718104fc0 msgr2=0x7f77181073b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 --2- 172.21.15.28:0/3439094344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718104fc0 0x7f77181073b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f7708009900 tx=0x7f770802bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 -- 172.21.15.28:0/3439094344 shutdown_connections 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 --2- 172.21.15.28:0/3439094344 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f77181078f0 0x7f7718109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 --2- 172.21.15.28:0/3439094344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718104fc0 0x7f77181073b0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 --2- 172.21.15.28:0/3439094344 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7718102690 0x7f7718104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 -- 172.21.15.28:0/3439094344 >> 172.21.15.28:0/3439094344 conn(0x7f77180fc020 msgr2=0x7f77180fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 -- 172.21.15.28:0/3439094344 shutdown_connections 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.075+0000 7f771fb83640 1 -- 172.21.15.28:0/3439094344 wait complete. 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 Processor -- start 2024-03-22T02:26:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 -- start start 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7718104fc0 0x7f7718198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77181078f0 0x7f771819f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f771810d830 con 0x7f7718102690 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f771810d6b0 con 0x7f77181078f0 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771fb83640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f771810d9b0 con 0x7f7718104fc0 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40082/0 (socket says 172.21.15.28:40082) 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 -- 172.21.15.28:0/3367252603 learned_addr learned my addr 172.21.15.28:0/3367252603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771e120640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77181078f0 0x7f771819f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d11e640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7718104fc0 0x7f7718198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 -- 172.21.15.28:0/3367252603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7718104fc0 msgr2=0x7f7718198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7718104fc0 0x7f7718198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 -- 172.21.15.28:0/3367252603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77181078f0 msgr2=0x7f771819f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77181078f0 0x7f771819f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 -- 172.21.15.28:0/3367252603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f771819fb20 con 0x7f7718102690 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.076+0000 7f771d91f640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f771400ea60 tx=0x7f771400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.077+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771400cde0 con 0x7f7718102690 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.077+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77140035f0 con 0x7f7718102690 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.077+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f771819fe10 con 0x7f7718102690 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.077+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7714010670 con 0x7f7718102690 2024-03-22T02:26:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.077+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181a0280 con 0x7f7718102690 2024-03-22T02:26:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.078+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77180fdb10 con 0x7f7718102690 2024-03-22T02:26:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.079+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f77140037a0 con 0x7f7718102690 2024-03-22T02:26:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.079+0000 7f770effd640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 0x7f77040a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.079+0000 7f771d11e640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 0x7f77040a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.079+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f77140cfa30 con 0x7f7718102690 2024-03-22T02:26:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.080+0000 7f771d11e640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 0x7f77040a5460 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7708000c00 tx=0x7f7708005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.084+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f771409c450 con 0x7f7718102690 2024-03-22T02:26:54.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 -- 172.21.15.28:0/521716682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8105020 msgr2=0x7ff7f8107410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/521716682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8105020 0x7ff7f8107410 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009860 tx=0x7ff7e002bb70 comp rx=0 tx=0).stop 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 -- 172.21.15.28:0/521716682 shutdown_connections 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/521716682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f8109d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/521716682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f8105020 0x7ff7f8107410 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/521716682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f81026f0 0x7ff7f8104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 -- 172.21.15.28:0/521716682 >> 172.21.15.28:0/521716682 conn(0x7ff7f80fc020 msgr2=0x7ff7f80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 -- 172.21.15.28:0/521716682 shutdown_connections 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.230+0000 7ff7fd88f640 1 -- 172.21.15.28:0/521716682 wait complete. 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 Processor -- start 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 -- start start 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 0x7ff7f8198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f819f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810d810 con 0x7ff7f81026f0 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810d690 con 0x7ff7f8105020 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7fd88f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f810d990 con 0x7ff7f8107950 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40104/0 (socket says 172.21.15.28:40104) 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f6ffd640 1 -- 172.21.15.28:0/1759019063 learned_addr learned my addr 172.21.15.28:0/1759019063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f819f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7effff640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 0x7ff7f8198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f6ffd640 1 -- 172.21.15.28:0/1759019063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 msgr2=0x7ff7f819f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.231+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f819f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7f6ffd640 1 -- 172.21.15.28:0/1759019063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 msgr2=0x7ff7f8198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 0x7ff7f8198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7f6ffd640 1 -- 172.21.15.28:0/1759019063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f819fa70 con 0x7ff7f81026f0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7effff640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 0x7ff7f8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800ea60 tx=0x7ff7e800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7fc88d640 1 -- 172.21.15.28:0/1759019063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e800cde0 con 0x7ff7f81026f0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f819fd60 con 0x7ff7f81026f0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7fc88d640 1 -- 172.21.15.28:0/1759019063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e80035f0 con 0x7ff7f81026f0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7fc88d640 1 -- 172.21.15.28:0/1759019063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8010670 con 0x7ff7f81026f0 2024-03-22T02:26:54.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.232+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a0210 con 0x7ff7f81026f0 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 Processor -- start 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 -- start start 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7fc88d640 1 -- 172.21.15.28:0/1759019063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff7e80037a0 con 0x7ff7f81026f0 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56060388c040 con 0x56060392d040 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5606038349e0 con 0x560603927ed0 2024-03-22T02:26:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff800472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560603887ff0 con 0x5606039274e0 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7f6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40118/0 (socket says 172.21.15.28:40118) 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7effff640 1 -- 172.21.15.28:0/2122267933 learned_addr learned my addr 172.21.15.28:0/2122267933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39444/0 (socket says 172.21.15.28:39444) 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.235+0000 7ff7f6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52482/0 (socket says 172.21.15.28:52482) 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 -- 172.21.15.28:0/2122267933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 msgr2=0x56060392c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 -- 172.21.15.28:0/2122267933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 msgr2=0x56060392f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 -- 172.21.15.28:0/2122267933 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606035a0210 con 0x5606039274e0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7effff640 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7fc88d640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 0x7ff7e40a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 0x7ff7e40a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000e820 tx=0x7ff7f00095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b9a465f6c9c9aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7fc88d640 1 -- 172.21.15.28:0/1759019063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ff7e80cfab0 con 0x7ff7f81026f0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7edffb640 1 -- 172.21.15.28:0/2122267933 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f001d070 con 0x5606039274e0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7edffb640 1 -- 172.21.15.28:0/2122267933 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f0003e50 con 0x5606039274e0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.236+0000 7ff7edffb640 1 -- 172.21.15.28:0/2122267933 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0018480 con 0x5606039274e0 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- 172.21.15.28:0/2122267933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 msgr2=0x560603927900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000e820 tx=0x7ff7f00095c0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff7f6ffd640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 0x7ff7e40a5480 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80cd510 tx=0x7ff7e80d3590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- 172.21.15.28:0/2122267933 shutdown_connections 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56060392d040 0x56060392f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x56060392c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 --2- 172.21.15.28:0/2122267933 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606039274e0 0x560603927900 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- 172.21.15.28:0/2122267933 >> 172.21.15.28:0/2122267933 conn(0x560603922a50 msgr2=0x560603922e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- 172.21.15.28:0/2122267933 shutdown_connections 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- 172.21.15.28:0/2122267933 wait complete. 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 -- start start 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.237+0000 7ff800472740 1 Processor -- start 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x5606039319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603938a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56060392fff0 con 0x5606039274e0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560603847db0 con 0x560603927ed0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560603930170 con 0x56060392d040 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff800472740 10 client.0 _ll_register_callbacks cb 0x56060359fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40120/0 (socket says 172.21.15.28:40120) 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3694053439 learned_addr learned my addr 172.21.15.28:0/3694053439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7effff640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x5606039319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603938a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3694053439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 msgr2=0x560603938a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603938a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 msgr2=0x5606039319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x5606039319f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3694053439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56060366c620 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7effff640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x5606039319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005b20 tx=0x7ff7f0005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cdaa59c1a1b032 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.238+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f002e060 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ff7f0003970 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7f0003e50 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff7f0035050 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 10 client.19311 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x56060393bda0 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7f0003e50 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7ff7f000b040 con 0x5606039274e0 2024-03-22T02:26:54.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 10 client.19311 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 10 client.19311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 20 client.19311 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x56060393f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 4 client.19311 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560603899580 con 0x56060393d1a0 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.239+0000 7ff7effff640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x56060393f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.240+0000 7ff7d6ffd640 10 client.19311 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.240+0000 7ff7effff640 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x56060393f590 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e0616ba2b651665 server_cookie=7a9ef489c91713ea in_seq=0 out_seq=0 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3694053439 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560603899580 con 0x56060393d1a0 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff7d6ffd640 10 client.19311 handle_command_reply: tid=0 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 client.19311 shutdown 2024-03-22T02:26:54.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 msgr2=0x5606039314b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005b20 tx=0x7ff7f0005a50 comp rx=0 tx=0).stop 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 shutdown_connections 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x56060393f590 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603938a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff7d6ffd640 0 client.19311 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x5606039319f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 --2- 172.21.15.28:0/3694053439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x5606039314b0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 >> 172.21.15.28:0/3694053439 conn(0x560603922a50 msgr2=0x560603921c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 shutdown_connections 2024-03-22T02:26:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 1 -- 172.21.15.28:0/3694053439 wait complete. 2024-03-22T02:26:54.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.242+0000 7ff800472740 20 client.19311 trim_cache size 0 max 16384 2024-03-22T02:26:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 Processor -- start 2024-03-22T02:26:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 -- start start 2024-03-22T02:26:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5606039274e0 0x560603931950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603921ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81ab080 con 0x560603927ed0 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81a8950 con 0x5606039274e0 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff800472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81a67a0 con 0x56060392d040 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff7effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff7effff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40136/0 (socket says 172.21.15.28:40136) 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5606039274e0 0x560603931950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff7effff640 1 -- 172.21.15.28:0/3681752394 learned_addr learned my addr 172.21.15.28:0/3681752394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.257+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603921ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 -- 172.21.15.28:0/3681752394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 msgr2=0x560603921ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603921ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 -- 172.21.15.28:0/3681752394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5606039274e0 msgr2=0x560603931950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5606039274e0 0x560603931950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 -- 172.21.15.28:0/3681752394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606039302a0 con 0x560603927ed0 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7effff640 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009830 tx=0x7ff7e002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9b5d77efc0d014 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7ed7fa640 1 -- 172.21.15.28:0/3681752394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e0039040 con 0x560603927ed0 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7ed7fa640 1 -- 172.21.15.28:0/3681752394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e0051070 con 0x560603927ed0 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.258+0000 7ff7ed7fa640 1 -- 172.21.15.28:0/3681752394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e004c360 con 0x560603927ed0 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- 172.21.15.28:0/3681752394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 msgr2=0x5606039389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0009830 tx=0x7ff7e002e040 comp rx=0 tx=0).stop 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- 172.21.15.28:0/3681752394 shutdown_connections 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603921ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560603927ed0 0x5606039389c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- 172.21.15.28:0/3681752394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5606039274e0 0x560603931950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- 172.21.15.28:0/3681752394 >> 172.21.15.28:0/3681752394 conn(0x560603922a50 msgr2=0x5606039213c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- 172.21.15.28:0/3681752394 shutdown_connections 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- 172.21.15.28:0/3681752394 wait complete. 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- start start 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 Processor -- start 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x560603936970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603937980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.259+0000 7ff800472740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81a6e10 con 0x5606039274e0 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81a8280 con 0x560603927ed0 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f81a74e0 con 0x56060392d040 2024-03-22T02:26:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 10 client.0 _ll_register_callbacks cb 0x7ff7f001b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40140/0 (socket says 172.21.15.28:40140) 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3433925956 learned_addr learned my addr 172.21.15.28:0/3433925956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7effff640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x560603936970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603937980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3433925956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 msgr2=0x560603937980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603937980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 msgr2=0x560603936970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x560603936970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 -- 172.21.15.28:0/3433925956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560603937f70 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7effff640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x560603936970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7f77fe640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00294b0 tx=0x7ff7f00021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd83aa24a823f3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0041060 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ff7f0029b50 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7e0051070 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7ff7f0048050 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 10 client.19317 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.260+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5606039a3640 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e0051070 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7ff7f000b050 con 0x5606039274e0 2024-03-22T02:26:54.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 10 client.19317 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 10 client.19317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 20 client.19317 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x5606039a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 4 client.19317 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5606038938d0 con 0x56060393d1a0 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff7effff640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x5606039a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff7d6ffd640 10 client.19317 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:54.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.261+0000 7ff7effff640 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x5606039a5f60 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8472f9789ee719bc server_cookie=7c131691a214cf5 in_seq=0 out_seq=0 2024-03-22T02:26:54.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.330+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f771802d3b0 con 0x7f7718102690 2024-03-22T02:26:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.331+0000 7f770effd640 1 -- 172.21.15.28:0/3367252603 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11863 (secure 0 0 0) 0x7f771802d3b0 con 0x7f7718102690 2024-03-22T02:26:54.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:26:54.332 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:26:28.349997+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29075":{"gid":29075,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/366330553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":366330553},{"type":"v1","addr":"172.21.15.83:6837","nonce":366330553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:26:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 38 2024-03-22T02:26:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 msgr2=0x7f77040a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 0x7f77040a5460 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7708000c00 tx=0x7f7708005fd0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 msgr2=0x7f7718197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f771400ea60 tx=0x7f771400ef20 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 shutdown_connections 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f77040a2fa0 0x7f77040a5460 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f77181078f0 0x7f771819f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7718104fc0 0x7f7718198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 --2- 172.21.15.28:0/3367252603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7718102690 0x7f7718197f00 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 >> 172.21.15.28:0/3367252603 conn(0x7f77180fc020 msgr2=0x7f77181030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 shutdown_connections 2024-03-22T02:26:54.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.333+0000 7f771fb83640 1 -- 172.21.15.28:0/3367252603 wait complete. 2024-03-22T02:26:54.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:26:54.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1382753 11.19 MB/sec execute 576 sec latency 63.888 ms 2024-03-22T02:26:54.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.600+0000 7f75055d0640 1 -- 172.21.15.28:0/2287623318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 msgr2=0x7f750010afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.600+0000 7f75055d0640 1 --2- 172.21.15.28:0/2287623318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f750010afc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009ca0 tx=0x7f74f802c120 comp rx=0 tx=0).stop 2024-03-22T02:26:54.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.606+0000 7f75055d0640 1 -- 172.21.15.28:0/2287623318 shutdown_connections 2024-03-22T02:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.606+0000 7f75055d0640 1 --2- 172.21.15.28:0/2287623318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010deb0 0x7f750010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.606+0000 7f75055d0640 1 --2- 172.21.15.28:0/2287623318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010b500 0x7f750010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.606+0000 7f75055d0640 1 --2- 172.21.15.28:0/2287623318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f750010afc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.606+0000 7f75055d0640 1 -- 172.21.15.28:0/2287623318 >> 172.21.15.28:0/2287623318 conn(0x7f750006e9f0 msgr2=0x7f7500070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.612+0000 7f75055d0640 1 -- 172.21.15.28:0/2287623318 shutdown_connections 2024-03-22T02:26:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.615+0000 7f75055d0640 1 -- 172.21.15.28:0/2287623318 wait complete. 2024-03-22T02:26:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.615+0000 7f75055d0640 1 Processor -- start 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 -- start start 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f75001a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010b500 0x7f75001a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 0x7f75001ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f750011c600 con 0x7f750010deb0 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f750011c480 con 0x7f7500108b90 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.618+0000 7f75055d0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f750011c780 con 0x7f750010b500 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.619+0000 7f74feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f75001a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.620+0000 7f74feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f75001a6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52518/0 (socket says 172.21.15.28:52518) 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.620+0000 7f74feffd640 1 -- 172.21.15.28:0/1459830281 learned_addr learned my addr 172.21.15.28:0/1459830281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.620+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 0x7f75001ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.620+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010b500 0x7f75001a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1459830281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010b500 msgr2=0x7f75001a72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010b500 0x7f75001a72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1459830281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 msgr2=0x7f75001a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f75001a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1459830281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75001103f0 con 0x7f750010deb0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.621+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 0x7f75001ae2b0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e940 tx=0x7f74f000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.624+0000 7f74e7fff640 1 -- 172.21.15.28:0/1459830281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f000ccc0 con 0x7f750010deb0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.624+0000 7f74e7fff640 1 -- 172.21.15.28:0/1459830281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f0005020 con 0x7f750010deb0 2024-03-22T02:26:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.625+0000 7f74e7fff640 1 -- 172.21.15.28:0/1459830281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f0004a00 con 0x7f750010deb0 2024-03-22T02:26:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.626+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75001106e0 con 0x7f750010deb0 2024-03-22T02:26:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.626+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7500110bb0 con 0x7f750010deb0 2024-03-22T02:26:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.633+0000 7f74e7fff640 1 -- 172.21.15.28:0/1459830281 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f74f0004ba0 con 0x7f750010deb0 2024-03-22T02:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.634+0000 7f74e7fff640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 0x7f74e80a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.635+0000 7f74e7fff640 1 -- 172.21.15.28:0/1459830281 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f74f00cfc40 con 0x7f750010deb0 2024-03-22T02:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.635+0000 7f75081b8740 1 Processor -- start 2024-03-22T02:26:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.636+0000 7f74feffd640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 0x7f74e80a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 -- start start 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0bdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0c0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642bb01d8d0 con 0x5642bb0b8820 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5642bafc5e00 con 0x5642bb0b9210 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.637+0000 7f75081b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642bb019880 con 0x5642bb0be380 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.639+0000 7f74ff7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.639+0000 7f74fe7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0bdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.639+0000 7f74feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.639+0000 7f74feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40156/0 (socket says 172.21.15.28:40156) 2024-03-22T02:26:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.639+0000 7f74feffd640 1 -- 172.21.15.28:0/3467716672 learned_addr learned my addr 172.21.15.28:0/3467716672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.640+0000 7f74feffd640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 0x7f74e80a5410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005e30 tx=0x7f74f8009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.642+0000 7f74feffd640 1 -- 172.21.15.28:0/3467716672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 msgr2=0x5642bb0c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.642+0000 7f74feffd640 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.642+0000 7f74feffd640 1 -- 172.21.15.28:0/3467716672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 msgr2=0x5642bb0bdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.642+0000 7f74feffd640 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0bdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.642+0000 7f74feffd640 1 -- 172.21.15.28:0/3467716672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642bad313f0 con 0x5642bb0b8820 2024-03-22T02:26:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.647+0000 7f74feffd640 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f74f8037040 tx=0x7f74f804a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464b9c7c82f04031 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.655+0000 7f74e57fa640 1 -- 172.21.15.28:0/3467716672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f8039040 con 0x5642bb0b8820 2024-03-22T02:26:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.655+0000 7f74e57fa640 1 -- 172.21.15.28:0/3467716672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f8034b20 con 0x5642bb0b8820 2024-03-22T02:26:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.655+0000 7f74e57fa640 1 -- 172.21.15.28:0/3467716672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f8043960 con 0x5642bb0b8820 2024-03-22T02:26:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.658+0000 7f75081b8740 1 -- 172.21.15.28:0/3467716672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 msgr2=0x5642bb0b8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.658+0000 7f75081b8740 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f74f8037040 tx=0x7f74f804a000 comp rx=0 tx=0).stop 2024-03-22T02:26:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.660+0000 7f75081b8740 1 -- 172.21.15.28:0/3467716672 shutdown_connections 2024-03-22T02:26:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.660+0000 7f75081b8740 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.661+0000 7f75081b8740 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0bdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.661+0000 7f75081b8740 1 --2- 172.21.15.28:0/3467716672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0b8c40 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.661+0000 7f75081b8740 1 -- 172.21.15.28:0/3467716672 >> 172.21.15.28:0/3467716672 conn(0x5642bb0b3d90 msgr2=0x5642bb0b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.664+0000 7f75081b8740 1 -- 172.21.15.28:0/3467716672 shutdown_connections 2024-03-22T02:26:54.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.664+0000 7f75081b8740 1 -- 172.21.15.28:0/3467716672 wait complete. 2024-03-22T02:26:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.665+0000 7f75081b8740 1 -- start start 2024-03-22T02:26:54.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.665+0000 7f75081b8740 1 Processor -- start 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b8820 0x5642bb0c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 0x5642bb0c3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5642bb0c12d0 con 0x5642bb0be380 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5642bafd9300 con 0x5642bb0b8820 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5642bb0c1450 con 0x5642bb0b9210 2024-03-22T02:26:54.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.668+0000 7f75081b8740 10 client.0 _ll_register_callbacks cb 0x5642bb014e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.669+0000 7f74feffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b8820 0x5642bb0c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.669+0000 7f74ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.669+0000 7f74ff7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0ca2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40170/0 (socket says 172.21.15.28:40170) 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.669+0000 7f74ff7fe640 1 -- 172.21.15.28:0/2063371853 learned_addr learned my addr 172.21.15.28:0/2063371853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.669+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 0x5642bb0c3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.670+0000 7f74fe7fc640 1 -- 172.21.15.28:0/2063371853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b8820 msgr2=0x5642bb0c2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.670+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b8820 0x5642bb0c2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.670+0000 7f74fe7fc640 1 -- 172.21.15.28:0/2063371853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 msgr2=0x5642bb0ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.670+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.671+0000 7f74fe7fc640 1 -- 172.21.15.28:0/2063371853 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642badfb7d0 con 0x5642bb0b9210 2024-03-22T02:26:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.671+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 0x5642bb0c3290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5642bb0b9610 tx=0x7f74f40039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8645eb8617bdfc02 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.672+0000 7f74e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.675+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f4033860 con 0x5642bb0b9210 2024-03-22T02:26:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.675+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f8034b20 con 0x5642bb0b9210 2024-03-22T02:26:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.675+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f74f4033a00 con 0x5642bb0b9210 2024-03-22T02:26:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.678+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f74f4033c00 con 0x5642bb0b9210 2024-03-22T02:26:54.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.679+0000 7f75081b8740 10 client.29146 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.679+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x5642bb0cd760 con 0x5642bb0b9210 2024-03-22T02:26:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.683+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f8034b20 con 0x5642bb0b9210 2024-03-22T02:26:54.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.683+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f74f403b070 con 0x5642bb0b9210 2024-03-22T02:26:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 10 client.29146 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 10 client.29146 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:26:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 20 client.29146 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb0cee60 0x5642bb0d1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 4 client.29146 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:54.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.687+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5642bb02ae10 con 0x5642bb0cee60 2024-03-22T02:26:54.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.691+0000 7f74feffd640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb0cee60 0x5642bb0d1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.700+0000 7f74feffd640 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb0cee60 0x5642bb0d1290 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=66c7b99ce9b8b00e server_cookie=8ac479e0272b34a2 in_seq=0 out_seq=0 2024-03-22T02:26:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.702+0000 7f74e2ff5640 10 client.29146 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.702+0000 7f74e2ff5640 1 -- 172.21.15.28:0/2063371853 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5642bb02ae10 con 0x5642bb0cee60 2024-03-22T02:26:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.702+0000 7f74e2ff5640 10 client.29146 handle_command_reply: tid=0 2024-03-22T02:26:54.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.702+0000 7f75081b8740 1 client.29146 shutdown 2024-03-22T02:26:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.709+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 msgr2=0x5642bb0c3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.709+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 0x5642bb0c3290 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5642bb0b9610 tx=0x7f74f40039c0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 shutdown_connections 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b9210 0x5642bb0c3290 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb0cee60 0x5642bb0d1290 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 --2- 172.21.15.28:0/2063371853 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b8820 0x5642bb0c2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 >> 172.21.15.28:0/2063371853 conn(0x5642bb0b3d90 msgr2=0x5642bb0b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.712+0000 7f74e2ff5640 0 client.29146 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:54.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.716+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 shutdown_connections 2024-03-22T02:26:54.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.716+0000 7f75081b8740 1 -- 172.21.15.28:0/2063371853 wait complete. 2024-03-22T02:26:54.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.716+0000 7f75081b8740 20 client.29146 trim_cache size 0 max 16384 2024-03-22T02:26:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 Processor -- start 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 -- start start 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75001b3890 con 0x5642bb0b8820 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75001b1160 con 0x5642bb0b9210 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f75081b8740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7500118950 con 0x5642bb0be380 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40176/0 (socket says 172.21.15.28:40176) 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 -- 172.21.15.28:0/3608277426 learned_addr learned my addr 172.21.15.28:0/3608277426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 -- 172.21.15.28:0/3608277426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 msgr2=0x5642bb0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 -- 172.21.15.28:0/3608277426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 msgr2=0x5642bb0ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74feffd640 1 -- 172.21.15.28:0/3608277426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642bb0c14b0 con 0x5642bb0b8820 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.741+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.742+0000 7f74feffd640 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f74f8037040 tx=0x7f74f8043400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4259093e3d78b093 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.742+0000 7f74e4ff9640 1 -- 172.21.15.28:0/3608277426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f8039070 con 0x5642bb0b8820 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.742+0000 7f74e4ff9640 1 -- 172.21.15.28:0/3608277426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74f8034b20 con 0x5642bb0b8820 2024-03-22T02:26:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.742+0000 7f74e4ff9640 1 -- 172.21.15.28:0/3608277426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f8053040 con 0x5642bb0b8820 2024-03-22T02:26:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- 172.21.15.28:0/3608277426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 msgr2=0x5642bb0d0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f74f8037040 tx=0x7f74f8043400 comp rx=0 tx=0).stop 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- 172.21.15.28:0/3608277426 shutdown_connections 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0be380 0x5642bb0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 --2- 172.21.15.28:0/3608277426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0b8820 0x5642bb0d0e70 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- 172.21.15.28:0/3608277426 >> 172.21.15.28:0/3608277426 conn(0x5642bb0b3d90 msgr2=0x5642bb0c1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- 172.21.15.28:0/3608277426 shutdown_connections 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- 172.21.15.28:0/3608277426 wait complete. 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.743+0000 7f75081b8740 1 -- start start 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f75081b8740 1 Processor -- start 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f75081b8740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 0x5642bb0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f75081b8740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0c8220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f75081b8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0c9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f74feffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 0x5642bb0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f74feffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 0x5642bb0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39534/0 (socket says 172.21.15.28:39534) 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f74feffd640 1 -- 172.21.15.28:0/1519807103 learned_addr learned my addr 172.21.15.28:0/1519807103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75001af620 con 0x5642bb0be380 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f74fe7fc640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.744+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f75001b0a90 con 0x5642bb0b9210 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f75001afcf0 con 0x5642bb0b8820 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f75081b8740 10 client.0 _ll_register_callbacks cb 0x7f74f4004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 msgr2=0x5642bb0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 0x5642bb0c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1519807103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 msgr2=0x5642bb0c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642bb0b4800 con 0x5642bb0be380 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74f8034b20 con 0x5642bb0be380 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74ff7fe640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0c9230 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f74f0004770 tx=0x7f74f0004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cdc08540fde1598 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74e2ff5640 10 client.19335 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74e2ff5640 1 -- 172.21.15.28:0/1519807103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74f00a4d60 con 0x5642bb0be380 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.745+0000 7f74e2ff5640 1 -- 172.21.15.28:0/1519807103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f74f009b350 con 0x5642bb0be380 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f74e2ff5640 1 -- 172.21.15.28:0/1519807103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f74f009b570 con 0x5642bb0be380 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 10 client.19335 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=38}) -- 0x5642bb135160 con 0x5642bb0be380 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f74e2ff5640 1 -- 172.21.15.28:0/1519807103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7f74f001f070 con 0x5642bb0be380 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 10 client.19335 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 10 client.19335 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 20 client.19335 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb135a50 0x5642bb137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 4 client.19335 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5642bb025160 con 0x5642bb135a50 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.746+0000 7f74feffd640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb135a50 0x5642bb137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f74feffd640 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb135a50 0x5642bb137e70 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68f4f82cb1d03644 server_cookie=7ceb2788f13650fb in_seq=0 out_seq=0 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f74e2ff5640 10 client.19335 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f74e2ff5640 1 -- 172.21.15.28:0/1519807103 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5642bb025160 con 0x5642bb135a50 2024-03-22T02:26:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f74e2ff5640 10 client.19335 handle_command_reply: tid=0 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 client.19335 shutdown 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 msgr2=0x5642bb0c9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0c9230 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f74f0004770 tx=0x7f74f0004700 comp rx=0 tx=0).stop 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 shutdown_connections 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5642bb0be380 0x5642bb0c9230 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x5642bb135a50 0x5642bb137e70 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5642bb0b9210 0x5642bb0c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.747+0000 7f74e2ff5640 0 client.19335 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.748+0000 7f75081b8740 1 --2- 172.21.15.28:0/1519807103 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5642bb0b8820 0x5642bb0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.748+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 >> 172.21.15.28:0/1519807103 conn(0x5642bb0b3d90 msgr2=0x5642bb0b6ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.748+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 shutdown_connections 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.748+0000 7f75081b8740 1 -- 172.21.15.28:0/1519807103 wait complete. 2024-03-22T02:26:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.748+0000 7f75081b8740 20 client.19335 trim_cache size 0 max 16384 2024-03-22T02:26:54.756 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:54.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:54 smithi120 ceph-mon[36803]: pgmap v760: 129 pgs: 129 active+clean; 53 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 333 op/s 2024-03-22T02:26:54.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:54 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3367252603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 msgr2=0x7f74e80a5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 0x7f74e80a5410 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f74f8005e30 tx=0x7f74f8009800 comp rx=0 tx=0).stop 2024-03-22T02:26:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 msgr2=0x7f75001ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 0x7f75001ae2b0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f74f000e940 tx=0x7f74f000ee00 comp rx=0 tx=0).stop 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 shutdown_connections 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f74e80a2f50 0x7f74e80a5410 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750010deb0 0x7f75001ae2b0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f750010b500 0x7f75001a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 --2- 172.21.15.28:0/1459830281 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7500108b90 0x7f75001a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.750+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 >> 172.21.15.28:0/1459830281 conn(0x7f750006e9f0 msgr2=0x7f750010b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.751+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 shutdown_connections 2024-03-22T02:26:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.751+0000 7f75055d0640 1 -- 172.21.15.28:0/1459830281 wait complete. 2024-03-22T02:26:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:54 smithi083 ceph-mon[36340]: pgmap v760: 129 pgs: 129 active+clean; 53 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 333 op/s 2024-03-22T02:26:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:54 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3367252603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:54.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:54 smithi028 ceph-mon[31916]: pgmap v760: 129 pgs: 129 active+clean; 53 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.5 MiB/s wr, 333 op/s 2024-03-22T02:26:54.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:54 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3367252603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:26:54.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:26:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.800+0000 7ff7d6ffd640 1 -- 172.21.15.28:0/3433925956 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5606038938d0 con 0x56060393d1a0 2024-03-22T02:26:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.800+0000 7ff7d6ffd640 10 client.19317 handle_command_reply: tid=0 2024-03-22T02:26:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.800+0000 7ff800472740 1 client.19317 shutdown 2024-03-22T02:26:54.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 msgr2=0x560603936430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7ff7f00294b0 tx=0x7ff7f00021d0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 shutdown_connections 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56060393d1a0 0x5606039a5f60 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56060392d040 0x560603937980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560603927ed0 0x560603936970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 --2- 172.21.15.28:0/3433925956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5606039274e0 0x560603936430 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 >> 172.21.15.28:0/3433925956 conn(0x560603922a50 msgr2=0x560603921c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff7d6ffd640 0 client.19317 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 shutdown_connections 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 1 -- 172.21.15.28:0/3433925956 wait complete. 2024-03-22T02:26:54.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.801+0000 7ff800472740 20 client.19317 trim_cache size 0 max 16384 2024-03-22T02:26:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:26:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:26:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:26:54.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 msgr2=0x7ff7e40a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 0x7ff7e40a5480 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80cd510 tx=0x7ff7e80d3590 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 msgr2=0x7ff7f8197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800ea60 tx=0x7ff7e800ef20 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 shutdown_connections 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff7e40a2fc0 0x7ff7e40a5480 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7f8107950 0x7ff7f819f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff7f8105020 0x7ff7f8198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 --2- 172.21.15.28:0/1759019063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f81026f0 0x7ff7f8197e50 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 >> 172.21.15.28:0/1759019063 conn(0x7ff7f80fc020 msgr2=0x7ff7f8103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 shutdown_connections 2024-03-22T02:26:54.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:26:54.802+0000 7ff7fd88f640 1 -- 172.21.15.28:0/1759019063 wait complete. 2024-03-22T02:26:54.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 -- 172.21.15.28:0/2544082913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 msgr2=0x7fb9cc1073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 --2- 172.21.15.28:0/2544082913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc1073f0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009900 tx=0x7fb9c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 -- 172.21.15.28:0/2544082913 shutdown_connections 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 --2- 172.21.15.28:0/2544082913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc107930 0x7fb9cc109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 --2- 172.21.15.28:0/2544082913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc1073f0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 --2- 172.21.15.28:0/2544082913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc104ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 -- 172.21.15.28:0/2544082913 >> 172.21.15.28:0/2544082913 conn(0x7fb9cc0fc020 msgr2=0x7fb9cc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 -- 172.21.15.28:0/2544082913 shutdown_connections 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.883+0000 7fb9d201c640 1 -- 172.21.15.28:0/2544082913 wait complete. 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 Processor -- start 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 -- start start 2024-03-22T02:26:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 0x7fb9cc100bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10d820 con 0x7fb9cc105000 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10d6a0 con 0x7fb9cc107930 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.884+0000 7fb9d201c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc10d9a0 con 0x7fb9cc1026d0 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40204/0 (socket says 172.21.15.28:40204) 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 -- 172.21.15.28:0/1791353242 learned_addr learned my addr 172.21.15.28:0/1791353242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 0x7fb9cc100bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 -- 172.21.15.28:0/1791353242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 msgr2=0x7fb9cc102040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc102040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 -- 172.21.15.28:0/1791353242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 msgr2=0x7fb9cc100bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 0x7fb9cc100bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 -- 172.21.15.28:0/1791353242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9cc101450 con 0x7fb9cc105000 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 0x7fb9cc100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0000c00 tx=0x7fb9c002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.885+0000 7fb9c8ff9640 1 -- 172.21.15.28:0/1791353242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0041070 con 0x7fb9cc105000 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.886+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9cc1a1ff0 con 0x7fb9cc105000 2024-03-22T02:26:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.886+0000 7fb9c8ff9640 1 -- 172.21.15.28:0/1791353242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c0004450 con 0x7fb9cc105000 2024-03-22T02:26:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.886+0000 7fb9c8ff9640 1 -- 172.21.15.28:0/1791353242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c0034440 con 0x7fb9cc105000 2024-03-22T02:26:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.886+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9cc1a2460 con 0x7fb9cc105000 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 Processor -- start 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 -- start start 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9c8ff9640 1 -- 172.21.15.28:0/1791353242 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb9c0044070 con 0x7fb9cc105000 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4ca20 0x561aa5b51520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 0x561aa5b54020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561aa5aa88d0 con 0x561aa5b4c030 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561aa5a50e00 con 0x561aa5b51b90 2024-03-22T02:26:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561aa5aa4880 con 0x561aa5b4ca20 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 0x561aa5b54020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40220/0 (socket says 172.21.15.28:40220) 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3809091089 learned_addr learned my addr 172.21.15.28:0/3809091089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4ca20 0x561aa5b51520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3809091089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4ca20 msgr2=0x561aa5b51520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4ca20 0x561aa5b51520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3809091089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 msgr2=0x561aa5b54020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 0x561aa5b54020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3809091089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa57bc3f0 con 0x561aa5b4c030 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9caffd640 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 0x561aa5b54020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.889+0000 7fb9c8ff9640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 0x7fb9b40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40117e0 tx=0x7fb9c4011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae551c91f262b2d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 0x7fb9b40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9c8ff9640 1 -- 172.21.15.28:0/1791353242 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fb9c00f1230 con 0x7fb9cc105000 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9b1ffb640 1 -- 172.21.15.28:0/3809091089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4011310 con 0x561aa5b4c030 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9b1ffb640 1 -- 172.21.15.28:0/3809091089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c4003e50 con 0x561aa5b4c030 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.890+0000 7fb9b1ffb640 1 -- 172.21.15.28:0/3809091089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4004620 con 0x561aa5b4c030 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 0x7fb9b40a53e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x561aa5b4ce20 tx=0x7fb9bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3809091089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 msgr2=0x561aa5b4c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fb9c40117e0 tx=0x7fb9c4011ca0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3809091089 shutdown_connections 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b51b90 0x561aa5b54020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4ca20 0x561aa5b51520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3809091089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b4c450 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3809091089 >> 172.21.15.28:0/3809091089 conn(0x561aa5b475a0 msgr2=0x561aa5b479a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3809091089 shutdown_connections 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3809091089 wait complete. 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.891+0000 7fb9d4c07740 1 -- start start 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 Processor -- start 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b56af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561aa5b54b40 con 0x561aa5b4c030 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561aa5a64300 con 0x561aa5b4ca20 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561aa5b54cc0 con 0x561aa5b51b90 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9d4c07740 10 client.0 _ll_register_callbacks cb 0x561aa5a9fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40236/0 (socket says 172.21.15.28:40236) 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/3284001600 learned_addr learned my addr 172.21.15.28:0/3284001600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9caffd640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b56af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3284001600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 msgr2=0x561aa5b56af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b56af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3284001600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 msgr2=0x561aa5b565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.892+0000 7fb9cbfff640 1 -- 172.21.15.28:0/3284001600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa58867d0 con 0x561aa5b51b90 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5db30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400f790 tx=0x7fb9c40113c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=388ef8d182aab09 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c4004a40 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c4003e50 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fb9c4004be0 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb9c4032050 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9d4c07740 10 client.29164 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561aa5b60ea0 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c4003e50 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.893+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb9c4018020 con 0x561aa5b51b90 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 10 client.29164 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 10 client.29164 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 20 client.29164 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5b622b0 0x561aa5b646c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 4 client.29164 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aa5ab5e10 con 0x561aa5b622b0 2024-03-22T02:26:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5b622b0 0x561aa5b646c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9aaffd640 10 client.29164 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:54.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.894+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5b622b0 0x561aa5b646c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b04c1f1069e92e4c server_cookie=5970f1552625e58f in_seq=0 out_seq=0 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.895+0000 7fb9aaffd640 1 -- 172.21.15.28:0/3284001600 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561aa5ab5e10 con 0x561aa5b622b0 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.895+0000 7fb9aaffd640 10 client.29164 handle_command_reply: tid=0 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.895+0000 7fb9d4c07740 1 client.29164 shutdown 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 msgr2=0x561aa5b5db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5db30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb9c400f790 tx=0x7fb9c40113c0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 shutdown_connections 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5b622b0 0x561aa5b646c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5db30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b56af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/3284001600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9aaffd640 0 client.29164 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 >> 172.21.15.28:0/3284001600 conn(0x561aa5b475a0 msgr2=0x561aa5b3dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 shutdown_connections 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 1 -- 172.21.15.28:0/3284001600 wait complete. 2024-03-22T02:26:54.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.896+0000 7fb9d4c07740 20 client.29164 trim_cache size 0 max 16384 2024-03-22T02:26:54.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.915+0000 7fb9d4c07740 1 Processor -- start 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 -- start start 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b51b90 0x561aa5b56af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1ab130 con 0x561aa5b51b90 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1a8a00 con 0x561aa5b4ca20 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1a6850 con 0x561aa5b4c030 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b51b90 0x561aa5b56af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39592/0 (socket says 172.21.15.28:39592) 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/1600618546 learned_addr learned my addr 172.21.15.28:0/1600618546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 -- 172.21.15.28:0/1600618546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 msgr2=0x561aa5b565b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 -- 172.21.15.28:0/1600618546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b51b90 msgr2=0x561aa5b56af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b51b90 0x561aa5b56af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.916+0000 7fb9caffd640 1 -- 172.21.15.28:0/1600618546 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa5b393e0 con 0x561aa5b4ca20 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5daf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00ee970 tx=0x7fb9c00f4d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf7b21e35c131c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9b17fa640 1 -- 172.21.15.28:0/1600618546 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c004b070 con 0x561aa5b4ca20 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9b17fa640 1 -- 172.21.15.28:0/1600618546 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c0004c80 con 0x561aa5b4ca20 2024-03-22T02:26:54.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9b17fa640 1 -- 172.21.15.28:0/1600618546 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c00a80b0 con 0x561aa5b4ca20 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1600618546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 msgr2=0x561aa5b5daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.917+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5daf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00ee970 tx=0x7fb9c00f4d10 comp rx=0 tx=0).stop 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1600618546 shutdown_connections 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b51b90 0x561aa5b56af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5daf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1600618546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b4c030 0x561aa5b565b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1600618546 >> 172.21.15.28:0/1600618546 conn(0x561aa5b475a0 msgr2=0x561aa5b55090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1600618546 shutdown_connections 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1600618546 wait complete. 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 -- start start 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.918+0000 7fb9d4c07740 1 Processor -- start 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1a6ec0 con 0x561aa5b4c030 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1a8330 con 0x561aa5b4ca20 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9cc1a7590 con 0x561aa5b51b90 2024-03-22T02:26:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9d4c07740 10 client.0 _ll_register_callbacks cb 0x7fb9c00f4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40262/0 (socket says 172.21.15.28:40262) 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/1777248408 learned_addr learned my addr 172.21.15.28:0/1777248408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cbfff640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 msgr2=0x561aa5b5c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/1777248408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 msgr2=0x561aa5b5b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 -- 172.21.15.28:0/1777248408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa5b5c840 con 0x561aa5b4c030 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9cb7fe640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009560 tx=0x7fb9bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b85e351b8076c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.919+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc00fdf0 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fb9bc02a430 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9c0004c80 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fb9bc039050 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 10 client.19353 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x561aa5b5dc10 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9c0004c80 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fb9bc003b00 con 0x561aa5b4c030 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 10 client.19353 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 10 client.19353 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 20 client.19353 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:54.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5bc8be0 0x561aa5bcb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 4 client.19353 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561aa5ab0160 con 0x561aa5bc8be0 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.920+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5bc8be0 0x561aa5bcb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9aaffd640 10 client.19353 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9caffd640 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5bc8be0 0x561aa5bcb000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d024a7d6ec80fc6 server_cookie=4c04ea6b7d621a26 in_seq=0 out_seq=0 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9aaffd640 1 -- 172.21.15.28:0/1777248408 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561aa5ab0160 con 0x561aa5bc8be0 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9aaffd640 10 client.19353 handle_command_reply: tid=0 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 client.19353 shutdown 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 msgr2=0x561aa5b5ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009560 tx=0x7fb9bc0046c0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 shutdown_connections 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x561aa5bc8be0 0x561aa5bcb000 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aa5b51b90 0x561aa5b5c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aa5b4ca20 0x561aa5b5b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9aaffd640 0 client.19353 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 --2- 172.21.15.28:0/1777248408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aa5b4c030 0x561aa5b5ad00 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 >> 172.21.15.28:0/1777248408 conn(0x561aa5b475a0 msgr2=0x561aa5b56590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 shutdown_connections 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 1 -- 172.21.15.28:0/1777248408 wait complete. 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.921+0000 7fb9d4c07740 20 client.19353 trim_cache size 0 max 16384 2024-03-22T02:26:54.933 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 msgr2=0x7fb9b40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 0x7fb9b40a53e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x561aa5b4ce20 tx=0x7fb9bc005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 msgr2=0x7fb9cc100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0000c00 tx=0x7fb9c002c640 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 shutdown_connections 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9b40a2f20 0x7fb9b40a53e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9cc107930 0x7fb9cc100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9cc105000 0x7fb9cc100680 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 --2- 172.21.15.28:0/1791353242 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9cc1026d0 0x7fb9cc102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 >> 172.21.15.28:0/1791353242 conn(0x7fb9cc0fc020 msgr2=0x7fb9cc103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:54.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 shutdown_connections 2024-03-22T02:26:54.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:54.922+0000 7fb9d201c640 1 -- 172.21.15.28:0/1791353242 wait complete. 2024-03-22T02:26:54.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:26:55.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 -- 172.21.15.28:0/2594124965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac103750 msgr2=0x7fd1ac109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/2594124965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac103750 0x7fd1ac109fe0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009ca0 tx=0x7fd1a402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 -- 172.21.15.28:0/2594124965 shutdown_connections 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/2594124965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac103750 0x7fd1ac109fe0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/2594124965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac102d90 0x7fd1ac103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/2594124965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 -- 172.21.15.28:0/2594124965 >> 172.21.15.28:0/2594124965 conn(0x7fd1ac0fd110 msgr2=0x7fd1ac0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 -- 172.21.15.28:0/2594124965 shutdown_connections 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.088+0000 7fd1b35f7640 1 -- 172.21.15.28:0/2594124965 wait complete. 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 Processor -- start 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 -- start start 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 0x7fd1ac073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac103750 0x7fd1ac0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac10d820 con 0x7fd1ac101ba0 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac10d6a0 con 0x7fd1ac102d90 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b35f7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac10d9a0 con 0x7fd1ac103750 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40284/0 (socket says 172.21.15.28:40284) 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 -- 172.21.15.28:0/938345294 learned_addr learned my addr 172.21.15.28:0/938345294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 0x7fd1ac073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1b94640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac103750 0x7fd1ac0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 -- 172.21.15.28:0/938345294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac103750 msgr2=0x7fd1ac0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac103750 0x7fd1ac0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 -- 172.21.15.28:0/938345294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 msgr2=0x7fd1ac073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 0x7fd1ac073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 -- 172.21.15.28:0/938345294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac074d50 con 0x7fd1ac101ba0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 0x7fd1ac073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1b1393640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9c0 tx=0x7fd1a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.089+0000 7fd1a27fc640 1 -- 172.21.15.28:0/938345294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a800cd80 con 0x7fd1ac101ba0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.090+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac1a1fb0 con 0x7fd1ac101ba0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.090+0000 7fd1a27fc640 1 -- 172.21.15.28:0/938345294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a80035f0 con 0x7fd1ac101ba0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.090+0000 7fd1a27fc640 1 -- 172.21.15.28:0/938345294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8010690 con 0x7fd1ac101ba0 2024-03-22T02:26:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.090+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ac1a2410 con 0x7fd1ac101ba0 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.092+0000 7fd1a27fc640 1 -- 172.21.15.28:0/938345294 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd1a80037a0 con 0x7fd1ac101ba0 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1a27fc640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 0x7fd1980a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 Processor -- start 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1a27fc640 1 -- 172.21.15.28:0/938345294 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd1a80cfbe0 con 0x7fd1ac101ba0 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 -- start start 2024-03-22T02:26:55.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 0x7fd1980a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90dcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90df810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e7d903c8d0 con 0x55e7d90d8210 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e7d8fe4e00 con 0x55e7d90d7820 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e7d9038880 con 0x55e7d90dd380 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b1b94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90df810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90dcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52620/0 (socket says 172.21.15.28:52620) 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.093+0000 7fd1b1393640 1 -- 172.21.15.28:0/2809741371 learned_addr learned my addr 172.21.15.28:0/2809741371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:55.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 0x7fd1980a53e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0756b0 tx=0x7fd19c011860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 -- 172.21.15.28:0/2809741371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 msgr2=0x55e7d90df810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90df810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 -- 172.21.15.28:0/2809741371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 msgr2=0x55e7d90dcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90dcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 -- 172.21.15.28:0/2809741371 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d8d503f0 con 0x55e7d90d7820 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1b1393640 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd1a809be90 tx=0x7fd1a809c600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d96b15d9a5264e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1977fe640 1 -- 172.21.15.28:0/2809741371 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80b4020 con 0x55e7d90d7820 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1977fe640 1 -- 172.21.15.28:0/2809741371 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a80a4910 con 0x55e7d90d7820 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.094+0000 7fd1977fe640 1 -- 172.21.15.28:0/2809741371 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a80a4480 con 0x55e7d90d7820 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- 172.21.15.28:0/2809741371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 msgr2=0x55e7d90d7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd1a809be90 tx=0x7fd1a809c600 comp rx=0 tx=0).stop 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- 172.21.15.28:0/2809741371 shutdown_connections 2024-03-22T02:26:55.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90dcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/2809741371 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90d7c40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- 172.21.15.28:0/2809741371 >> 172.21.15.28:0/2809741371 conn(0x55e7d90d2d90 msgr2=0x55e7d90d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- 172.21.15.28:0/2809741371 shutdown_connections 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- 172.21.15.28:0/2809741371 wait complete. 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 -- start start 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.095+0000 7fd1b61dd740 1 Processor -- start 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e7d90e0330 con 0x55e7d90d8210 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e7d8ff8300 con 0x55e7d90d7820 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e7d90e04b0 con 0x55e7d90dd380 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 10 client.0 _ll_register_callbacks cb 0x55e7d9033e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52628/0 (socket says 172.21.15.28:52628) 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1b94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 -- 172.21.15.28:0/1925751884 learned_addr learned my addr 172.21.15.28:0/1925751884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 -- 172.21.15.28:0/1925751884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 msgr2=0x55e7d90e9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 -- 172.21.15.28:0/1925751884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 msgr2=0x55e7d90e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 -- 172.21.15.28:0/1925751884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d8e1a7d0 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b1393640 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8098310 tx=0x7fd1a8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=631e0da107ab0174 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd194ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a807e020 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a80a4910 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.096+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd1a80a4680 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd1a80b4070 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 10 client.29183 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e7d90ec5a0 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a80a4910 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fd1a809b350 con 0x55e7d90d7820 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 10 client.29183 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 10 client.29183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 20 client.29183 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d90ed8f0 0x55e7d90efce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 4 client.29183 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7d9049e10 con 0x55e7d90ed8f0 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.097+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d90ed8f0 0x55e7d90efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.098+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d90ed8f0 0x55e7d90efce0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44abf0045586ba31 server_cookie=c9c0a75f67599f8b in_seq=0 out_seq=0 2024-03-22T02:26:55.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.098+0000 7fd194ff9640 10 client.29183 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.099+0000 7fd194ff9640 1 -- 172.21.15.28:0/1925751884 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e7d9049e10 con 0x55e7d90ed8f0 2024-03-22T02:26:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.099+0000 7fd194ff9640 10 client.29183 handle_command_reply: tid=0 2024-03-22T02:26:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.099+0000 7fd1b61dd740 1 client.29183 shutdown 2024-03-22T02:26:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 msgr2=0x55e7d90e1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8098310 tx=0x7fd1a8004730 comp rx=0 tx=0).stop 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 shutdown_connections 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d90ed8f0 0x55e7d90efce0 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/1925751884 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e1cb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 >> 172.21.15.28:0/1925751884 conn(0x55e7d90d2d90 msgr2=0x55e7d90d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd194ff9640 0 client.29183 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 shutdown_connections 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 1 -- 172.21.15.28:0/1925751884 wait complete. 2024-03-22T02:26:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.100+0000 7fd1b61dd740 20 client.29183 trim_cache size 0 max 16384 2024-03-22T02:26:55.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 Processor -- start 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 -- start start 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90d7820 0x55e7d90e1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1af560 con 0x55e7d90dd380 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1ace30 con 0x55e7d90d8210 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1aac80 con 0x55e7d90d7820 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.115+0000 7fd1b1b94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40306/0 (socket says 172.21.15.28:40306) 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52642/0 (socket says 172.21.15.28:52642) 2024-03-22T02:26:55.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 -- 172.21.15.28:0/3396585662 learned_addr learned my addr 172.21.15.28:0/3396585662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1393640 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90d7820 0x55e7d90e1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 -- 172.21.15.28:0/3396585662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90d7820 msgr2=0x55e7d90e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90d7820 0x55e7d90e1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 -- 172.21.15.28:0/3396585662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 msgr2=0x55e7d90e91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 -- 172.21.15.28:0/3396585662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7d90dd140 con 0x55e7d90dd380 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd1b1b94640 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40324c0 tx=0x7fd1a4032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b0f5aa06e0c0ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd196ffd640 1 -- 172.21.15.28:0/3396585662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4039040 con 0x55e7d90dd380 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd196ffd640 1 -- 172.21.15.28:0/3396585662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a4032c90 con 0x55e7d90dd380 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.116+0000 7fd196ffd640 1 -- 172.21.15.28:0/3396585662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4004040 con 0x55e7d90dd380 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- 172.21.15.28:0/3396585662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 msgr2=0x55e7d90d1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd1a40324c0 tx=0x7fd1a4032590 comp rx=0 tx=0).stop 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- 172.21.15.28:0/3396585662 shutdown_connections 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90dd380 0x55e7d90d1740 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d8210 0x55e7d90e91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/3396585662 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90d7820 0x55e7d90e1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- 172.21.15.28:0/3396585662 >> 172.21.15.28:0/3396585662 conn(0x55e7d90d2d90 msgr2=0x55e7d90e18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- 172.21.15.28:0/3396585662 shutdown_connections 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- 172.21.15.28:0/3396585662 wait complete. 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 -- start start 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.117+0000 7fd1b61dd740 1 Processor -- start 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1ab2f0 con 0x55e7d90d8210 2024-03-22T02:26:55.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1ac760 con 0x55e7d90d7820 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac1ab9c0 con 0x55e7d90dd380 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b61dd740 10 client.0 _ll_register_callbacks cb 0x7fd19c009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b1393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b1b94640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40318/0 (socket says 172.21.15.28:40318) 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 -- 172.21.15.28:0/974308285 learned_addr learned my addr 172.21.15.28:0/974308285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 -- 172.21.15.28:0/974308285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 msgr2=0x55e7d90e83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 -- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 msgr2=0x55e7d90e6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:55.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 -- 172.21.15.28:0/974308285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7d90e8990 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b1b94640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b1393640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd1b0b92640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd19c011e00 tx=0x7fd19c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7404d7a206bdf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd194ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c017040 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.118+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd19c003d80 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1a4032c90 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7fd19c039050 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 10 client.19371 fetch_fsmap learned FSMap version 38 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55e7d9154140 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a4032c90 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) ==== 8346+0+0 (secure 0 0 0) 0x7fd19c00f840 con 0x55e7d90d8210 2024-03-22T02:26:55.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 10 client.19371 fetch_fsmap finished waiting for FSMap version 38 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 10 client.19371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 20 client.19371 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d9154b10 0x55e7d9156f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 4 client.19371 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e7d9044160 con 0x55e7d9154b10 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b1393640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d9154b10 0x55e7d9156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd194ff9640 10 client.19371 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.119+0000 7fd1b1393640 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d9154b10 0x55e7d9156f30 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17e5971ae4b0d8f8 server_cookie=a60ee99821db0576 in_seq=0 out_seq=0 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd194ff9640 1 -- 172.21.15.28:0/974308285 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e7d9044160 con 0x55e7d9154b10 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd194ff9640 10 client.19371 handle_command_reply: tid=0 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 client.19371 shutdown 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 msgr2=0x55e7d90e7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd19c011e00 tx=0x7fd19c004730 comp rx=0 tx=0).stop 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 shutdown_connections 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e7d9154b10 0x55e7d9156f30 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e7d90dd380 0x55e7d90e83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7d90d8210 0x55e7d90e7390 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 --2- 172.21.15.28:0/974308285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e7d90d7820 0x55e7d90e6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 >> 172.21.15.28:0/974308285 conn(0x55e7d90d2d90 msgr2=0x55e7d90d6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd194ff9640 0 client.19371 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 shutdown_connections 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 1 -- 172.21.15.28:0/974308285 wait complete. 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.120+0000 7fd1b61dd740 20 client.19371 trim_cache size 0 max 16384 2024-03-22T02:26:55.129 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:26:55.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 msgr2=0x7fd1980a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 0x7fd1980a53e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0756b0 tx=0x7fd19c011860 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 msgr2=0x7fd1ac075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800e9c0 tx=0x7fd1a800ee80 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 shutdown_connections 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1980a2f20 0x7fd1980a53e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1ac103750 0x7fd1ac0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1ac102d90 0x7fd1ac073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 --2- 172.21.15.28:0/938345294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1ac101ba0 0x7fd1ac075940 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 >> 172.21.15.28:0/938345294 conn(0x7fd1ac0fd110 msgr2=0x7fd1ac1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 shutdown_connections 2024-03-22T02:26:55.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:55.121+0000 7fd1b35f7640 1 -- 172.21.15.28:0/938345294 wait complete. 2024-03-22T02:26:55.439 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1384536 11.19 MB/sec execute 577 sec latency 270.984 ms 2024-03-22T02:26:56.135 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:26:56.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:26:56.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:26:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 -- 172.21.15.28:0/3435466162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc100a90 msgr2=0x7f43bc100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/3435466162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc100a90 0x7f43bc100eb0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009ca0 tx=0x7f43b402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 -- 172.21.15.28:0/3435466162 shutdown_connections 2024-03-22T02:26:56.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/3435466162 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc102640 0x7f43bc108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/3435466162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc101c80 0x7f43bc102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/3435466162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc100a90 0x7f43bc100eb0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.241+0000 7f43c2f4b640 1 -- 172.21.15.28:0/3435466162 >> 172.21.15.28:0/3435466162 conn(0x7f43bc0fc020 msgr2=0x7f43bc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- 172.21.15.28:0/3435466162 shutdown_connections 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- 172.21.15.28:0/3435466162 wait complete. 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 Processor -- start 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- start start 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc100a90 0x7f43bc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc102640 0x7f43bc19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc10c700 con 0x7f43bc101c80 2024-03-22T02:26:56.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc10c580 con 0x7f43bc100a90 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c2f4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc10c880 con 0x7f43bc102640 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40334/0 (socket says 172.21.15.28:40334) 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43bbfff640 1 -- 172.21.15.28:0/1395155071 learned_addr learned my addr 172.21.15.28:0/1395155071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc100a90 0x7f43bc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.242+0000 7f43c14e8640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc102640 0x7f43bc19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 -- 172.21.15.28:0/1395155071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc102640 msgr2=0x7f43bc19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc102640 0x7f43bc19f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 -- 172.21.15.28:0/1395155071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc100a90 msgr2=0x7f43bc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc100a90 0x7f43bc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 -- 172.21.15.28:0/1395155071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43bc19fb40 con 0x7f43bc101c80 2024-03-22T02:26:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e9f0 tx=0x7f43b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43b9ffb640 1 -- 172.21.15.28:0/1395155071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b000cd80 con 0x7f43bc101c80 2024-03-22T02:26:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43b9ffb640 1 -- 172.21.15.28:0/1395155071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b0005500 con 0x7f43bc101c80 2024-03-22T02:26:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43b9ffb640 1 -- 172.21.15.28:0/1395155071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b0010690 con 0x7f43bc101c80 2024-03-22T02:26:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43bc19fe30 con 0x7f43bc101c80 2024-03-22T02:26:56.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.243+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43bc1a02f0 con 0x7f43bc101c80 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.245+0000 7f43b9ffb640 1 -- 172.21.15.28:0/1395155071 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f43b00046c0 con 0x7f43bc101c80 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.245+0000 7f43c5c0b740 1 Processor -- start 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 -- start start 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x559752865fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x55975286dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5597527ca570 con 0x559752865bc0 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559752772e10 con 0x5597528665b0 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5597527c6520 con 0x55975286b720 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43b9ffb640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 0x7f43a40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c14e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x559752865fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52688/0 (socket says 172.21.15.28:52688) 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 -- 172.21.15.28:0/3985995537 learned_addr learned my addr 172.21.15.28:0/3985995537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c14e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x559752865fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40344/0 (socket says 172.21.15.28:40344) 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43bbfff640 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x55975286dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c14e8640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 0x7f43a40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43b9ffb640 1 -- 172.21.15.28:0/1395155071 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f43b00cfba0 con 0x7f43bc101c80 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 -- 172.21.15.28:0/3985995537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 msgr2=0x55975286dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x55975286dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 -- 172.21.15.28:0/3985995537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 msgr2=0x559752865fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x559752865fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 -- 172.21.15.28:0/3985995537 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597525a67e0 con 0x5597528665b0 2024-03-22T02:26:56.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43bbfff640 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x55975286dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f43bc101ae0 tx=0x7f43b40323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6071d4b920c3d3a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.246+0000 7f43c14e8640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 0x7f43a40a5460 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009710 tx=0x7f43ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43a2ffd640 1 -- 172.21.15.28:0/3985995537 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4043070 con 0x5597528665b0 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43a2ffd640 1 -- 172.21.15.28:0/3985995537 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b4032c90 con 0x5597528665b0 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43a2ffd640 1 -- 172.21.15.28:0/3985995537 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4039040 con 0x5597528665b0 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 -- 172.21.15.28:0/3985995537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 msgr2=0x55975286b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f43bc101ae0 tx=0x7f43b40323d0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 -- 172.21.15.28:0/3985995537 shutdown_connections 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x55975286dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x55975286b0b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/3985995537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x559752865fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 -- 172.21.15.28:0/3985995537 >> 172.21.15.28:0/3985995537 conn(0x559752861130 msgr2=0x559752861530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 -- 172.21.15.28:0/3985995537 shutdown_connections 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.247+0000 7f43c5c0b740 1 -- 172.21.15.28:0/3985995537 wait complete. 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 -- start start 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 Processor -- start 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752870240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55975286e670 con 0x55975286b720 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559752786110 con 0x5597528665b0 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55975286e7f0 con 0x559752865bc0 2024-03-22T02:26:56.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c5c0b740 10 client.0 _ll_register_callbacks cb 0x5597527c2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c14e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752870240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c14e8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40356/0 (socket says 172.21.15.28:40356) 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c14e8640 1 -- 172.21.15.28:0/4156638476 learned_addr learned my addr 172.21.15.28:0/4156638476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.248+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39690/0 (socket says 172.21.15.28:39690) 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 -- 172.21.15.28:0/4156638476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 msgr2=0x55975286fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 -- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 msgr2=0x559752870240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752870240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 -- 172.21.15.28:0/4156638476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559752606f70 con 0x55975286b720 2024-03-22T02:26:56.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43bbfff640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752870240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c14e8640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00ffd0 tx=0x7f43ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7112f99d65397c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac02f480 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f43ac023450 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b4032c90 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f43ac023650 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 10 client.19383 fetch_fsmap learned FSMap version 39 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55975249e170 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b4032c90 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f43ac021040 con 0x55975286b720 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 10 client.19383 fetch_fsmap finished waiting for FSMap version 39 2024-03-22T02:26:56.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 10 client.19383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 20 client.19383 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55975287bc00 0x55975287e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 4 client.19383 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.249+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597527d7ab0 con 0x55975287bc00 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.250+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55975287bc00 0x55975287e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.250+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55975287bc00 0x55975287e030 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b98ee30bc1951b8 server_cookie=43964dfef35d54a3 in_seq=0 out_seq=0 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.250+0000 7f439bfff640 10 client.19383 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f439bfff640 1 -- 172.21.15.28:0/4156638476 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597527d7ab0 con 0x55975287bc00 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f439bfff640 10 client.19383 handle_command_reply: tid=0 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 client.19383 shutdown 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 msgr2=0x559752877280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f43ac00ffd0 tx=0x7f43ac0046c0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 shutdown_connections 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55975287bc00 0x55975287e030 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55975286b720 0x559752877280 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f439bfff640 0 client.19383 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752870240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/4156638476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559752865bc0 0x55975286fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 >> 172.21.15.28:0/4156638476 conn(0x559752861130 msgr2=0x5597528603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 shutdown_connections 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 1 -- 172.21.15.28:0/4156638476 wait complete. 2024-03-22T02:26:56.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.251+0000 7f43c5c0b740 20 client.19383 trim_cache size 0 max 16384 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.268+0000 7f43c5c0b740 1 Processor -- start 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 -- start start 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559752865bc0 0x559752864aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752867050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1ab110 con 0x5597528665b0 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1a89e0 con 0x559752865bc0 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1a6830 con 0x55975286b720 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559752865bc0 0x559752864aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 -- 172.21.15.28:0/1849862423 learned_addr learned my addr 172.21.15.28:0/1849862423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43c14e8640 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752867050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 -- 172.21.15.28:0/1849862423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 msgr2=0x559752867050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752867050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 -- 172.21.15.28:0/1849862423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559752865bc0 msgr2=0x559752864aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559752865bc0 0x559752864aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 -- 172.21.15.28:0/1849862423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55975286edf0 con 0x5597528665b0 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43bbfff640 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f43b00097d0 tx=0x7f43b006ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf698694d461f79a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43a27fc640 1 -- 172.21.15.28:0/1849862423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b009be60 con 0x5597528665b0 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43a27fc640 1 -- 172.21.15.28:0/1849862423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b009ffb0 con 0x5597528665b0 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.269+0000 7f43a27fc640 1 -- 172.21.15.28:0/1849862423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b00babd0 con 0x5597528665b0 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- 172.21.15.28:0/1849862423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 msgr2=0x559752877240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f43b00097d0 tx=0x7f43b006ba40 comp rx=0 tx=0).stop 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- 172.21.15.28:0/1849862423 shutdown_connections 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752867050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5597528665b0 0x559752877240 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/1849862423 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559752865bc0 0x559752864aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- 172.21.15.28:0/1849862423 >> 172.21.15.28:0/1849862423 conn(0x559752861130 msgr2=0x559752864390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- 172.21.15.28:0/1849862423 shutdown_connections 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- 172.21.15.28:0/1849862423 wait complete. 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 -- start start 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.270+0000 7f43c5c0b740 1 Processor -- start 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752874e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752875e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1a6ea0 con 0x559752865bc0 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1a8310 con 0x5597528665b0 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f43bc1a7570 con 0x55975286b720 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 10 client.0 _ll_register_callbacks cb 0x7f43ac009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:56.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40378/0 (socket says 172.21.15.28:40378) 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 -- 172.21.15.28:0/100203002 learned_addr learned my addr 172.21.15.28:0/100203002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43bbfff640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752874e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c14e8640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752875e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 -- 172.21.15.28:0/100203002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 msgr2=0x559752875e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752875e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 -- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 msgr2=0x559752874e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752874e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 -- 172.21.15.28:0/100203002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559752876410 con 0x559752865bc0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c0ce7640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f43b40324c0 tx=0x7f43b40093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12787dccff3498db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f439bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:56.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4039040 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f43b402cca0 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43b009ffb0 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.271+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f43b405c050 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 10 client.19389 fetch_fsmap learned FSMap version 39 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5597528e1f30 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b009ffb0 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f43b4050510 con 0x559752865bc0 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 10 client.19389 fetch_fsmap finished waiting for FSMap version 39 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 10 client.19389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 20 client.19389 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5597528e2440 0x5597528e4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 4 client.19389 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:26:56.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5597527d1e00 con 0x5597528e2440 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43bbfff640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5597528e2440 0x5597528e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f439bfff640 10 client.19389 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.272+0000 7f43bbfff640 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5597528e2440 0x5597528e4830 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f48ec70f412645f9 server_cookie=9a496d5d2c32df55 in_seq=0 out_seq=0 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.273+0000 7f439bfff640 1 -- 172.21.15.28:0/100203002 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5597527d1e00 con 0x5597528e2440 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.273+0000 7f439bfff640 10 client.19389 handle_command_reply: tid=0 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.273+0000 7f43c5c0b740 1 client.19389 shutdown 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 msgr2=0x5597528748d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f43b40324c0 tx=0x7f43b40093e0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 shutdown_connections 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5597528e2440 0x5597528e4830 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55975286b720 0x559752875e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5597528665b0 0x559752874e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 --2- 172.21.15.28:0/100203002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559752865bc0 0x5597528748d0 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 >> 172.21.15.28:0/100203002 conn(0x559752861130 msgr2=0x559752864c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f439bfff640 0 client.19389 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 shutdown_connections 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 1 -- 172.21.15.28:0/100203002 wait complete. 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c5c0b740 20 client.19389 trim_cache size 0 max 16384 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:26:56.283 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0bec9fb2-ba8e-446b-a77e-66d84172c0a8", 2024-03-22T02:26:56.284 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:26:56.284 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:56.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 msgr2=0x7f43a40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 0x7f43a40a5460 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f43ac009710 tx=0x7f43ac005f90 comp rx=0 tx=0).stop 2024-03-22T02:26:56.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 msgr2=0x7f43bc198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.274+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f43b000e9f0 tx=0x7f43b000edf0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 shutdown_connections 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f43a40a2fa0 0x7f43a40a5460 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43bc102640 0x7f43bc19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43bc101c80 0x7f43bc198460 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 --2- 172.21.15.28:0/1395155071 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f43bc100a90 0x7f43bc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 >> 172.21.15.28:0/1395155071 conn(0x7f43bc0fc020 msgr2=0x7f43bc0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 shutdown_connections 2024-03-22T02:26:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.275+0000 7f43c2f4b640 1 -- 172.21.15.28:0/1395155071 wait complete. 2024-03-22T02:26:56.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:26:56.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:26:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 -- 172.21.15.28:0/1654512671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50102d90 msgr2=0x7f8e50103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/1654512671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50102d90 0x7f8e50103210 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009900 tx=0x7f8e3c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:26:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 -- 172.21.15.28:0/1654512671 shutdown_connections 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/1654512671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50103750 0x7f8e50109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/1654512671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50102d90 0x7f8e50103210 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/1654512671 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50101ba0 0x7f8e50101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 -- 172.21.15.28:0/1654512671 >> 172.21.15.28:0/1654512671 conn(0x7f8e500fd130 msgr2=0x7f8e500ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 -- 172.21.15.28:0/1654512671 shutdown_connections 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.391+0000 7f8e56e0e640 1 -- 172.21.15.28:0/1654512671 wait complete. 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 Processor -- start 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 -- start start 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50102d90 0x7f8e5019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50103750 0x7f8e501a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5010d800 con 0x7f8e50101ba0 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5010d680 con 0x7f8e50103750 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e56e0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5010d980 con 0x7f8e50102d90 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40388/0 (socket says 172.21.15.28:40388) 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 -- 172.21.15.28:0/413964566 learned_addr learned my addr 172.21.15.28:0/413964566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50102d90 0x7f8e5019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e553ab640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50103750 0x7f8e501a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 -- 172.21.15.28:0/413964566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50102d90 msgr2=0x7f8e5019a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50102d90 0x7f8e5019a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 -- 172.21.15.28:0/413964566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50103750 msgr2=0x7f8e501a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50103750 0x7f8e501a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 -- 172.21.15.28:0/413964566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e501a1d70 con 0x7f8e50101ba0 2024-03-22T02:26:56.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.392+0000 7f8e54baa640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400e9c0 tx=0x7f8e4400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.393+0000 7f8e4dffb640 1 -- 172.21.15.28:0/413964566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400ccc0 con 0x7f8e50101ba0 2024-03-22T02:26:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.393+0000 7f8e4dffb640 1 -- 172.21.15.28:0/413964566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e440035f0 con 0x7f8e50101ba0 2024-03-22T02:26:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.393+0000 7f8e4dffb640 1 -- 172.21.15.28:0/413964566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4400ce60 con 0x7f8e50101ba0 2024-03-22T02:26:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.393+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e501a2060 con 0x7f8e50101ba0 2024-03-22T02:26:56.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.393+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e501a25d0 con 0x7f8e50101ba0 2024-03-22T02:26:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.395+0000 7f8e4dffb640 1 -- 172.21.15.28:0/413964566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8e440037a0 con 0x7f8e50101ba0 2024-03-22T02:26:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 Processor -- start 2024-03-22T02:26:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 -- start start 2024-03-22T02:26:56.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2d8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2dd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2da23d8c0 con 0x55c2da2d8430 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2da1e5c20 con 0x55c2da2ddf90 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e599f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c2da239870 con 0x55c2da2d8e20 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4dffb640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 0x7f8e400a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2dd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2d8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2d8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40394/0 (socket says 172.21.15.28:40394) 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:52736/0 (socket says 172.21.15.28:52736) 2024-03-22T02:26:56.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e553ab640 1 -- 172.21.15.28:0/2336520401 learned_addr learned my addr 172.21.15.28:0/2336520401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e553ab640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 0x7f8e400a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4dffb640 1 -- 172.21.15.28:0/413964566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8e440cfaa0 con 0x7f8e50101ba0 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 -- 172.21.15.28:0/2336520401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 msgr2=0x55c2da2dd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2dd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 -- 172.21.15.28:0/2336520401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 msgr2=0x55c2da2d8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2d8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.396+0000 7f8e4ffff640 1 -- 172.21.15.28:0/2336520401 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2da20a150 con 0x55c2da2ddf90 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.397+0000 7f8e553ab640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 0x7f8e400a5460 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005c40 tx=0x7f8e48009c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.397+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009230 tx=0x7f8e3c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c14c7489ffbfa4a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.397+0000 7f8e3affd640 1 -- 172.21.15.28:0/2336520401 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c02cb80 con 0x55c2da2ddf90 2024-03-22T02:26:56.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.397+0000 7f8e3affd640 1 -- 172.21.15.28:0/2336520401 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e3c005040 con 0x55c2da2ddf90 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.397+0000 7f8e3affd640 1 -- 172.21.15.28:0/2336520401 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c004920 con 0x55c2da2ddf90 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- 172.21.15.28:0/2336520401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 msgr2=0x55c2da2e0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c009230 tx=0x7f8e3c009560 comp rx=0 tx=0).stop 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- 172.21.15.28:0/2336520401 shutdown_connections 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e0420 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2dd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2336520401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2d8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- 172.21.15.28:0/2336520401 >> 172.21.15.28:0/2336520401 conn(0x55c2da2d39a0 msgr2=0x55c2da2d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- 172.21.15.28:0/2336520401 shutdown_connections 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- 172.21.15.28:0/2336520401 wait complete. 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- start start 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 Processor -- start 2024-03-22T02:26:56.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2da2e0ee0 con 0x55c2da2ddf90 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55c2da1f9130 con 0x55c2da2d8e20 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55c2da2e1060 con 0x55c2da2d8430 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.398+0000 7f8e599f4740 10 client.0 _ll_register_callbacks cb 0x55c2da1f8b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e4ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39734/0 (socket says 172.21.15.28:39734) 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40406/0 (socket says 172.21.15.28:40406) 2024-03-22T02:26:56.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 -- 172.21.15.28:0/313655612 learned_addr learned my addr 172.21.15.28:0/313655612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 -- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 msgr2=0x55c2da2e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 -- 172.21.15.28:0/313655612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 msgr2=0x55c2da2e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 -- 172.21.15.28:0/313655612 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2da01b7d0 con 0x55c2da2d8430 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e553ab640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e54baa640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e4409bd50 tx=0x7f8e4409b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5d04c78455ade15 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4409bdc0 con 0x55c2da2d8430 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e3c005040 con 0x55c2da2d8430 2024-03-22T02:26:56.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8e440a4340 con 0x55c2da2d8430 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8e4409ff10 con 0x55c2da2d8430 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e599f4740 10 client.29218 fetch_fsmap learned FSMap version 39 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.399+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c2d9f513f0 con 0x55c2da2d8430 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e440b7030 con 0x55c2da2d8430 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f8e440bacc0 con 0x55c2da2d8430 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 10 client.29218 fetch_fsmap finished waiting for FSMap version 39 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 10 client.29218 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 20 client.29218 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da2ee790 0x55c2da2f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 4 client.29218 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2da24ae00 con 0x55c2da2ee790 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da2ee790 0x55c2da2f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da2ee790 0x55c2da2f0c20 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28a9e9303371a4e2 server_cookie=5c3dd1a8cc297b11 in_seq=0 out_seq=0 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.400+0000 7f8e2ffff640 10 client.29218 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e2ffff640 1 -- 172.21.15.28:0/313655612 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2da24ae00 con 0x55c2da2ee790 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e2ffff640 10 client.29218 handle_command_reply: tid=0 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 client.29218 shutdown 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 msgr2=0x55c2da2e2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8e4409bd50 tx=0x7f8e4409b9c0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 shutdown_connections 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da2ee790 0x55c2da2f0c20 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 --2- 172.21.15.28:0/313655612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e2940 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e2ffff640 0 client.29218 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 >> 172.21.15.28:0/313655612 conn(0x55c2da2d39a0 msgr2=0x55c2da2d2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 shutdown_connections 2024-03-22T02:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 1 -- 172.21.15.28:0/313655612 wait complete. 2024-03-22T02:26:56.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.401+0000 7f8e599f4740 20 client.29218 trim_cache size 0 max 16384 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.418+0000 7f8e599f4740 1 Processor -- start 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.418+0000 7f8e599f4740 1 -- start start 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501af470 con 0x55c2da2d8430 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501acd40 con 0x55c2da2ddf90 2024-03-22T02:26:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e599f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501aab90 con 0x55c2da2d8e20 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40420/0 (socket says 172.21.15.28:40420) 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 -- 172.21.15.28:0/2544909471 learned_addr learned my addr 172.21.15.28:0/2544909471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e4ffff640 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e553ab640 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 -- 172.21.15.28:0/2544909471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 msgr2=0x55c2da2e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2e9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 -- 172.21.15.28:0/2544909471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 msgr2=0x55c2da2e2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 -- 172.21.15.28:0/2544909471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2d9f110f0 con 0x55c2da2d8430 2024-03-22T02:26:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e54baa640 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009830 tx=0x7f8e44098190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1c5c3e976902e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e3a7fc640 1 -- 172.21.15.28:0/2544909471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4409b520 con 0x55c2da2d8430 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e3a7fc640 1 -- 172.21.15.28:0/2544909471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e440b7070 con 0x55c2da2d8430 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.419+0000 7f8e3a7fc640 1 -- 172.21.15.28:0/2544909471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e440bad90 con 0x55c2da2d8430 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- 172.21.15.28:0/2544909471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 msgr2=0x55c2da2e2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009830 tx=0x7f8e44098190 comp rx=0 tx=0).stop 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- 172.21.15.28:0/2544909471 shutdown_connections 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2ddf90 0x55c2da2e2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8e20 0x55c2da2e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 --2- 172.21.15.28:0/2544909471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2d8430 0x55c2da2e2550 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- 172.21.15.28:0/2544909471 >> 172.21.15.28:0/2544909471 conn(0x55c2da2d39a0 msgr2=0x55c2da2da3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- 172.21.15.28:0/2544909471 shutdown_connections 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- 172.21.15.28:0/2544909471 wait complete. 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 -- start start 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.420+0000 7f8e599f4740 1 Processor -- start 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501ab200 con 0x55c2da2ddf90 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501ac670 con 0x55c2da2d8e20 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8e501ab8d0 con 0x55c2da2d8430 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e599f4740 10 client.0 _ll_register_callbacks cb 0x7f8e440cff50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e4ffff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e54baa640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 -- 172.21.15.28:0/478184611 learned_addr learned my addr 172.21.15.28:0/478184611 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 -- 172.21.15.28:0/478184611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 msgr2=0x55c2da2e9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 -- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 msgr2=0x55c2da2e6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 -- 172.21.15.28:0/478184611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2da352ae0 con 0x55c2da2ddf90 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e553ab640 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8e4801fbf0 tx=0x7f8e4801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0bc345b6534f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.421+0000 7f8e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48020ba0 con 0x55c2da2ddf90 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8e48020d40 con 0x55c2da2ddf90 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e440b7070 con 0x55c2da2ddf90 2024-03-22T02:26:56.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8e48038050 con 0x55c2da2ddf90 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 10 client.19407 fetch_fsmap learned FSMap version 39 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55c2da353d00 con 0x55c2da2ddf90 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e440b7070 con 0x55c2da2ddf90 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) ==== 8350+0+0 (secure 0 0 0) 0x7f8e4802b400 con 0x55c2da2ddf90 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 10 client.19407 fetch_fsmap finished waiting for FSMap version 39 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 10 client.19407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 20 client.19407 populate_metadata read hostname 'smithi028' 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da354760 0x55c2da356b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 4 client.19407 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c2da245150 con 0x55c2da354760 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.422+0000 7f8e54baa640 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da354760 0x55c2da356b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:26:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.423+0000 7f8e54baa640 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da354760 0x55c2da356b80 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed6217f1d09f8796 server_cookie=40355822db8669ee in_seq=0 out_seq=0 2024-03-22T02:26:56.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.423+0000 7f8e2ffff640 10 client.19407 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e2ffff640 1 -- 172.21.15.28:0/478184611 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c2da245150 con 0x55c2da354760 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e2ffff640 10 client.19407 handle_command_reply: tid=0 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 client.19407 shutdown 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 msgr2=0x55c2da2e6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f8e4801fbf0 tx=0x7f8e4801d000 comp rx=0 tx=0).stop 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 shutdown_connections 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55c2da354760 0x55c2da356b80 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2da2ddf90 0x55c2da2e6cb0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55c2da2d8e20 0x55c2da2e6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e2ffff640 0 client.19407 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 --2- 172.21.15.28:0/478184611 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c2da2d8430 0x55c2da2e9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 >> 172.21.15.28:0/478184611 conn(0x55c2da2d39a0 msgr2=0x55c2da2e24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 shutdown_connections 2024-03-22T02:26:56.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 1 -- 172.21.15.28:0/478184611 wait complete. 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.425+0000 7f8e599f4740 20 client.19407 trim_cache size 0 max 16384 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "0bec9fb2-ba8e-446b-a77e-66d84172c0a8": { 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0bec9fb2-ba8e-446b-a77e-66d84172c0a8", 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:26:56.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 msgr2=0x7f8e400a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 0x7f8e400a5460 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8e48005c40 tx=0x7f8e48009c90 comp rx=0 tx=0).stop 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 msgr2=0x7f8e5019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:26:56.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8e4400e9c0 tx=0x7f8e4400edc0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 shutdown_connections 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8e400a2fa0 0x7f8e400a5460 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8e50103750 0x7f8e501a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8e50102d90 0x7f8e5019a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 --2- 172.21.15.28:0/413964566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e50101ba0 0x7f8e5019a150 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 >> 172.21.15.28:0/413964566 conn(0x7f8e500fd130 msgr2=0x7f8e50073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 shutdown_connections 2024-03-22T02:26:56.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:26:56.426+0000 7f8e56e0e640 1 -- 172.21.15.28:0/413964566 wait complete. 2024-03-22T02:26:56.439 INFO:tasks.cephfs.filesystem:scrub status for tag:0bec9fb2-ba8e-446b-a77e-66d84172c0a8 - {'path': '/', 'tag': '0bec9fb2-ba8e-446b-a77e-66d84172c0a8', 'options': 'recursive,force'} 2024-03-22T02:26:56.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1385496 11.18 MB/sec execute 578 sec latency 349.484 ms 2024-03-22T02:26:56.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:56 smithi083 ceph-mon[36340]: pgmap v761: 129 pgs: 129 active+clean; 48 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-22T02:26:56.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:56 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:56 smithi028 ceph-mon[31916]: pgmap v761: 129 pgs: 129 active+clean; 48 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-22T02:26:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:56 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:56 smithi120 ceph-mon[36803]: pgmap v761: 129 pgs: 129 active+clean; 48 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.0 MiB/s wr, 359 op/s 2024-03-22T02:26:56.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:56 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:26:57.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1385496 11.16 MB/sec execute 579 sec latency 1141.996 ms 2024-03-22T02:26:57.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:57 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:26:57.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:57 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:57.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:57 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:26:57.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:57 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:57.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:57 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:26:57.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:57 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:57.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:57 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:26:57.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:57 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:58.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:57 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:26:58.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:57 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:58.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:57 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:26:58.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:57 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:26:58.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1385496 11.14 MB/sec execute 580 sec latency 2142.047 ms 2024-03-22T02:26:58.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:26:58 smithi028 ceph-mon[31916]: pgmap v762: 129 pgs: 129 active+clean; 47 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-22T02:26:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:26:58 smithi120 ceph-mon[36803]: pgmap v762: 129 pgs: 129 active+clean; 47 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-22T02:26:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:26:58 smithi083 ceph-mon[36340]: pgmap v762: 129 pgs: 129 active+clean; 47 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 278 op/s 2024-03-22T02:26:59.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1385496 11.12 MB/sec execute 581 sec latency 3142.148 ms 2024-03-22T02:27:00.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1386420 11.11 MB/sec execute 582 sec latency 3465.781 ms 2024-03-22T02:27:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:00 smithi028 ceph-mon[31916]: pgmap v763: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 390 op/s 2024-03-22T02:27:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:00 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:27:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:00 smithi120 ceph-mon[36803]: pgmap v763: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 390 op/s 2024-03-22T02:27:00.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:00 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:27:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:00 smithi083 ceph-mon[36340]: pgmap v763: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.9 MiB/s wr, 390 op/s 2024-03-22T02:27:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:00 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:27:01.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1388584 11.11 MB/sec execute 583 sec latency 88.094 ms 2024-03-22T02:27:01.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:01 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:01.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:01 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:02.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:01 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:02.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1390017 11.11 MB/sec execute 584 sec latency 34.149 ms 2024-03-22T02:27:02.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:02 smithi028 ceph-mon[31916]: pgmap v764: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 351 op/s 2024-03-22T02:27:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:02 smithi120 ceph-mon[36803]: pgmap v764: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 351 op/s 2024-03-22T02:27:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:02 smithi083 ceph-mon[36340]: pgmap v764: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.3 MiB/s wr, 351 op/s 2024-03-22T02:27:03.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1392589 11.11 MB/sec execute 585 sec latency 12.646 ms 2024-03-22T02:27:04.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1394211 11.11 MB/sec execute 586 sec latency 39.235 ms 2024-03-22T02:27:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:04 smithi028 ceph-mon[31916]: pgmap v765: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:27:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:04 smithi120 ceph-mon[36803]: pgmap v765: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:27:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:04 smithi083 ceph-mon[36340]: pgmap v765: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 2.4 MiB/s wr, 353 op/s 2024-03-22T02:27:05.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1396681 11.11 MB/sec execute 587 sec latency 23.768 ms 2024-03-22T02:27:06.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1398024 11.11 MB/sec execute 588 sec latency 94.487 ms 2024-03-22T02:27:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:06 smithi028 ceph-mon[31916]: pgmap v766: 129 pgs: 129 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.8 MiB/s wr, 395 op/s 2024-03-22T02:27:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:06 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:06 smithi120 ceph-mon[36803]: pgmap v766: 129 pgs: 129 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.8 MiB/s wr, 395 op/s 2024-03-22T02:27:06.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:06 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:06 smithi083 ceph-mon[36340]: pgmap v766: 129 pgs: 129 active+clean; 41 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 2.8 MiB/s wr, 395 op/s 2024-03-22T02:27:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:06 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:07.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1399707 11.10 MB/sec execute 589 sec latency 266.454 ms 2024-03-22T02:27:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- 172.21.15.28:0/3187536507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988100ab0 msgr2=0x7f2988100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 --2- 172.21.15.28:0/3187536507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988100ab0 0x7f2988100ed0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f298002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- 172.21.15.28:0/3187536507 shutdown_connections 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 --2- 172.21.15.28:0/3187536507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988102660 0x7f2988108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 --2- 172.21.15.28:0/3187536507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988101ca0 0x7f2988102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 --2- 172.21.15.28:0/3187536507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988100ab0 0x7f2988100ed0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- 172.21.15.28:0/3187536507 >> 172.21.15.28:0/3187536507 conn(0x7f29880fc020 msgr2=0x7f29880fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- 172.21.15.28:0/3187536507 shutdown_connections 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- 172.21.15.28:0/3187536507 wait complete. 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 Processor -- start 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.901+0000 7f298fdbc640 1 -- start start 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 0x7f29881049d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988102660 0x7f29881035c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f298810c720 con 0x7f2988101ca0 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f298810c5a0 con 0x7f2988100ab0 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298fdbc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f298810c8a0 con 0x7f2988102660 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52920/0 (socket says 172.21.15.28:52920) 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 -- 172.21.15.28:0/1866066967 learned_addr learned my addr 172.21.15.28:0/1866066967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298db58640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 0x7f29881049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298e359640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988102660 0x7f29881035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 -- 172.21.15.28:0/1866066967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988102660 msgr2=0x7f29881035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988102660 0x7f29881035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 -- 172.21.15.28:0/1866066967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 msgr2=0x7f29881049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 0x7f29881049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 -- 172.21.15.28:0/1866066967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2988103de0 con 0x7f2988101ca0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298db58640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 0x7f29881049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f298d357640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f297800e9f0 tx=0x7f297800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.902+0000 7f297effd640 1 -- 172.21.15.28:0/1866066967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297800ccc0 con 0x7f2988101ca0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.903+0000 7f297effd640 1 -- 172.21.15.28:0/1866066967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f297800ce60 con 0x7f2988101ca0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.903+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29881a1fc0 con 0x7f2988101ca0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.903+0000 7f297effd640 1 -- 172.21.15.28:0/1866066967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978010690 con 0x7f2988101ca0 2024-03-22T02:27:07.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.903+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29881a2480 con 0x7f2988101ca0 2024-03-22T02:27:07.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.905+0000 7f297effd640 1 -- 172.21.15.28:0/1866066967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f29780046c0 con 0x7f2988101ca0 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.905+0000 7f29929a2740 1 Processor -- start 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 -- start start 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f297effd640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 0x7f29740a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x5596710359d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559671036040 0x5596710384d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559670f95040 con 0x5596710304e0 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559670f3d9e0 con 0x559671036040 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f29929a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559670f90ff0 con 0x559671030ed0 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298db58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298e359640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 0x7f29740a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298d357640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x5596710359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298db58640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52936/0 (socket says 172.21.15.28:52936) 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298db58640 1 -- 172.21.15.28:0/2035796550 learned_addr learned my addr 172.21.15.28:0/2035796550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f297effd640 1 -- 172.21.15.28:0/1866066967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f29780cfb60 con 0x7f2988101ca0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.906+0000 7f298e359640 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559671036040 0x5596710384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 -- 172.21.15.28:0/2035796550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 msgr2=0x5596710359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x5596710359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 -- 172.21.15.28:0/2035796550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559671036040 msgr2=0x5596710384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559671036040 0x5596710384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 -- 172.21.15.28:0/2035796550 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559670ca9210 con 0x5596710304e0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298d357640 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x5596710359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298db58640 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f2980005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a618df32bd8169b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f298e359640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 0x7f29740a5500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2984009fd0 tx=0x7f2984009a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f2973fff640 1 -- 172.21.15.28:0/2035796550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980043070 con 0x5596710304e0 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f2973fff640 1 -- 172.21.15.28:0/2035796550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2980032c90 con 0x5596710304e0 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.907+0000 7f2973fff640 1 -- 172.21.15.28:0/2035796550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2980039040 con 0x5596710304e0 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- 172.21.15.28:0/2035796550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 msgr2=0x559671030900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f2980009ca0 tx=0x7f2980005ef0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- 172.21.15.28:0/2035796550 shutdown_connections 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559671036040 0x5596710384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x5596710359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 --2- 172.21.15.28:0/2035796550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5596710304e0 0x559671030900 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- 172.21.15.28:0/2035796550 >> 172.21.15.28:0/2035796550 conn(0x55967102ba50 msgr2=0x55967102be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- 172.21.15.28:0/2035796550 shutdown_connections 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- 172.21.15.28:0/2035796550 wait complete. 2024-03-22T02:27:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 -- start start 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.908+0000 7f29929a2740 1 Processor -- start 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x55967103a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671041980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559671038f90 con 0x559671036040 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559670f50db0 con 0x5596710304e0 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559671039110 con 0x559671030ed0 2024-03-22T02:27:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29929a2740 10 client.0 _ll_register_callbacks cb 0x559670ca8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298d357640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x55967103a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298e359640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671041980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44340/0 (socket says 172.21.15.28:44340) 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298d357640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x55967103a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45198/0 (socket says 172.21.15.28:45198) 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 -- 172.21.15.28:0/3334301522 learned_addr learned my addr 172.21.15.28:0/3334301522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298e359640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671041980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52952/0 (socket says 172.21.15.28:52952) 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 -- 172.21.15.28:0/3334301522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 msgr2=0x55967103a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x55967103a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 -- 172.21.15.28:0/3334301522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 msgr2=0x559671041980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671041980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 -- 172.21.15.28:0/3334301522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559670d75620 con 0x5596710304e0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f298db58640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f298002c630 tx=0x7f298002c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b426003b851d49ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.909+0000 7f29717fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298004c030 con 0x5596710304e0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2980032c90 con 0x5596710304e0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f29800347f0 con 0x5596710304e0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f2980059050 con 0x5596710304e0 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 10 client.29237 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x559671044cf0 con 0x5596710304e0 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2980032c90 con 0x5596710304e0 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f2980048af0 con 0x5596710304e0 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 10 client.29237 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 10 client.29237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 20 client.29237 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559671046040 0x559671048430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 4 client.29237 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:07.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559670fa2580 con 0x559671046040 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.910+0000 7f298d357640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559671046040 0x559671048430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29717fa640 10 client.29237 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f298d357640 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559671046040 0x559671048430 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=725176353b0b4979 server_cookie=270be203de17cadc in_seq=0 out_seq=0 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29717fa640 1 -- 172.21.15.28:0/3334301522 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559670fa2580 con 0x559671046040 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29717fa640 10 client.29237 handle_command_reply: tid=0 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29929a2740 1 client.29237 shutdown 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 msgr2=0x55967103a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.911+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f298002c630 tx=0x7f298002c810 comp rx=0 tx=0).stop 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 shutdown_connections 2024-03-22T02:27:07.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559671046040 0x559671048430 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671041980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x55967103a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29717fa640 0 client.29237 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 --2- 172.21.15.28:0/3334301522 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103a400 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 >> 172.21.15.28:0/3334301522 conn(0x55967102ba50 msgr2=0x55967102ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 shutdown_connections 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 1 -- 172.21.15.28:0/3334301522 wait complete. 2024-03-22T02:27:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.912+0000 7f29929a2740 20 client.29237 trim_cache size 0 max 16384 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 Processor -- start 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 -- start start 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967102f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x559671041940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29881af470 con 0x559671036040 2024-03-22T02:27:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29881acce0 con 0x5596710304e0 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f29929a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f29881aab30 con 0x559671030ed0 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52966/0 (socket says 172.21.15.28:52966) 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 -- 172.21.15.28:0/3873584591 learned_addr learned my addr 172.21.15.28:0/3873584591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298db58640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967102f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298d357640 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x559671041940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 -- 172.21.15.28:0/3873584591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 msgr2=0x559671041940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x559671041940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 -- 172.21.15.28:0/3873584591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 msgr2=0x55967102f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967102f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298e359640 1 -- 172.21.15.28:0/3873584591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55967102c4c0 con 0x559671036040 2024-03-22T02:27:07.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.927+0000 7f298db58640 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967102f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f298e359640 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f29840136d0 tx=0x7f2984005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d055f8f43b1448 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29737fe640 1 -- 172.21.15.28:0/3873584591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29840130c0 con 0x559671036040 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29737fe640 1 -- 172.21.15.28:0/3873584591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2984013260 con 0x559671036040 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29737fe640 1 -- 172.21.15.28:0/3873584591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298402b880 con 0x559671036040 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 -- 172.21.15.28:0/3873584591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 msgr2=0x559671039a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f29840136d0 tx=0x7f2984005f90 comp rx=0 tx=0).stop 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 -- 172.21.15.28:0/3873584591 shutdown_connections 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671036040 0x559671039a20 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671030ed0 0x559671041940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 --2- 172.21.15.28:0/3873584591 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967102f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 -- 172.21.15.28:0/3873584591 >> 172.21.15.28:0/3873584591 conn(0x55967102ba50 msgr2=0x5596710321d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 -- 172.21.15.28:0/3873584591 shutdown_connections 2024-03-22T02:27:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.928+0000 7f29929a2740 1 -- 172.21.15.28:0/3873584591 wait complete. 2024-03-22T02:27:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 -- start start 2024-03-22T02:27:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 Processor -- start 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 0x5596710403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29881ab1a0 con 0x559671030ed0 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f29881ac610 con 0x5596710304e0 2024-03-22T02:27:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f29881ab870 con 0x559671036040 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f29929a2740 10 client.0 _ll_register_callbacks cb 0x7f2984013460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f298d357640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f298d357640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52980/0 (socket says 172.21.15.28:52980) 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f298d357640 1 -- 172.21.15.28:0/4231994682 learned_addr learned my addr 172.21.15.28:0/4231994682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f298e359640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 0x5596710403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.929+0000 7f298db58640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 -- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 msgr2=0x55967103eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 -- 172.21.15.28:0/4231994682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 msgr2=0x55967103f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 -- 172.21.15.28:0/4231994682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596710409b0 con 0x559671036040 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298d357640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f298e359640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 0x5596710403c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2984009a30 tx=0x7f29840098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f162c91b53b3ed server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298402f070 con 0x559671036040 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2984013260 con 0x559671036040 2024-03-22T02:27:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f29840037f0 con 0x559671036040 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f29840130c0 con 0x559671036040 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29929a2740 10 client.29242 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5596710ac370 con 0x559671036040 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2984013260 con 0x559671036040 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.930+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f2984020050 con 0x559671036040 2024-03-22T02:27:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 10 client.29242 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 10 client.29242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 20 client.29242 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5596710acb90 0x5596710aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 4 client.29242 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:27:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559670f9c8d0 con 0x5596710acb90 2024-03-22T02:27:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f298db58640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5596710acb90 0x5596710aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f29717fa640 10 client.29242 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:07.931+0000 7f298db58640 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5596710acb90 0x5596710aefb0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d6c3a96034569d8 server_cookie=f3a60d03e6ac9b37 in_seq=0 out_seq=0 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29717fa640 1 -- 172.21.15.28:0/4231994682 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559670f9c8d0 con 0x5596710acb90 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29717fa640 10 client.29242 handle_command_reply: tid=0 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 client.29242 shutdown 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 msgr2=0x5596710403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 0x5596710403c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2984009a30 tx=0x7f29840098b0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 shutdown_connections 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5596710acb90 0x5596710aefb0 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559671036040 0x5596710403c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.162 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559671030ed0 0x55967103f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 --2- 172.21.15.28:0/4231994682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5596710304e0 0x55967103eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 >> 172.21.15.28:0/4231994682 conn(0x55967102ba50 msgr2=0x5596710315c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.160+0000 7f29717fa640 0 client.29242 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 shutdown_connections 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f29929a2740 1 -- 172.21.15.28:0/4231994682 wait complete. 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f29929a2740 20 client.29242 trim_cache size 0 max 16384 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:27:08.163 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:27:08.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:27:08.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:27:08.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 msgr2=0x7f29740a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:08.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 0x7f29740a5500 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f2984009fd0 tx=0x7f2984009a60 comp rx=0 tx=0).stop 2024-03-22T02:27:08.164 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 msgr2=0x7f2988103080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f297800e9f0 tx=0x7f297800edf0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 shutdown_connections 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f29740a3040 0x7f29740a5500 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2988102660 0x7f29881035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2988101ca0 0x7f2988103080 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 --2- 172.21.15.28:0/1866066967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2988100ab0 0x7f29881049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 >> 172.21.15.28:0/1866066967 conn(0x7f29880fc020 msgr2=0x7f29880fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 shutdown_connections 2024-03-22T02:27:08.165 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:08.161+0000 7f298fdbc640 1 -- 172.21.15.28:0/1866066967 wait complete. 2024-03-22T02:27:08.440 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1399707 11.08 MB/sec execute 590 sec latency 1266.548 ms 2024-03-22T02:27:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:08 smithi120 ceph-mon[36803]: pgmap v767: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.2 MiB/s wr, 336 op/s 2024-03-22T02:27:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:08 smithi083 ceph-mon[36340]: pgmap v767: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.2 MiB/s wr, 336 op/s 2024-03-22T02:27:09.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:08 smithi028 ceph-mon[31916]: pgmap v767: 129 pgs: 129 active+clean; 40 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 2.2 MiB/s wr, 336 op/s 2024-03-22T02:27:09.442 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1399707 11.06 MB/sec execute 591 sec latency 2266.601 ms 2024-03-22T02:27:10.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1399733 11.05 MB/sec execute 592 sec latency 3219.132 ms 2024-03-22T02:27:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:10 smithi120 ceph-mon[36803]: pgmap v768: 129 pgs: 129 active+clean; 37 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:27:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:10 smithi083 ceph-mon[36340]: pgmap v768: 129 pgs: 129 active+clean; 37 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:27:11.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:10 smithi028 ceph-mon[31916]: pgmap v768: 129 pgs: 129 active+clean; 37 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-03-22T02:27:11.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1401014 11.04 MB/sec execute 593 sec latency 145.483 ms 2024-03-22T02:27:12.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1403254 11.04 MB/sec execute 594 sec latency 104.905 ms 2024-03-22T02:27:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:12 smithi120 ceph-mon[36803]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-22T02:27:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:12 smithi083 ceph-mon[36340]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-22T02:27:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:12 smithi028 ceph-mon[31916]: pgmap v769: 129 pgs: 129 active+clean; 38 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.5 MiB/s wr, 262 op/s 2024-03-22T02:27:13.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1403254 11.02 MB/sec execute 595 sec latency 1105.027 ms 2024-03-22T02:27:14.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1403254 11.00 MB/sec execute 596 sec latency 2105.077 ms 2024-03-22T02:27:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:14 smithi120 ceph-mon[36803]: pgmap v770: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-03-22T02:27:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:14 smithi083 ceph-mon[36340]: pgmap v770: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-03-22T02:27:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:14 smithi028 ceph-mon[31916]: pgmap v770: 129 pgs: 129 active+clean; 39 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 2.4 MiB/s wr, 264 op/s 2024-03-22T02:27:15.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1403280 10.99 MB/sec execute 597 sec latency 3056.930 ms 2024-03-22T02:27:16.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1404631 10.98 MB/sec execute 598 sec latency 65.898 ms 2024-03-22T02:27:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:16 smithi120 ceph-mon[36803]: pgmap v771: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.7 MiB/s wr, 287 op/s 2024-03-22T02:27:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:16 smithi083 ceph-mon[36340]: pgmap v771: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.7 MiB/s wr, 287 op/s 2024-03-22T02:27:17.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:16 smithi028 ceph-mon[31916]: pgmap v771: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 2.7 MiB/s wr, 287 op/s 2024-03-22T02:27:17.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 1406799 10.98 MB/sec execute 599 sec latency 230.856 ms 2024-03-22T02:27:18.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 cleanup 600 sec 2024-03-22T02:27:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:18 smithi120 ceph-mon[36803]: pgmap v772: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.2 MiB/s wr, 242 op/s 2024-03-22T02:27:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:18 smithi083 ceph-mon[36340]: pgmap v772: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.2 MiB/s wr, 242 op/s 2024-03-22T02:27:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:18 smithi028 ceph-mon[31916]: pgmap v772: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 2.2 MiB/s wr, 242 op/s 2024-03-22T02:27:19.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 cleanup 601 sec 2024-03-22T02:27:20.441 INFO:tasks.workunit.client.0.smithi028.stdout: 1 cleanup 602 sec 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: 0 cleanup 602 sec 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: Operation Count AvgLat MaxLat 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: ---------------------------------------- 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: NTCreateX 209180 1.123 4175.328 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: Close 153691 0.003 0.097 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: Rename 8856 4.887 216.259 2024-03-22T02:27:20.570 INFO:tasks.workunit.client.0.smithi028.stdout: Unlink 42208 1.803 292.314 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Deltree 6 85.938 116.657 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Mkdir 3 0.005 0.006 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Qpathinfo 189608 0.742 259.776 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Qfileinfo 33270 0.002 0.032 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Qfsinfo 34754 0.164 8.993 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Sfileinfo 17032 0.022 47.683 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Find 73334 0.237 145.475 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: WriteX 104443 0.014 6.488 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: ReadX 328047 0.031 167.662 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: LockX 682 1.087 31.875 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: UnlockX 682 1.022 30.771 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: Flush 14657 4.575 259.499 2024-03-22T02:27:20.571 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:27:20.572 INFO:tasks.workunit.client.0.smithi028.stdout:Throughput 10.9806 MB/sec 1 clients 1 procs max_latency=4175.335 ms 2024-03-22T02:27:20.572 INFO:tasks.workunit.client.0.smithi028.stderr:+ dbench 10 2024-03-22T02:27:20.572 INFO:tasks.workunit.client.0.smithi028.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-03-22T02:27:20.572 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:27:20.572 INFO:tasks.workunit.client.0.smithi028.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-03-22T02:27:20.573 INFO:tasks.workunit.client.0.smithi028.stdout:9 of 10 processes prepared for launch 0 sec 2024-03-22T02:27:20.672 INFO:tasks.workunit.client.0.smithi028.stdout:10 of 10 processes prepared for launch 0 sec 2024-03-22T02:27:20.672 INFO:tasks.workunit.client.0.smithi028.stdout:releasing clients 2024-03-22T02:27:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:20 smithi120 ceph-mon[36803]: pgmap v773: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-03-22T02:27:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:20 smithi083 ceph-mon[36340]: pgmap v773: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-03-22T02:27:21.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:20 smithi028 ceph-mon[31916]: pgmap v773: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 -- 172.21.15.28:0/1494845301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 msgr2=0x7fdc0010afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1494845301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0010afc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009860 tx=0x7fdbf002bb70 comp rx=0 tx=0).stop 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 -- 172.21.15.28:0/1494845301 shutdown_connections 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1494845301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc0010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1494845301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 0x7fdc0010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1494845301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0010afc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.301+0000 7fdc0818a640 1 -- 172.21.15.28:0/1494845301 >> 172.21.15.28:0/1494845301 conn(0x7fdc0006ebd0 msgr2=0x7fdc00071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.306+0000 7fdc0818a640 1 -- 172.21.15.28:0/1494845301 shutdown_connections 2024-03-22T02:27:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.308+0000 7fdc0818a640 1 -- 172.21.15.28:0/1494845301 wait complete. 2024-03-22T02:27:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.308+0000 7fdc0818a640 1 Processor -- start 2024-03-22T02:27:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 -- start start 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0019e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 0x7fdc0019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00111e10 con 0x7fdc0010b500 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00111c90 con 0x7fdc0010deb0 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.312+0000 7fdc0818a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00111f90 con 0x7fdc00108b90 2024-03-22T02:27:21.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.313+0000 7fdc06727640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.313+0000 7fdc06727640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57650/0 (socket says 172.21.15.28:57650) 2024-03-22T02:27:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.313+0000 7fdc06727640 1 -- 172.21.15.28:0/1802952802 learned_addr learned my addr 172.21.15.28:0/1802952802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:21.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.315+0000 7fdc05f26640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0019e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.316+0000 7fdc05725640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 0x7fdc0019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.317+0000 7fdc06727640 1 -- 172.21.15.28:0/1802952802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 msgr2=0x7fdc0019e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.317+0000 7fdc06727640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0019e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.320 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.317+0000 7fdc06727640 1 -- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 msgr2=0x7fdc0019e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.317+0000 7fdc06727640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 0x7fdc0019e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.317+0000 7fdc06727640 1 -- 172.21.15.28:0/1802952802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc001a5fe0 con 0x7fdc0010deb0 2024-03-22T02:27:21.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.318+0000 7fdc06727640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800cad0 tx=0x7fdbf800ced0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.320+0000 7fdbf6ffd640 1 -- 172.21.15.28:0/1802952802 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800ad60 con 0x7fdc0010deb0 2024-03-22T02:27:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.321+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc001a62d0 con 0x7fdc0010deb0 2024-03-22T02:27:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.321+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001a6820 con 0x7fdc0010deb0 2024-03-22T02:27:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.321+0000 7fdbf6ffd640 1 -- 172.21.15.28:0/1802952802 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf8005020 con 0x7fdc0010deb0 2024-03-22T02:27:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.321+0000 7fdbf6ffd640 1 -- 172.21.15.28:0/1802952802 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800f6b0 con 0x7fdc0010deb0 2024-03-22T02:27:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.326+0000 7fdc0ad75740 1 Processor -- start 2024-03-22T02:27:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.326+0000 7fdbf6ffd640 1 -- 172.21.15.28:0/1802952802 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdbf800f850 con 0x7fdc0010deb0 2024-03-22T02:27:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.327+0000 7fdbf6ffd640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 0x7fdbec0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.328+0000 7fdbf6ffd640 1 -- 172.21.15.28:0/1802952802 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdbf80d0740 con 0x7fdc0010deb0 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 -- start start 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a1ed0 0x564b755a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a7040 0x564b755a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b75506040 con 0x564b755a14e0 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b754ae9e0 con 0x564b755a7040 2024-03-22T02:27:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b75501ff0 con 0x564b755a1ed0 2024-03-22T02:27:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.329+0000 7fdc05725640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 0x7fdbec0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.331+0000 7fdc05f26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.331+0000 7fdc05f26640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44162/0 (socket says 172.21.15.28:44162) 2024-03-22T02:27:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.331+0000 7fdc05f26640 1 -- 172.21.15.28:0/237115887 learned_addr learned my addr 172.21.15.28:0/237115887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:21.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.331+0000 7fdc06727640 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a7040 0x564b755a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05f26640 1 -- 172.21.15.28:0/237115887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a1ed0 msgr2=0x564b755a69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05725640 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a1ed0 0x564b755a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05f26640 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a1ed0 0x564b755a69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05f26640 1 -- 172.21.15.28:0/237115887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a7040 msgr2=0x564b755a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05f26640 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a7040 0x564b755a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.332+0000 7fdc05f26640 1 -- 172.21.15.28:0/237115887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b7521a210 con 0x564b755a14e0 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.333+0000 7fdc05f26640 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0037040 tx=0x7fdbf0009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e705323473c81a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.334+0000 7fdbebfff640 1 -- 172.21.15.28:0/237115887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0039040 con 0x564b755a14e0 2024-03-22T02:27:21.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.334+0000 7fdbebfff640 1 -- 172.21.15.28:0/237115887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0034db0 con 0x564b755a14e0 2024-03-22T02:27:21.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 -- 172.21.15.28:0/237115887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 msgr2=0x564b755a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0037040 tx=0x7fdbf0009510 comp rx=0 tx=0).stop 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc05725640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 0x7fdbec0a5670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdc0019f8d0 tx=0x7fdbfc01a800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 -- 172.21.15.28:0/237115887 shutdown_connections 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a7040 0x564b755a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a1ed0 0x564b755a69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/237115887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755a1900 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.335+0000 7fdc0ad75740 1 -- 172.21.15.28:0/237115887 >> 172.21.15.28:0/237115887 conn(0x564b7559ca50 msgr2=0x564b7559ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.339 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.338+0000 7fdc0ad75740 1 -- 172.21.15.28:0/237115887 shutdown_connections 2024-03-22T02:27:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.341+0000 7fdc0ad75740 1 -- 172.21.15.28:0/237115887 wait complete. 2024-03-22T02:27:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.341+0000 7fdc0ad75740 1 -- start start 2024-03-22T02:27:21.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.341+0000 7fdc0ad75740 1 Processor -- start 2024-03-22T02:27:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a7040 0x564b755b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b755a9fe0 con 0x564b755a14e0 2024-03-22T02:27:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b754c1db0 con 0x564b755a1ed0 2024-03-22T02:27:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b755aa160 con 0x564b755a7040 2024-03-22T02:27:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.342+0000 7fdc0ad75740 10 client.0 _ll_register_callbacks cb 0x564b75219a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:21.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.343+0000 7fdc05725640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.343+0000 7fdc05725640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57670/0 (socket says 172.21.15.28:57670) 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.343+0000 7fdc05725640 1 -- 172.21.15.28:0/2583608543 learned_addr learned my addr 172.21.15.28:0/2583608543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.343+0000 7fdc05f26640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 -- 172.21.15.28:0/2583608543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a7040 msgr2=0x564b755b29e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a7040 0x564b755b29e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 -- 172.21.15.28:0/2583608543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 msgr2=0x564b755ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 -- 172.21.15.28:0/2583608543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b752e6620 con 0x564b755a1ed0 2024-03-22T02:27:21.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05f26640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdc05725640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc01acf0 tx=0x7fdbfc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a5d5b6c44c60418 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.344+0000 7fdbe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:27:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.345+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc003d50 con 0x564b755a1ed0 2024-03-22T02:27:21.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.345+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdbfc005020 con 0x564b755a1ed0 2024-03-22T02:27:21.347 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.346+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf0034db0 con 0x564b755a1ed0 2024-03-22T02:27:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.347+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdbfc005220 con 0x564b755a1ed0 2024-03-22T02:27:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.347+0000 7fdc0ad75740 10 client.29252 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:21.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.348+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564b755b5df0 con 0x564b755a1ed0 2024-03-22T02:27:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.350+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf0034db0 con 0x564b755a1ed0 2024-03-22T02:27:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.350+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fdbfc042610 con 0x564b755a1ed0 2024-03-22T02:27:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.351+0000 7fdc0ad75740 10 client.29252 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.351+0000 7fdc0ad75740 10 client.29252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:21.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.351+0000 7fdc0ad75740 20 client.29252 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.352+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b755b7480 0x564b755b9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.352+0000 7fdc0ad75740 4 client.29252 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.352+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b75513580 con 0x564b755b7480 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.352+0000 7fdc05f26640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b755b7480 0x564b755b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.353+0000 7fdc05f26640 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b755b7480 0x564b755b9940 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5f8c05de45b839d server_cookie=b268193b570381e4 in_seq=0 out_seq=0 2024-03-22T02:27:21.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.353+0000 7fdbe97fa640 10 client.29252 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.356+0000 7fdbe97fa640 1 -- 172.21.15.28:0/2583608543 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b75513580 con 0x564b755b7480 2024-03-22T02:27:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.356+0000 7fdbe97fa640 10 client.29252 handle_command_reply: tid=0 2024-03-22T02:27:21.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.356+0000 7fdc0ad75740 1 client.29252 shutdown 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.356+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 msgr2=0x564b755ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.356+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc01acf0 tx=0x7fdbfc004730 comp rx=0 tx=0).stop 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 shutdown_connections 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b755b7480 0x564b755b9940 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a7040 0x564b755b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755ab9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/2583608543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a14e0 0x564b755ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 >> 172.21.15.28:0/2583608543 conn(0x564b7559ca50 msgr2=0x564b7559bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdbe97fa640 0 client.29252 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 shutdown_connections 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 1 -- 172.21.15.28:0/2583608543 wait complete. 2024-03-22T02:27:21.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.357+0000 7fdc0ad75740 20 client.29252 trim_cache size 0 max 16384 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 Processor -- start 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 -- start start 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe4012850 con 0x564b755a7040 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe40100c0 con 0x564b755a1ed0 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe400df10 con 0x564b755a14e0 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44180/0 (socket says 172.21.15.28:44180) 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 -- 172.21.15.28:0/55240648 learned_addr learned my addr 172.21.15.28:0/55240648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc05725640 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc05f26640 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 -- 172.21.15.28:0/55240648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 msgr2=0x564b755a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 -- 172.21.15.28:0/55240648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 msgr2=0x564b755b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.382+0000 7fdc06727640 1 -- 172.21.15.28:0/55240648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b75598280 con 0x564b755a7040 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.383+0000 7fdc06727640 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8098e60 tx=0x7fdbf809cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e5fb30d006c05b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.383+0000 7fdbeb7fe640 1 -- 172.21.15.28:0/55240648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf801d040 con 0x564b755a7040 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.383+0000 7fdbeb7fe640 1 -- 172.21.15.28:0/55240648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf80cdee0 con 0x564b755a7040 2024-03-22T02:27:21.385 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.383+0000 7fdbeb7fe640 1 -- 172.21.15.28:0/55240648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf800fda0 con 0x564b755a7040 2024-03-22T02:27:21.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- 172.21.15.28:0/55240648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 msgr2=0x564b7559b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.389 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8098e60 tx=0x7fdbf809cb40 comp rx=0 tx=0).stop 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- 172.21.15.28:0/55240648 shutdown_connections 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b7559b3c0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/55240648 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- 172.21.15.28:0/55240648 >> 172.21.15.28:0/55240648 conn(0x564b7559ca50 msgr2=0x564b7559fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- 172.21.15.28:0/55240648 shutdown_connections 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- 172.21.15.28:0/55240648 wait complete. 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.384+0000 7fdc0ad75740 1 -- start start 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 Processor -- start 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b755b1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe400e580 con 0x564b755a7040 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe400f9f0 con 0x564b755a1ed0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe400ec50 con 0x564b755a14e0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53058/0 (socket says 172.21.15.28:53058) 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 -- 172.21.15.28:0/111299584 learned_addr learned my addr 172.21.15.28:0/111299584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc06727640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b755b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05725640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc0ad75740 10 client.0 _ll_register_callbacks cb 0x7fdbfc024f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 -- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 msgr2=0x564b755b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 -- 172.21.15.28:0/111299584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 msgr2=0x564b755b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b755b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05f26640 1 -- 172.21.15.28:0/111299584 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b755b1bf0 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.385+0000 7fdc05725640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdc05f26640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005b60 tx=0x7fdbf0032500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=304ef4b5de0c9376 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdbe97fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf005a060 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdbf002ca70 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf80cdee0 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fdbf0061050 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdc0ad75740 10 client.29254 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564b7561d880 con 0x564b755a14e0 2024-03-22T02:27:21.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.386+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf80cdee0 con 0x564b755a14e0 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fdbf00050e0 con 0x564b755a14e0 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 10 client.29254 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 10 client.29254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 20 client.29254 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b7561dea0 0x564b75620290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 4 client.29254 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b7550d8d0 con 0x564b7561dea0 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc05725640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b7561dea0 0x564b75620290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdc05725640 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b7561dea0 0x564b75620290 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95a78166f9a581d3 server_cookie=5358a892f5e54f95 in_seq=0 out_seq=0 2024-03-22T02:27:21.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.387+0000 7fdbe97fa640 10 client.29254 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:21.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 233 89.62 MB/sec warmup 1 sec latency 196.097 ms 2024-03-22T02:27:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdbe97fa640 1 -- 172.21.15.28:0/111299584 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b7550d8d0 con 0x564b7561dea0 2024-03-22T02:27:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdbe97fa640 10 client.29254 handle_command_reply: tid=0 2024-03-22T02:27:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 client.29254 shutdown 2024-03-22T02:27:21.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 msgr2=0x564b755b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0005b60 tx=0x7fdbf0032500 comp rx=0 tx=0).stop 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 shutdown_connections 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b7561dea0 0x564b75620290 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b755a7040 0x564b755b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b755a1ed0 0x564b755b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 --2- 172.21.15.28:0/111299584 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b755a14e0 0x564b755b00b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 >> 172.21.15.28:0/111299584 conn(0x564b7559ca50 msgr2=0x564b755a0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.928+0000 7fdbe97fa640 0 client.29254 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.929+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 shutdown_connections 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.929+0000 7fdc0ad75740 1 -- 172.21.15.28:0/111299584 wait complete. 2024-03-22T02:27:21.930 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.929+0000 7fdc0ad75740 20 client.29254 trim_cache size 0 max 16384 2024-03-22T02:27:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:27:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:27:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:27:21.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 msgr2=0x7fdbec0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 0x7fdbec0a5670 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fdc0019f8d0 tx=0x7fdbfc01a800 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 msgr2=0x7fdc001a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdbf800cad0 tx=0x7fdbf800ced0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 shutdown_connections 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdbec0a31b0 0x7fdbec0a5670 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc0010deb0 0x7fdc001a5970 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0010b500 0x7fdc0019e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 --2- 172.21.15.28:0/1802952802 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc00108b90 0x7fdc0019e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:21.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 >> 172.21.15.28:0/1802952802 conn(0x7fdc0006ebd0 msgr2=0x7fdc00109280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 shutdown_connections 2024-03-22T02:27:21.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:21.930+0000 7fdc0818a640 1 -- 172.21.15.28:0/1802952802 wait complete. 2024-03-22T02:27:21.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:27:21.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:27:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:27:22.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:27:22.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:27:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:27:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:27:22.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:27:22.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 423 74.72 MB/sec warmup 2 sec latency 298.270 ms 2024-03-22T02:27:22.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:22 smithi028 ceph-mon[31916]: pgmap v774: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-03-22T02:27:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:22 smithi120 ceph-mon[36803]: pgmap v774: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-03-22T02:27:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:22 smithi083 ceph-mon[36340]: pgmap v774: 129 pgs: 129 active+clean; 35 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 2.2 MiB/s wr, 220 op/s 2024-03-22T02:27:23.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 763 70.12 MB/sec warmup 3 sec latency 841.231 ms 2024-03-22T02:27:24.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 1148 56.27 MB/sec warmup 4 sec latency 1841.259 ms 2024-03-22T02:27:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:24 smithi120 ceph-mon[36803]: pgmap v775: 129 pgs: 129 active+clean; 34 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-03-22T02:27:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:24 smithi083 ceph-mon[36340]: pgmap v775: 129 pgs: 129 active+clean; 34 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-03-22T02:27:25.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:24 smithi028 ceph-mon[31916]: pgmap v775: 129 pgs: 129 active+clean; 34 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 2.3 MiB/s wr, 189 op/s 2024-03-22T02:27:25.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 1624 51.56 MB/sec warmup 5 sec latency 2023.842 ms 2024-03-22T02:27:26.440 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:27:26.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:27:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- 172.21.15.28:0/3192032180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94101ba0 msgr2=0x7fda94101fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 --2- 172.21.15.28:0/3192032180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94101ba0 0x7fda94101fc0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fda90009900 tx=0x7fda9002bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- 172.21.15.28:0/3192032180 shutdown_connections 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 --2- 172.21.15.28:0/3192032180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94103750 0x7fda94109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 --2- 172.21.15.28:0/3192032180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94102d90 0x7fda94103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 --2- 172.21.15.28:0/3192032180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94101ba0 0x7fda94101fc0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- 172.21.15.28:0/3192032180 >> 172.21.15.28:0/3192032180 conn(0x7fda940fd130 msgr2=0x7fda940ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- 172.21.15.28:0/3192032180 shutdown_connections 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- 172.21.15.28:0/3192032180 wait complete. 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 Processor -- start 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.574+0000 7fda9c356640 1 -- start start 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94101ba0 0x7fda94197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94103750 0x7fda9419f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda9410d7f0 con 0x7fda94102d90 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fda9410d670 con 0x7fda94101ba0 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda9410d970 con 0x7fda94103750 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56238/0 (socket says 172.21.15.28:56238) 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 -- 172.21.15.28:0/1055991731 learned_addr learned my addr 172.21.15.28:0/1055991731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94101ba0 0x7fda94197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9a8f3640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94103750 0x7fda9419f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 -- 172.21.15.28:0/1055991731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94103750 msgr2=0x7fda9419f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94103750 0x7fda9419f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 -- 172.21.15.28:0/1055991731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94101ba0 msgr2=0x7fda94197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94101ba0 0x7fda94197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 -- 172.21.15.28:0/1055991731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9419fb50 con 0x7fda94102d90 2024-03-22T02:27:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda998f1640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fda8400e9f0 tx=0x7fda8400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda8b7fe640 1 -- 172.21.15.28:0/1055991731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8400ccc0 con 0x7fda94102d90 2024-03-22T02:27:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9419fe40 con 0x7fda94102d90 2024-03-22T02:27:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda8b7fe640 1 -- 172.21.15.28:0/1055991731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda84010910 con 0x7fda94102d90 2024-03-22T02:27:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.575+0000 7fda8b7fe640 1 -- 172.21.15.28:0/1055991731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840188a0 con 0x7fda94102d90 2024-03-22T02:27:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.576+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda941a0350 con 0x7fda94102d90 2024-03-22T02:27:26.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.577+0000 7fda8b7fe640 1 -- 172.21.15.28:0/1055991731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fda84010ac0 con 0x7fda94102d90 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 Processor -- start 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 -- start start 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d50850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d58420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557b08cb58c0 con 0x557b08d50430 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557b08c5dc20 con 0x557b08d50e20 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda9f01d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557b08cb1870 con 0x557b08d55f90 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.578+0000 7fda8b7fe640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 0x7fda800a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a8f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d50850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53748/0 (socket says 172.21.15.28:53748) 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 -- 172.21.15.28:0/1932904896 learned_addr learned my addr 172.21.15.28:0/1932904896 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda998f1640 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d58420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a8f3640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 0x7fda800a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda8b7fe640 1 -- 172.21.15.28:0/1055991731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fda840cfc10 con 0x7fda94102d90 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 -- 172.21.15.28:0/1932904896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 msgr2=0x557b08d58420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d58420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 -- 172.21.15.28:0/1932904896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 msgr2=0x557b08d50850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d50850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 -- 172.21.15.28:0/1932904896 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b08c82150 con 0x557b08d50e20 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a8f3640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 0x7fda800a53e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x557b08d51680 tx=0x7fda8c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.579+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fda90009900 tx=0x7fda90009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2d96bb97508a73 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda88bfe640 1 -- 172.21.15.28:0/1932904896 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9002cb80 con 0x557b08d50e20 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda88bfe640 1 -- 172.21.15.28:0/1932904896 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda90005040 con 0x557b08d50e20 2024-03-22T02:27:26.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda88bfe640 1 -- 172.21.15.28:0/1932904896 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda90004920 con 0x557b08d50e20 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 -- 172.21.15.28:0/1932904896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 msgr2=0x557b08d55920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fda90009900 tx=0x7fda90009560 comp rx=0 tx=0).stop 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 -- 172.21.15.28:0/1932904896 shutdown_connections 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d58420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d55920 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 --2- 172.21.15.28:0/1932904896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d50850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 -- 172.21.15.28:0/1932904896 >> 172.21.15.28:0/1932904896 conn(0x557b08d4b9a0 msgr2=0x557b08d4bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.580+0000 7fda9f01d740 1 -- 172.21.15.28:0/1932904896 shutdown_connections 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 -- 172.21.15.28:0/1932904896 wait complete. 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 -- start start 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 Processor -- start 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d5a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d61ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557b08d58ee0 con 0x557b08d50e20 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x557b08c71130 con 0x557b08d50430 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56250/0 (socket says 172.21.15.28:56250) 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda998f1640 1 -- 172.21.15.28:0/2535759985 learned_addr learned my addr 172.21.15.28:0/2535759985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d5a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x557b08d59060 con 0x557b08d55f90 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9a8f3640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d61ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.581+0000 7fda9f01d740 10 client.0 _ll_register_callbacks cb 0x557b08c70b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 -- 172.21.15.28:0/2535759985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 msgr2=0x557b08d61ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d61ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 -- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 msgr2=0x557b08d5a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d5a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 -- 172.21.15.28:0/2535759985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b08a937d0 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d5a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda90005040 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda998f1640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda840cd300 tx=0x7fda8400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf32db52eaf24404 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda7dffb640 10 client.19455 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda7dffb640 1 -- 172.21.15.28:0/2535759985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda84026040 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda7dffb640 1 -- 172.21.15.28:0/2535759985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fda840850b0 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda7dffb640 1 -- 172.21.15.28:0/2535759985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fda8409c630 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 10 client.19455 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=41}) -- 0x557b089c93f0 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda7dffb640 1 -- 172.21.15.28:0/2535759985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fda840a44b0 con 0x557b08d50e20 2024-03-22T02:27:26.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 10 client.19455 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 10 client.19455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 20 client.19455 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08d68a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 4 client.19455 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.582+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b08cc2e00 con 0x557b08d66590 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.583+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08d68a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.583+0000 7fda7dffb640 10 client.19455 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.583+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08d68a00 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b03e10bbb4838aab server_cookie=58e81a07f86b001b in_seq=0 out_seq=0 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda7dffb640 1 -- 172.21.15.28:0/2535759985 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b08cc2e00 con 0x557b08d66590 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda7dffb640 10 client.19455 handle_command_reply: tid=0 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 client.19455 shutdown 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 msgr2=0x557b08d5ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fda840cd300 tx=0x7fda8400cef0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 shutdown_connections 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08d68a00 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d61ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda7dffb640 0 client.19455 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d5ae90 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 --2- 172.21.15.28:0/2535759985 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d5a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 >> 172.21.15.28:0/2535759985 conn(0x557b08d4b9a0 msgr2=0x557b08d4ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.584+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 shutdown_connections 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.585+0000 7fda9f01d740 1 -- 172.21.15.28:0/2535759985 wait complete. 2024-03-22T02:27:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.585+0000 7fda9f01d740 20 client.19455 trim_cache size 0 max 16384 2024-03-22T02:27:26.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 Processor -- start 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 -- start start 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d4f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d6d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda941ab070 con 0x557b08d50e20 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fda941a88e0 con 0x557b08d50430 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.602+0000 7fda9f01d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda941a6730 con 0x557b08d55f90 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda9a0f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d4f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56258/0 (socket says 172.21.15.28:56258) 2024-03-22T02:27:26.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 -- 172.21.15.28:0/4217629931 learned_addr learned my addr 172.21.15.28:0/4217629931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda9a8f3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d6d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 -- 172.21.15.28:0/4217629931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 msgr2=0x557b08d6d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d6d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 -- 172.21.15.28:0/4217629931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 msgr2=0x557b08d4f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d4f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 -- 172.21.15.28:0/4217629931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b089890f0 con 0x557b08d50e20 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d4f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda998f1640 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c6a0 tx=0x7fda840bc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff2853f564c7277 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda7ffff640 1 -- 172.21.15.28:0/4217629931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8409b890 con 0x557b08d50e20 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda7ffff640 1 -- 172.21.15.28:0/4217629931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda90005040 con 0x557b08d50e20 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.603+0000 7fda7ffff640 1 -- 172.21.15.28:0/4217629931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda840a44b0 con 0x557b08d50e20 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- 172.21.15.28:0/4217629931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 msgr2=0x557b08d61e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fda8400c6a0 tx=0x7fda840bc000 comp rx=0 tx=0).stop 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- 172.21.15.28:0/4217629931 shutdown_connections 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d6d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50e20 0x557b08d61e90 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 --2- 172.21.15.28:0/4217629931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50430 0x557b08d4f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- 172.21.15.28:0/4217629931 >> 172.21.15.28:0/4217629931 conn(0x557b08d4b9a0 msgr2=0x557b08d4ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- 172.21.15.28:0/4217629931 shutdown_connections 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- 172.21.15.28:0/4217629931 wait complete. 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.604+0000 7fda9f01d740 1 -- start start 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 Processor -- start 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d5f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d60d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fda941a6da0 con 0x557b08d50430 2024-03-22T02:27:26.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fda941a8210 con 0x557b08d50e20 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fda941a7470 con 0x557b08d55f90 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9f01d740 10 client.0 _ll_register_callbacks cb 0x7fda8409bd00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9a0f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d5f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53778/0 (socket says 172.21.15.28:53778) 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 -- 172.21.15.28:0/3438147556 learned_addr learned my addr 172.21.15.28:0/3438147556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9a8f3640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d60d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 -- 172.21.15.28:0/3438147556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 msgr2=0x557b08d60d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d60d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 -- 172.21.15.28:0/3438147556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 msgr2=0x557b08d5f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d5f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 -- 172.21.15.28:0/3438147556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b08d61350 con 0x557b08d50e20 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d5f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda998f1640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fda840a48b0 tx=0x7fda8400cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcd3b95bad111b99 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.605+0000 7fda7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8407f070 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fda8409b580 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda90005040 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fda840a0080 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 10 client.29276 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x557b08dccab0 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda90005040 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fda8409af60 con 0x557b08d50e20 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 10 client.29276 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 10 client.29276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:26.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 20 client.29276 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08dcf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 4 client.29276 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.610+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557b08cbd150 con 0x557b08d66590 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.611+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08dcf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.611+0000 7fda9a0f2640 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08dcf240 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77d094c35e6cce28 server_cookie=63c23536b75ff3d3 in_seq=0 out_seq=0 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.611+0000 7fda7dffb640 10 client.29276 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda7dffb640 1 -- 172.21.15.28:0/3438147556 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557b08cbd150 con 0x557b08d66590 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda7dffb640 10 client.29276 handle_command_reply: tid=0 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 client.29276 shutdown 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 msgr2=0x557b08d5fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fda840a48b0 tx=0x7fda8400cef0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 shutdown_connections 2024-03-22T02:27:26.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x557b08d66590 0x557b08dcf240 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x557b08d55f90 0x557b08d60d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x557b08d50e20 0x557b08d5fd50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 --2- 172.21.15.28:0/3438147556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557b08d50430 0x557b08d5f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda7dffb640 0 client.29276 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 >> 172.21.15.28:0/3438147556 conn(0x557b08d4b9a0 msgr2=0x557b08d4f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 shutdown_connections 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 1 -- 172.21.15.28:0/3438147556 wait complete. 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9f01d740 20 client.29276 trim_cache size 0 max 16384 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:27:26.617 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:27:26.618 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:27:26.618 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:27:26.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 msgr2=0x7fda800a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 0x7fda800a53e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x557b08d51680 tx=0x7fda8c009500 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 msgr2=0x7fda94198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fda8400e9f0 tx=0x7fda8400edf0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 shutdown_connections 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fda800a2f20 0x7fda800a53e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fda94103750 0x7fda9419f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda94102d90 0x7fda94198470 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 --2- 172.21.15.28:0/1055991731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fda94101ba0 0x7fda94197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.612+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 >> 172.21.15.28:0/1055991731 conn(0x7fda940fd130 msgr2=0x7fda941061b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.613+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 shutdown_connections 2024-03-22T02:27:26.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.613+0000 7fda9c356640 1 -- 172.21.15.28:0/1055991731 wait complete. 2024-03-22T02:27:26.626 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:27:26.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:27:26.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 2281 45.83 MB/sec warmup 6 sec latency 129.727 ms 2024-03-22T02:27:26.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 -- 172.21.15.28:0/3429909679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308101760 msgr2=0x7f2308109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 --2- 172.21.15.28:0/3429909679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308101760 0x7f2308109000 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009ca0 tx=0x7f22f802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 -- 172.21.15.28:0/3429909679 shutdown_connections 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 --2- 172.21.15.28:0/3429909679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308101760 0x7f2308109000 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 --2- 172.21.15.28:0/3429909679 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308100da0 0x7f2308101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 --2- 172.21.15.28:0/3429909679 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308102070 0x7f2308102490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 -- 172.21.15.28:0/3429909679 >> 172.21.15.28:0/3429909679 conn(0x7f23080fb7d0 msgr2=0x7f23080fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 -- 172.21.15.28:0/3429909679 shutdown_connections 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 -- 172.21.15.28:0/3429909679 wait complete. 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.737+0000 7f230d365640 1 Processor -- start 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 -- start start 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308101760 0x7f23081983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308102070 0x7f230819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f230810d890 con 0x7f2308100da0 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f230810d710 con 0x7f2308102070 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f230d365640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f230810da10 con 0x7f2308101760 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56296/0 (socket says 172.21.15.28:56296) 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 -- 172.21.15.28:0/2315495694 learned_addr learned my addr 172.21.15.28:0/2315495694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f23067fc640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308101760 0x7f23081983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f23077fe640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308102070 0x7f230819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 -- 172.21.15.28:0/2315495694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308101760 msgr2=0x7f23081983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308101760 0x7f23081983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 -- 172.21.15.28:0/2315495694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308102070 msgr2=0x7f230819f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308102070 0x7f230819f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 -- 172.21.15.28:0/2315495694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230819faa0 con 0x7f2308100da0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f2306ffd640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9c0 tx=0x7f230000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.738+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230000cd80 con 0x7f2308100da0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.739+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230819fd90 con 0x7f2308100da0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.739+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23000035f0 con 0x7f2308100da0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.739+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300010690 con 0x7f2308100da0 2024-03-22T02:27:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.739+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23081a01d0 con 0x7f2308100da0 2024-03-22T02:27:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.740+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308107e80 con 0x7f2308100da0 2024-03-22T02:27:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.741+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f23000037a0 con 0x7f2308100da0 2024-03-22T02:27:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.741+0000 7f22fffff640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 0x7f22f40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.741+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f23000cfbf0 con 0x7f2308100da0 2024-03-22T02:27:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.741+0000 7f23067fc640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 0x7f22f40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.742+0000 7f23067fc640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 0x7f22f40a53e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2308199390 tx=0x7f22f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:26.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.745+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f230009c610 con 0x7f2308100da0 2024-03-22T02:27:26.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:26 smithi028 ceph-mon[31916]: pgmap v776: 129 pgs: 129 active+clean; 32 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-03-22T02:27:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.984+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f230802d3b0 con 0x7f2308100da0 2024-03-22T02:27:26.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.985+0000 7f22fffff640 1 -- 172.21.15.28:0/2315495694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+11867 (secure 0 0 0) 0x7f230009bf10 con 0x7f2308100da0 2024-03-22T02:27:26.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:27:26.987 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":41,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:27:04.751463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29075":{"gid":29075,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/366330553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":366330553},{"type":"v1","addr":"172.21.15.83:6837","nonce":366330553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24508,24509,24478]},"id":1}]} 2024-03-22T02:27:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 msgr2=0x7f22f40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 0x7f22f40a53e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2308199390 tx=0x7f22f000b390 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 msgr2=0x7f2308197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f230000e9c0 tx=0x7f230000ee80 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 shutdown_connections 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f22f40a2f20 0x7f22f40a53e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2308102070 0x7f230819f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2308101760 0x7f23081983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 --2- 172.21.15.28:0/2315495694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2308100da0 0x7f2308197e80 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 >> 172.21.15.28:0/2315495694 conn(0x7f23080fb7d0 msgr2=0x7f2308105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:26.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 shutdown_connections 2024-03-22T02:27:26.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:26.987+0000 7f230d365640 1 -- 172.21.15.28:0/2315495694 wait complete. 2024-03-22T02:27:26.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:27:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:26 smithi120 ceph-mon[36803]: pgmap v776: 129 pgs: 129 active+clean; 32 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-03-22T02:27:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:26 smithi083 ceph-mon[36340]: pgmap v776: 129 pgs: 129 active+clean; 32 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 3.8 MiB/s wr, 256 op/s 2024-03-22T02:27:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 -- 172.21.15.28:0/3719455516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc102640 msgr2=0x7fe1bc108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3719455516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc102640 0x7fe1bc108ed0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009ca0 tx=0x7fe1b402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 -- 172.21.15.28:0/3719455516 shutdown_connections 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3719455516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc102640 0x7fe1bc108ed0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3719455516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 0x7fe1bc102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3719455516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc100a90 0x7fe1bc100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 -- 172.21.15.28:0/3719455516 >> 172.21.15.28:0/3719455516 conn(0x7fe1bc0fc020 msgr2=0x7fe1bc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 -- 172.21.15.28:0/3719455516 shutdown_connections 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.107+0000 7fe1c315e640 1 -- 172.21.15.28:0/3719455516 wait complete. 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 Processor -- start 2024-03-22T02:27:27.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 -- start start 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 0x7fe1bc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc102640 0x7fe1bc19f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc10c710 con 0x7fe1bc100a90 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc10c590 con 0x7fe1bc101c80 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c315e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc10c890 con 0x7fe1bc102640 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56310/0 (socket says 172.21.15.28:56310) 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 -- 172.21.15.28:0/3314786028 learned_addr learned my addr 172.21.15.28:0/3314786028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 0x7fe1bc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc102640 0x7fe1bc19f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 -- 172.21.15.28:0/3314786028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc102640 msgr2=0x7fe1bc19f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc102640 0x7fe1bc19f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 -- 172.21.15.28:0/3314786028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 msgr2=0x7fe1bc1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 0x7fe1bc1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.108+0000 7fe1c0efa640 1 -- 172.21.15.28:0/3314786028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1bc19fad0 con 0x7fe1bc100a90 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000e9c0 tx=0x7fe1b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1b9ffb640 1 -- 172.21.15.28:0/3314786028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000cd80 con 0x7fe1bc100a90 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1b9ffb640 1 -- 172.21.15.28:0/3314786028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b00035f0 con 0x7fe1bc100a90 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1bc19fdc0 con 0x7fe1bc100a90 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1b9ffb640 1 -- 172.21.15.28:0/3314786028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0010690 con 0x7fe1bc100a90 2024-03-22T02:27:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.109+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1bc1a0230 con 0x7fe1bc100a90 2024-03-22T02:27:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.111+0000 7fe1b9ffb640 1 -- 172.21.15.28:0/3314786028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe1b00037a0 con 0x7fe1bc100a90 2024-03-22T02:27:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.111+0000 7fe1c5e28740 1 Processor -- start 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.111+0000 7fe1b9ffb640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 0x7fe1ac0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.111+0000 7fe1c5e28740 1 -- start start 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 0x7fe1ac0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1b9ffb640 1 -- 172.21.15.28:0/3314786028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe1b00cfb60 con 0x7fe1bc100a90 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc86c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc8bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b8dbeb8d0 con 0x564b8dc8c380 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b8db93e00 con 0x564b8dc86820 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b8dbe7880 con 0x564b8dc87210 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56316/0 (socket says 172.21.15.28:56316) 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 -- 172.21.15.28:0/21538010 learned_addr learned my addr 172.21.15.28:0/21538010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc8bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc86c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 0x7fe1ac0a53e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4006010 tx=0x7fe1b402ce20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 -- 172.21.15.28:0/21538010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 msgr2=0x564b8dc8bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc8bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 -- 172.21.15.28:0/21538010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 msgr2=0x564b8dc8e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 -- 172.21.15.28:0/21538010 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8d8ff3f0 con 0x564b8dc86820 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc8bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc86c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cd2a0 tx=0x7fe1b009c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cde1b30f81e214a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1a6ffd640 1 -- 172.21.15.28:0/21538010 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b009ffd0 con 0x564b8dc86820 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.112+0000 7fe1a6ffd640 1 -- 172.21.15.28:0/21538010 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b00a4890 con 0x564b8dc86820 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1a6ffd640 1 -- 172.21.15.28:0/21538010 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00a4660 con 0x564b8dc86820 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- 172.21.15.28:0/21538010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 msgr2=0x564b8dc86c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc86c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00cd2a0 tx=0x7fe1b009c580 comp rx=0 tx=0).stop 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- 172.21.15.28:0/21538010 shutdown_connections 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc8e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc8bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/21538010 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc86c40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- 172.21.15.28:0/21538010 >> 172.21.15.28:0/21538010 conn(0x564b8dc81d90 msgr2=0x564b8dc82190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- 172.21.15.28:0/21538010 shutdown_connections 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- 172.21.15.28:0/21538010 wait complete. 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.113+0000 7fe1c5e28740 1 -- start start 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 Processor -- start 2024-03-22T02:27:27.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc91270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc982b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b8dc8f2d0 con 0x564b8dc86820 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b8dba7300 con 0x564b8dc8c380 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b8dc8f450 con 0x564b8dc87210 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c5e28740 10 client.0 _ll_register_callbacks cb 0x564b8dbe2e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56324/0 (socket says 172.21.15.28:56324) 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4059715301 learned_addr learned my addr 172.21.15.28:0/4059715301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc91270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 msgr2=0x564b8dc91270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc91270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4059715301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 msgr2=0x564b8dc982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.114+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4059715301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8d9c97d0 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe1b009bb80 tx=0x7fe1b00cd1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2862c2aa3629efdd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b007f060 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe1b009b2d0 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b00a4890 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fe1b009ffd0 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 10 client.19479 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564b8dc9b620 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b00a4890 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fe1b00b8070 con 0x564b8dc86820 2024-03-22T02:27:27.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 10 client.19479 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 10 client.19479 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 20 client.19479 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dc9ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 4 client.19479 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.115+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b8dbf8e10 con 0x564b8dc9c970 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.116+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dc9ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.116+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dc9ed60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=674e8f108fec7e7b server_cookie=6aefbf729846a488 in_seq=0 out_seq=0 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.116+0000 7fe19bfff640 10 client.19479 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe19bfff640 1 -- 172.21.15.28:0/4059715301 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b8dbf8e10 con 0x564b8dc9c970 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe19bfff640 10 client.19479 handle_command_reply: tid=0 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 client.19479 shutdown 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 msgr2=0x564b8dc90d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fe1b009bb80 tx=0x7fe1b00cd1e0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 shutdown_connections 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dc9ed60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe19bfff640 0 client.19479 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc91270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4059715301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc90d30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 >> 172.21.15.28:0/4059715301 conn(0x564b8dc81d90 msgr2=0x564b8dc80fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 shutdown_connections 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4059715301 wait complete. 2024-03-22T02:27:27.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.117+0000 7fe1c5e28740 20 client.19479 trim_cache size 0 max 16384 2024-03-22T02:27:27.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 Processor -- start 2024-03-22T02:27:27.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 -- start start 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc98240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc80740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1ad290 con 0x564b8dc8c380 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1aab00 con 0x564b8dc86820 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1a8950 con 0x564b8dc87210 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c16fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc80740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc98240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53846/0 (socket says 172.21.15.28:53846) 2024-03-22T02:27:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4267841845 learned_addr learned my addr 172.21.15.28:0/4267841845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.131+0000 7fe1c16fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc80740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56326/0 (socket says 172.21.15.28:56326) 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4267841845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 msgr2=0x564b8dc98240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc98240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4267841845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 msgr2=0x564b8dc80740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc80740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 -- 172.21.15.28:0/4267841845 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b8dc8ecd0 con 0x564b8dc86820 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000c6a0 tx=0x7fe1b00045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3c777d089fb195e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1a67fc640 1 -- 172.21.15.28:0/4267841845 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b007f070 con 0x564b8dc86820 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1a67fc640 1 -- 172.21.15.28:0/4267841845 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b0098560 con 0x564b8dc86820 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.132+0000 7fe1a67fc640 1 -- 172.21.15.28:0/4267841845 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00a4ae0 con 0x564b8dc86820 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4267841845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 msgr2=0x564b8dc91050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000c6a0 tx=0x7fe1b00045f0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4267841845 shutdown_connections 2024-03-22T02:27:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc8c380 0x564b8dc80740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc98240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/4267841845 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc86820 0x564b8dc91050 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4267841845 >> 172.21.15.28:0/4267841845 conn(0x564b8dc81d90 msgr2=0x564b8dc90940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4267841845 shutdown_connections 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- 172.21.15.28:0/4267841845 wait complete. 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- start start 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 Processor -- start 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc961e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1a8fc0 con 0x564b8dc86820 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1aa430 con 0x564b8dc8c380 2024-03-22T02:27:27.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.133+0000 7fe1c5e28740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe1bc1a9690 con 0x564b8dc87210 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c5e28740 10 client.0 _ll_register_callbacks cb 0x7fe1a8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56328/0 (socket says 172.21.15.28:56328) 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1bbfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 -- 172.21.15.28:0/2746599174 learned_addr learned my addr 172.21.15.28:0/2746599174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c16fb640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 -- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 msgr2=0x564b8dc961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 -- 172.21.15.28:0/2746599174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 msgr2=0x564b8dc971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 -- 172.21.15.28:0/2746599174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b8dc977e0 con 0x564b8dc86820 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c0efa640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00a4360 tx=0x7fe1b009ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14192b84b0c9c70 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe19bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1b00a9070 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe1b00a4ce0 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1b0098560 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fe1b007f070 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c5e28740 10 client.19485 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.134+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x564b8dd02e30 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1b0098560 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7fe1b00ac020 con 0x564b8dc86820 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 10 client.19485 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 10 client.19485 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 20 client.19485 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dd05650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 4 client.19485 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564b8dbf3160 con 0x564b8dc9c970 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dd05650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1bbfff640 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dd05650 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ed09fc5ba1098a3a server_cookie=c74c28ea14f6ee8f in_seq=0 out_seq=0 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe19bfff640 10 client.19485 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe19bfff640 1 -- 172.21.15.28:0/2746599174 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564b8dbf3160 con 0x564b8dc9c970 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe19bfff640 10 client.19485 handle_command_reply: tid=0 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.135+0000 7fe1c5e28740 1 client.19485 shutdown 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 msgr2=0x564b8dc95ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe1b00a4360 tx=0x7fe1b009ba70 comp rx=0 tx=0).stop 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 shutdown_connections 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x564b8dc9c970 0x564b8dd05650 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b8dc8c380 0x564b8dc971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe19bfff640 0 client.19485 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b8dc87210 0x564b8dc961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 --2- 172.21.15.28:0/2746599174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b8dc86820 0x564b8dc95ca0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 >> 172.21.15.28:0/2746599174 conn(0x564b8dc81d90 msgr2=0x564b8dc84ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 shutdown_connections 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 1 -- 172.21.15.28:0/2746599174 wait complete. 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c5e28740 20 client.19485 trim_cache size 0 max 16384 2024-03-22T02:27:27.144 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 msgr2=0x7fe1ac0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 0x7fe1ac0a53e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4006010 tx=0x7fe1b402ce20 comp rx=0 tx=0).stop 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 msgr2=0x7fe1bc197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.136+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000e9c0 tx=0x7fe1b000ee80 comp rx=0 tx=0).stop 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 shutdown_connections 2024-03-22T02:27:27.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe1ac0a2f20 0x7fe1ac0a53e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe1bc102640 0x7fe1bc19f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe1bc101c80 0x7fe1bc1983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 --2- 172.21.15.28:0/3314786028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1bc100a90 0x7fe1bc197eb0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 >> 172.21.15.28:0/3314786028 conn(0x7fe1bc0fc020 msgr2=0x7fe1bc0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 shutdown_connections 2024-03-22T02:27:27.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.137+0000 7fe1c315e640 1 -- 172.21.15.28:0/3314786028 wait complete. 2024-03-22T02:27:27.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 -- 172.21.15.28:0/1735966532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c410e190 msgr2=0x7f50c4110580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 --2- 172.21.15.28:0/1735966532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c410e190 0x7f50c4110580 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009cd0 tx=0x7f50bc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 -- 172.21.15.28:0/1735966532 shutdown_connections 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 --2- 172.21.15.28:0/1735966532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c410e190 0x7f50c4110580 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 --2- 172.21.15.28:0/1735966532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c40745f0 0x7f50c4074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 --2- 172.21.15.28:0/1735966532 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 0x7f50c4074020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 -- 172.21.15.28:0/1735966532 >> 172.21.15.28:0/1735966532 conn(0x7f50c40fd130 msgr2=0x7f50c40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 -- 172.21.15.28:0/1735966532 shutdown_connections 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.255+0000 7f50c9925640 1 -- 172.21.15.28:0/1735966532 wait complete. 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 Processor -- start 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 -- start start 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 0x7f50c419ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c410e190 0x7f50c41a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113e50 con 0x7f50c40745f0 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113cd0 con 0x7f50c4075bd0 2024-03-22T02:27:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c9925640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4113fd0 con 0x7f50c410e190 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56334/0 (socket says 172.21.15.28:56334) 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 -- 172.21.15.28:0/2785330612 learned_addr learned my addr 172.21.15.28:0/2785330612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50bbfff640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 0x7f50c419ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c37fe640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c410e190 0x7f50c41a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 -- 172.21.15.28:0/2785330612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c410e190 msgr2=0x7f50c41a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c410e190 0x7f50c41a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 -- 172.21.15.28:0/2785330612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 msgr2=0x7f50c419ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 0x7f50c419ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.256+0000 7f50c2ffd640 1 -- 172.21.15.28:0/2785330612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c41a6110 con 0x7f50c40745f0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f50b400e9f0 tx=0x7f50b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c8923640 1 -- 172.21.15.28:0/2785330612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b400cd80 con 0x7f50c40745f0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c8923640 1 -- 172.21.15.28:0/2785330612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50b4010910 con 0x7f50c40745f0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c41a6400 con 0x7f50c40745f0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c8923640 1 -- 172.21.15.28:0/2785330612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b4018990 con 0x7f50c40745f0 2024-03-22T02:27:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.257+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c41a6950 con 0x7f50c40745f0 2024-03-22T02:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.259+0000 7f50c8923640 1 -- 172.21.15.28:0/2785330612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f50b4010ac0 con 0x7f50c40745f0 2024-03-22T02:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.259+0000 7f50cc5eb740 1 Processor -- start 2024-03-22T02:27:27.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.259+0000 7f50cc5eb740 1 -- start start 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1ccad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1ccd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558b1c2a8d0 con 0x5558b1cc6210 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5558b1bd2e00 con 0x5558b1ccb380 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5558b1c26880 con 0x5558b1cc5820 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1ccad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1ccd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33852/0 (socket says 172.21.15.28:33852) 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1ccad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56344/0 (socket says 172.21.15.28:56344) 2024-03-22T02:27:27.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 -- 172.21.15.28:0/3467194968 learned_addr learned my addr 172.21.15.28:0/3467194968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c8923640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 0x7f50b00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 -- 172.21.15.28:0/3467194968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 msgr2=0x5558b1ccd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1ccd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 -- 172.21.15.28:0/3467194968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 msgr2=0x5558b1ccad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1ccad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 -- 172.21.15.28:0/3467194968 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558b193e3f0 con 0x5558b1cc5820 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c8923640 1 -- 172.21.15.28:0/2785330612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f50b40cfb50 con 0x7f50c40745f0 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 0x7f50b00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.260+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f50bc02c700 tx=0x7f50bc005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c77f23e3ba2f43 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50b9ffb640 1 -- 172.21.15.28:0/3467194968 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc039040 con 0x5558b1cc5820 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50b9ffb640 1 -- 172.21.15.28:0/3467194968 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50bc003e50 con 0x5558b1cc5820 2024-03-22T02:27:27.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50b9ffb640 1 -- 172.21.15.28:0/3467194968 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc0047f0 con 0x5558b1cc5820 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3467194968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 msgr2=0x5558b1cc5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50c37fe640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 0x7f50b00a53e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f50bc04c000 tx=0x7f50bc045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f50bc02c700 tx=0x7f50bc005f70 comp rx=0 tx=0).stop 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3467194968 shutdown_connections 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1ccd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1ccad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3467194968 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cc5c40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3467194968 >> 172.21.15.28:0/3467194968 conn(0x5558b1cc0d90 msgr2=0x5558b1cc1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3467194968 shutdown_connections 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.261+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3467194968 wait complete. 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 -- start start 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 Processor -- start 2024-03-22T02:27:27.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1ccfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc6210 0x5558b1cd02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558b1cce330 con 0x5558b1ccb380 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5558b1be6300 con 0x5558b1cc6210 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5558b1cce4b0 con 0x5558b1cc5820 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50cc5eb740 10 client.0 _ll_register_callbacks cb 0x5558b1c21e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1ccfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc6210 0x5558b1cd02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1ccfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33868/0 (socket says 172.21.15.28:33868) 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56358/0 (socket says 172.21.15.28:56358) 2024-03-22T02:27:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c2ffd640 1 -- 172.21.15.28:0/3381337758 learned_addr learned my addr 172.21.15.28:0/3381337758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 -- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 msgr2=0x5558b1ccfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1ccfd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 -- 172.21.15.28:0/3381337758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc6210 msgr2=0x5558b1cd02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.262+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc6210 0x5558b1cd02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50c37fe640 1 -- 172.21.15.28:0/3381337758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558b1a087d0 con 0x5558b1ccb380 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0324c0 tx=0x7f50bc032ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f993c602430e565d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc039070 con 0x5558b1ccb380 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f50bc004770 con 0x5558b1ccb380 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50bc003e50 con 0x5558b1ccb380 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f50bc003970 con 0x5558b1ccb380 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50cc5eb740 10 client.19497 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5558b1cda650 con 0x5558b1ccb380 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50bc003e50 con 0x5558b1ccb380 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f50bc03f040 con 0x5558b1ccb380 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50cc5eb740 10 client.19497 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.263+0000 7f50cc5eb740 10 client.19497 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50cc5eb740 20 client.19497 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1cdba90 0x5558b1cdde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50cc5eb740 4 client.19497 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558b1c37e10 con 0x5558b1cdba90 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1cdba90 0x5558b1cdde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1cdba90 0x5558b1cdde80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1bc4b4c1af85b0f server_cookie=a3d09c2f1a3be7cf in_seq=0 out_seq=0 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.264+0000 7f50a2ffd640 10 client.19497 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:27.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50a2ffd640 1 -- 172.21.15.28:0/3381337758 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558b1c37e10 con 0x5558b1cdba90 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50a2ffd640 10 client.19497 handle_command_reply: tid=0 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 client.19497 shutdown 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 msgr2=0x5558b1cd72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0324c0 tx=0x7f50bc032ae0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 shutdown_connections 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1cdba90 0x5558b1cdde80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd72e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc6210 0x5558b1cd02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3381337758 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1ccfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50a2ffd640 0 client.19497 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 >> 172.21.15.28:0/3381337758 conn(0x5558b1cc0d90 msgr2=0x5558b1cbffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.265+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 shutdown_connections 2024-03-22T02:27:27.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.266+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3381337758 wait complete. 2024-03-22T02:27:27.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.266+0000 7f50cc5eb740 20 client.19497 trim_cache size 0 max 16384 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.279+0000 7f50cc5eb740 1 Processor -- start 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.279+0000 7f50cc5eb740 1 -- start start 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cbf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1cbfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41b38f0 con 0x5558b1cc6210 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41b1160 con 0x5558b1ccb380 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41aefb0 con 0x5558b1cc5820 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56368/0 (socket says 172.21.15.28:56368) 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 -- 172.21.15.28:0/3757541175 learned_addr learned my addr 172.21.15.28:0/3757541175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cbf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50c37fe640 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1cbfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 -- 172.21.15.28:0/3757541175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 msgr2=0x5558b1cbf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cbf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 -- 172.21.15.28:0/3757541175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 msgr2=0x5558b1cbfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1cbfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 -- 172.21.15.28:0/3757541175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558b1cde410 con 0x5558b1cc6210 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50bbfff640 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009cf0 tx=0x7f50ac005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8e7c3660c12fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50b97fa640 1 -- 172.21.15.28:0/3757541175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac022070 con 0x5558b1cc6210 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50b97fa640 1 -- 172.21.15.28:0/3757541175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac0035f0 con 0x5558b1cc6210 2024-03-22T02:27:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.280+0000 7f50b97fa640 1 -- 172.21.15.28:0/3757541175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac01d6c0 con 0x5558b1cc6210 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3757541175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 msgr2=0x5558b1cd72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f50ac009cf0 tx=0x7f50ac005f90 comp rx=0 tx=0).stop 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3757541175 shutdown_connections 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1ccb380 0x5558b1cbfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1cc6210 0x5558b1cd72a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/3757541175 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc5820 0x5558b1cbf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3757541175 >> 172.21.15.28:0/3757541175 conn(0x5558b1cc0d90 msgr2=0x5558b1cc7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3757541175 shutdown_connections 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- 172.21.15.28:0/3757541175 wait complete. 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.281+0000 7f50cc5eb740 1 -- start start 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 Processor -- start 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 0x5558b1cd4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc6210 0x5558b1cd5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41af620 con 0x5558b1ccb380 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41b0a90 con 0x5558b1cc5820 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f50c41afcf0 con 0x5558b1cc6210 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50cc5eb740 10 client.0 _ll_register_callbacks cb 0x7f50b40981e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c2ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 0x5558b1cd4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56382/0 (socket says 172.21.15.28:56382) 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 -- 172.21.15.28:0/4096843634 learned_addr learned my addr 172.21.15.28:0/4096843634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50bbfff640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc6210 0x5558b1cd5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 -- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc6210 msgr2=0x5558b1cd5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc6210 0x5558b1cd5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 -- 172.21.15.28:0/4096843634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 msgr2=0x5558b1cd4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 0x5558b1cd4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 -- 172.21.15.28:0/4096843634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558b1cc5460 con 0x5558b1ccb380 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 0x5558b1cd4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50c37fe640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f50bc032420 tx=0x7f50bc032340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4629bfd5253cc123 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:27.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.282+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc06d060 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f50bc003d10 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50ac0035f0 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f50bc039040 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 10 client.19503 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5558b1d42000 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50ac0035f0 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f50bc004350 con 0x5558b1ccb380 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 10 client.19503 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 10 client.19503 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 20 client.19503 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1d428a0 0x5558b1d44cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 4 client.19503 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558b1c32160 con 0x5558b1d428a0 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.283+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1d428a0 0x5558b1d44cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50a2ffd640 10 client.19503 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50c2ffd640 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1d428a0 0x5558b1d44cc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b6ad97c9a304019 server_cookie=490331d53d6db80f in_seq=0 out_seq=0 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50a2ffd640 1 -- 172.21.15.28:0/4096843634 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558b1c32160 con 0x5558b1d428a0 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50a2ffd640 10 client.19503 handle_command_reply: tid=0 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 client.19503 shutdown 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 msgr2=0x5558b1cd6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f50bc032420 tx=0x7f50bc032340 comp rx=0 tx=0).stop 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 shutdown_connections 2024-03-22T02:27:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x5558b1d428a0 0x5558b1d44cc0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558b1ccb380 0x5558b1cd6470 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5558b1cc6210 0x5558b1cd5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 --2- 172.21.15.28:0/4096843634 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5558b1cc5820 0x5558b1cd4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50a2ffd640 0 client.19503 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 >> 172.21.15.28:0/4096843634 conn(0x5558b1cc0d90 msgr2=0x5558b1cc3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 shutdown_connections 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 1 -- 172.21.15.28:0/4096843634 wait complete. 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.284+0000 7f50cc5eb740 20 client.19503 trim_cache size 0 max 16384 2024-03-22T02:27:27.294 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 msgr2=0x7f50b00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 0x7f50b00a53e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f50bc04c000 tx=0x7f50bc045000 comp rx=0 tx=0).stop 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 msgr2=0x7f50c419e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f50b400e9f0 tx=0x7f50b400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 shutdown_connections 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f50b00a2f20 0x7f50b00a53e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50c410e190 0x7f50c41a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f50c4075bd0 0x7f50c419ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 --2- 172.21.15.28:0/2785330612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50c40745f0 0x7f50c419e4f0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 >> 172.21.15.28:0/2785330612 conn(0x7f50c40fd130 msgr2=0x7f50c4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 shutdown_connections 2024-03-22T02:27:27.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.285+0000 7f50c9925640 1 -- 172.21.15.28:0/2785330612 wait complete. 2024-03-22T02:27:27.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 -- 172.21.15.28:0/3546952556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 msgr2=0x7f889c101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 --2- 172.21.15.28:0/3546952556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c101fe0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f888c009900 tx=0x7f888c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 -- 172.21.15.28:0/3546952556 shutdown_connections 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 --2- 172.21.15.28:0/3546952556 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c103770 0x7f889c10a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 --2- 172.21.15.28:0/3546952556 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c102db0 0x7f889c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 --2- 172.21.15.28:0/3546952556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c101fe0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 -- 172.21.15.28:0/3546952556 >> 172.21.15.28:0/3546952556 conn(0x7f889c0fd130 msgr2=0x7f889c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 -- 172.21.15.28:0/3546952556 shutdown_connections 2024-03-22T02:27:27.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.407+0000 7f88a08c9640 1 -- 172.21.15.28:0/3546952556 wait complete. 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 Processor -- start 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 -- start start 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c102db0 0x7f889c073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 0x7f889c0744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10d830 con 0x7f889c101bc0 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10d6b0 con 0x7f889c102db0 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.408+0000 7f88a08c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f889c10d9b0 con 0x7f889c103770 2024-03-22T02:27:27.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56406/0 (socket says 172.21.15.28:56406) 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 -- 172.21.15.28:0/1808254384 learned_addr learned my addr 172.21.15.28:0/1808254384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f8893fff640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c102db0 0x7f889c073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 0x7f889c0744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 -- 172.21.15.28:0/1808254384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 msgr2=0x7f889c0744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 0x7f889c0744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 -- 172.21.15.28:0/1808254384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c102db0 msgr2=0x7f889c073f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c102db0 0x7f889c073f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 -- 172.21.15.28:0/1808254384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c074a00 con 0x7f889c101bc0 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 0x7f889c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f889a59c640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f888c000c00 tx=0x7f888c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.409+0000 7f88937fe640 1 -- 172.21.15.28:0/1808254384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c041070 con 0x7f889c101bc0 2024-03-22T02:27:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.410+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c074c90 con 0x7f889c101bc0 2024-03-22T02:27:27.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.410+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c1a6930 con 0x7f889c101bc0 2024-03-22T02:27:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.410+0000 7f88937fe640 1 -- 172.21.15.28:0/1808254384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f888c004450 con 0x7f889c101bc0 2024-03-22T02:27:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.410+0000 7f88937fe640 1 -- 172.21.15.28:0/1808254384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c034440 con 0x7f889c101bc0 2024-03-22T02:27:27.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.412+0000 7f88937fe640 1 -- 172.21.15.28:0/1808254384 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f888c044070 con 0x7f889c101bc0 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 Processor -- start 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 -- start start 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa058d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa05b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88937fe640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 0x7f88880a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561df9fb88d0 con 0x561dfa054210 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561df9f60e00 con 0x561dfa059380 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88a3591740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561df9fb4880 con 0x561dfa053820 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f889a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa058d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33920/0 (socket says 172.21.15.28:33920) 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f889ad9d640 1 -- 172.21.15.28:0/2749889917 learned_addr learned my addr 172.21.15.28:0/2749889917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 0x7f88880a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.413+0000 7f88937fe640 1 -- 172.21.15.28:0/1808254384 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f888c0f1150 con 0x7f889c101bc0 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 -- 172.21.15.28:0/2749889917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 msgr2=0x561dfa053c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 -- 172.21.15.28:0/2749889917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 msgr2=0x561dfa05b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa05b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 -- 172.21.15.28:0/2749889917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561df9ccc3f0 con 0x561dfa054210 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889ad9d640 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889a59c640 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa058d10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f888c0bd540 tx=0x7f888c08c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a687e2682c256212 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f8890ff9640 1 -- 172.21.15.28:0/2749889917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c04b070 con 0x561dfa054210 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 0x7f88880a5460 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x561dfa054a70 tx=0x7f8894009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f8890ff9640 1 -- 172.21.15.28:0/2749889917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f888c0bd5b0 con 0x561dfa054210 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.414+0000 7f8890ff9640 1 -- 172.21.15.28:0/2749889917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0c5e80 con 0x561dfa054210 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 -- 172.21.15.28:0/2749889917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 msgr2=0x561dfa058d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa058d10 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f888c0bd540 tx=0x7f888c08c1a0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 -- 172.21.15.28:0/2749889917 shutdown_connections 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa05b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa058d10 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 --2- 172.21.15.28:0/2749889917 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa053c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 -- 172.21.15.28:0/2749889917 >> 172.21.15.28:0/2749889917 conn(0x561dfa04ed90 msgr2=0x561dfa04f190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 -- 172.21.15.28:0/2749889917 shutdown_connections 2024-03-22T02:27:27.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.415+0000 7f88a3591740 1 -- 172.21.15.28:0/2749889917 wait complete. 2024-03-22T02:27:27.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 -- start start 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 Processor -- start 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa065060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561dfa05c0f0 con 0x561dfa054210 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561df9f74300 con 0x561dfa059380 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561dfa05c270 con 0x561dfa053820 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f88a3591740 10 client.0 _ll_register_callbacks cb 0x561df9fafe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f8893fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f889a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f889a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33922/0 (socket says 172.21.15.28:33922) 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f889a59c640 1 -- 172.21.15.28:0/337549362 learned_addr learned my addr 172.21.15.28:0/337549362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.416+0000 7f889ad9d640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa065060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 -- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 msgr2=0x561dfa065060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa065060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 -- 172.21.15.28:0/337549362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 msgr2=0x561dfa05e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 -- 172.21.15.28:0/337549362 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561df9d967d0 con 0x561dfa053820 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f889a59c640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f888c0eebb0 tx=0x7f888c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f780e76c831625 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f8881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c04b070 con 0x561dfa053820 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f888c0035f0 con 0x561dfa053820 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f888c0bd5b0 con 0x561dfa053820 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f888c0bd0f0 con 0x561dfa053820 2024-03-22T02:27:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f88a3591740 10 client.29326 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561dfa05c0f0 con 0x561dfa053820 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.417+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f888c0bd5b0 con 0x561dfa053820 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f888c0c59f0 con 0x561dfa053820 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 10 client.29326 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 10 client.29326 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 20 client.29326 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa069910 0x561dfa06bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 4 client.29326 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561df9fc5e10 con 0x561dfa069910 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f8893fff640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa069910 0x561dfa06bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f8881ffb640 10 client.29326 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.418+0000 7f8893fff640 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa069910 0x561dfa06bd90 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=344204ea639a2430 server_cookie=281379a58ce24bc in_seq=0 out_seq=0 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f8881ffb640 1 -- 172.21.15.28:0/337549362 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561df9fc5e10 con 0x561dfa069910 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f8881ffb640 10 client.29326 handle_command_reply: tid=0 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 client.29326 shutdown 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 msgr2=0x561dfa05dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f888c0eebb0 tx=0x7f888c005fd0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 shutdown_connections 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa069910 0x561dfa06bd90 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa059380 0x561dfa065060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 --2- 172.21.15.28:0/337549362 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa05dae0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f8881ffb640 0 client.29326 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 >> 172.21.15.28:0/337549362 conn(0x561dfa04ed90 msgr2=0x561dfa04dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 shutdown_connections 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 1 -- 172.21.15.28:0/337549362 wait complete. 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.421+0000 7f88a3591740 20 client.29326 trim_cache size 0 max 16384 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 Processor -- start 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 -- start start 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa05c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa064ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f887c0125b0 con 0x561dfa054210 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00fe20 con 0x561dfa04ed90 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f88a3591740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00dc70 con 0x561dfa053820 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56420/0 (socket says 172.21.15.28:56420) 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 -- 172.21.15.28:0/1401686842 learned_addr learned my addr 172.21.15.28:0/1401686842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889a59c640 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa05c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f8893fff640 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa064ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 -- 172.21.15.28:0/1401686842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 msgr2=0x561dfa064ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa064ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 -- 172.21.15.28:0/1401686842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 msgr2=0x561dfa05c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa05c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.435+0000 7f889ad9d640 1 -- 172.21.15.28:0/1401686842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dfa06c410 con 0x561dfa054210 2024-03-22T02:27:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f889ad9d640 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8894011d30 tx=0x7f8894013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b71294336e6bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f8883fff640 1 -- 172.21.15.28:0/1401686842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940045c0 con 0x561dfa054210 2024-03-22T02:27:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f8883fff640 1 -- 172.21.15.28:0/1401686842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8894004760 con 0x561dfa054210 2024-03-22T02:27:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f8883fff640 1 -- 172.21.15.28:0/1401686842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894020780 con 0x561dfa054210 2024-03-22T02:27:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 -- 172.21.15.28:0/1401686842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 msgr2=0x561dfa05cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f8894011d30 tx=0x7f8894013070 comp rx=0 tx=0).stop 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 -- 172.21.15.28:0/1401686842 shutdown_connections 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa054210 0x561dfa05cb10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa053820 0x561dfa064ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 --2- 172.21.15.28:0/1401686842 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa05c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 -- 172.21.15.28:0/1401686842 >> 172.21.15.28:0/1401686842 conn(0x561dfa054ff0 msgr2=0x561dfa04dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 -- 172.21.15.28:0/1401686842 shutdown_connections 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.436+0000 7f88a3591740 1 -- 172.21.15.28:0/1401686842 wait complete. 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 -- start start 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 Processor -- start 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa062bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa054210 0x561dfa064110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00e2e0 con 0x561dfa053820 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00f750 con 0x561dfa04ed90 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00e9b0 con 0x561dfa054210 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f88a3591740 10 client.0 _ll_register_callbacks cb 0x7f8894011810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f889a59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa062bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56428/0 (socket says 172.21.15.28:56428) 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 -- 172.21.15.28:0/3337812562 learned_addr learned my addr 172.21.15.28:0/3337812562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f889ad9d640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa054210 0x561dfa064110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 -- 172.21.15.28:0/3337812562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa054210 msgr2=0x561dfa064110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa054210 0x561dfa064110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 -- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 msgr2=0x561dfa062bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa062bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f8893fff640 1 -- 172.21.15.28:0/3337812562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dfa064700 con 0x561dfa053820 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.437+0000 7f889a59c640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa062bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8893fff640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8884009c40 tx=0x7f8884010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7838aa0e3f685741 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888401de60 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f888401d580 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8894004760 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f888402f050 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 10 client.19521 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x561dfa0d0250 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8894004760 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f888400f4c0 con 0x561dfa053820 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 10 client.19521 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 10 client.19521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 20 client.19521 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa059380 0x561dfa0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 4 client.19521 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.438+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561df9fc0160 con 0x561dfa059380 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.439+0000 7f889a59c640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa059380 0x561dfa0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.439+0000 7f889a59c640 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa059380 0x561dfa0d2cd0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7543c09e11284833 server_cookie=2239f1c88b7a23dd in_seq=0 out_seq=0 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.439+0000 7f8881ffb640 10 client.19521 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f8881ffb640 1 -- 172.21.15.28:0/3337812562 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561df9fc0160 con 0x561dfa059380 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f8881ffb640 10 client.19521 handle_command_reply: tid=0 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 client.19521 shutdown 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 msgr2=0x561dfa063100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f8884009c40 tx=0x7f8884010800 comp rx=0 tx=0).stop 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 shutdown_connections 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561dfa059380 0x561dfa0d2cd0 crc :-1 s=READY pgs=351 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561dfa054210 0x561dfa064110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561dfa053820 0x561dfa063100 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 --2- 172.21.15.28:0/3337812562 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561dfa04ed90 0x561dfa062bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 >> 172.21.15.28:0/3337812562 conn(0x561dfa054ff0 msgr2=0x561dfa05c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f8881ffb640 0 client.19521 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 shutdown_connections 2024-03-22T02:27:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.440+0000 7f88a3591740 1 -- 172.21.15.28:0/3337812562 wait complete. 2024-03-22T02:27:27.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a3591740 20 client.19521 trim_cache size 0 max 16384 2024-03-22T02:27:27.459 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 msgr2=0x7f88880a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 0x7f88880a5460 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x561dfa054a70 tx=0x7f8894009500 comp rx=0 tx=0).stop 2024-03-22T02:27:27.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 msgr2=0x7f889c075940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f888c000c00 tx=0x7f888c02c640 comp rx=0 tx=0).stop 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 shutdown_connections 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f88880a2fa0 0x7f88880a5460 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f889c103770 0x7f889c0744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f889c102db0 0x7f889c073f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 --2- 172.21.15.28:0/1808254384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f889c101bc0 0x7f889c075940 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 >> 172.21.15.28:0/1808254384 conn(0x7f889c0fd130 msgr2=0x7f889c106150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 shutdown_connections 2024-03-22T02:27:27.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:27.441+0000 7f88a08c9640 1 -- 172.21.15.28:0/1808254384 wait complete. 2024-03-22T02:27:27.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 2736 45.06 MB/sec warmup 7 sec latency 65.088 ms 2024-03-22T02:27:27.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:27 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2315495694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:27:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:27 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2315495694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:27:28.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:27 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2315495694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:27:28.462 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-22T02:27:28.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:27:28.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 -- 172.21.15.28:0/5380791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 msgr2=0x7f6f8c109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 --2- 172.21.15.28:0/5380791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c109fe0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009ca0 tx=0x7f6f8402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 -- 172.21.15.28:0/5380791 shutdown_connections 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 --2- 172.21.15.28:0/5380791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c109fe0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 --2- 172.21.15.28:0/5380791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 0x7f6f8c103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 --2- 172.21.15.28:0/5380791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 -- 172.21.15.28:0/5380791 >> 172.21.15.28:0/5380791 conn(0x7f6f8c0fd130 msgr2=0x7f6f8c0ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 -- 172.21.15.28:0/5380791 shutdown_connections 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.573+0000 7f6f911b6640 1 -- 172.21.15.28:0/5380791 wait complete. 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.574+0000 7f6f911b6640 1 Processor -- start 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.574+0000 7f6f911b6640 1 -- start start 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.574+0000 7f6f911b6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.574+0000 7f6f911b6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 0x7f6f8c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10d820 con 0x7f6f8c103750 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10d6a0 con 0x7f6f8c101ba0 2024-03-22T02:27:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c10d9a0 con 0x7f6f8c102d90 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8a59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 0x7f6f8c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56440/0 (socket says 172.21.15.28:56440) 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53998/0 (socket says 172.21.15.28:53998) 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 -- 172.21.15.28:0/3830224364 learned_addr learned my addr 172.21.15.28:0/3830224364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 -- 172.21.15.28:0/3830224364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 msgr2=0x7f6f8c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 0x7f6f8c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 -- 172.21.15.28:0/3830224364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 msgr2=0x7f6f8c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 -- 172.21.15.28:0/3830224364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f8c19fae0 con 0x7f6f8c103750 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f6f84032530 tx=0x7f6f84032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f73fff640 1 -- 172.21.15.28:0/3830224364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84041020 con 0x7f6f8c103750 2024-03-22T02:27:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f8c19fd70 con 0x7f6f8c103750 2024-03-22T02:27:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f73fff640 1 -- 172.21.15.28:0/3830224364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f84034cd0 con 0x7f6f8c103750 2024-03-22T02:27:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f73fff640 1 -- 172.21.15.28:0/3830224364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f84039040 con 0x7f6f8c103750 2024-03-22T02:27:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.575+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f8c1a01d0 con 0x7f6f8c103750 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.577+0000 7f6f73fff640 1 -- 172.21.15.28:0/3830224364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6f84044020 con 0x7f6f8c103750 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 Processor -- start 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 -- start start 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f73fff640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 0x7f6f740a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca754f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca7551fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cca74aa570 con 0x55cca7549fd0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cca7452e10 con 0x55cca754fb30 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f93d98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cca74a6520 con 0x55cca754a9c0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 0x7f6f740a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f73fff640 1 -- 172.21.15.28:0/3830224364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6f840f15b0 con 0x7f6f8c103750 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56456/0 (socket says 172.21.15.28:56456) 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/3035245936 learned_addr learned my addr 172.21.15.28:0/3035245936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca754f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca7551fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/3035245936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 msgr2=0x55cca754f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca754f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/3035245936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 msgr2=0x55cca7551fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca7551fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.578+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/3035245936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca72867e0 con 0x55cca7549fd0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.579+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 0x7f6f740a53e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009710 tx=0x7f6f7c009ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.579+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009ca0 tx=0x7f6f80011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51def426c199bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.579+0000 7f6f717fa640 1 -- 172.21.15.28:0/3035245936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80004220 con 0x55cca7549fd0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.579+0000 7f6f717fa640 1 -- 172.21.15.28:0/3035245936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f80009e50 con 0x55cca7549fd0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.579+0000 7f6f717fa640 1 -- 172.21.15.28:0/3035245936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80018740 con 0x55cca7549fd0 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- 172.21.15.28:0/3035245936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 msgr2=0x55cca754a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009ca0 tx=0x7f6f80011810 comp rx=0 tx=0).stop 2024-03-22T02:27:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- 172.21.15.28:0/3035245936 shutdown_connections 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca7551fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca754f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3035245936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca754a3f0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- 172.21.15.28:0/3035245936 >> 172.21.15.28:0/3035245936 conn(0x55cca7541130 msgr2=0x55cca7541530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- 172.21.15.28:0/3035245936 shutdown_connections 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- 172.21.15.28:0/3035245936 wait complete. 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- start start 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 Processor -- start 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca75546f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cca7552ae0 con 0x55cca7549fd0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cca7466110 con 0x55cca754fb30 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cca7552c60 con 0x55cca754a9c0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f93d98740 10 client.0 _ll_register_callbacks cb 0x55cca74a2be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f8ad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56462/0 (socket says 172.21.15.28:56462) 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/2888754927 learned_addr learned my addr 172.21.15.28:0/2888754927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca75546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.580+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/2888754927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 msgr2=0x55cca75546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca75546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 msgr2=0x55cca755b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 -- 172.21.15.28:0/2888754927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca72e6f70 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6f80011d10 tx=0x7f6f800052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997cc8da38dc47a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f80027020 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6f800035f0 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f80009e50 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6f80032050 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f93d98740 10 client.19533 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55cca717e170 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.581+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f80009e50 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f6f80004c50 con 0x55cca7549fd0 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 10 client.19533 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 10 client.19533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:28.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 20 client.19533 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca755fea0 0x55cca7562340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 4 client.19533 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cca74b7ab0 con 0x55cca755fea0 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca755fea0 0x55cca7562340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca755fea0 0x55cca7562340 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77f07768b8c952a server_cookie=727868313420eed3 in_seq=0 out_seq=0 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.582+0000 7f6f6eff5640 10 client.19533 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2888754927 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cca74b7ab0 con 0x55cca755fea0 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f6eff5640 10 client.19533 handle_command_reply: tid=0 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 client.19533 shutdown 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 msgr2=0x55cca75541b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6f80011d10 tx=0x7f6f800052f0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 shutdown_connections 2024-03-22T02:27:28.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca755fea0 0x55cca7562340 crc :-1 s=READY pgs=352 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f6eff5640 0 client.19533 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754a9c0 0x55cca75546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2888754927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca7549fd0 0x55cca75541b0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 >> 172.21.15.28:0/2888754927 conn(0x55cca7541130 msgr2=0x55cca75403f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 shutdown_connections 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.584+0000 7f6f93d98740 1 -- 172.21.15.28:0/2888754927 wait complete. 2024-03-22T02:27:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.585+0000 7f6f93d98740 20 client.19533 trim_cache size 0 max 16384 2024-03-22T02:27:28.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 Processor -- start 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 -- start start 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca7549fd0 0x55cca75537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 0x55cca7553d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1ab120 con 0x55cca754a9c0 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1a8990 con 0x55cca7549fd0 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f93d98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1a67e0 con 0x55cca754fb30 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56468/0 (socket says 172.21.15.28:56468) 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f8a59c640 1 -- 172.21.15.28:0/3918497868 learned_addr learned my addr 172.21.15.28:0/3918497868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca7549fd0 0x55cca75537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.603+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 0x55cca7553d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 -- 172.21.15.28:0/3918497868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 msgr2=0x55cca7553d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 0x55cca7553d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 -- 172.21.15.28:0/3918497868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca7549fd0 msgr2=0x55cca75537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca7549fd0 0x55cca75537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 -- 172.21.15.28:0/3918497868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca75451d0 con 0x55cca754a9c0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 0x55cca7553d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a8c5ec22124531 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f70ff9640 1 -- 172.21.15.28:0/3918497868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c0044c0 con 0x55cca754a9c0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f70ff9640 1 -- 172.21.15.28:0/3918497868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f7c005500 con 0x55cca754a9c0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f70ff9640 1 -- 172.21.15.28:0/3918497868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004e10 con 0x55cca754a9c0 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f93d98740 1 -- 172.21.15.28:0/3918497868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 msgr2=0x55cca755b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.604+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c000c00 tx=0x7f6f7c021000 comp rx=0 tx=0).stop 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- 172.21.15.28:0/3918497868 shutdown_connections 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca754fb30 0x55cca7553d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca755b650 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- 172.21.15.28:0/3918497868 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca7549fd0 0x55cca75537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- 172.21.15.28:0/3918497868 >> 172.21.15.28:0/3918497868 conn(0x55cca7541130 msgr2=0x55cca75403f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- 172.21.15.28:0/3918497868 shutdown_connections 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- 172.21.15.28:0/3918497868 wait complete. 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- start start 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 Processor -- start 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca7549fd0 0x55cca7559020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1a6e50 con 0x55cca754a9c0 2024-03-22T02:27:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1a82c0 con 0x55cca754fb30 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8c1a7520 con 0x55cca7549fd0 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f93d98740 10 client.0 _ll_register_callbacks cb 0x7f6f7c0134e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56482/0 (socket says 172.21.15.28:56482) 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 -- 172.21.15.28:0/2154842904 learned_addr learned my addr 172.21.15.28:0/2154842904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca7549fd0 0x55cca7559020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8b59e640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 -- 172.21.15.28:0/2154842904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca7549fd0 msgr2=0x55cca7559020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca7549fd0 0x55cca7559020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 -- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 msgr2=0x55cca755a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.605+0000 7f6f8a59c640 1 -- 172.21.15.28:0/2154842904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca755ab60 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f8a59c640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009e70 tx=0x7f6f7c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41906c445b276181 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c004d90 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6f7c004f30 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f7c005500 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f6f7c0132c0 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 10 client.19539 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55cca75c63f0 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f7c005500 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f6f7c01f040 con 0x55cca754a9c0 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 10 client.19539 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 10 client.19539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 20 client.19539 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:28.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca75c6e80 0x55cca75c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 4 client.19539 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-22T02:27:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.606+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cca74b1e00 con 0x55cca75c6e80 2024-03-22T02:27:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.607+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca75c6e80 0x55cca75c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.607+0000 7f6f8ad9d640 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca75c6e80 0x55cca75c92a0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b86c69dac9a0aa0 server_cookie=d161fe812b92fb8d in_seq=0 out_seq=0 2024-03-22T02:27:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.607+0000 7f6f6eff5640 10 client.19539 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.610+0000 7f6f6eff5640 1 -- 172.21.15.28:0/2154842904 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cca74b1e00 con 0x55cca75c6e80 2024-03-22T02:27:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.610+0000 7f6f6eff5640 10 client.19539 handle_command_reply: tid=0 2024-03-22T02:27:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.610+0000 7f6f93d98740 1 client.19539 shutdown 2024-03-22T02:27:28.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 msgr2=0x55cca7559560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009e70 tx=0x7f6f7c0044c0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 shutdown_connections 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cca75c6e80 0x55cca75c92a0 crc :-1 s=READY pgs=353 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cca754fb30 0x55cca755a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cca754a9c0 0x55cca7559560 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 --2- 172.21.15.28:0/2154842904 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cca7549fd0 0x55cca7559020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 >> 172.21.15.28:0/2154842904 conn(0x55cca7541130 msgr2=0x55cca754b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f6eff5640 0 client.19539 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 shutdown_connections 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 1 -- 172.21.15.28:0/2154842904 wait complete. 2024-03-22T02:27:28.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f93d98740 20 client.19539 trim_cache size 0 max 16384 2024-03-22T02:27:28.617 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:27:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-03-22T02:27:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "4af51de4-8142-4f5a-be4d-b2673e5d5ea7", 2024-03-22T02:27:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-03-22T02:27:28.619 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 msgr2=0x7f6f740a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 0x7f6f740a53e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009710 tx=0x7f6f7c009ad0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 msgr2=0x7f6f8c19f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f6f84032530 tx=0x7f6f84032f20 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 shutdown_connections 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6f740a2f20 0x7f6f740a53e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f8c103750 0x7f6f8c19f470 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6f8c102d90 0x7f6f8c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 --2- 172.21.15.28:0/3830224364 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6f8c101ba0 0x7f6f8c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.611+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 >> 172.21.15.28:0/3830224364 conn(0x7f6f8c0fd130 msgr2=0x7f6f8c106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.612+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 shutdown_connections 2024-03-22T02:27:28.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.612+0000 7f6f911b6640 1 -- 172.21.15.28:0/3830224364 wait complete. 2024-03-22T02:27:28.624 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:27:28.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:27:28.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3275 43.83 MB/sec warmup 8 sec latency 66.455 ms 2024-03-22T02:27:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 -- 172.21.15.28:0/4054647148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 msgr2=0x7f3954100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 --2- 172.21.15.28:0/4054647148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f3954100eb0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f3944009be0 tx=0x7f394402bef0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 -- 172.21.15.28:0/4054647148 shutdown_connections 2024-03-22T02:27:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 --2- 172.21.15.28:0/4054647148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954102640 0x7f3954108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 --2- 172.21.15.28:0/4054647148 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954101c80 0x7f3954102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 --2- 172.21.15.28:0/4054647148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f3954100eb0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 -- 172.21.15.28:0/4054647148 >> 172.21.15.28:0/4054647148 conn(0x7f39540fc020 msgr2=0x7f39540fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 -- 172.21.15.28:0/4054647148 shutdown_connections 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.734+0000 7f3959381640 1 -- 172.21.15.28:0/4054647148 wait complete. 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 Processor -- start 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 -- start start 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 0x7f395419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 0x7f39541a5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f395410c710 con 0x7f3954100a90 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f395410c590 con 0x7f3954102640 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3959381640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f395410c890 con 0x7f3954101c80 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56510/0 (socket says 172.21.15.28:56510) 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 -- 172.21.15.28:0/2304761193 learned_addr learned my addr 172.21.15.28:0/2304761193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f39527fc640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 0x7f395419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f39537fe640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 0x7f39541a5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 -- 172.21.15.28:0/2304761193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 msgr2=0x7f395419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 0x7f395419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 -- 172.21.15.28:0/2304761193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 msgr2=0x7f39541a5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 0x7f39541a5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 -- 172.21.15.28:0/2304761193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39541a6120 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f39527fc640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 0x7f395419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f39537fe640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 0x7f39541a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.735+0000 7f3952ffd640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f3944036040 tx=0x7f3944033580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.736+0000 7f394bfff640 1 -- 172.21.15.28:0/2304761193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944043020 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.736+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39541a63b0 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.736+0000 7f394bfff640 1 -- 172.21.15.28:0/2304761193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3944003e50 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.736+0000 7f394bfff640 1 -- 172.21.15.28:0/2304761193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3944034690 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.736+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39541a6950 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f394bfff640 1 -- 172.21.15.28:0/2304761193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3944046020 con 0x7f3954100a90 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f395bf6b740 1 Processor -- start 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f395bf6b740 1 -- start start 2024-03-22T02:27:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f394bfff640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 0x7f39400a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f394bfff640 1 -- 172.21.15.28:0/2304761193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f39440f1f60 con 0x7f3954100a90 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.738+0000 7f39537fe640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 0x7f39400a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afffcf90 0x55f2affff420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2aff5c8c0 con 0x55f2afff7e20 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f2aff04c20 con 0x55f2afff7430 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f395bf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2aff58870 con 0x55f2afffcf90 2024-03-22T02:27:28.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56512/0 (socket says 172.21.15.28:56512) 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 -- 172.21.15.28:0/2084148256 learned_addr learned my addr 172.21.15.28:0/2084148256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54074/0 (socket says 172.21.15.28:54074) 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39537fe640 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afffcf90 0x55f2affff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39537fe640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 0x7f39400a53e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f394c005fd0 tx=0x7f394c0113e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 -- 172.21.15.28:0/2084148256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afffcf90 msgr2=0x55f2affff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afffcf90 0x55f2affff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 -- 172.21.15.28:0/2084148256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 msgr2=0x55f2afff7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 -- 172.21.15.28:0/2084148256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2aff29150 con 0x55f2afff7e20 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f3952ffd640 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39527fc640 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f393c009d50 tx=0x7f393c0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd54f04b31bd86e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39497fa640 1 -- 172.21.15.28:0/2084148256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c012650 con 0x55f2afff7e20 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.739+0000 7f39497fa640 1 -- 172.21.15.28:0/2084148256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f393c009e40 con 0x55f2afff7e20 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f39497fa640 1 -- 172.21.15.28:0/2084148256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c00fc00 con 0x55f2afff7e20 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 -- 172.21.15.28:0/2084148256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 msgr2=0x55f2afffc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f393c009d50 tx=0x7f393c0117b0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 -- 172.21.15.28:0/2084148256 shutdown_connections 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afffcf90 0x55f2affff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2afffc920 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 --2- 172.21.15.28:0/2084148256 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 -- 172.21.15.28:0/2084148256 >> 172.21.15.28:0/2084148256 conn(0x55f2afff29a0 msgr2=0x55f2afff2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 -- 172.21.15.28:0/2084148256 shutdown_connections 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.740+0000 7f395bf6b740 1 -- 172.21.15.28:0/2084148256 wait complete. 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 -- start start 2024-03-22T02:27:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 Processor -- start 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0001e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0008ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f2afffff40 con 0x55f2afff7430 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55f2aff18130 con 0x55f2afffcf90 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55f2b00000c0 con 0x55f2afff7e20 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f395bf6b740 10 client.0 _ll_register_callbacks cb 0x55f2aff17b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56514/0 (socket says 172.21.15.28:56514) 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 -- 172.21.15.28:0/403424867 learned_addr learned my addr 172.21.15.28:0/403424867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f39527fc640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0001e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f39537fe640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0008ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 -- 172.21.15.28:0/403424867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 msgr2=0x55f2b0001e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0001e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 -- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 msgr2=0x55f2b0008ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0008ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.741+0000 7f3952ffd640 1 -- 172.21.15.28:0/403424867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2afd3a7d0 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f3952ffd640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f39440bac50 tx=0x7f39440bdbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178d838e393a4c81 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440c23d0 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f39440bc580 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f2aff69e00 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f39440c6c90 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 10 client.19551 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f2afc703f0 con 0x55f2afff7430 2024-03-22T02:27:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f2aff69e00 con 0x55f2afff7430 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f3944004cd0 con 0x55f2afff7430 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 10 client.19551 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 10 client.19551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 20 client.19551 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b000d8b0 0x55f2b000fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 4 client.19551 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.742+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2aff64150 con 0x55f2b000d8b0 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.743+0000 7f39527fc640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b000d8b0 0x55f2b000fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.743+0000 7f39527fc640 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b000d8b0 0x55f2b000fd50 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8652b2d4a56cc440 server_cookie=6312bb5139464648 in_seq=0 out_seq=0 2024-03-22T02:27:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.743+0000 7f393a7fc640 10 client.19551 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f393a7fc640 1 -- 172.21.15.28:0/403424867 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2aff64150 con 0x55f2b000d8b0 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f393a7fc640 10 client.19551 handle_command_reply: tid=0 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 client.19551 shutdown 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 msgr2=0x55f2b0001940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f39440bac50 tx=0x7f39440bdbe0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 shutdown_connections 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b000d8b0 0x55f2b000fd50 crc :-1 s=READY pgs=354 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0008ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f393a7fc640 0 client.19551 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0001e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 --2- 172.21.15.28:0/403424867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7430 0x55f2b0001940 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 >> 172.21.15.28:0/403424867 conn(0x55f2afff29a0 msgr2=0x55f2afff1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 shutdown_connections 2024-03-22T02:27:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 1 -- 172.21.15.28:0/403424867 wait complete. 2024-03-22T02:27:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.748+0000 7f395bf6b740 20 client.19551 trim_cache size 0 max 16384 2024-03-22T02:27:28.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 Processor -- start 2024-03-22T02:27:28.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 -- start start 2024-03-22T02:27:28.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0008e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39541b38a0 con 0x55f2afffcf90 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39541b1110 con 0x55f2afff7430 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.768+0000 7f395bf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39541aef60 con 0x55f2afff7e20 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0008e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39537fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56516/0 (socket says 172.21.15.28:56516) 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39537fe640 1 -- 172.21.15.28:0/3840031991 learned_addr learned my addr 172.21.15.28:0/3840031991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f3952ffd640 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 -- 172.21.15.28:0/3840031991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 msgr2=0x55f2afff1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 -- 172.21.15.28:0/3840031991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 msgr2=0x55f2afff18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 -- 172.21.15.28:0/3840031991 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2afffcce0 con 0x55f2afff7e20 2024-03-22T02:27:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39537fe640 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f39527fc640 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0008e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55f2afff3910 tx=0x7f393c0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bef4e936869eee server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f3948ff9640 1 -- 172.21.15.28:0/3840031991 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c032070 con 0x55f2afff7e20 2024-03-22T02:27:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f3948ff9640 1 -- 172.21.15.28:0/3840031991 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f393c0055a0 con 0x55f2afff7e20 2024-03-22T02:27:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.769+0000 7f3948ff9640 1 -- 172.21.15.28:0/3840031991 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c0035f0 con 0x55f2afff7e20 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 -- 172.21.15.28:0/3840031991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 msgr2=0x55f2b0008e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0008e80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55f2afff3910 tx=0x7f393c0058d0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 -- 172.21.15.28:0/3840031991 shutdown_connections 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afffcf90 0x55f2afff18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7e20 0x55f2b0008e80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 --2- 172.21.15.28:0/3840031991 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afff7430 0x55f2afff1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 -- 172.21.15.28:0/3840031991 >> 172.21.15.28:0/3840031991 conn(0x55f2afff29a0 msgr2=0x55f2b0001550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 -- 172.21.15.28:0/3840031991 shutdown_connections 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.770+0000 7f395bf6b740 1 -- 172.21.15.28:0/3840031991 wait complete. 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 -- start start 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 Processor -- start 2024-03-22T02:27:28.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 0x55f2b0006860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2b0006da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39541af5d0 con 0x55f2afff7e20 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f39541b0a40 con 0x55f2afffcf90 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f39541afca0 con 0x55f2afff7430 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f395bf6b740 10 client.0 _ll_register_callbacks cb 0x7f39440bddc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39527fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2b0006da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 0x55f2b0006860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f3952ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 0x55f2b0006860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34046/0 (socket says 172.21.15.28:34046) 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f3952ffd640 1 -- 172.21.15.28:0/1667238970 learned_addr learned my addr 172.21.15.28:0/1667238970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39537fe640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39527fc640 1 -- 172.21.15.28:0/1667238970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 msgr2=0x55f2b0006860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39527fc640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 0x55f2b0006860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39527fc640 1 -- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 msgr2=0x55f2b0007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.771+0000 7f39527fc640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f39527fc640 1 -- 172.21.15.28:0/1667238970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2afc300f0 con 0x55f2afff7e20 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f39527fc640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2b0006da0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f393c00f5e0 tx=0x7f393c0099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc57cc735f2a3e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f393a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c032070 con 0x55f2afff7e20 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f393c005050 con 0x55f2afff7e20 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f393c0055a0 con 0x55f2afff7e20 2024-03-22T02:27:28.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f393c00fe50 con 0x55f2afff7e20 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f395bf6b740 10 client.19557 fetch_fsmap learned FSMap version 41 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55f2b0073d80 con 0x55f2afff7e20 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.772+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f393c0055a0 con 0x55f2afff7e20 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 41) ==== 8358+0+0 (secure 0 0 0) 0x7f393c02d070 con 0x55f2afff7e20 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 10 client.19557 fetch_fsmap finished waiting for FSMap version 41 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 10 client.19557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 20 client.19557 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b0074550 0x55f2b0076970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 4 client.19557 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f2afee1540 con 0x55f2b0074550 2024-03-22T02:27:28.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f3952ffd640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b0074550 0x55f2b0076970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f393a7fc640 10 client.19557 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.773+0000 7f3952ffd640 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b0074550 0x55f2b0076970 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fed832644c70c38 server_cookie=94499ad307831da4 in_seq=0 out_seq=0 2024-03-22T02:27:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f393a7fc640 1 -- 172.21.15.28:0/1667238970 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55f2afee1540 con 0x55f2b0074550 2024-03-22T02:27:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f393a7fc640 10 client.19557 handle_command_reply: tid=0 2024-03-22T02:27:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 client.19557 shutdown 2024-03-22T02:27:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 msgr2=0x55f2b0006da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2b0006da0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f393c00f5e0 tx=0x7f393c0099a0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 shutdown_connections 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55f2b0074550 0x55f2b0076970 crc :-1 s=READY pgs=355 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55f2afffcf90 0x55f2b0007db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f2afff7e20 0x55f2b0006da0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 --2- 172.21.15.28:0/1667238970 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f2afff7430 0x55f2b0006860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 >> 172.21.15.28:0/1667238970 conn(0x55f2afff29a0 msgr2=0x55f2afff5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f393a7fc640 0 client.19557 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 shutdown_connections 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 1 -- 172.21.15.28:0/1667238970 wait complete. 2024-03-22T02:27:28.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.777+0000 7f395bf6b740 20 client.19557 trim_cache size 0 max 16384 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (506 inodes in the stack)", 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "4af51de4-8142-4f5a-be4d-b2673e5d5ea7": { 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "4af51de4-8142-4f5a-be4d-b2673e5d5ea7", 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:27:28.783 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:27:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 msgr2=0x7f39400a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 0x7f39400a53e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f394c005fd0 tx=0x7f394c0113e0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 msgr2=0x7f395419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f3944036040 tx=0x7f3944033580 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 shutdown_connections 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f39400a2f20 0x7f39400a53e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3954102640 0x7f39541a5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3954101c80 0x7f395419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 --2- 172.21.15.28:0/2304761193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3954100a90 0x7f395419ec30 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 >> 172.21.15.28:0/2304761193 conn(0x7f39540fc020 msgr2=0x7f39540fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 shutdown_connections 2024-03-22T02:27:28.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:28.778+0000 7f3959381640 1 -- 172.21.15.28:0/2304761193 wait complete. 2024-03-22T02:27:28.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:28 smithi028 ceph-mon[31916]: pgmap v777: 129 pgs: 129 active+clean; 33 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-03-22T02:27:28.789 INFO:tasks.cephfs.filesystem:scrub status for tag:4af51de4-8142-4f5a-be4d-b2673e5d5ea7 - {'path': '/', 'tag': '4af51de4-8142-4f5a-be4d-b2673e5d5ea7', 'options': 'recursive,force'} 2024-03-22T02:27:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-mon[36803]: pgmap v777: 129 pgs: 129 active+clean; 33 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-03-22T02:27:28.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:27:28.745+0000 7fc080fe9700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000b87e [2,head] /volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25282=0-4194304@1} caps={25282=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25282 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55fdca73a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000b87e:[<0x1000000b87a/PCBENCHM.PPT v2>,<0x1000000b7f0/PWRPNT v26>,<0x1000000b7ed/~dmtmp v20>,<0x10000000007/client9 v1491>,<0x10000000006/clients v1244>,<0x10000000005/tmp v1205>,<0x100000001fb/client.0 v1200>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1192>,<0x10000000001/sv_1 v1214>,<0x10000000000/qa v1180>,<0x1/volumes v1161>]//[3]","memoryvalue":"(4)0x1000000b87e:[<0x1000000b87a/PCBENCHM.PPT v2>,<0x1000000b7f0/PWRPNT v228>,<0x1000000b7ed/~dmtmp v30>,<0x10000000007/client9 v1485>,<0x10000000006/clients v1247>,<0x10000000005/tmp v1208>,<0x100000001fb/client.0 v1203>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1195>,<0x10000000001/sv_1 v1217>,<0x10000000000/qa v1183>,<0x1/volumes v1163>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-22T02:27:28.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:27:28.746+0000 7fc080fe9700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000b888 [2,head] /volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP auth v4 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25282=0-4194304@1} caps={25282=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25282 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55fdca73ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000b888:[<0x1000000b87a/ZD16.BMP v4>,<0x1000000b7f0/PWRPNT v26>,<0x1000000b7ed/~dmtmp v20>,<0x10000000007/client9 v1491>,<0x10000000006/clients v1244>,<0x10000000005/tmp v1205>,<0x100000001fb/client.0 v1200>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1192>,<0x10000000001/sv_1 v1214>,<0x10000000000/qa v1180>,<0x1/volumes v1161>]//[3]","memoryvalue":"(4)0x1000000b888:[<0x1000000b87a/ZD16.BMP v4>,<0x1000000b7f0/PWRPNT v228>,<0x1000000b7ed/~dmtmp v30>,<0x10000000007/client9 v1485>,<0x10000000006/clients v1247>,<0x10000000005/tmp v1208>,<0x100000001fb/client.0 v1203>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1195>,<0x10000000001/sv_1 v1217>,<0x10000000000/qa v1183>,<0x1/volumes v1163>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-22T02:27:28.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:27:28.746+0000 7fc080fe9700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000b8ac [2,head] /volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT auth v10 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25282=0-4194304@1} caps={25282=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25282 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55fdca73b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000b8ac:[<0x1000000b87a/TRIDOTS.POT v10>,<0x1000000b7f0/PWRPNT v26>,<0x1000000b7ed/~dmtmp v20>,<0x10000000007/client9 v1491>,<0x10000000006/clients v1244>,<0x10000000005/tmp v1205>,<0x100000001fb/client.0 v1200>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1192>,<0x10000000001/sv_1 v1214>,<0x10000000000/qa v1180>,<0x1/volumes v1161>]//[3]","memoryvalue":"(4)0x1000000b8ac:[<0x1000000b87a/TRIDOTS.POT v10>,<0x1000000b7f0/PWRPNT v228>,<0x1000000b7ed/~dmtmp v30>,<0x10000000007/client9 v1485>,<0x10000000006/clients v1247>,<0x10000000005/tmp v1208>,<0x100000001fb/client.0 v1203>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1195>,<0x10000000001/sv_1 v1217>,<0x10000000000/qa v1183>,<0x1/volumes v1163>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-22T02:27:28.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:27:28.746+0000 7fc080fe9700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000b8a1 [2,head] /volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT auth v8 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25282=0-4194304@1} caps={25282=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25282 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55fdca73bb80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000b8a1:[<0x1000000b87a/TIPS.PPT v8>,<0x1000000b7f0/PWRPNT v26>,<0x1000000b7ed/~dmtmp v20>,<0x10000000007/client9 v1491>,<0x10000000006/clients v1244>,<0x10000000005/tmp v1205>,<0x100000001fb/client.0 v1200>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1192>,<0x10000000001/sv_1 v1214>,<0x10000000000/qa v1180>,<0x1/volumes v1161>]//[3]","memoryvalue":"(4)0x1000000b8a1:[<0x1000000b87a/TIPS.PPT v8>,<0x1000000b7f0/PWRPNT v228>,<0x1000000b7ed/~dmtmp v30>,<0x10000000007/client9 v1485>,<0x10000000006/clients v1247>,<0x10000000005/tmp v1208>,<0x100000001fb/client.0 v1203>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1195>,<0x10000000001/sv_1 v1217>,<0x10000000000/qa v1183>,<0x1/volumes v1163>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-22T02:27:28.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:27:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:27:28.746+0000 7fc080fe9700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000b895 [2,head] /volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA auth v6 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={25282=0-4194304@1} caps={25282=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@1},l=25282 | importingcaps=0 caps=1 replicated=0 authpin=1 scrubqueue=0 0x55fdca73c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000b895:[<0x1000000b87a/PPTOOLS1.PPA v6>,<0x1000000b7f0/PWRPNT v26>,<0x1000000b7ed/~dmtmp v20>,<0x10000000007/client9 v1491>,<0x10000000006/clients v1244>,<0x10000000005/tmp v1205>,<0x100000001fb/client.0 v1200>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1192>,<0x10000000001/sv_1 v1214>,<0x10000000000/qa v1180>,<0x1/volumes v1161>]//[3]","memoryvalue":"(4)0x1000000b895:[<0x1000000b87a/PPTOOLS1.PPA v6>,<0x1000000b7f0/PWRPNT v228>,<0x1000000b7ed/~dmtmp v30>,<0x10000000007/client9 v1485>,<0x10000000006/clients v1247>,<0x10000000005/tmp v1208>,<0x100000001fb/client.0 v1203>,<0x100000001fa/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4 v1195>,<0x10000000001/sv_1 v1217>,<0x10000000000/qa v1183>,<0x1/volumes v1163>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-03-22T02:27:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:28 smithi083 ceph-mon[36340]: pgmap v777: 129 pgs: 129 active+clean; 33 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 3.7 MiB/s wr, 255 op/s 2024-03-22T02:27:29.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3623 42.63 MB/sec warmup 9 sec latency 74.711 ms 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub queued for path: / 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: Scrub error on inode 0x1000000b87e (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: Scrub error on inode 0x1000000b888 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: Scrub error on inode 0x1000000b8ac (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:29.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: Scrub error on inode 0x1000000b8a1 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: Scrub error on inode 0x1000000b895 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: active paths [/] 2024-03-22T02:27:30.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:29 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub queued for path: / 2024-03-22T02:27:30.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: Scrub error on inode 0x1000000b87e (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: Scrub error on inode 0x1000000b888 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: Scrub error on inode 0x1000000b8ac (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: Scrub error on inode 0x1000000b8a1 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: Scrub error on inode 0x1000000b895 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: active paths [/] 2024-03-22T02:27:30.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:29 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub queued for path: / 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: Scrub error on inode 0x1000000b87e (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: Scrub error on inode 0x1000000b888 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: Scrub error on inode 0x1000000b8ac (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: Scrub error on inode 0x1000000b8a1 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: Scrub error on inode 0x1000000b895 (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.c log and `damage ls` output for details 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: active paths [/] 2024-03-22T02:27:30.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:29 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:30.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3805 39.28 MB/sec warmup 10 sec latency 93.912 ms 2024-03-22T02:27:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:30 smithi120 ceph-mon[36803]: pgmap v778: 129 pgs: 129 active+clean; 42 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-22T02:27:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:30 smithi120 ceph-mon[36803]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:30 smithi083 ceph-mon[36340]: pgmap v778: 129 pgs: 129 active+clean; 42 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-22T02:27:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:30 smithi083 ceph-mon[36340]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:31.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:30 smithi028 ceph-mon[31916]: pgmap v778: 129 pgs: 129 active+clean; 42 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.8 MiB/s wr, 322 op/s 2024-03-22T02:27:31.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:30 smithi028 ceph-mon[31916]: scrub summary: idle+waiting paths [/] 2024-03-22T02:27:31.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3968 36.29 MB/sec warmup 11 sec latency 128.580 ms 2024-03-22T02:27:32.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 3990 33.28 MB/sec warmup 12 sec latency 862.766 ms 2024-03-22T02:27:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:32 smithi120 ceph-mon[36803]: pgmap v779: 129 pgs: 129 active+clean; 44 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 4.5 MiB/s wr, 312 op/s 2024-03-22T02:27:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:32 smithi083 ceph-mon[36340]: pgmap v779: 129 pgs: 129 active+clean; 44 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 4.5 MiB/s wr, 312 op/s 2024-03-22T02:27:33.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:32 smithi028 ceph-mon[31916]: pgmap v779: 129 pgs: 129 active+clean; 44 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 4.5 MiB/s wr, 312 op/s 2024-03-22T02:27:33.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4139 30.88 MB/sec warmup 13 sec latency 269.938 ms 2024-03-22T02:27:34.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4376 29.09 MB/sec warmup 14 sec latency 91.754 ms 2024-03-22T02:27:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:34 smithi120 ceph-mon[36803]: pgmap v780: 129 pgs: 129 active+clean; 78 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 7.3 MiB/s wr, 342 op/s 2024-03-22T02:27:35.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:34 smithi083 ceph-mon[36340]: pgmap v780: 129 pgs: 129 active+clean; 78 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 7.3 MiB/s wr, 342 op/s 2024-03-22T02:27:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- 172.21.15.28:0/3815936516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c102660 msgr2=0x7f962c108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 --2- 172.21.15.28:0/3815936516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c102660 0x7f962c108ef0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f9624009ca0 tx=0x7f962402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- 172.21.15.28:0/3815936516 shutdown_connections 2024-03-22T02:27:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 --2- 172.21.15.28:0/3815936516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c102660 0x7f962c108ef0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 --2- 172.21.15.28:0/3815936516 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c101ca0 0x7f962c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 --2- 172.21.15.28:0/3815936516 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 0x7f962c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- 172.21.15.28:0/3815936516 >> 172.21.15.28:0/3815936516 conn(0x7f962c0fc020 msgr2=0x7f962c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- 172.21.15.28:0/3815936516 shutdown_connections 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- 172.21.15.28:0/3815936516 wait complete. 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 Processor -- start 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.025+0000 7f9633df6640 1 -- start start 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 0x7f962c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c102660 0x7f962c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f962c10c730 con 0x7f962c101ca0 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f962c10c5b0 con 0x7f962c102660 2024-03-22T02:27:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9633df6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f962c10c8b0 con 0x7f962c100ab0 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 0x7f962c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37362/0 (socket says 172.21.15.28:37362) 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 -- 172.21.15.28:0/770378226 learned_addr learned my addr 172.21.15.28:0/770378226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9632393640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c102660 0x7f962c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 -- 172.21.15.28:0/770378226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 msgr2=0x7f962c069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 0x7f962c069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 -- 172.21.15.28:0/770378226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c102660 msgr2=0x7f962c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c102660 0x7f962c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 -- 172.21.15.28:0/770378226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962c19fa30 con 0x7f962c101ca0 2024-03-22T02:27:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9631391640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f961c00e9f0 tx=0x7f961c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.026+0000 7f9622ffd640 1 -- 172.21.15.28:0/770378226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c00cd80 con 0x7f962c101ca0 2024-03-22T02:27:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.027+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962c19fcc0 con 0x7f962c101ca0 2024-03-22T02:27:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.027+0000 7f9622ffd640 1 -- 172.21.15.28:0/770378226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f961c010910 con 0x7f962c101ca0 2024-03-22T02:27:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.027+0000 7f9622ffd640 1 -- 172.21.15.28:0/770378226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c0188f0 con 0x7f962c101ca0 2024-03-22T02:27:35.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.027+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f962c1a0200 con 0x7f962c101ca0 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.029+0000 7f9622ffd640 1 -- 172.21.15.28:0/770378226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f961c010ac0 con 0x7f962c101ca0 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.029+0000 7f96369dc740 1 Processor -- start 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 -- start start 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9622ffd640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 0x7f96180a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b7a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560ad4ad7040 con 0x560ad4b72ed0 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560ad4a7f9e0 con 0x560ad4b78040 2024-03-22T02:27:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f96369dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560ad4ad2ff0 con 0x560ad4b724e0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9632393640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 0x7f96180a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9622ffd640 1 -- 172.21.15.28:0/770378226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f961c0cfbf0 con 0x7f962c101ca0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33622/0 (socket says 172.21.15.28:33622) 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37376/0 (socket says 172.21.15.28:37376) 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 -- 172.21.15.28:0/2709135133 learned_addr learned my addr 172.21.15.28:0/2709135133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9632393640 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b7a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 -- 172.21.15.28:0/2709135133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 msgr2=0x560ad4b7a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b7a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 -- 172.21.15.28:0/2709135133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 msgr2=0x560ad4b779d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631b92640 1 -- 172.21.15.28:0/2709135133 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad47eb210 con 0x560ad4b724e0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.030+0000 7f9631391640 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f9632393640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 0x7f96180a5500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9624005e30 tx=0x7f9624009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f9631b92640 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9628009ca0 tx=0x7f9628011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5823033564fb4e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f9617fff640 1 -- 172.21.15.28:0/2709135133 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628004220 con 0x560ad4b724e0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f9617fff640 1 -- 172.21.15.28:0/2709135133 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96280043c0 con 0x560ad4b724e0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f9617fff640 1 -- 172.21.15.28:0/2709135133 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628018810 con 0x560ad4b724e0 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f96369dc740 1 -- 172.21.15.28:0/2709135133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 msgr2=0x560ad4b72900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.031+0000 7f96369dc740 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9628009ca0 tx=0x7f9628011810 comp rx=0 tx=0).stop 2024-03-22T02:27:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- 172.21.15.28:0/2709135133 shutdown_connections 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b7a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b779d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- 172.21.15.28:0/2709135133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b72900 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- 172.21.15.28:0/2709135133 >> 172.21.15.28:0/2709135133 conn(0x560ad4b6da50 msgr2=0x560ad4b6de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- 172.21.15.28:0/2709135133 shutdown_connections 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- 172.21.15.28:0/2709135133 wait complete. 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- start start 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 Processor -- start 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b839a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560ad4b7b040 con 0x560ad4b72ed0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560ad4a92db0 con 0x560ad4b78040 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560ad4b7b1c0 con 0x560ad4b724e0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.032+0000 7f96369dc740 10 client.0 _ll_register_callbacks cb 0x560ad47eaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33636/0 (socket says 172.21.15.28:33636) 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37392/0 (socket says 172.21.15.28:37392) 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631b92640 1 -- 172.21.15.28:0/833344385 learned_addr learned my addr 172.21.15.28:0/833344385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9632393640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.042 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 -- 172.21.15.28:0/833344385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 msgr2=0x560ad4b7c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 -- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 msgr2=0x560ad4b839a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b839a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 -- 172.21.15.28:0/833344385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad48b7620 con 0x560ad4b72ed0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f9631391640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f961c098930 tx=0x7f961c01d800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1d67780a72f3e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96157fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c026040 con 0x560ad4b72ed0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f961c0af020 con 0x560ad4b72ed0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96280043c0 con 0x560ad4b72ed0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f961c0bc0e0 con 0x560ad4b72ed0 2024-03-22T02:27:35.043 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96369dc740 10 client.19569 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.033+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560ad4b86d80 con 0x560ad4b72ed0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96280043c0 con 0x560ad4b72ed0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f961c0af4d0 con 0x560ad4b72ed0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 10 client.19569 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 10 client.19569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 20 client.19569 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4b884e0 0x560ad4b8a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 4 client.19569 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ad4ae4580 con 0x560ad4b884e0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f9631b92640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4b884e0 0x560ad4b8a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f9631b92640 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4b884e0 0x560ad4b8a910 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8d8b2b3321901b8 server_cookie=f5cddb6664caad66 in_seq=0 out_seq=0 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.034+0000 7f96157fa640 10 client.19569 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:35.044 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96157fa640 1 -- 172.21.15.28:0/833344385 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560ad4ae4580 con 0x560ad4b884e0 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96157fa640 10 client.19569 handle_command_reply: tid=0 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 client.19569 shutdown 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 msgr2=0x560ad4b7c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f961c098930 tx=0x7f961c01d800 comp rx=0 tx=0).stop 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 shutdown_connections 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4b884e0 0x560ad4b8a910 crc :-1 s=READY pgs=356 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b839a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b7c960 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 --2- 172.21.15.28:0/833344385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96157fa640 0 client.19569 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 >> 172.21.15.28:0/833344385 conn(0x560ad4b6da50 msgr2=0x560ad4b6cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 shutdown_connections 2024-03-22T02:27:35.045 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 1 -- 172.21.15.28:0/833344385 wait complete. 2024-03-22T02:27:35.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.036+0000 7f96369dc740 20 client.19569 trim_cache size 0 max 16384 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 Processor -- start 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 -- start start 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b83960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1ad210 con 0x560ad4b72ed0 2024-03-22T02:27:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1aaa80 con 0x560ad4b78040 2024-03-22T02:27:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.052+0000 7f96369dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1a88d0 con 0x560ad4b724e0 2024-03-22T02:27:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33648/0 (socket says 172.21.15.28:33648) 2024-03-22T02:27:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 -- 172.21.15.28:0/3313524739 learned_addr learned my addr 172.21.15.28:0/3313524739 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631391640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b83960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9632393640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 -- 172.21.15.28:0/3313524739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 msgr2=0x560ad4b738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 -- 172.21.15.28:0/3313524739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 msgr2=0x560ad4b83960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b83960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 -- 172.21.15.28:0/3313524739 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ad4b732b0 con 0x560ad4b724e0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631391640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b83960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.053+0000 7f9631b92640 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9628014040 tx=0x7f9628009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42cb9c5f7fe15d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.054+0000 7f96177fe640 1 -- 172.21.15.28:0/3313524739 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f962801e040 con 0x560ad4b724e0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.054+0000 7f96177fe640 1 -- 172.21.15.28:0/3313524739 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9628009e50 con 0x560ad4b724e0 2024-03-22T02:27:35.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.054+0000 7f96177fe640 1 -- 172.21.15.28:0/3313524739 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9628003850 con 0x560ad4b724e0 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 -- 172.21.15.28:0/3313524739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 msgr2=0x560ad4b7ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9628014040 tx=0x7f9628009ba0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 -- 172.21.15.28:0/3313524739 shutdown_connections 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b78040 0x560ad4b738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b72ed0 0x560ad4b83960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 --2- 172.21.15.28:0/3313524739 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b724e0 0x560ad4b7ba40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.060 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 -- 172.21.15.28:0/3313524739 >> 172.21.15.28:0/3313524739 conn(0x560ad4b6da50 msgr2=0x560ad4b773d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.055+0000 7f96369dc740 1 -- 172.21.15.28:0/3313524739 shutdown_connections 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 -- 172.21.15.28:0/3313524739 wait complete. 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 -- start start 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 Processor -- start 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 0x560ad4b81b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 0x560ad4b82ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1a8f40 con 0x560ad4b78040 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1aa3b0 con 0x560ad4b724e0 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f962c1a9610 con 0x560ad4b72ed0 2024-03-22T02:27:35.061 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f96369dc740 10 client.0 _ll_register_callbacks cb 0x7f961c09bd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.056+0000 7f9631b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 0x560ad4b81b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631b92640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50394/0 (socket says 172.21.15.28:50394) 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631b92640 1 -- 172.21.15.28:0/1506781557 learned_addr learned my addr 172.21.15.28:0/1506781557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9632393640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 0x560ad4b82ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 -- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 msgr2=0x560ad4b815f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 -- 172.21.15.28:0/1506781557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 msgr2=0x560ad4b82ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 0x560ad4b82ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 -- 172.21.15.28:0/1506781557 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ad4b830c0 con 0x560ad4b72ed0 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9632393640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 0x560ad4b82ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631b92640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f9631391640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 0x560ad4b81b30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f961c09bc60 tx=0x7f961c09b3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdaf2a787761c6af server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f96157fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c0ae930 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9628009e50 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.057+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f961c0cd300 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f961c09bf60 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 10 client.29386 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560ad4beed40 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9628009e50 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f961c0b8070 con 0x560ad4b72ed0 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 10 client.29386 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 10 client.29386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 20 client.29386 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4bef5d0 0x560ad4bf19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 4 client.29386 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560ad4b7c210 con 0x560ad4bef5d0 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.058+0000 7f9631b92640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4bef5d0 0x560ad4bf19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.059+0000 7f96157fa640 10 client.29386 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.059+0000 7f9631b92640 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4bef5d0 0x560ad4bf19f0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f52651107892adde server_cookie=7edaab5790e559ce in_seq=0 out_seq=0 2024-03-22T02:27:35.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:34 smithi028 ceph-mon[31916]: pgmap v780: 129 pgs: 129 active+clean; 78 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 7.3 MiB/s wr, 342 op/s 2024-03-22T02:27:35.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:27:34.626+0000 7fa3d2c81700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-22T02:27:35.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4521 27.45 MB/sec warmup 15 sec latency 511.432 ms 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96157fa640 1 -- 172.21.15.28:0/1506781557 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560ad4b7c210 con 0x560ad4bef5d0 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96157fa640 10 client.29386 handle_command_reply: tid=0 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 client.29386 shutdown 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 msgr2=0x560ad4b81b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 0x560ad4b81b30 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f961c09bc60 tx=0x7f961c09b3b0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 shutdown_connections 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560ad4bef5d0 0x560ad4bf19f0 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ad4b78040 0x560ad4b82ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560ad4b72ed0 0x560ad4b81b30 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 --2- 172.21.15.28:0/1506781557 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560ad4b724e0 0x560ad4b815f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96157fa640 0 client.29386 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 >> 172.21.15.28:0/1506781557 conn(0x560ad4b6da50 msgr2=0x560ad4b74160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 shutdown_connections 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 1 -- 172.21.15.28:0/1506781557 wait complete. 2024-03-22T02:27:35.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.992+0000 7f96369dc740 20 client.29386 trim_cache size 0 max 16384 2024-03-22T02:27:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:27:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:27:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:27:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 msgr2=0x7f96180a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 0x7f96180a5500 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9624005e30 tx=0x7f9624009800 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 msgr2=0x7f962c19cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f961c00e9f0 tx=0x7f961c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 shutdown_connections 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f96180a3040 0x7f96180a5500 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f962c102660 0x7f962c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f962c101ca0 0x7f962c19cb00 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 --2- 172.21.15.28:0/770378226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f962c100ab0 0x7f962c069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 >> 172.21.15.28:0/770378226 conn(0x7f962c0fc020 msgr2=0x7f962c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 shutdown_connections 2024-03-22T02:27:35.997 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:35.993+0000 7f9633df6640 1 -- 172.21.15.28:0/770378226 wait complete. 2024-03-22T02:27:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-22T02:27:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:27:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:27:35.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:27:35.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:35 smithi120 ceph-mon[36803]: scrub summary: idle 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:27:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:35 smithi083 ceph-mon[36340]: scrub summary: idle 2024-03-22T02:27:36.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-03-22T02:27:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] up:active 2024-03-22T02:27:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:27:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:27:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:27:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:35 smithi028 ceph-mon[31916]: scrub summary: idle 2024-03-22T02:27:36.672 INFO:tasks.workunit.client.0.smithi028.stdout: 10 4811 26.97 MB/sec warmup 16 sec latency 654.306 ms 2024-03-22T02:27:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:36 smithi120 ceph-mon[36803]: pgmap v781: 129 pgs: 129 active+clean; 174 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 16 MiB/s wr, 389 op/s 2024-03-22T02:27:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:36 smithi083 ceph-mon[36340]: pgmap v781: 129 pgs: 129 active+clean; 174 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 16 MiB/s wr, 389 op/s 2024-03-22T02:27:37.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:36 smithi028 ceph-mon[31916]: pgmap v781: 129 pgs: 129 active+clean; 174 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 16 MiB/s wr, 389 op/s 2024-03-22T02:27:37.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 5212 26.41 MB/sec warmup 17 sec latency 108.173 ms 2024-03-22T02:27:38.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 5772 25.79 MB/sec warmup 18 sec latency 98.939 ms 2024-03-22T02:27:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:38 smithi120 ceph-mon[36803]: pgmap v782: 129 pgs: 129 active+clean; 212 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 18 MiB/s wr, 354 op/s 2024-03-22T02:27:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:38 smithi083 ceph-mon[36340]: pgmap v782: 129 pgs: 129 active+clean; 212 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 18 MiB/s wr, 354 op/s 2024-03-22T02:27:39.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:38 smithi028 ceph-mon[31916]: pgmap v782: 129 pgs: 129 active+clean; 212 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 18 MiB/s wr, 354 op/s 2024-03-22T02:27:39.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 6222 26.27 MB/sec warmup 19 sec latency 59.745 ms 2024-03-22T02:27:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:39 smithi120 ceph-mon[36803]: pgmap v783: 129 pgs: 129 active+clean; 206 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 578 op/s 2024-03-22T02:27:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:39 smithi083 ceph-mon[36340]: pgmap v783: 129 pgs: 129 active+clean; 206 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 578 op/s 2024-03-22T02:27:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:39 smithi028 ceph-mon[31916]: pgmap v783: 129 pgs: 129 active+clean; 206 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 18 MiB/s wr, 578 op/s 2024-03-22T02:27:40.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 6743 26.92 MB/sec warmup 20 sec latency 59.726 ms 2024-03-22T02:27:41.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7095 27.09 MB/sec warmup 21 sec latency 80.235 ms 2024-03-22T02:27:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:41 smithi120 ceph-mon[36803]: pgmap v784: 129 pgs: 129 active+clean; 208 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 595 op/s 2024-03-22T02:27:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:41 smithi083 ceph-mon[36340]: pgmap v784: 129 pgs: 129 active+clean; 208 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 595 op/s 2024-03-22T02:27:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:41 smithi028 ceph-mon[31916]: pgmap v784: 129 pgs: 129 active+clean; 208 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 17 MiB/s wr, 595 op/s 2024-03-22T02:27:42.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7298 26.50 MB/sec warmup 22 sec latency 88.241 ms 2024-03-22T02:27:43.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7472 25.63 MB/sec warmup 23 sec latency 89.658 ms 2024-03-22T02:27:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:43 smithi120 ceph-mon[36803]: pgmap v785: 129 pgs: 129 active+clean; 205 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 17 MiB/s wr, 581 op/s 2024-03-22T02:27:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:43 smithi083 ceph-mon[36340]: pgmap v785: 129 pgs: 129 active+clean; 205 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 17 MiB/s wr, 581 op/s 2024-03-22T02:27:44.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:43 smithi028 ceph-mon[31916]: pgmap v785: 129 pgs: 129 active+clean; 205 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 17 MiB/s wr, 581 op/s 2024-03-22T02:27:44.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7621 24.67 MB/sec warmup 24 sec latency 71.831 ms 2024-03-22T02:27:45.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 7792 23.79 MB/sec warmup 25 sec latency 153.007 ms 2024-03-22T02:27:46.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:45 smithi120 ceph-mon[36803]: pgmap v786: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 589 op/s 2024-03-22T02:27:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:45 smithi083 ceph-mon[36340]: pgmap v786: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 589 op/s 2024-03-22T02:27:46.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:45 smithi028 ceph-mon[31916]: pgmap v786: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 14 MiB/s wr, 589 op/s 2024-03-22T02:27:46.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 8080 23.36 MB/sec warmup 26 sec latency 76.598 ms 2024-03-22T02:27:47.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 8471 23.30 MB/sec warmup 27 sec latency 54.848 ms 2024-03-22T02:27:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:47 smithi120 ceph-mon[36803]: pgmap v787: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.4 MiB/s wr, 522 op/s 2024-03-22T02:27:48.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:47 smithi083 ceph-mon[36340]: pgmap v787: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.4 MiB/s wr, 522 op/s 2024-03-22T02:27:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:47 smithi028 ceph-mon[31916]: pgmap v787: 129 pgs: 129 active+clean; 212 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.4 MiB/s wr, 522 op/s 2024-03-22T02:27:48.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 9007 23.11 MB/sec warmup 28 sec latency 95.954 ms 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 -- 172.21.15.28:0/744791375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 msgr2=0x7f09e4102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 --2- 172.21.15.28:0/744791375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e4102130 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009900 tx=0x7f09d802bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 -- 172.21.15.28:0/744791375 shutdown_connections 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 --2- 172.21.15.28:0/744791375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4102670 0x7f09e4108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 --2- 172.21.15.28:0/744791375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e4102130 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 --2- 172.21.15.28:0/744791375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4100ac0 0x7f09e4100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 -- 172.21.15.28:0/744791375 >> 172.21.15.28:0/744791375 conn(0x7f09e40fc050 msgr2=0x7f09e40fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 -- 172.21.15.28:0/744791375 shutdown_connections 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 -- 172.21.15.28:0/744791375 wait complete. 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.093+0000 7f09e96fb640 1 Processor -- start 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 -- start start 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4100ac0 0x7f09e4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4102670 0x7f09e41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09e410c740 con 0x7f09e4101cb0 2024-03-22T02:27:49.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f09e410c5c0 con 0x7f09e4102670 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e96fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f09e410c8c0 con 0x7f09e4100ac0 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51982/0 (socket says 172.21.15.28:51982) 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 -- 172.21.15.28:0/1209673828 learned_addr learned my addr 172.21.15.28:0/1209673828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4100ac0 0x7f09e4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4102670 0x7f09e41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 -- 172.21.15.28:0/1209673828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4100ac0 msgr2=0x7f09e4199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4100ac0 0x7f09e4199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 -- 172.21.15.28:0/1209673828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4102670 msgr2=0x7f09e41a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4102670 0x7f09e41a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 -- 172.21.15.28:0/1209673828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e41a1c10 con 0x7f09e4101cb0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.094+0000 7f09e27fc640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009230 tx=0x7f09d802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.095+0000 7f09cbfff640 1 -- 172.21.15.28:0/1209673828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8041070 con 0x7f09e4101cb0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.095+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e41a1ea0 con 0x7f09e4101cb0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.095+0000 7f09cbfff640 1 -- 172.21.15.28:0/1209673828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09d8004450 con 0x7f09e4101cb0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.095+0000 7f09cbfff640 1 -- 172.21.15.28:0/1209673828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d8003e20 con 0x7f09e4101cb0 2024-03-22T02:27:49.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.095+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e41a2310 con 0x7f09e4101cb0 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.097+0000 7f09cbfff640 1 -- 172.21.15.28:0/1209673828 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f09d8044070 con 0x7f09e4101cb0 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 Processor -- start 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 -- start start 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 0x5648abc209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09cbfff640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 0x7f09cc0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648abb80040 con 0x5648abc21040 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648abb289e0 con 0x5648abc1b4e0 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648abb7bff0 con 0x5648abc1bed0 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e27fc640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 0x7f09cc0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:37854/0 (socket says 172.21.15.28:37854) 2024-03-22T02:27:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 -- 172.21.15.28:0/17092052 learned_addr learned my addr 172.21.15.28:0/17092052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 0x5648abc209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09cbfff640 1 -- 172.21.15.28:0/1209673828 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f09d80f1320 con 0x7f09e4101cb0 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 -- 172.21.15.28:0/17092052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 msgr2=0x5648abc209d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 0x5648abc209d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 -- 172.21.15.28:0/17092052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 msgr2=0x5648abc234d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc234d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e37fe640 1 -- 172.21.15.28:0/17092052 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648ab894210 con 0x5648abc1b4e0 2024-03-22T02:27:49.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.098+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 0x5648abc209d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.099+0000 7f09e27fc640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 0x7f09cc0a5480 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f09d80ba0c0 tx=0x7f09d80b8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.099+0000 7f09e37fe640 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09dc005f40 tx=0x7f09dc009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63edf538274638ef server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.099+0000 7f09c97fa640 1 -- 172.21.15.28:0/17092052 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc011070 con 0x5648abc1b4e0 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.099+0000 7f09c97fa640 1 -- 172.21.15.28:0/17092052 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc003e50 con 0x5648abc1b4e0 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.099+0000 7f09c97fa640 1 -- 172.21.15.28:0/17092052 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc004290 con 0x5648abc1b4e0 2024-03-22T02:27:49.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- 172.21.15.28:0/17092052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 msgr2=0x5648abc1b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09dc005f40 tx=0x7f09dc009b80 comp rx=0 tx=0).stop 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- 172.21.15.28:0/17092052 shutdown_connections 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc234d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1bed0 0x5648abc209d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/17092052 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1b4e0 0x5648abc1b900 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- 172.21.15.28:0/17092052 >> 172.21.15.28:0/17092052 conn(0x5648abc16a50 msgr2=0x5648abc16e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- 172.21.15.28:0/17092052 shutdown_connections 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- 172.21.15.28:0/17092052 wait complete. 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- start start 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 Processor -- start 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc25ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5648abc23ff0 con 0x5648abc1bed0 2024-03-22T02:27:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5648abb3bdb0 con 0x5648abc21040 2024-03-22T02:27:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5648abc24170 con 0x5648abc1b4e0 2024-03-22T02:27:49.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09ec3c2740 10 client.0 _ll_register_callbacks cb 0x5648ab893a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55630/0 (socket says 172.21.15.28:55630) 2024-03-22T02:27:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09e37fe640 1 -- 172.21.15.28:0/2034147244 learned_addr learned my addr 172.21.15.28:0/2034147244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09e37fe640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.119 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.100+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc25ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 -- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 msgr2=0x5648abc2ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 -- 172.21.15.28:0/2034147244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 msgr2=0x5648abc25ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc25ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 -- 172.21.15.28:0/2034147244 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648ab960620 con 0x5648abc1b4e0 2024-03-22T02:27:49.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09e37fe640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00fa40 tx=0x7f09dc009f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b627383ab948c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc004a40 con 0x5648abc1b4e0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09dc003e50 con 0x5648abc1b4e0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f09dc004be0 con 0x5648abc1b4e0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f09dc004de0 con 0x5648abc1b4e0 2024-03-22T02:27:49.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09ec3c2740 10 client.29398 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5648abc2fe00 con 0x5648abc1b4e0 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.101+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc003e50 con 0x5648abc1b4e0 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f09dc0051e0 con 0x5648abc1b4e0 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 10 client.29398 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 10 client.29398 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:49.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 20 client.29398 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abc312f0 0x5648abc336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 4 client.29398 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:49.123 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648abb8d580 con 0x5648abc312f0 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abc312f0 0x5648abc336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abc312f0 0x5648abc336e0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5805a5325555c5b server_cookie=c0079622e40862ef in_seq=0 out_seq=0 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.102+0000 7f09c6ff5640 10 client.29398 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09c6ff5640 1 -- 172.21.15.28:0/2034147244 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5648abb8d580 con 0x5648abc312f0 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09c6ff5640 10 client.29398 handle_command_reply: tid=0 2024-03-22T02:27:49.124 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 client.29398 shutdown 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 msgr2=0x5648abc25580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00fa40 tx=0x7f09dc009f20 comp rx=0 tx=0).stop 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 shutdown_connections 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abc312f0 0x5648abc336e0 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc25ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/2034147244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25580 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09c6ff5640 0 client.29398 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.103+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 >> 172.21.15.28:0/2034147244 conn(0x5648abc16a50 msgr2=0x5648abc15c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.104+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 shutdown_connections 2024-03-22T02:27:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.104+0000 7f09ec3c2740 1 -- 172.21.15.28:0/2034147244 wait complete. 2024-03-22T02:27:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.104+0000 7f09ec3c2740 20 client.29398 trim_cache size 0 max 16384 2024-03-22T02:27:49.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.120+0000 7f09ec3c2740 1 Processor -- start 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.120+0000 7f09ec3c2740 1 -- start start 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.120+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1bed0 0x5648abc2ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41af4a0 con 0x5648abc21040 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41acd10 con 0x5648abc1bed0 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41aab60 con 0x5648abc1b4e0 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52024/0 (socket says 172.21.15.28:52024) 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 -- 172.21.15.28:0/1468769624 learned_addr learned my addr 172.21.15.28:0/1468769624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1bed0 0x5648abc2ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 -- 172.21.15.28:0/1468769624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 msgr2=0x5648abc25a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 -- 172.21.15.28:0/1468769624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1bed0 msgr2=0x5648abc2ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1bed0 0x5648abc2ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 -- 172.21.15.28:0/1468769624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648abc33d40 con 0x5648abc21040 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009ef0 tx=0x7f09dc000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46c12a3f4ca15aa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09c8ff9640 1 -- 172.21.15.28:0/1468769624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc005780 con 0x5648abc21040 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09c8ff9640 1 -- 172.21.15.28:0/1468769624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc036070 con 0x5648abc21040 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.121+0000 7f09c8ff9640 1 -- 172.21.15.28:0/1468769624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00fcd0 con 0x5648abc21040 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1468769624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 msgr2=0x5648abc1a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f09dc009ef0 tx=0x7f09dc000f40 comp rx=0 tx=0).stop 2024-03-22T02:27:49.134 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1468769624 shutdown_connections 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc21040 0x5648abc1a3c0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc1bed0 0x5648abc2ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1468769624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc25a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1468769624 >> 172.21.15.28:0/1468769624 conn(0x5648abc16a50 msgr2=0x5648abc19cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1468769624 shutdown_connections 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1468769624 wait complete. 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 -- start start 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.122+0000 7f09ec3c2740 1 Processor -- start 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc2a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.135 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41ab1d0 con 0x5648abc1bed0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41ac640 con 0x5648abc21040 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f09e41ab8a0 con 0x5648abc1b4e0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 10 client.0 _ll_register_callbacks cb 0x7f09dc011630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52034/0 (socket says 172.21.15.28:52034) 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 -- 172.21.15.28:0/1467935114 learned_addr learned my addr 172.21.15.28:0/1467935114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc2a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 -- 172.21.15.28:0/1467935114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 msgr2=0x5648abc2a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc2a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 -- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 msgr2=0x5648abc2b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.136 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 -- 172.21.15.28:0/1467935114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648abc2be30 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09e2ffd640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4020630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f8c9ebb341f1b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4024030 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f09d4005020 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.123+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09dc036070 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f09d402e050 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 10 client.19593 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5648abc9ffd0 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09dc036070 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f09d40035f0 con 0x5648abc1bed0 2024-03-22T02:27:49.137 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 10 client.19593 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 10 client.19593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 20 client.19593 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abca0a30 0x5648abca2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 4 client.19593 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5648abb878d0 con 0x5648abca0a30 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.124+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abca0a30 0x5648abca2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.125+0000 7f09c6ff5640 10 client.19593 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:49.138 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.125+0000 7f09e37fe640 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abca0a30 0x5648abca2e50 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c2f0c0378fe552 server_cookie=56a6903fc7f7d1fb in_seq=0 out_seq=0 2024-03-22T02:27:49.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 9255 22.60 MB/sec warmup 29 sec latency 519.731 ms 2024-03-22T02:27:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09c6ff5640 1 -- 172.21.15.28:0/1467935114 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5648abb878d0 con 0x5648abca0a30 2024-03-22T02:27:49.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09c6ff5640 10 client.19593 handle_command_reply: tid=0 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 client.19593 shutdown 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 msgr2=0x5648abc2a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f09d4000c00 tx=0x7f09d4020630 comp rx=0 tx=0).stop 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 shutdown_connections 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5648abca0a30 0x5648abca2e50 crc :-1 s=READY pgs=359 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5648abc21040 0x5648abc2b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5648abc1bed0 0x5648abc2a830 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09c6ff5640 0 client.19593 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 --2- 172.21.15.28:0/1467935114 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5648abc1b4e0 0x5648abc2a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 >> 172.21.15.28:0/1467935114 conn(0x5648abc16a50 msgr2=0x5648abc1a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 shutdown_connections 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 1 -- 172.21.15.28:0/1467935114 wait complete. 2024-03-22T02:27:49.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.897+0000 7f09ec3c2740 20 client.19593 trim_cache size 0 max 16384 2024-03-22T02:27:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:27:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:27:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:27:49.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 msgr2=0x7f09cc0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 0x7f09cc0a5480 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f09d80ba0c0 tx=0x7f09d80b8040 comp rx=0 tx=0).stop 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 msgr2=0x7f09e419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009230 tx=0x7f09d802c650 comp rx=0 tx=0).stop 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 shutdown_connections 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f09cc0a2fc0 0x7f09cc0a5480 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f09e4102670 0x7f09e41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e4101cb0 0x7f09e419a530 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 --2- 172.21.15.28:0/1209673828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09e4100ac0 0x7f09e4199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 >> 172.21.15.28:0/1209673828 conn(0x7f09e40fc050 msgr2=0x7f09e40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 shutdown_connections 2024-03-22T02:27:49.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:27:49.898+0000 7f09e96fb640 1 -- 172.21.15.28:0/1209673828 wait complete. 2024-03-22T02:27:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:49 smithi120 ceph-mon[36803]: pgmap v788: 129 pgs: 129 active+clean; 214 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 474 op/s 2024-03-22T02:27:50.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:49 smithi083 ceph-mon[36340]: pgmap v788: 129 pgs: 129 active+clean; 214 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 474 op/s 2024-03-22T02:27:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:49 smithi028 ceph-mon[31916]: pgmap v788: 129 pgs: 129 active+clean; 214 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 2.1 MiB/s wr, 474 op/s 2024-03-22T02:27:50.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 9733 23.06 MB/sec warmup 30 sec latency 672.103 ms 2024-03-22T02:27:51.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10301 23.63 MB/sec warmup 31 sec latency 75.090 ms 2024-03-22T02:27:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:51 smithi120 ceph-mon[36803]: pgmap v789: 129 pgs: 129 active+clean; 220 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-03-22T02:27:52.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:51 smithi083 ceph-mon[36340]: pgmap v789: 129 pgs: 129 active+clean; 220 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-03-22T02:27:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:51 smithi028 ceph-mon[31916]: pgmap v789: 129 pgs: 129 active+clean; 220 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.9 MiB/s wr, 297 op/s 2024-03-22T02:27:52.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10623 23.63 MB/sec warmup 32 sec latency 61.330 ms 2024-03-22T02:27:53.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 10850 23.43 MB/sec warmup 33 sec latency 89.765 ms 2024-03-22T02:27:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:53 smithi120 ceph-mon[36803]: pgmap v790: 129 pgs: 129 active+clean; 220 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 305 op/s 2024-03-22T02:27:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:53 smithi083 ceph-mon[36340]: pgmap v790: 129 pgs: 129 active+clean; 220 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 305 op/s 2024-03-22T02:27:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:53 smithi028 ceph-mon[31916]: pgmap v790: 129 pgs: 129 active+clean; 220 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 2.1 MiB/s wr, 305 op/s 2024-03-22T02:27:54.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 11022 22.96 MB/sec warmup 34 sec latency 79.373 ms 2024-03-22T02:27:55.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 11167 22.38 MB/sec warmup 35 sec latency 129.131 ms 2024-03-22T02:27:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:55 smithi120 ceph-mon[36803]: pgmap v791: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 441 op/s 2024-03-22T02:27:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:55 smithi083 ceph-mon[36340]: pgmap v791: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 441 op/s 2024-03-22T02:27:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:55 smithi028 ceph-mon[31916]: pgmap v791: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 2.2 MiB/s wr, 441 op/s 2024-03-22T02:27:56.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 11404 21.89 MB/sec warmup 36 sec latency 97.551 ms 2024-03-22T02:27:57.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 11702 21.75 MB/sec warmup 37 sec latency 100.688 ms 2024-03-22T02:27:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:57 smithi120 ceph-mon[36803]: pgmap v792: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 396 op/s 2024-03-22T02:27:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:57 smithi083 ceph-mon[36340]: pgmap v792: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 396 op/s 2024-03-22T02:27:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:57 smithi028 ceph-mon[31916]: pgmap v792: 129 pgs: 129 active+clean; 210 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 1.7 MiB/s wr, 396 op/s 2024-03-22T02:27:58.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 12138 21.57 MB/sec warmup 38 sec latency 66.791 ms 2024-03-22T02:27:58.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-22T02:27:58.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-22T02:27:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 -- 172.21.15.28:0/1701844857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 msgr2=0x7ffa7c103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 --2- 172.21.15.28:0/1701844857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c103230 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009900 tx=0x7ffa6c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 -- 172.21.15.28:0/1701844857 shutdown_connections 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 --2- 172.21.15.28:0/1701844857 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c10a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 --2- 172.21.15.28:0/1701844857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c103230 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 --2- 172.21.15.28:0/1701844857 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 0x7ffa7c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 -- 172.21.15.28:0/1701844857 >> 172.21.15.28:0/1701844857 conn(0x7ffa7c0fd130 msgr2=0x7ffa7c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 -- 172.21.15.28:0/1701844857 shutdown_connections 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.909+0000 7ffa834a4640 1 -- 172.21.15.28:0/1701844857 wait complete. 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 Processor -- start 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 -- start start 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 0x7ffa7c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c19f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c10d820 con 0x7ffa7c102db0 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c10d6a0 con 0x7ffa7c101bc0 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa834a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c10d9a0 con 0x7ffa7c103770 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59730/0 (socket says 172.21.15.28:59730) 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa80a3f640 1 -- 172.21.15.28:0/4125253262 learned_addr learned my addr 172.21.15.28:0/4125253262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81240640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 0x7ffa7c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c19f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81a41640 1 -- 172.21.15.28:0/4125253262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 msgr2=0x7ffa7c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 0x7ffa7c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81a41640 1 -- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 msgr2=0x7ffa7c198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.910+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa81a41640 1 -- 172.21.15.28:0/4125253262 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c19fae0 con 0x7ffa7c103770 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c19f470 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009fd0 tx=0x7ffa7400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa727fc640 1 -- 172.21.15.28:0/4125253262 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400ce20 con 0x7ffa7c103770 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa727fc640 1 -- 172.21.15.28:0/4125253262 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa74005020 con 0x7ffa7c103770 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7c19fdd0 con 0x7ffa7c103770 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa727fc640 1 -- 172.21.15.28:0/4125253262 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74010720 con 0x7ffa7c103770 2024-03-22T02:27:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.911+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7c1a0290 con 0x7ffa7c103770 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 Processor -- start 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa727fc640 1 -- 172.21.15.28:0/4125253262 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ffa74005540 con 0x7ffa7c103770 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 -- start start 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba428850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 0x55eeba42d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba430420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eeba38d8c0 con 0x55eeba428430 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eeba335c20 con 0x55eeba42df90 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa8616d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eeba389870 con 0x55eeba428e20 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba428850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba428850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59746/0 (socket says 172.21.15.28:59746) 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa80a3f640 1 -- 172.21.15.28:0/959356524 learned_addr learned my addr 172.21.15.28:0/959356524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa81240640 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 0x55eeba42d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.914+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba430420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa727fc640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 0x7ffa680a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 -- 172.21.15.28:0/959356524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 msgr2=0x55eeba430420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba430420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 -- 172.21.15.28:0/959356524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 msgr2=0x55eeba428850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba428850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 -- 172.21.15.28:0/959356524 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeba35a150 con 0x55eeba428e20 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 0x7ffa680a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa81240640 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 0x55eeba42d920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009bf0 tx=0x7ffa78011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71bdcef201c564de server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa727fc640 1 -- 172.21.15.28:0/4125253262 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ffa740cfe80 con 0x7ffa7c103770 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa677fe640 1 -- 172.21.15.28:0/959356524 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78004030 con 0x55eeba428e20 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.915+0000 7ffa677fe640 1 -- 172.21.15.28:0/959356524 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa78005500 con 0x55eeba428e20 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa677fe640 1 -- 172.21.15.28:0/959356524 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78004e40 con 0x55eeba428e20 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa8616d740 1 -- 172.21.15.28:0/959356524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 msgr2=0x55eeba42d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa81a41640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 0x7ffa680a5460 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a4680 tx=0x7ffa740054b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa8616d740 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 0x55eeba42d920 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009bf0 tx=0x7ffa78011810 comp rx=0 tx=0).stop 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa8616d740 1 -- 172.21.15.28:0/959356524 shutdown_connections 2024-03-22T02:27:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.916+0000 7ffa8616d740 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba430420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428e20 0x55eeba42d920 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 --2- 172.21.15.28:0/959356524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba428850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- 172.21.15.28:0/959356524 >> 172.21.15.28:0/959356524 conn(0x55eeba4239a0 msgr2=0x55eeba423da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- 172.21.15.28:0/959356524 shutdown_connections 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- 172.21.15.28:0/959356524 wait complete. 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- start start 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 Processor -- start 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba4328d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba42df90 0x55eeba439e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eeba430f90 con 0x55eeba428430 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55eeba349130 con 0x55eeba428e20 2024-03-22T02:27:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.917+0000 7ffa8616d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55eeba431110 con 0x55eeba42df90 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa8616d740 10 client.0 _ll_register_callbacks cb 0x55eeba348b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33980/0 (socket says 172.21.15.28:33980) 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 -- 172.21.15.28:0/3996196295 learned_addr learned my addr 172.21.15.28:0/3996196295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba4328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 -- 172.21.15.28:0/3996196295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba42df90 msgr2=0x55eeba439e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba42df90 0x55eeba439e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 -- 172.21.15.28:0/3996196295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 msgr2=0x55eeba4328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba4328d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 -- 172.21.15.28:0/3996196295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeba16b7d0 con 0x55eeba428e20 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba4328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa81240640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffa78000c00 tx=0x7ffa78005260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17877c9821ec675f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.918+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78028020 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa78005500 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ffa78019430 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffa78033050 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 10 client.29417 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55eeba0a13f0 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa78005500 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7ffa780244b0 con 0x55eeba428e20 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 10 client.29417 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 10 client.29417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 20 client.29417 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:58.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba43eb30 0x55eeba440fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 4 client.29417 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.919+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eeba39ae00 con 0x55eeba43eb30 2024-03-22T02:27:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.920+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba43eb30 0x55eeba440fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.921+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba43eb30 0x55eeba440fb0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=659535863786bb2d server_cookie=7cc3e3454dee0ad0 in_seq=0 out_seq=0 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.921+0000 7ffa64ff9640 10 client.29417 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa64ff9640 1 -- 172.21.15.28:0/3996196295 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55eeba39ae00 con 0x55eeba43eb30 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa64ff9640 10 client.29417 handle_command_reply: tid=0 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 client.29417 shutdown 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 msgr2=0x55eeba432e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffa78000c00 tx=0x7ffa78005260 comp rx=0 tx=0).stop 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 shutdown_connections 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba43eb30 0x55eeba440fb0 crc :-1 s=READY pgs=360 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba42df90 0x55eeba439e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba432e10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 --2- 172.21.15.28:0/3996196295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428430 0x55eeba4328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 >> 172.21.15.28:0/3996196295 conn(0x55eeba4239a0 msgr2=0x55eeba422c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa64ff9640 0 client.29417 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 shutdown_connections 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 1 -- 172.21.15.28:0/3996196295 wait complete. 2024-03-22T02:27:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.924+0000 7ffa8616d740 20 client.29417 trim_cache size 0 max 16384 2024-03-22T02:27:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 Processor -- start 2024-03-22T02:27:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 -- start start 2024-03-22T02:27:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba423d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba42a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1ad320 con 0x55eeba428e20 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1aab90 con 0x55eeba42df90 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa8616d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1a89e0 con 0x55eeba428430 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59768/0 (socket says 172.21.15.28:59768) 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 -- 172.21.15.28:0/1511813931 learned_addr learned my addr 172.21.15.28:0/1511813931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba423d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba42a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 -- 172.21.15.28:0/1511813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 msgr2=0x55eeba423d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba423d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 -- 172.21.15.28:0/1511813931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 msgr2=0x55eeba42a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba42a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa81240640 1 -- 172.21.15.28:0/1511813931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eeba0610f0 con 0x55eeba428e20 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.945+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba42a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.946+0000 7ffa81240640 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009250 tx=0x7ffa78005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb08757544a9f94 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.946+0000 7ffa66ffd640 1 -- 172.21.15.28:0/1511813931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78028070 con 0x55eeba428e20 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.946+0000 7ffa66ffd640 1 -- 172.21.15.28:0/1511813931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa7801f040 con 0x55eeba428e20 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.946+0000 7ffa66ffd640 1 -- 172.21.15.28:0/1511813931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78024680 con 0x55eeba428e20 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- 172.21.15.28:0/1511813931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 msgr2=0x55eeba439e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009250 tx=0x7ffa78005fd0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- 172.21.15.28:0/1511813931 shutdown_connections 2024-03-22T02:27:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba42df90 0x55eeba42a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba428e20 0x55eeba439e10 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1511813931 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba423d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- 172.21.15.28:0/1511813931 >> 172.21.15.28:0/1511813931 conn(0x55eeba4239a0 msgr2=0x55eeba41ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- 172.21.15.28:0/1511813931 shutdown_connections 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- 172.21.15.28:0/1511813931 wait complete. 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- start start 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 Processor -- start 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba439a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba4364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1a9050 con 0x55eeba42df90 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1aa4c0 con 0x55eeba428e20 2024-03-22T02:27:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c1a9720 con 0x55eeba428430 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.947+0000 7ffa8616d740 10 client.0 _ll_register_callbacks cb 0x7ffa78009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa81240640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba4364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59770/0 (socket says 172.21.15.28:59770) 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 -- 172.21.15.28:0/1881499580 learned_addr learned my addr 172.21.15.28:0/1881499580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba439a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 -- 172.21.15.28:0/1881499580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 msgr2=0x55eeba439a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba439a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 -- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 msgr2=0x55eeba4364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba4364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 -- 172.21.15.28:0/1881499580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eeba436f40 con 0x55eeba42df90 2024-03-22T02:27:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa81240640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba4364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa80a3f640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c006010 tx=0x7ffa78004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fa337456a9d44 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c04d730 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7ffa6c0047a0 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa7801f040 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7ffa6c05e050 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa8616d740 10 client.19611 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.948+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55eeba4a3ef0 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa7801f040 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7ffa6c0035f0 con 0x55eeba42df90 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 10 client.19611 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 10 client.19611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 20 client.19611 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba4a4830 0x55eeba4a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 4 client.19611 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-22T02:27:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eeba395150 con 0x55eeba4a4830 2024-03-22T02:27:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa81240640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba4a4830 0x55eeba4a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa81240640 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba4a4830 0x55eeba4a6c50 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e984ed5c85e6b3e8 server_cookie=b57dfb02eddfdf39 in_seq=0 out_seq=0 2024-03-22T02:27:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.949+0000 7ffa64ff9640 10 client.19611 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:58.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa64ff9640 1 -- 172.21.15.28:0/1881499580 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55eeba395150 con 0x55eeba4a4830 2024-03-22T02:27:58.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa64ff9640 10 client.19611 handle_command_reply: tid=0 2024-03-22T02:27:58.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 client.19611 shutdown 2024-03-22T02:27:58.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 msgr2=0x55eeba436a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c006010 tx=0x7ffa78004e90 comp rx=0 tx=0).stop 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 shutdown_connections 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55eeba4a4830 0x55eeba4a6c50 crc :-1 s=READY pgs=361 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eeba42df90 0x55eeba436a00 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55eeba428e20 0x55eeba4364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa64ff9640 0 client.19611 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 --2- 172.21.15.28:0/1881499580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eeba428430 0x55eeba439a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 >> 172.21.15.28:0/1881499580 conn(0x55eeba4239a0 msgr2=0x55eeba440b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 shutdown_connections 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 1 -- 172.21.15.28:0/1881499580 wait complete. 2024-03-22T02:27:58.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.951+0000 7ffa8616d740 20 client.19611 trim_cache size 0 max 16384 2024-03-22T02:27:58.959 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:27:58.959 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-03-22T02:27:58.959 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-03-22T02:27:58.959 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 msgr2=0x7ffa680a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 0x7ffa680a5460 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ffa740a4680 tx=0x7ffa740054b0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 msgr2=0x7ffa7c19f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c19f470 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ffa74009fd0 tx=0x7ffa7400ef10 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 shutdown_connections 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ffa680a2fa0 0x7ffa680a5460 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa7c103770 0x7ffa7c19f470 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa7c102db0 0x7ffa7c198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 --2- 172.21.15.28:0/4125253262 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ffa7c101bc0 0x7ffa7c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 >> 172.21.15.28:0/4125253262 conn(0x7ffa7c0fd130 msgr2=0x7ffa7c106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 shutdown_connections 2024-03-22T02:27:58.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:58.952+0000 7ffa834a4640 1 -- 172.21.15.28:0/4125253262 wait complete. 2024-03-22T02:27:58.966 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-22T02:27:58.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:27:59.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- 172.21.15.28:0/1143339218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90101cb0 msgr2=0x7f7e90102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- 172.21.15.28:0/1143339218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90101cb0 0x7f7e90102130 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f7e84009900 tx=0x7f7e8402bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- 172.21.15.28:0/1143339218 shutdown_connections 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- 172.21.15.28:0/1143339218 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 0x7f7e90108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- 172.21.15.28:0/1143339218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90101cb0 0x7f7e90102130 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- 172.21.15.28:0/1143339218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90100ac0 0x7f7e90100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- 172.21.15.28:0/1143339218 >> 172.21.15.28:0/1143339218 conn(0x7f7e900fc070 msgr2=0x7f7e900fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- 172.21.15.28:0/1143339218 shutdown_connections 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- 172.21.15.28:0/1143339218 wait complete. 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 Processor -- start 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 -- start start 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.078+0000 7f7e95563640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90101cb0 0x7f7e9019cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 0x7f7e9019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9010c720 con 0x7f7e90100ac0 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9010c5a0 con 0x7f7e90102670 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9010c8a0 con 0x7f7e90101cb0 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59782/0 (socket says 172.21.15.28:59782) 2024-03-22T02:27:59.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 -- 172.21.15.28:0/4092824838 learned_addr learned my addr 172.21.15.28:0/4092824838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8f7fe640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 0x7f7e9019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8e7fc640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90101cb0 0x7f7e9019cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 -- 172.21.15.28:0/4092824838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90101cb0 msgr2=0x7f7e9019cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90101cb0 0x7f7e9019cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 -- 172.21.15.28:0/4092824838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 msgr2=0x7f7e9019f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 0x7f7e9019f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 -- 172.21.15.28:0/4092824838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9019fb50 con 0x7f7e90100ac0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e8effd640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000e9c0 tx=0x7f7e8000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8000cd80 con 0x7f7e90100ac0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9019fde0 con 0x7f7e90100ac0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e800035f0 con 0x7f7e90100ac0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80010690 con 0x7f7e90100ac0 2024-03-22T02:27:59.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.079+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e901a0250 con 0x7f7e90100ac0 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.080+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e90107de0 con 0x7f7e90100ac0 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.082+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7e800037a0 con 0x7f7e90100ac0 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.082+0000 7f7e77fff640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 0x7f7e780a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.082+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f7e800cfc20 con 0x7f7e90100ac0 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.082+0000 7f7e8e7fc640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 0x7f7e780a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.083+0000 7f7e8e7fc640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 0x7f7e780a53e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7e8402c7b0 tx=0x7f7e84005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.086+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e8009c6d0 con 0x7f7e90100ac0 2024-03-22T02:27:59.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.334+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e9002d3b0 con 0x7f7e90100ac0 2024-03-22T02:27:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.334+0000 7f7e77fff640 1 -- 172.21.15.28:0/4092824838 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11868 (secure 0 0 0) 0x7f7e9002d3b0 con 0x7f7e90100ac0 2024-03-22T02:27:59.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:27:59.335 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":42,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:27:34.006260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":246,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29075":{"gid":29075,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/366330553","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":366330553},{"type":"v1","addr":"172.21.15.83:6837","nonce":366330553}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:27:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 42 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 msgr2=0x7f7e780a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 0x7f7e780a53e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7e8402c7b0 tx=0x7f7e84005fd0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 msgr2=0x7f7e9006ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f7e8000e9c0 tx=0x7f7e8000ee80 comp rx=0 tx=0).stop 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 shutdown_connections 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7e780a2f20 0x7f7e780a53e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7e90102670 0x7f7e9019f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7e90101cb0 0x7f7e9019cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 --2- 172.21.15.28:0/4092824838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e90100ac0 0x7f7e9006ad50 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 >> 172.21.15.28:0/4092824838 conn(0x7f7e900fc070 msgr2=0x7f7e90104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.336+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 shutdown_connections 2024-03-22T02:27:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.337+0000 7f7e95563640 1 -- 172.21.15.28:0/4092824838 wait complete. 2024-03-22T02:27:59.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 -- 172.21.15.28:0/630450644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c102670 msgr2=0x7fdf5c104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 --2- 172.21.15.28:0/630450644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c102670 0x7fdf5c104a60 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fdf54009900 tx=0x7fdf5402bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 -- 172.21.15.28:0/630450644 shutdown_connections 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 --2- 172.21.15.28:0/630450644 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 --2- 172.21.15.28:0/630450644 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 --2- 172.21.15.28:0/630450644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c102670 0x7fdf5c104a60 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 -- 172.21.15.28:0/630450644 >> 172.21.15.28:0/630450644 conn(0x7fdf5c0fc020 msgr2=0x7fdf5c0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 -- 172.21.15.28:0/630450644 shutdown_connections 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.459+0000 7fdf64381640 1 -- 172.21.15.28:0/630450644 wait complete. 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 Processor -- start 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 -- start start 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 0x7fdf5c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c10d880 con 0x7fdf5c1078d0 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c10d700 con 0x7fdf5c102670 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf64381640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c10da00 con 0x7fdf5c104fa0 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6291e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 0x7fdf5c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6291e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59800/0 (socket says 172.21.15.28:59800) 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37938/0 (socket says 172.21.15.28:37938) 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6291e640 1 -- 172.21.15.28:0/3551092622 learned_addr learned my addr 172.21.15.28:0/3551092622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 -- 172.21.15.28:0/3551092622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 msgr2=0x7fdf5c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 -- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 msgr2=0x7fdf5c19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6211d640 1 -- 172.21.15.28:0/3551092622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5c19fb40 con 0x7fdf5c102670 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6191c640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.460+0000 7fdf6291e640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 0x7fdf5c197f20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdf54036040 tx=0x7fdf5402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf537fe640 1 -- 172.21.15.28:0/3551092622 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54043020 con 0x7fdf5c102670 2024-03-22T02:27:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5c19fdd0 con 0x7fdf5c102670 2024-03-22T02:27:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf537fe640 1 -- 172.21.15.28:0/3551092622 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf540054e0 con 0x7fdf5c102670 2024-03-22T02:27:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf537fe640 1 -- 172.21.15.28:0/3551092622 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54003c30 con 0x7fdf5c102670 2024-03-22T02:27:59.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.461+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf5c1a0240 con 0x7fdf5c102670 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.463+0000 7fdf66f6e740 1 Processor -- start 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.463+0000 7fdf537fe640 1 -- 172.21.15.28:0/3551092622 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdf54046020 con 0x7fdf5c102670 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 -- start start 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c41c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c46d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c49810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559967ba68d0 con 0x559967c42210 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559967b4ee00 con 0x559967c41820 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559967ba2880 con 0x559967c47380 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c46d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6291e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c41c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c46d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59804/0 (socket says 172.21.15.28:59804) 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c49810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6211d640 1 -- 172.21.15.28:0/1780587727 learned_addr learned my addr 172.21.15.28:0/1780587727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf537fe640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 -- 172.21.15.28:0/1780587727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 msgr2=0x559967c41c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c41c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 -- 172.21.15.28:0/1780587727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 msgr2=0x559967c46d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c46d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6191c640 1 -- 172.21.15.28:0/1780587727 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599678ba3f0 con 0x559967c47380 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf6291e640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.464+0000 7fdf537fe640 1 -- 172.21.15.28:0/3551092622 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdf540f17c0 con 0x7fdf5c102670 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf6191c640 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c49810 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0697f0 tx=0x7fdf4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d8508c2fbf9175 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf50bfe640 1 -- 172.21.15.28:0/1780587727 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c004310 con 0x559967c47380 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf50bfe640 1 -- 172.21.15.28:0/1780587727 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf4c0044b0 con 0x559967c47380 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf50bfe640 1 -- 172.21.15.28:0/1780587727 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c0035f0 con 0x559967c47380 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf6291e640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 0x7fdf480a53e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x559967c42a70 tx=0x7fdf58004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 -- 172.21.15.28:0/1780587727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 msgr2=0x559967c49810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c49810 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0697f0 tx=0x7fdf4c011810 comp rx=0 tx=0).stop 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 -- 172.21.15.28:0/1780587727 shutdown_connections 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c49810 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c46d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/1780587727 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c41c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.465+0000 7fdf66f6e740 1 -- 172.21.15.28:0/1780587727 >> 172.21.15.28:0/1780587727 conn(0x559967c3cd90 msgr2=0x559967c3d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- 172.21.15.28:0/1780587727 shutdown_connections 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- 172.21.15.28:0/1780587727 wait complete. 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- start start 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 Processor -- start 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c4bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c53260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559967c4a330 con 0x559967c42210 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x559967b62300 con 0x559967c41820 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x559967c4a4b0 con 0x559967c47380 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf66f6e740 10 client.0 _ll_register_callbacks cb 0x559967b9de30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf6291e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c53260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c4bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59808/0 (socket says 172.21.15.28:59808) 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.466+0000 7fdf6191c640 1 -- 172.21.15.28:0/468729720 learned_addr learned my addr 172.21.15.28:0/468729720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 -- 172.21.15.28:0/468729720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 msgr2=0x559967c4bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c4bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 -- 172.21.15.28:0/468729720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 msgr2=0x559967c4c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 -- 172.21.15.28:0/468729720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599679847d0 con 0x559967c47380 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6191c640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf6291e640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c53260 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdf58011880 tx=0x7fdf58004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecd0b1b2bf457a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf58004d20 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf4c0044b0 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdf5800e680 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fdf58032050 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf66f6e740 10 client.29437 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559967c56690 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.467+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf4c0044b0 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fdf580054e0 con 0x559967c47380 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 10 client.29437 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 10 client.29437 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 20 client.29437 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967c57de0 0x559967c5a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 4 client.29437 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:59.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559967bb3e10 con 0x559967c57de0 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf6211d640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967c57de0 0x559967c5a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf6211d640 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967c57de0 0x559967c5a240 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=41115896af59a6c server_cookie=947f61499b9d1f9c in_seq=0 out_seq=0 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.468+0000 7fdf45ffb640 10 client.29437 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf45ffb640 1 -- 172.21.15.28:0/468729720 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559967bb3e10 con 0x559967c57de0 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf45ffb640 10 client.29437 handle_command_reply: tid=0 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 client.29437 shutdown 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 msgr2=0x559967c53260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c53260 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fdf58011880 tx=0x7fdf58004ab0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 shutdown_connections 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967c53260 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c4c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967c57de0 0x559967c5a240 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/468729720 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967c4bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf45ffb640 0 client.29437 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 >> 172.21.15.28:0/468729720 conn(0x559967c3cd90 msgr2=0x559967c3bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 shutdown_connections 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 1 -- 172.21.15.28:0/468729720 wait complete. 2024-03-22T02:27:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.469+0000 7fdf66f6e740 20 client.29437 trim_cache size 0 max 16384 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.485+0000 7fdf66f6e740 1 Processor -- start 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 -- start start 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c53220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c47380 0x559967c3bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1ad2d0 con 0x559967c42210 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1aab40 con 0x559967c47380 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.486+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1a8990 con 0x559967c41820 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37974/0 (socket says 172.21.15.28:37974) 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 -- 172.21.15.28:0/3846850101 learned_addr learned my addr 172.21.15.28:0/3846850101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 -- 172.21.15.28:0/3846850101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c47380 msgr2=0x559967c3bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c47380 0x559967c3bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 -- 172.21.15.28:0/3846850101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 msgr2=0x559967c53220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:27:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c53220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 -- 172.21.15.28:0/3846850101 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559967c4a5f0 con 0x559967c41820 2024-03-22T02:27:59.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.487+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdf540b9f40 tx=0x7fdf540eed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ccdedc03465afa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.488+0000 7fdf47fff640 1 -- 172.21.15.28:0/3846850101 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf540b9d80 con 0x559967c41820 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.488+0000 7fdf47fff640 1 -- 172.21.15.28:0/3846850101 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf54005690 con 0x559967c41820 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.488+0000 7fdf47fff640 1 -- 172.21.15.28:0/3846850101 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf540bd440 con 0x559967c41820 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3846850101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 msgr2=0x559967c3b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdf540b9f40 tx=0x7fdf540eed80 comp rx=0 tx=0).stop 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3846850101 shutdown_connections 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c47380 0x559967c3bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967c53220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3846850101 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c41820 0x559967c3b740 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.489+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3846850101 >> 172.21.15.28:0/3846850101 conn(0x559967c3cd90 msgr2=0x559967c4bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3846850101 shutdown_connections 2024-03-22T02:27:59.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3846850101 wait complete. 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 -- start start 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 Processor -- start 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967cb9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967cc09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1a9000 con 0x559967c42210 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.490+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1aa470 con 0x559967c41820 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf66f6e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5c1a96d0 con 0x559967c47380 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf66f6e740 10 client.0 _ll_register_callbacks cb 0x7fdf58011750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59834/0 (socket says 172.21.15.28:59834) 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 -- 172.21.15.28:0/3630197785 learned_addr learned my addr 172.21.15.28:0/3630197785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967cb9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 -- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 msgr2=0x559967cc09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967cc09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 -- 172.21.15.28:0/3630197785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 msgr2=0x559967cb9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967cb9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 -- 172.21.15.28:0/3630197785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559967cc0fd0 con 0x559967c42210 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf6191c640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009710 tx=0x7fdf4c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fa26e27c35cf3a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.491+0000 7fdf45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c027020 con 0x559967c42210 2024-03-22T02:27:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fdf4c003da0 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559967bae160 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fdf4c032050 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf66f6e740 10 client.19632 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.492+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559967cc2340 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x559967bae160 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fdf4c004420 con 0x559967c42210 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 10 client.19632 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 10 client.19632 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 20 client.19632 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967cc2c10 0x559967cc5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 4 client.19632 mds_command: new command op to 24478 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.493+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 --> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559967b2b6a0 con 0x559967cc2c10 2024-03-22T02:27:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.497+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967cc2c10 0x559967cc5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf45ffb640 10 client.19632 ms_handle_connect on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf6211d640 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967cc2c10 0x559967cc5030 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0a7a1ee5855a36e server_cookie=9398d5dedfa15267 in_seq=0 out_seq=0 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf45ffb640 1 -- 172.21.15.28:0/3630197785 <== mds.2 v2:172.21.15.120:6836/2149554336 1 ==== command_reply(tid 0: 0 ) ==== 8+0+1159 (crc 0 0 0) 0x559967b2b6a0 con 0x559967cc2c10 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf45ffb640 10 client.19632 handle_command_reply: tid=0 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 client.19632 shutdown 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 msgr2=0x559967cb99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c009710 tx=0x7fdf4c003820 comp rx=0 tx=0).stop 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 shutdown_connections 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559967c47380 0x559967cc09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559967c42210 0x559967cb99a0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.120:6836/2149554336,v1:172.21.15.120:6837/2149554336] conn(0x559967cc2c10 0x559967cc5030 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 --2- 172.21.15.28:0/3630197785 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559967c41820 0x559967cb9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 >> 172.21.15.28:0/3630197785 conn(0x559967c3cd90 msgr2=0x559967c3fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.498+0000 7fdf45ffb640 0 client.19632 ms_handle_reset on v2:172.21.15.120:6836/2149554336 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 shutdown_connections 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf66f6e740 1 -- 172.21.15.28:0/3630197785 wait complete. 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf66f6e740 20 client.19632 trim_cache size 0 max 16384 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1309504951, 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511675006, 2024-03-22T02:27:59.501 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT" 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 1790431142, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511675041, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT" 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2124312569, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511675016, 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP" 2024-03-22T02:27:59.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2948196508, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511675029, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA" 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3827943303, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511675052, 2024-03-22T02:27:59.503 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT" 2024-03-22T02:27:59.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:27:59.504 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-03-22T02:27:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 msgr2=0x7fdf480a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 0x7fdf480a53e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x559967c42a70 tx=0x7fdf58004a40 comp rx=0 tx=0).stop 2024-03-22T02:27:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 msgr2=0x7fdf5c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.499+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 0x7fdf5c197f20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdf54036040 tx=0x7fdf5402c650 comp rx=0 tx=0).stop 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 shutdown_connections 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdf480a2f20 0x7fdf480a53e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf5c1078d0 0x7fdf5c19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdf5c104fa0 0x7fdf5c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 --2- 172.21.15.28:0/3551092622 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdf5c102670 0x7fdf5c197f20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 >> 172.21.15.28:0/3551092622 conn(0x7fdf5c0fc020 msgr2=0x7fdf5c103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 shutdown_connections 2024-03-22T02:27:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.500+0000 7fdf64381640 1 -- 172.21.15.28:0/3551092622 wait complete. 2024-03-22T02:27:59.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-22T02:27:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- 172.21.15.28:0/2424079318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c100970 msgr2=0x7f6c6c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 --2- 172.21.15.28:0/2424079318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c100970 0x7f6c6c100d90 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009900 tx=0x7f6c6802bc30 comp rx=0 tx=0).stop 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- 172.21.15.28:0/2424079318 shutdown_connections 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 --2- 172.21.15.28:0/2424079318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 0x7f6c6c108db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 --2- 172.21.15.28:0/2424079318 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c101b60 0x7f6c6c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 --2- 172.21.15.28:0/2424079318 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c100970 0x7f6c6c100d90 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- 172.21.15.28:0/2424079318 >> 172.21.15.28:0/2424079318 conn(0x7f6c6c0fbf00 msgr2=0x7f6c6c0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- 172.21.15.28:0/2424079318 shutdown_connections 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- 172.21.15.28:0/2424079318 wait complete. 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 Processor -- start 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.624+0000 7f6c7287d640 1 -- start start 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c100970 0x7f6c6c197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 0x7f6c6c19f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c10d710 con 0x7f6c6c101b60 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c10d590 con 0x7f6c6c100970 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c10d890 con 0x7f6c6c102520 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59850/0 (socket says 172.21.15.28:59850) 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 -- 172.21.15.28:0/1695362297 learned_addr learned my addr 172.21.15.28:0/1695362297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7187b640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c100970 0x7f6c6c197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7207c640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 0x7f6c6c19f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 -- 172.21.15.28:0/1695362297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 msgr2=0x7f6c6c19f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 0x7f6c6c19f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 -- 172.21.15.28:0/1695362297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c100970 msgr2=0x7f6c6c197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c100970 0x7f6c6c197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 -- 172.21.15.28:0/1695362297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c6c19f980 con 0x7f6c6c101b60 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7107a640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00e9f0 tx=0x7f6c5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c62ffd640 1 -- 172.21.15.28:0/1695362297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c00cd80 con 0x7f6c6c101b60 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c62ffd640 1 -- 172.21.15.28:0/1695362297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c5c010910 con 0x7f6c6c101b60 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c6c19fc70 con 0x7f6c6c101b60 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c62ffd640 1 -- 172.21.15.28:0/1695362297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c018990 con 0x7f6c6c101b60 2024-03-22T02:27:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.625+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c6c1a01c0 con 0x7f6c6c101b60 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.628+0000 7f6c62ffd640 1 -- 172.21.15.28:0/1695362297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6c5c010ac0 con 0x7f6c6c101b60 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.628+0000 7f6c767a2740 1 Processor -- start 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.628+0000 7f6c767a2740 1 -- start start 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcabcbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcabcea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ebcab2b8d0 con 0x55ebcabc6a10 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ebcaad3e00 con 0x55ebcabcc570 2024-03-22T02:27:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c767a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ebcab27880 con 0x55ebcabc7400 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c62ffd640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 0x7f6c580a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59864/0 (socket says 172.21.15.28:59864) 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7207c640 1 -- 172.21.15.28:0/3148587645 learned_addr learned my addr 172.21.15.28:0/3148587645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcabcea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcabcbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c62ffd640 1 -- 172.21.15.28:0/1695362297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6c5c0cfaf0 con 0x7f6c6c101b60 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7207c640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 0x7f6c580a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 -- 172.21.15.28:0/3148587645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 msgr2=0x55ebcabcbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcabcbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 -- 172.21.15.28:0/3148587645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 msgr2=0x55ebcabc6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7107a640 1 -- 172.21.15.28:0/3148587645 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebca83f3f0 con 0x55ebcabcc570 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcabcbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.629+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcabcea00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0cd230 tx=0x7f6c5c06ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43192e8648175028 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c7207c640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 0x7f6c580a53e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55ebcabc7c60 tx=0x7f6c64004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c57fff640 1 -- 172.21.15.28:0/3148587645 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c026040 con 0x55ebcabcc570 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c57fff640 1 -- 172.21.15.28:0/3148587645 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c5c09ff60 con 0x55ebcabcc570 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c57fff640 1 -- 172.21.15.28:0/3148587645 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c5c0bae50 con 0x55ebcabcc570 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c767a2740 1 -- 172.21.15.28:0/3148587645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 msgr2=0x55ebcabcea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.630+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcabcea00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c0cd230 tx=0x7f6c5c06ecb0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- 172.21.15.28:0/3148587645 shutdown_connections 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcabcea00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcabcbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3148587645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcabc6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- 172.21.15.28:0/3148587645 >> 172.21.15.28:0/3148587645 conn(0x55ebcabc1f80 msgr2=0x55ebcabc2380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- 172.21.15.28:0/3148587645 shutdown_connections 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- 172.21.15.28:0/3148587645 wait complete. 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- start start 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 Processor -- start 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabd0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabd1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.631+0000 7f6c767a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ebcabcf4c0 con 0x55ebcabcc570 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c767a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ebcaae7300 con 0x55ebcabc7400 2024-03-22T02:27:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c767a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ebcabcf640 con 0x55ebcabc6a10 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c767a2740 10 client.0 _ll_register_callbacks cb 0x55ebcab22e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7187b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabd0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59878/0 (socket says 172.21.15.28:59878) 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 -- 172.21.15.28:0/818301632 learned_addr learned my addr 172.21.15.28:0/818301632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7107a640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabd1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 -- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 msgr2=0x55ebcabd0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabd0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 -- 172.21.15.28:0/818301632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 msgr2=0x55ebcabd1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabd1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 -- 172.21.15.28:0/818301632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebca9097d0 con 0x55ebcabcc570 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7187b640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabd0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c7207c640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6c64013420 tx=0x7f6c64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c0961f447e8afa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c557fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64010a40 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6c64010be0 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.632+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c5c09ff60 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6c64027020 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 10 client.19644 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ebcabdb7a0 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c5c09ff60 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f6c6401f3c0 con 0x55ebcabcc570 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 10 client.19644 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 10 client.19644 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 20 client.19644 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcabdcf10 0x55ebcabdf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 4 client.19644 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebcab38e10 con 0x55ebcabdcf10 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.633+0000 7f6c7187b640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcabdcf10 0x55ebcabdf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.634+0000 7f6c557fa640 10 client.19644 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.634+0000 7f6c7187b640 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcabdcf10 0x55ebcabdf380 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bc24efd7c96baa9e server_cookie=1b638e51dc9b7773 in_seq=0 out_seq=0 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c557fa640 1 -- 172.21.15.28:0/818301632 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ebcab38e10 con 0x55ebcabdcf10 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c557fa640 10 client.19644 handle_command_reply: tid=0 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 client.19644 shutdown 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 msgr2=0x55ebcabd83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f6c64013420 tx=0x7f6c64004a40 comp rx=0 tx=0).stop 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 shutdown_connections 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcabdcf10 0x55ebcabdf380 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabd83c0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabd1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c557fa640 0 client.19644 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 --2- 172.21.15.28:0/818301632 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabd0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 >> 172.21.15.28:0/818301632 conn(0x55ebcabc1f80 msgr2=0x55ebcabc1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 shutdown_connections 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 1 -- 172.21.15.28:0/818301632 wait complete. 2024-03-22T02:27:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.635+0000 7f6c767a2740 20 client.19644 trim_cache size 0 max 16384 2024-03-22T02:27:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 Processor -- start 2024-03-22T02:27:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 -- start start 2024-03-22T02:27:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabc0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabc0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1ad120 con 0x55ebcabcc570 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1aa990 con 0x55ebcabc7400 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c767a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1a87e0 con 0x55ebcabc6a10 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59884/0 (socket says 172.21.15.28:59884) 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 -- 172.21.15.28:0/3793421596 learned_addr learned my addr 172.21.15.28:0/3793421596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabc0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabc0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 -- 172.21.15.28:0/3793421596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 msgr2=0x55ebcabc0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabc0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 -- 172.21.15.28:0/3793421596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 msgr2=0x55ebcabc0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabc0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 -- 172.21.15.28:0/3793421596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebcabc77e0 con 0x55ebcabcc570 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.650+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6c6401f830 tx=0x7f6c64009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c540fc4d45e7b014 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c577fe640 1 -- 172.21.15.28:0/3793421596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6401faa0 con 0x55ebcabcc570 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c577fe640 1 -- 172.21.15.28:0/3793421596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6403c070 con 0x55ebcabcc570 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c577fe640 1 -- 172.21.15.28:0/3793421596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64037800 con 0x55ebcabcc570 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 -- 172.21.15.28:0/3793421596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 msgr2=0x55ebcabe3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f6c6401f830 tx=0x7f6c64009fd0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 -- 172.21.15.28:0/3793421596 shutdown_connections 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabcc570 0x55ebcabe3c40 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabc7400 0x55ebcabc0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3793421596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc6a10 0x55ebcabc0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.651+0000 7f6c767a2740 1 -- 172.21.15.28:0/3793421596 >> 172.21.15.28:0/3793421596 conn(0x55ebcabc1f80 msgr2=0x55ebcabc8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- 172.21.15.28:0/3793421596 shutdown_connections 2024-03-22T02:27:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- 172.21.15.28:0/3793421596 wait complete. 2024-03-22T02:27:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- start start 2024-03-22T02:27:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 Processor -- start 2024-03-22T02:27:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcac3f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1a8e50 con 0x55ebcabc6a10 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1aa2c0 con 0x55ebcabcc570 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.652+0000 7f6c767a2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f6c6c1a9520 con 0x55ebcabc7400 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c767a2740 10 client.0 _ll_register_callbacks cb 0x7f6c64003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7187b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34094/0 (socket says 172.21.15.28:34094) 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 -- 172.21.15.28:0/3537710940 learned_addr learned my addr 172.21.15.28:0/3537710940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7187b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59894/0 (socket says 172.21.15.28:59894) 2024-03-22T02:27:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7107a640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcac3f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 -- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 msgr2=0x55ebcac3f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcac3f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 -- 172.21.15.28:0/3537710940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 msgr2=0x55ebcac3eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 -- 172.21.15.28:0/3537710940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebcac46710 con 0x55ebcabcc570 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c7207c640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005cc0 tx=0x7f6c6401fc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bebfbb6f40590d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c557fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.653+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c64044070 con 0x55ebcabcc570 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f6c64037580 con 0x55ebcabcc570 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c6403c070 con 0x55ebcabcc570 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f6c64010e20 con 0x55ebcabcc570 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 10 client.29462 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55ebcac47a10 con 0x55ebcabcc570 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6403c070 con 0x55ebcabcc570 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f6c64049560 con 0x55ebcabcc570 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 10 client.29462 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 10 client.29462 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 20 client.29462 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcac48300 0x55ebcac4a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 4 client.29462 mds_command: new command op to 24508 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.654+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 --> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebcab33160 con 0x55ebcac48300 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcac48300 0x55ebcac4a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c557fa640 10 client.29462 ms_handle_connect on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c7187b640 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcac48300 0x55ebcac4a720 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=db566f4d21f44661 server_cookie=9ad4660f71a469e in_seq=0 out_seq=0 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c557fa640 1 -- 172.21.15.28:0/3537710940 <== mds.1 v2:172.21.15.120:6838/990751976 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebcab33160 con 0x55ebcac48300 2024-03-22T02:27:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c557fa640 10 client.29462 handle_command_reply: tid=0 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c767a2740 1 client.29462 shutdown 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 msgr2=0x55ebcac46120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005cc0 tx=0x7f6c6401fc50 comp rx=0 tx=0).stop 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 shutdown_connections 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.655+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:6838/990751976,v1:172.21.15.120:6839/990751976] conn(0x55ebcac48300 0x55ebcac4a720 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ebcabcc570 0x55ebcac46120 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebcabc7400 0x55ebcac3f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 --2- 172.21.15.28:0/3537710940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ebcabc6a10 0x55ebcac3eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 >> 172.21.15.28:0/3537710940 conn(0x55ebcabc1f80 msgr2=0x55ebcabc0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c557fa640 0 client.29462 ms_handle_reset on v2:172.21.15.120:6838/990751976 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 shutdown_connections 2024-03-22T02:27:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 1 -- 172.21.15.28:0/3537710940 wait complete. 2024-03-22T02:27:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c767a2740 20 client.29462 trim_cache size 0 max 16384 2024-03-22T02:27:59.663 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:27:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 msgr2=0x7f6c580a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 0x7f6c580a53e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55ebcabc7c60 tx=0x7f6c64004ab0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 msgr2=0x7f6c6c1982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.656+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c00e9f0 tx=0x7f6c5c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 shutdown_connections 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f6c580a2f20 0x7f6c580a53e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6c6c102520 0x7f6c6c19f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c6c101b60 0x7f6c6c1982a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 --2- 172.21.15.28:0/1695362297 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6c6c100970 0x7f6c6c197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 >> 172.21.15.28:0/1695362297 conn(0x7f6c6c0fbf00 msgr2=0x7f6c6c104ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 shutdown_connections 2024-03-22T02:27:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.657+0000 7f6c7287d640 1 -- 172.21.15.28:0/1695362297 wait complete. 2024-03-22T02:27:59.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-22T02:27:59.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 12641 21.68 MB/sec warmup 39 sec latency 82.295 ms 2024-03-22T02:27:59.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 -- 172.21.15.28:0/1039250588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18107870 msgr2=0x7f8b18109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1039250588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18107870 0x7f8b18109cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b10009ca0 tx=0x7f8b1002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 -- 172.21.15.28:0/1039250588 shutdown_connections 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1039250588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18107870 0x7f8b18109cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1039250588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b18107330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1039250588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18102610 0x7f8b18104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 -- 172.21.15.28:0/1039250588 >> 172.21.15.28:0/1039250588 conn(0x7f8b180fc020 msgr2=0x7f8b180fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 -- 172.21.15.28:0/1039250588 shutdown_connections 2024-03-22T02:27:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.791+0000 7f8b1f266640 1 -- 172.21.15.28:0/1039250588 wait complete. 2024-03-22T02:27:59.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 Processor -- start 2024-03-22T02:27:59.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 -- start start 2024-03-22T02:27:59.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18102610 0x7f8b18197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18107870 0x7f8b1819f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1810d830 con 0x7f8b18104f40 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1810d6b0 con 0x7f8b18107870 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1f266640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1810d9b0 con 0x7f8b18102610 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59926/0 (socket says 172.21.15.28:59926) 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 -- 172.21.15.28:0/1356162588 learned_addr learned my addr 172.21.15.28:0/1356162588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1d002640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18102610 0x7f8b18197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1d803640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18107870 0x7f8b1819f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 -- 172.21.15.28:0/1356162588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18102610 msgr2=0x7f8b18197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18102610 0x7f8b18197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 -- 172.21.15.28:0/1356162588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18107870 msgr2=0x7f8b1819f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18107870 0x7f8b1819f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 -- 172.21.15.28:0/1356162588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1819fad0 con 0x7f8b18104f40 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.792+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9c0 tx=0x7f8b0800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.793+0000 7f8b0e7fc640 1 -- 172.21.15.28:0/1356162588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0800cd80 con 0x7f8b18104f40 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.793+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1819fdc0 con 0x7f8b18104f40 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.793+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b181a0290 con 0x7f8b18104f40 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.793+0000 7f8b0e7fc640 1 -- 172.21.15.28:0/1356162588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b08005500 con 0x7f8b18104f40 2024-03-22T02:27:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.793+0000 7f8b0e7fc640 1 -- 172.21.15.28:0/1356162588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08010430 con 0x7f8b18104f40 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b0e7fc640 1 -- 172.21.15.28:0/1356162588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8b08005020 con 0x7f8b18104f40 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 Processor -- start 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 -- start start 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcafcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcaff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3dca5c8d0 con 0x55a3dcafd380 2024-03-22T02:27:59.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a3dca04e00 con 0x55a3dcaf8210 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.796+0000 7f8b21e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a3dca58880 con 0x55a3dcaf7820 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcaff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d803640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcafcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d803640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38080/0 (socket says 172.21.15.28:38080) 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d803640 1 -- 172.21.15.28:0/1058689033 learned_addr learned my addr 172.21.15.28:0/1058689033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b0e7fc640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 0x7f8b040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 -- 172.21.15.28:0/1058689033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 msgr2=0x55a3dcaf7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 -- 172.21.15.28:0/1058689033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 msgr2=0x55a3dcafcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcafcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 -- 172.21.15.28:0/1058689033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3dc7703f0 con 0x55a3dcafd380 2024-03-22T02:27:59.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d803640 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d002640 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcafcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b0e7fc640 1 -- 172.21.15.28:0/1356162588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8b080cfc10 con 0x7f8b18104f40 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1d803640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 0x7f8b040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b1c801640 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcaff810 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cd2d0 tx=0x7f8b08004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eb26a320bf4db2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b037fe640 1 -- 172.21.15.28:0/1058689033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0809bd00 con 0x55a3dcafd380 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b037fe640 1 -- 172.21.15.28:0/1058689033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b0809c630 con 0x55a3dcafd380 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.797+0000 7f8b037fe640 1 -- 172.21.15.28:0/1058689033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b0809b890 con 0x55a3dcafd380 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 -- 172.21.15.28:0/1058689033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 msgr2=0x55a3dcaff810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b1d803640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 0x7f8b040a53e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006010 tx=0x7f8b1002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcaff810 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8b080cd2d0 tx=0x7f8b08004a20 comp rx=0 tx=0).stop 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 -- 172.21.15.28:0/1058689033 shutdown_connections 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcaff810 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcafcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 --2- 172.21.15.28:0/1058689033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcaf7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 -- 172.21.15.28:0/1058689033 >> 172.21.15.28:0/1058689033 conn(0x55a3dcaf2d90 msgr2=0x55a3dcaf3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 -- 172.21.15.28:0/1058689033 shutdown_connections 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.798+0000 7f8b21e46740 1 -- 172.21.15.28:0/1058689033 wait complete. 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 -- start start 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 Processor -- start 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 0x55a3dcb021f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcafd380 0x55a3dcb09230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a3dcb002d0 con 0x55a3dcaf8210 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a3dca18300 con 0x55a3dcafd380 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a3dcb00450 con 0x55a3dcaf7820 2024-03-22T02:27:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b21e46740 10 client.0 _ll_register_callbacks cb 0x55a3dca53e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 0x55a3dcb021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38088/0 (socket says 172.21.15.28:38088) 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 0x55a3dcb021f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59934/0 (socket says 172.21.15.28:59934) 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1d002640 1 -- 172.21.15.28:0/2844205473 learned_addr learned my addr 172.21.15.28:0/2844205473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.799+0000 7f8b1d803640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcafd380 0x55a3dcb09230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 -- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcafd380 msgr2=0x55a3dcb09230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcafd380 0x55a3dcb09230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 -- 172.21.15.28:0/2844205473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 msgr2=0x55a3dcb021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 0x55a3dcb021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 -- 172.21.15.28:0/2844205473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3dc83a7d0 con 0x55a3dcaf7820 2024-03-22T02:27:59.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b14012700 tx=0x7f8b14012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36a17f4a6ed2a65f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14004050 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8b140041f0 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b0809c630 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8b140043f0 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b21e46740 10 client.29470 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a3dcb0c670 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.800+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b0809c630 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f8b14003c20 con 0x55a3dcaf7820 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 10 client.29470 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 10 client.29470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 20 client.29470 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb0ddc0 0x55a3dcb10260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 4 client.29470 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:27:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3dca69e10 con 0x55a3dcb0ddc0 2024-03-22T02:27:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b1c801640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb0ddc0 0x55a3dcb10260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b1c801640 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb0ddc0 0x55a3dcb10260 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a0c1e0b7ae150aa server_cookie=1796686648e6fafa in_seq=0 out_seq=0 2024-03-22T02:27:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.801+0000 7f8b00ff9640 10 client.29470 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.803+0000 7f8b00ff9640 1 -- 172.21.15.28:0/2844205473 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3dca69e10 con 0x55a3dcb0ddc0 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.803+0000 7f8b00ff9640 10 client.29470 handle_command_reply: tid=0 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.803+0000 7f8b21e46740 1 client.29470 shutdown 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 msgr2=0x55a3dcb01cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b14012700 tx=0x7f8b14012bc0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 shutdown_connections 2024-03-22T02:27:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb0ddc0 0x55a3dcb10260 crc :-1 s=READY pgs=362 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcafd380 0x55a3dcb09230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b00ff9640 0 client.29470 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf8210 0x55a3dcb021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2844205473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb01cb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 >> 172.21.15.28:0/2844205473 conn(0x55a3dcaf2d90 msgr2=0x55a3dcaf1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 shutdown_connections 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 1 -- 172.21.15.28:0/2844205473 wait complete. 2024-03-22T02:27:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.804+0000 7f8b21e46740 20 client.29470 trim_cache size 0 max 16384 2024-03-22T02:27:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 Processor -- start 2024-03-22T02:27:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 -- start start 2024-03-22T02:27:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb01810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcafd380 0x55a3dcb01d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181ad3b0 con 0x55a3dcaf7820 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181aac20 con 0x55a3dcaf8210 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b21e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181a8a70 con 0x55a3dcafd380 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59936/0 (socket says 172.21.15.28:59936) 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 -- 172.21.15.28:0/2751806909 learned_addr learned my addr 172.21.15.28:0/2751806909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1c801640 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb01810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d803640 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcafd380 0x55a3dcb01d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 -- 172.21.15.28:0/2751806909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcafd380 msgr2=0x55a3dcb01d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcafd380 0x55a3dcb01d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 -- 172.21.15.28:0/2751806909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 msgr2=0x55a3dcb01810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb01810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 -- 172.21.15.28:0/2751806909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3dcaf85f0 con 0x55a3dcaf7820 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.819+0000 7f8b1d002640 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005840 tx=0x7f8b14010980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c255b71b097d16a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b02ffd640 1 -- 172.21.15.28:0/2751806909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1402d070 con 0x55a3dcaf7820 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b02ffd640 1 -- 172.21.15.28:0/2751806909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14005040 con 0x55a3dcaf7820 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b02ffd640 1 -- 172.21.15.28:0/2751806909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1400fbf0 con 0x55a3dcaf7820 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 -- 172.21.15.28:0/2751806909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 msgr2=0x55a3dcb012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005840 tx=0x7f8b14010980 comp rx=0 tx=0).stop 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 -- 172.21.15.28:0/2751806909 shutdown_connections 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcafd380 0x55a3dcb01d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb01810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 --2- 172.21.15.28:0/2751806909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcaf7820 0x55a3dcb012d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 -- 172.21.15.28:0/2751806909 >> 172.21.15.28:0/2751806909 conn(0x55a3dcaf2d90 msgr2=0x55a3dcaf6880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 -- 172.21.15.28:0/2751806909 shutdown_connections 2024-03-22T02:27:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.820+0000 7f8b21e46740 1 -- 172.21.15.28:0/2751806909 wait complete. 2024-03-22T02:27:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 -- start start 2024-03-22T02:27:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 Processor -- start 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb06d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb07290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181a90e0 con 0x55a3dcafd380 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181aa550 con 0x55a3dcaf8210 2024-03-22T02:27:59.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8b181a97b0 con 0x55a3dcaf7820 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b21e46740 10 client.0 _ll_register_callbacks cb 0x7f8b14009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b1d002640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb06d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b1d803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b1d803640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59944/0 (socket says 172.21.15.28:59944) 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b1c801640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb07290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.821+0000 7f8b1d803640 1 -- 172.21.15.28:0/3874546316 learned_addr learned my addr 172.21.15.28:0/3874546316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 -- 172.21.15.28:0/3874546316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 msgr2=0x55a3dcb06d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb06d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 -- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 msgr2=0x55a3dcb07290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb07290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 -- 172.21.15.28:0/3874546316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3dcb08820 con 0x55a3dcafd380 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b1d803640 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8b1002c630 tx=0x7f8b10032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a60edc899eea38 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:27:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b10039040 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f8b10003d80 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b14005040 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8b10032a10 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b21e46740 10 client.19668 fetch_fsmap learned FSMap version 42 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55a3dcb7d0a0 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14005040 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.822+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f8b1003f040 con 0x55a3dcafd380 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 10 client.19668 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 10 client.19668 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 20 client.19668 populate_metadata read hostname 'smithi028' 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb7db80 0x55a3dcb7ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 4 client.19668 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-22T02:27:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a3dca64160 con 0x55a3dcb7db80 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b1d002640 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb7db80 0x55a3dcb7ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b1d002640 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb7db80 0x55a3dcb7ffa0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6357937287f8ab3 server_cookie=20175a7eba0f8848 in_seq=0 out_seq=0 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.823+0000 7f8b00ff9640 10 client.19668 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b00ff9640 1 -- 172.21.15.28:0/3874546316 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a3dca64160 con 0x55a3dcb7db80 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b00ff9640 10 client.19668 handle_command_reply: tid=0 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 client.19668 shutdown 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 msgr2=0x55a3dcb08230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f8b1002c630 tx=0x7f8b10032590 comp rx=0 tx=0).stop 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 shutdown_connections 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a3dcb7db80 0x55a3dcb7ffa0 crc :-1 s=READY pgs=363 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a3dcafd380 0x55a3dcb08230 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b00ff9640 0 client.19668 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a3dcaf8210 0x55a3dcb07290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 --2- 172.21.15.28:0/3874546316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a3dcaf7820 0x55a3dcb06d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 >> 172.21.15.28:0/3874546316 conn(0x55a3dcaf2d90 msgr2=0x55a3dcaf1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 shutdown_connections 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.825+0000 7f8b21e46740 1 -- 172.21.15.28:0/3874546316 wait complete. 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b21e46740 20 client.19668 trim_cache size 0 max 16384 2024-03-22T02:27:59.831 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 msgr2=0x7f8b040a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 0x7f8b040a53e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8b10006010 tx=0x7f8b1002e040 comp rx=0 tx=0).stop 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 msgr2=0x7f8b181983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9c0 tx=0x7f8b0800ee80 comp rx=0 tx=0).stop 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 shutdown_connections 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8b040a2f20 0x7f8b040a53e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8b18107870 0x7f8b1819f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b18104f40 0x7f8b181983f0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 --2- 172.21.15.28:0/1356162588 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b18102610 0x7f8b18197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:27:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 >> 172.21.15.28:0/1356162588 conn(0x7f8b180fc020 msgr2=0x7f8b18102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:27:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 shutdown_connections 2024-03-22T02:27:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:27:59.826+0000 7f8b1f266640 1 -- 172.21.15.28:0/1356162588 wait complete. 2024-03-22T02:27:59.838 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 1309504951, 'ino': 1099511675006, 'path': '/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT'}, {'damage_type': 'backtrace', 'id': 1790431142, 'ino': 1099511675041, 'path': '/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TIPS.PPT'}, {'damage_type': 'backtrace', 'id': 2124312569, 'ino': 1099511675016, 'path': '/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/ZD16.BMP'}, {'damage_type': 'backtrace', 'id': 2948196508, 'ino': 1099511675029, 'path': '/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PPTOOLS1.PPA'}, {'damage_type': 'backtrace', 'id': 3827943303, 'ino': 1099511675052, 'path': '/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT'}] 2024-03-22T02:27:59.838 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-03-22T02:28:00.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:59 smithi028 ceph-mon[31916]: pgmap v793: 129 pgs: 129 active+clean; 216 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 438 op/s 2024-03-22T02:28:00.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:27:59 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4092824838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:28:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:59 smithi120 ceph-mon[36803]: pgmap v793: 129 pgs: 129 active+clean; 216 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 438 op/s 2024-03-22T02:28:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:27:59 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4092824838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:28:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:59 smithi083 ceph-mon[36340]: pgmap v793: 129 pgs: 129 active+clean; 216 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.3 MiB/s wr, 438 op/s 2024-03-22T02:28:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:27:59 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4092824838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:28:00.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 13216 21.94 MB/sec warmup 40 sec latency 62.859 ms 2024-03-22T02:28:01.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 13659 22.15 MB/sec warmup 41 sec latency 136.402 ms 2024-03-22T02:28:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:01 smithi120 ceph-mon[36803]: pgmap v794: 129 pgs: 129 active+clean; 217 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 427 op/s 2024-03-22T02:28:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:01 smithi083 ceph-mon[36340]: pgmap v794: 129 pgs: 129 active+clean; 217 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 427 op/s 2024-03-22T02:28:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:01 smithi028 ceph-mon[31916]: pgmap v794: 129 pgs: 129 active+clean; 217 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 427 op/s 2024-03-22T02:28:02.673 INFO:tasks.workunit.client.0.smithi028.stdout: 10 13876 22.08 MB/sec warmup 42 sec latency 382.442 ms 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/1321854340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18103df0 msgr2=0x7f4d18104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/1321854340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18103df0 0x7f4d18104210 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009900 tx=0x7f4d0802bc30 comp rx=0 tx=0).stop 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/1321854340 shutdown_connections 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/1321854340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d181059a0 0x7f4d1810c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/1321854340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d18104fe0 0x7f4d18105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/1321854340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18103df0 0x7f4d18104210 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.013+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/1321854340 >> 172.21.15.28:0/1321854340 conn(0x7f4d180ff380 msgr2=0x7f4d181017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/1321854340 shutdown_connections 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/1321854340 wait complete. 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 Processor -- start 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- start start 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d18103df0 0x7f4d1819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d181059a0 0x7f4d181a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810fa60 con 0x7f4d18104fe0 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810f8e0 con 0x7f4d18103df0 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ef1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1810fbe0 con 0x7f4d181059a0 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42432/0 (socket says 172.21.15.28:42432) 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d18103df0 0x7f4d1819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d17fff640 1 -- 172.21.15.28:0/3538481413 learned_addr learned my addr 172.21.15.28:0/3538481413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.014+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d181059a0 0x7f4d181a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 -- 172.21.15.28:0/3538481413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d181059a0 msgr2=0x7f4d181a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d181059a0 0x7f4d181a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 -- 172.21.15.28:0/3538481413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d18103df0 msgr2=0x7f4d1819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d18103df0 0x7f4d1819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 -- 172.21.15.28:0/3538481413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d181a1c90 con 0x7f4d18104fe0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d17fff640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e9f0 tx=0x7f4d0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d15ffb640 1 -- 172.21.15.28:0/3538481413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0000cd80 con 0x7f4d18104fe0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d15ffb640 1 -- 172.21.15.28:0/3538481413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d00005500 con 0x7f4d18104fe0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d181a1f80 con 0x7f4d18104fe0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d15ffb640 1 -- 172.21.15.28:0/3538481413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d00010690 con 0x7f4d18104fe0 2024-03-22T02:28:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.015+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d181a23f0 con 0x7f4d18104fe0 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.017+0000 7f4d15ffb640 1 -- 172.21.15.28:0/3538481413 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4d000046c0 con 0x7f4d18104fe0 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 Processor -- start 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 -- start start 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d15ffb640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 0x7f4d040a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b318e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b31914d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583b30ee040 con 0x5583b31894e0 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5583b30969e0 con 0x5583b3189ed0 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d21be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5583b30e9ff0 con 0x5583b318f040 2024-03-22T02:28:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 0x7f4d040a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d15ffb640 1 -- 172.21.15.28:0/3538481413 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4d000cfbc0 con 0x7f4d18104fe0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b318e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42434/0 (socket says 172.21.15.28:42434) 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d1ccba640 1 -- 172.21.15.28:0/1988471303 learned_addr learned my addr 172.21.15.28:0/1988471303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.018+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b31914d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 -- 172.21.15.28:0/1988471303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 msgr2=0x5583b31914d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b31914d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 -- 172.21.15.28:0/1988471303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 msgr2=0x5583b318e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b318e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 -- 172.21.15.28:0/1988471303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b2e02210 con 0x5583b31894e0 2024-03-22T02:28:03.025 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d17fff640 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b318e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 0x7f4d040a5480 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4d10005fd0 tx=0x7f4d10009b70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009900 tx=0x7f4d08009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb87c8df110df131 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d0effd640 1 -- 172.21.15.28:0/1988471303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0802cb80 con 0x5583b31894e0 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d0effd640 1 -- 172.21.15.28:0/1988471303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d08005040 con 0x5583b31894e0 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.019+0000 7f4d0effd640 1 -- 172.21.15.28:0/1988471303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08004920 con 0x5583b31894e0 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 -- 172.21.15.28:0/1988471303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 msgr2=0x5583b3189900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009900 tx=0x7f4d08009560 comp rx=0 tx=0).stop 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 -- 172.21.15.28:0/1988471303 shutdown_connections 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b31914d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b318e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 --2- 172.21.15.28:0/1988471303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3189900 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 -- 172.21.15.28:0/1988471303 >> 172.21.15.28:0/1988471303 conn(0x5583b3184a50 msgr2=0x5583b3184e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 -- 172.21.15.28:0/1988471303 shutdown_connections 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.020+0000 7f4d21be6740 1 -- 172.21.15.28:0/1988471303 wait complete. 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 -- start start 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 Processor -- start 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3193490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b319a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5583b3191f90 con 0x5583b31894e0 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5583b30a9db0 con 0x5583b3189ed0 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d21be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5583b3192110 con 0x5583b318f040 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.021+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3193490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57158/0 (socket says 172.21.15.28:57158) 2024-03-22T02:28:03.027 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 -- 172.21.15.28:0/145345777 learned_addr learned my addr 172.21.15.28:0/145345777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3193490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42446/0 (socket says 172.21.15.28:42446) 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b319a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d21be6740 10 client.0 _ll_register_callbacks cb 0x5583b2e01a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 -- 172.21.15.28:0/145345777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 msgr2=0x5583b319a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b319a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 -- 172.21.15.28:0/145345777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 msgr2=0x5583b3193490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3193490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 -- 172.21.15.28:0/145345777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b2ece620 con 0x5583b3189ed0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d17fff640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d000988b0 tx=0x7f4d0009c5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9452d9dd463418 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4cf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d000a48f0 con 0x5583b3189ed0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d08005040 con 0x5583b3189ed0 2024-03-22T02:28:03.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.022+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4d00095c30 con 0x5583b3189ed0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4d000bc0c0 con 0x5583b3189ed0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 10 client.29492 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5583b319dd60 con 0x5583b3189ed0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d08005040 con 0x5583b3189ed0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f4d0009b450 con 0x5583b3189ed0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 10 client.29492 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 10 client.29492 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 20 client.29492 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b319f3f0 0x5583b31a1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 4 client.29492 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583b30fb580 con 0x5583b319f3f0 2024-03-22T02:28:03.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.023+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b319f3f0 0x5583b31a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.024+0000 7f4cf7fff640 10 client.29492 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.024+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b319f3f0 0x5583b31a1880 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a14d86494c34d23 server_cookie=bdfd70a69bead559 in_seq=0 out_seq=0 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4cf7fff640 1 -- 172.21.15.28:0/145345777 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f4d080509c0 con 0x5583b319f3f0 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4cf7fff640 10 client.29492 handle_command_reply: tid=0 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 client.29492 shutdown 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 msgr2=0x5583b31939d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4d000988b0 tx=0x7f4d0009c5e0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 shutdown_connections 2024-03-22T02:28:03.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b319f3f0 0x5583b31a1880 crc :-1 s=READY pgs=364 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b319a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4cf7fff640 0 client.29492 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31939d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 --2- 172.21.15.28:0/145345777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b31894e0 0x5583b3193490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 >> 172.21.15.28:0/145345777 conn(0x5583b3184a50 msgr2=0x5583b3183c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 shutdown_connections 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 1 -- 172.21.15.28:0/145345777 wait complete. 2024-03-22T02:28:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.028+0000 7f4d21be6740 20 client.29492 trim_cache size 0 max 16384 2024-03-22T02:28:03.046 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 Processor -- start 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 -- start start 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 0x5583b318a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b318aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181af3c0 con 0x5583b3189ed0 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181acc30 con 0x5583b31894e0 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d21be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181aaa80 con 0x5583b318f040 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 0x5583b318a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42458/0 (socket says 172.21.15.28:42458) 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1d4bb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b318aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.047 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d17fff640 1 -- 172.21.15.28:0/2370218999 learned_addr learned my addr 172.21.15.28:0/2370218999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 -- 172.21.15.28:0/2370218999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 msgr2=0x5583b318aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b318aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 -- 172.21.15.28:0/2370218999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 msgr2=0x5583b319a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d1ccba640 1 -- 172.21.15.28:0/2370218999 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583b318a2b0 con 0x5583b31894e0 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.045+0000 7f4d17fff640 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 0x5583b318a760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4d08036040 tx=0x7f4d08000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=792fe0008778c632 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d0e7fc640 1 -- 172.21.15.28:0/2370218999 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08005d50 con 0x5583b31894e0 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d0e7fc640 1 -- 172.21.15.28:0/2370218999 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0804b040 con 0x5583b31894e0 2024-03-22T02:28:03.048 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d0e7fc640 1 -- 172.21.15.28:0/2370218999 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08004bd0 con 0x5583b31894e0 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d21be6740 1 -- 172.21.15.28:0/2370218999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 msgr2=0x5583b318a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.046+0000 7f4d21be6740 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 0x5583b318a760 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4d08036040 tx=0x7f4d08000bc0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- 172.21.15.28:0/2370218999 shutdown_connections 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b318f040 0x5583b318aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b3189ed0 0x5583b319a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- 172.21.15.28:0/2370218999 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b31894e0 0x5583b318a760 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- 172.21.15.28:0/2370218999 >> 172.21.15.28:0/2370218999 conn(0x5583b3184a50 msgr2=0x5583b3188540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.052 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- 172.21.15.28:0/2370218999 shutdown_connections 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- 172.21.15.28:0/2370218999 wait complete. 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- start start 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 Processor -- start 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b31894e0 0x5583b31983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31988f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181ab0f0 con 0x5583b318f040 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181ac560 con 0x5583b3189ed0 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4d181ab7c0 con 0x5583b31894e0 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d21be6740 10 client.0 _ll_register_callbacks cb 0x7f4d0000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d1d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.053 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d1ccba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b31894e0 0x5583b31983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d1d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42474/0 (socket says 172.21.15.28:42474) 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.047+0000 7f4d1d4bb640 1 -- 172.21.15.28:0/4015731677 learned_addr learned my addr 172.21.15.28:0/4015731677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 -- 172.21.15.28:0/4015731677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b31894e0 msgr2=0x5583b31983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b31894e0 0x5583b31983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 -- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 msgr2=0x5583b31988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 -- 172.21.15.28:0/4015731677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583b3199ea0 con 0x5583b318f040 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d17fff640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d1d4bb640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f4d100127b0 tx=0x7f4d1001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2bf4ae0ceb7c86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4cf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10020870 con 0x5583b318f040 2024-03-22T02:28:03.054 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4d10020a10 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d0804b040 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4d10020c10 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d21be6740 10 client.19686 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5583b3205d90 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.048+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d0804b040 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f4d10029410 con 0x5583b318f040 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 10 client.19686 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 10 client.19686 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 20 client.19686 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b32068a0 0x5583b3208cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 4 client.19686 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583b30f58d0 con 0x5583b32068a0 2024-03-22T02:28:03.055 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b32068a0 0x5583b3208cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4cf7fff640 10 client.19686 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:03.056 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.049+0000 7f4d1ccba640 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b32068a0 0x5583b3208cc0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0c486b309c0416f server_cookie=830e3176256f559 in_seq=0 out_seq=0 2024-03-22T02:28:03.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 13972 21.67 MB/sec warmup 43 sec latency 480.538 ms 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.819+0000 7f4cf7fff640 1 -- 172.21.15.28:0/4015731677 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583b30f58d0 con 0x5583b32068a0 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.819+0000 7f4cf7fff640 10 client.19686 handle_command_reply: tid=0 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.819+0000 7f4d21be6740 1 client.19686 shutdown 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.819+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 msgr2=0x5583b31998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.819+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f4d100127b0 tx=0x7f4d1001a000 comp rx=0 tx=0).stop 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 shutdown_connections 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5583b32068a0 0x5583b3208cc0 crc :-1 s=READY pgs=365 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4cf7fff640 0 client.19686 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5583b318f040 0x5583b31998b0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5583b3189ed0 0x5583b31988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 --2- 172.21.15.28:0/4015731677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5583b31894e0 0x5583b31983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 >> 172.21.15.28:0/4015731677 conn(0x5583b3184a50 msgr2=0x5583b31833c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 shutdown_connections 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 1 -- 172.21.15.28:0/4015731677 wait complete. 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.820+0000 7f4d21be6740 20 client.19686 trim_cache size 0 max 16384 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:28:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:28:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:28:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 msgr2=0x7f4d040a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 0x7f4d040a5480 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4d10005fd0 tx=0x7f4d10009b70 comp rx=0 tx=0).stop 2024-03-22T02:28:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 msgr2=0x7f4d1819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000e9f0 tx=0x7f4d0000edf0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 shutdown_connections 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d040a2fc0 0x7f4d040a5480 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4d181059a0 0x7f4d181a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d18104fe0 0x7f4d1819a5b0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 --2- 172.21.15.28:0/3538481413 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4d18103df0 0x7f4d1819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 >> 172.21.15.28:0/3538481413 conn(0x7f4d180ff380 msgr2=0x7f4d181082e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 shutdown_connections 2024-03-22T02:28:03.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:03.821+0000 7f4d1ef1e640 1 -- 172.21.15.28:0/3538481413 wait complete. 2024-03-22T02:28:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:03 smithi120 ceph-mon[36803]: pgmap v795: 129 pgs: 129 active+clean; 230 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 387 op/s 2024-03-22T02:28:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:03 smithi083 ceph-mon[36340]: pgmap v795: 129 pgs: 129 active+clean; 230 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 387 op/s 2024-03-22T02:28:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:03 smithi028 ceph-mon[31916]: pgmap v795: 129 pgs: 129 active+clean; 230 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.7 MiB/s wr, 387 op/s 2024-03-22T02:28:04.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 14206 21.62 MB/sec warmup 44 sec latency 580.277 ms 2024-03-22T02:28:05.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 14451 21.45 MB/sec warmup 45 sec latency 72.750 ms 2024-03-22T02:28:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:05 smithi120 ceph-mon[36803]: pgmap v796: 129 pgs: 129 active+clean; 266 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-22T02:28:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:05 smithi083 ceph-mon[36340]: pgmap v796: 129 pgs: 129 active+clean; 266 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-22T02:28:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:05 smithi028 ceph-mon[31916]: pgmap v796: 129 pgs: 129 active+clean; 266 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.6 MiB/s wr, 433 op/s 2024-03-22T02:28:06.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 14617 21.14 MB/sec warmup 46 sec latency 130.400 ms 2024-03-22T02:28:07.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 14757 20.76 MB/sec warmup 47 sec latency 225.950 ms 2024-03-22T02:28:08.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:07 smithi120 ceph-mon[36803]: pgmap v797: 129 pgs: 129 active+clean; 270 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.7 MiB/s wr, 407 op/s 2024-03-22T02:28:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:07 smithi083 ceph-mon[36340]: pgmap v797: 129 pgs: 129 active+clean; 270 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.7 MiB/s wr, 407 op/s 2024-03-22T02:28:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:07 smithi028 ceph-mon[31916]: pgmap v797: 129 pgs: 129 active+clean; 270 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 6.7 MiB/s wr, 407 op/s 2024-03-22T02:28:08.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 14985 20.52 MB/sec warmup 48 sec latency 99.728 ms 2024-03-22T02:28:09.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 15241 20.31 MB/sec warmup 49 sec latency 93.160 ms 2024-03-22T02:28:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:09 smithi120 ceph-mon[36803]: pgmap v798: 129 pgs: 129 active+clean; 250 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 570 op/s 2024-03-22T02:28:10.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:09 smithi083 ceph-mon[36340]: pgmap v798: 129 pgs: 129 active+clean; 250 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 570 op/s 2024-03-22T02:28:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:09 smithi028 ceph-mon[31916]: pgmap v798: 129 pgs: 129 active+clean; 250 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 7.6 MiB/s wr, 570 op/s 2024-03-22T02:28:10.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 15621 20.18 MB/sec warmup 50 sec latency 106.082 ms 2024-03-22T02:28:11.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16090 20.23 MB/sec warmup 51 sec latency 108.166 ms 2024-03-22T02:28:12.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:11 smithi120 ceph-mon[36803]: pgmap v799: 129 pgs: 129 active+clean; 249 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.2 MiB/s wr, 552 op/s 2024-03-22T02:28:12.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:11 smithi083 ceph-mon[36340]: pgmap v799: 129 pgs: 129 active+clean; 249 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.2 MiB/s wr, 552 op/s 2024-03-22T02:28:12.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:11 smithi028 ceph-mon[31916]: pgmap v799: 129 pgs: 129 active+clean; 249 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.2 MiB/s wr, 552 op/s 2024-03-22T02:28:12.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16537 20.31 MB/sec warmup 52 sec latency 289.762 ms 2024-03-22T02:28:13.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 16881 20.34 MB/sec warmup 53 sec latency 320.610 ms 2024-03-22T02:28:14.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:13 smithi120 ceph-mon[36803]: pgmap v800: 129 pgs: 129 active+clean; 253 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 577 op/s 2024-03-22T02:28:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:13 smithi083 ceph-mon[36340]: pgmap v800: 129 pgs: 129 active+clean; 253 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 577 op/s 2024-03-22T02:28:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:13 smithi028 ceph-mon[31916]: pgmap v800: 129 pgs: 129 active+clean; 253 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 7.5 MiB/s wr, 577 op/s 2024-03-22T02:28:14.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 17287 20.55 MB/sec warmup 54 sec latency 89.067 ms 2024-03-22T02:28:15.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 17649 20.61 MB/sec warmup 55 sec latency 66.734 ms 2024-03-22T02:28:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:15 smithi120 ceph-mon[36803]: pgmap v801: 129 pgs: 129 active+clean; 268 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 595 op/s 2024-03-22T02:28:16.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:15 smithi083 ceph-mon[36340]: pgmap v801: 129 pgs: 129 active+clean; 268 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 595 op/s 2024-03-22T02:28:16.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:15 smithi028 ceph-mon[31916]: pgmap v801: 129 pgs: 129 active+clean; 268 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 8.0 MiB/s wr, 595 op/s 2024-03-22T02:28:16.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 17892 20.53 MB/sec warmup 56 sec latency 74.494 ms 2024-03-22T02:28:16.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.940+0000 7f1ca90b1640 1 -- 172.21.15.28:0/2006728044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c093d70 msgr2=0x7f1c9c0941d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.940+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/2006728044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c093d70 0x7f1c9c0941d0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009900 tx=0x7f1c9002bc50 comp rx=0 tx=0).stop 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 -- 172.21.15.28:0/2006728044 shutdown_connections 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/2006728044 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c094710 0x7f1c9c094b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/2006728044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c093d70 0x7f1c9c0941d0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/2006728044 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c096230 0x7f1c9c096650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 -- 172.21.15.28:0/2006728044 >> 172.21.15.28:0/2006728044 conn(0x7f1c9c08f950 msgr2=0x7f1c9c091db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 -- 172.21.15.28:0/2006728044 shutdown_connections 2024-03-22T02:28:16.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.941+0000 7f1ca90b1640 1 -- 172.21.15.28:0/2006728044 wait complete. 2024-03-22T02:28:16.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.942+0000 7f1ca90b1640 1 Processor -- start 2024-03-22T02:28:16.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 -- start start 2024-03-22T02:28:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c093d70 0x7f1c9c129f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c094710 0x7f1c9c12a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c0a3cb0 con 0x7f1c9c096230 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c0a3b30 con 0x7f1c9c094710 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca90b1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c0a3e30 con 0x7f1c9c093d70 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.943+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c094710 0x7f1c9c12a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33132/0 (socket says 172.21.15.28:33132) 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3625758978 learned_addr learned my addr 172.21.15.28:0/3625758978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c093d70 0x7f1c9c129f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3625758978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c093d70 msgr2=0x7f1c9c129f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c093d70 0x7f1c9c129f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3625758978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c094710 msgr2=0x7f1c9c12a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:16.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c094710 0x7f1c9c12a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.944+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3625758978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c142bb0 con 0x7f1c9c096230 2024-03-22T02:28:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.945+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40511f0 tx=0x7f1ca4068440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.945+0000 7f1ca17fa640 1 -- 172.21.15.28:0/3625758978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4081070 con 0x7f1c9c096230 2024-03-22T02:28:16.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.945+0000 7f1ca17fa640 1 -- 172.21.15.28:0/3625758978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ca407ac70 con 0x7f1c9c096230 2024-03-22T02:28:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.945+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c142e40 con 0x7f1c9c096230 2024-03-22T02:28:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.945+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c143250 con 0x7f1c9c096230 2024-03-22T02:28:16.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.946+0000 7f1ca17fa640 1 -- 172.21.15.28:0/3625758978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4081070 con 0x7f1c9c096230 2024-03-22T02:28:16.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.949+0000 7f1ca17fa640 1 -- 172.21.15.28:0/3625758978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f1ca407dac0 con 0x7f1c9c096230 2024-03-22T02:28:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.950+0000 7f1ca17fa640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 0x7f1c940a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.950+0000 7f1ca17fa640 1 -- 172.21.15.28:0/3625758978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f1ca413b300 con 0x7f1c9c096230 2024-03-22T02:28:16.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.950+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 0x7f1c940a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.951+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 0x7f1c940a5620 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800a7d0 tx=0x7f1c98008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.951+0000 7f1cacfdd740 1 Processor -- start 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 -- start start 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccf1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf20a0 0x55882ccf6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55882cc56040 con 0x55882ccf7210 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55882cbfe9e0 con 0x55882ccf20a0 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55882cc51ff0 con 0x55882ccf16b0 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33138/0 (socket says 172.21.15.28:33138) 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3089339051 learned_addr learned my addr 172.21.15.28:0/3089339051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccf1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf20a0 0x55882ccf6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3089339051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 msgr2=0x55882ccf1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:16.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccf1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3089339051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf20a0 msgr2=0x55882ccf6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf20a0 0x55882ccf6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.952+0000 7f1ca88b0640 1 -- 172.21.15.28:0/3089339051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55882c96a210 con 0x55882ccf7210 2024-03-22T02:28:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.953+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40506a0 tx=0x7f1ca4067ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6d0076bcc01dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.953+0000 7f1c8e7fc640 1 -- 172.21.15.28:0/3089339051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca408a040 con 0x55882ccf7210 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.953+0000 7f1c8e7fc640 1 -- 172.21.15.28:0/3089339051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ca4107620 con 0x55882ccf7210 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.953+0000 7f1c8e7fc640 1 -- 172.21.15.28:0/3089339051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca4106a70 con 0x55882ccf7210 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 -- 172.21.15.28:0/3089339051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 msgr2=0x55882ccf96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40506a0 tx=0x7f1ca4067ad0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 -- 172.21.15.28:0/3089339051 shutdown_connections 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882ccf96a0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf20a0 0x55882ccf6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/3089339051 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccf1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.954+0000 7f1cacfdd740 1 -- 172.21.15.28:0/3089339051 >> 172.21.15.28:0/3089339051 conn(0x55882ccecc20 msgr2=0x55882cced020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.957+0000 7f1cacfdd740 1 -- 172.21.15.28:0/3089339051 shutdown_connections 2024-03-22T02:28:16.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.958+0000 7f1cacfdd740 1 -- 172.21.15.28:0/3089339051 wait complete. 2024-03-22T02:28:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.958+0000 7f1cacfdd740 1 -- start start 2024-03-22T02:28:16.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 Processor -- start 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf20a0 0x55882ccfbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882cd02bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55882ccfa1c0 con 0x55882ccf7210 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55882cc11db0 con 0x55882ccf16b0 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55882ccfa340 con 0x55882ccf20a0 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1cacfdd740 10 client.0 _ll_register_callbacks cb 0x55882c969a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.959+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:48850/0 (socket says 172.21.15.28:48850) 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1431240639 learned_addr learned my addr 172.21.15.28:0/1431240639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf20a0 0x55882ccfbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882cd02bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1431240639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf20a0 msgr2=0x55882ccfbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf20a0 0x55882ccfbbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1431240639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 msgr2=0x55882cd02bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882cd02bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1431240639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55882ca36620 con 0x55882ccf16b0 2024-03-22T02:28:16.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882cd02bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800cd10 tx=0x7f1c98004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f84aac35fbb93a6e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:16.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.960+0000 7f1c837fe640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.961+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98023070 con 0x55882ccf16b0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.961+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ca4107620 con 0x55882ccf16b0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.961+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f1c9801f640 con 0x55882ccf16b0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.962+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1c98019040 con 0x55882ccf16b0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.962+0000 7f1cacfdd740 10 client.29507 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.962+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55882cd05e10 con 0x55882ccf16b0 2024-03-22T02:28:16.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.962+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ca4107620 con 0x55882ccf16b0 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f1c9801e800 con 0x55882ccf16b0 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 10 client.29507 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 10 client.29507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 20 client.29507 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd09e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 4 client.29507 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:28:16.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.963+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55882cc63580 con 0x55882cd079a0 2024-03-22T02:28:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.965+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd09e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.966+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd09e00 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40dc0ae2d92e5619 server_cookie=a65d0a9933541310 in_seq=0 out_seq=0 2024-03-22T02:28:16.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.966+0000 7f1c837fe640 10 client.29507 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.969+0000 7f1c837fe640 1 -- 172.21.15.28:0/1431240639 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55882cc63580 con 0x55882cd079a0 2024-03-22T02:28:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.969+0000 7f1c837fe640 10 client.29507 handle_command_reply: tid=0 2024-03-22T02:28:16.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.969+0000 7f1cacfdd740 1 client.29507 shutdown 2024-03-22T02:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.970+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 msgr2=0x55882ccfb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:16.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.970+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800cd10 tx=0x7f1c98004750 comp rx=0 tx=0).stop 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 shutdown_connections 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd09e00 crc :-1 s=READY pgs=366 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf7210 0x55882cd02bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1c837fe640 0 client.29507 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf20a0 0x55882ccfbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.972+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1431240639 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfb670 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:16.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.973+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 >> 172.21.15.28:0/1431240639 conn(0x55882ccecc20 msgr2=0x55882ccebea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.974+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 shutdown_connections 2024-03-22T02:28:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.975+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1431240639 wait complete. 2024-03-22T02:28:16.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:16.975+0000 7f1cacfdd740 20 client.29507 trim_cache size 0 max 16384 2024-03-22T02:28:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 Processor -- start 2024-03-22T02:28:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 -- start start 2024-03-22T02:28:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd02b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf7210 0x55882ccf0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c150340 con 0x55882ccf20a0 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c14dbb0 con 0x55882ccf16b0 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.002+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c14ba00 con 0x55882ccf7210 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:48852/0 (socket says 172.21.15.28:48852) 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1318913638 learned_addr learned my addr 172.21.15.28:0/1318913638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd02b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf7210 0x55882ccf0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1318913638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf7210 msgr2=0x55882ccf0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf7210 0x55882ccf0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1318913638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 msgr2=0x55882cd02b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd02b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.009 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1318913638 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55882ccfa730 con 0x55882ccf16b0 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.003+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1c98000c00 tx=0x7f1c9801eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd85007257cc40d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1c8dffb640 1 -- 172.21.15.28:0/1318913638 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98023040 con 0x55882ccf16b0 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1c8dffb640 1 -- 172.21.15.28:0/1318913638 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c98013ba0 con 0x55882ccf16b0 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1c8dffb640 1 -- 172.21.15.28:0/1318913638 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98003690 con 0x55882ccf16b0 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1318913638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 msgr2=0x55882ccfbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f1c98000c00 tx=0x7f1c9801eee0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1318913638 shutdown_connections 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf7210 0x55882ccf0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd02b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1318913638 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf16b0 0x55882ccfbb10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.010 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.004+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1318913638 >> 172.21.15.28:0/1318913638 conn(0x55882ccecc20 msgr2=0x55882ccefe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1318913638 shutdown_connections 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1318913638 wait complete. 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- start start 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 Processor -- start 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd001a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf7210 0x55882cd011b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c14c070 con 0x55882ccf20a0 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c14d4e0 con 0x55882ccf7210 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1ca37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1cacfdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c14c740 con 0x55882ccf16b0 2024-03-22T02:28:17.011 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1ca3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40578/0 (socket says 172.21.15.28:40578) 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.005+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1214496989 learned_addr learned my addr 172.21.15.28:0/1214496989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1cacfdd740 10 client.0 _ll_register_callbacks cb 0x7f1c9800c990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca88b0640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf7210 0x55882cd011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf7210 msgr2=0x55882cd011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf7210 0x55882cd011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1214496989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 msgr2=0x55882cd001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 -- 172.21.15.28:0/1214496989 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55882cd017a0 con 0x55882ccf16b0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1ca3fff640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c520 tx=0x7f1c9801e5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff2bdca5b49a91dd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1c837fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98042060 con 0x55882ccf16b0 2024-03-22T02:28:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c98013ba0 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f1c98004030 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f1c98049050 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1cacfdd740 10 client.29509 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.006+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55882cd02a90 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c98013ba0 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f1c98006040 con 0x55882ccf16b0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 10 client.29509 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 10 client.29509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 20 client.29509 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd70190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 4 client.29509 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55882cc5d8d0 con 0x55882cd079a0 2024-03-22T02:28:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd70190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1ca37fe640 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd70190 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=258b5e4dab8b94b6 server_cookie=1284aa7d3d97d56d in_seq=0 out_seq=0 2024-03-22T02:28:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.007+0000 7f1c837fe640 10 client.29509 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:17.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 17981 20.26 MB/sec warmup 57 sec latency 584.311 ms 2024-03-22T02:28:17.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1c837fe640 1 -- 172.21.15.28:0/1214496989 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55882cc5d8d0 con 0x55882cd079a0 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1c837fe640 10 client.29509 handle_command_reply: tid=0 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 client.29509 shutdown 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 msgr2=0x55882ccffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c520 tx=0x7f1c9801e5f0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 shutdown_connections 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55882cd079a0 0x55882cd70190 crc :-1 s=READY pgs=367 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55882ccf7210 0x55882cd011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55882ccf20a0 0x55882cd001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 --2- 172.21.15.28:0/1214496989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55882ccf16b0 0x55882ccffc60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1c837fe640 0 client.29509 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 >> 172.21.15.28:0/1214496989 conn(0x55882ccecc20 msgr2=0x55882ccf0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:17.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 shutdown_connections 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.941+0000 7f1cacfdd740 1 -- 172.21.15.28:0/1214496989 wait complete. 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1cacfdd740 20 client.29509 trim_cache size 0 max 16384 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:28:17.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 msgr2=0x7f1c940a5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 0x7f1c940a5620 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800a7d0 tx=0x7f1c98008040 comp rx=0 tx=0).stop 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 msgr2=0x7f1c9c1314d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1ca40511f0 tx=0x7f1ca4068440 comp rx=0 tx=0).stop 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 shutdown_connections 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f1c940a3160 0x7f1c940a5620 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c096230 0x7f1c9c1314d0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1c9c094710 0x7f1c9c12a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 --2- 172.21.15.28:0/3625758978 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1c9c093d70 0x7f1c9c129f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:17.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 >> 172.21.15.28:0/3625758978 conn(0x7f1c9c08f950 msgr2=0x7f1c9c09b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:17.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 shutdown_connections 2024-03-22T02:28:17.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:17.942+0000 7f1ca90b1640 1 -- 172.21.15.28:0/3625758978 wait complete. 2024-03-22T02:28:18.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:17 smithi120 ceph-mon[36803]: pgmap v802: 129 pgs: 129 active+clean; 271 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 494 op/s 2024-03-22T02:28:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:17 smithi083 ceph-mon[36340]: pgmap v802: 129 pgs: 129 active+clean; 271 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 494 op/s 2024-03-22T02:28:18.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:17 smithi028 ceph-mon[31916]: pgmap v802: 129 pgs: 129 active+clean; 271 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.4 MiB/s wr, 494 op/s 2024-03-22T02:28:18.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 18158 20.04 MB/sec warmup 58 sec latency 640.168 ms 2024-03-22T02:28:19.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 18331 19.81 MB/sec warmup 59 sec latency 231.263 ms 2024-03-22T02:28:20.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:19 smithi120 ceph-mon[36803]: pgmap v803: 129 pgs: 129 active+clean; 261 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 589 op/s 2024-03-22T02:28:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:19 smithi083 ceph-mon[36340]: pgmap v803: 129 pgs: 129 active+clean; 261 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 589 op/s 2024-03-22T02:28:20.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:19 smithi028 ceph-mon[31916]: pgmap v803: 129 pgs: 129 active+clean; 261 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 589 op/s 2024-03-22T02:28:20.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 18607 19.65 MB/sec warmup 60 sec latency 58.773 ms 2024-03-22T02:28:21.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 18880 19.48 MB/sec warmup 61 sec latency 128.634 ms 2024-03-22T02:28:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:21 smithi120 ceph-mon[36803]: pgmap v804: 129 pgs: 129 active+clean; 256 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 514 op/s 2024-03-22T02:28:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:21 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:21 smithi083 ceph-mon[36340]: pgmap v804: 129 pgs: 129 active+clean; 256 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 514 op/s 2024-03-22T02:28:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:21 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:22.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:21 smithi028 ceph-mon[31916]: pgmap v804: 129 pgs: 129 active+clean; 256 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 514 op/s 2024-03-22T02:28:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:21 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:28:22.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 19222 19.38 MB/sec warmup 62 sec latency 75.659 ms 2024-03-22T02:28:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:23.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:23.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:22 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:28:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:23.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:22 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:28:23.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:28:23.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:28:23.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:22 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:28:23.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 19589 19.35 MB/sec warmup 63 sec latency 242.012 ms 2024-03-22T02:28:24.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 19902 19.30 MB/sec warmup 64 sec latency 113.756 ms 2024-03-22T02:28:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:24 smithi120 ceph-mon[36803]: pgmap v805: 129 pgs: 129 active+clean; 253 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 530 op/s 2024-03-22T02:28:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:24 smithi083 ceph-mon[36340]: pgmap v805: 129 pgs: 129 active+clean; 253 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 530 op/s 2024-03-22T02:28:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:24 smithi028 ceph-mon[31916]: pgmap v805: 129 pgs: 129 active+clean; 253 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 530 op/s 2024-03-22T02:28:25.674 INFO:tasks.workunit.client.0.smithi028.stdout: 10 20437 19.51 MB/sec warmup 65 sec latency 99.426 ms 2024-03-22T02:28:25.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:25 smithi120 ceph-mon[36803]: pgmap v806: 129 pgs: 129 active+clean; 266 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-03-22T02:28:26.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:25 smithi083 ceph-mon[36340]: pgmap v806: 129 pgs: 129 active+clean; 266 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-03-22T02:28:26.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:25 smithi028 ceph-mon[31916]: pgmap v806: 129 pgs: 129 active+clean; 266 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.5 MiB/s wr, 557 op/s 2024-03-22T02:28:26.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 20898 19.70 MB/sec warmup 66 sec latency 93.764 ms 2024-03-22T02:28:27.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 21212 19.71 MB/sec warmup 67 sec latency 62.327 ms 2024-03-22T02:28:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:27 smithi120 ceph-mon[36803]: pgmap v807: 129 pgs: 129 active+clean; 272 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 575 op/s 2024-03-22T02:28:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:27 smithi083 ceph-mon[36340]: pgmap v807: 129 pgs: 129 active+clean; 272 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 575 op/s 2024-03-22T02:28:28.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:27 smithi028 ceph-mon[31916]: pgmap v807: 129 pgs: 129 active+clean; 272 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 575 op/s 2024-03-22T02:28:28.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 21433 19.70 MB/sec warmup 68 sec latency 124.665 ms 2024-03-22T02:28:29.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 21640 19.60 MB/sec warmup 69 sec latency 78.485 ms 2024-03-22T02:28:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:29 smithi120 ceph-mon[36803]: pgmap v808: 129 pgs: 129 active+clean; 263 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 585 op/s 2024-03-22T02:28:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:29 smithi083 ceph-mon[36340]: pgmap v808: 129 pgs: 129 active+clean; 263 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 585 op/s 2024-03-22T02:28:30.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:29 smithi028 ceph-mon[31916]: pgmap v808: 129 pgs: 129 active+clean; 263 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.7 MiB/s wr, 585 op/s 2024-03-22T02:28:30.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 21809 19.38 MB/sec warmup 70 sec latency 99.839 ms 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 -- 172.21.15.28:0/1241966685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c101ca0 msgr2=0x7fe58c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 --2- 172.21.15.28:0/1241966685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c101ca0 0x7fe58c102120 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe58000d910 tx=0x7fe58002dbb0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 -- 172.21.15.28:0/1241966685 shutdown_connections 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 --2- 172.21.15.28:0/1241966685 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 0x7fe58c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 --2- 172.21.15.28:0/1241966685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c101ca0 0x7fe58c102120 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 --2- 172.21.15.28:0/1241966685 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c100ab0 0x7fe58c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 -- 172.21.15.28:0/1241966685 >> 172.21.15.28:0/1241966685 conn(0x7fe58c0fc020 msgr2=0x7fe58c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 -- 172.21.15.28:0/1241966685 shutdown_connections 2024-03-22T02:28:31.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.061+0000 7fe591bf7640 1 -- 172.21.15.28:0/1241966685 wait complete. 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 Processor -- start 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 -- start start 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c101ca0 0x7fe58c1961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 0x7fe58c19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c10b6e0 con 0x7fe58c100ab0 2024-03-22T02:28:31.064 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.062+0000 7fe591bf7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c10b560 con 0x7fe58c102660 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 0x7fe58c19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40096/0 (socket says 172.21.15.28:40096) 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 -- 172.21.15.28:0/854113126 learned_addr learned my addr 172.21.15.28:0/854113126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c10b860 con 0x7fe58c101ca0 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 -- 172.21.15.28:0/854113126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c101ca0 msgr2=0x7fe58c1961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c101ca0 0x7fe58c1961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 -- 172.21.15.28:0/854113126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 msgr2=0x7fe58c19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 0x7fe58c19d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.063+0000 7fe58b7fe640 1 -- 172.21.15.28:0/854113126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c19d8d0 con 0x7fe58c100ab0 2024-03-22T02:28:31.065 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b450 tx=0x7fe57c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe588ff9640 1 -- 172.21.15.28:0/854113126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c015020 con 0x7fe58c100ab0 2024-03-22T02:28:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe588ff9640 1 -- 172.21.15.28:0/854113126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c011d70 con 0x7fe58c100ab0 2024-03-22T02:28:31.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58c19dbc0 con 0x7fe58c100ab0 2024-03-22T02:28:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe588ff9640 1 -- 172.21.15.28:0/854113126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00f340 con 0x7fe58c100ab0 2024-03-22T02:28:31.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.064+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58c19e110 con 0x7fe58c100ab0 2024-03-22T02:28:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.067+0000 7fe588ff9640 1 -- 172.21.15.28:0/854113126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe57c018020 con 0x7fe58c100ab0 2024-03-22T02:28:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.068+0000 7fe588ff9640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 0x7fe5740a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.068+0000 7fe588ff9640 1 -- 172.21.15.28:0/854113126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe57c0cf110 con 0x7fe58c100ab0 2024-03-22T02:28:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.069+0000 7fe58affd640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 0x7fe5740a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.070+0000 7fe5947e6740 1 Processor -- start 2024-03-22T02:28:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.070+0000 7fe5947e6740 1 -- start start 2024-03-22T02:28:31.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.070+0000 7fe58affd640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 0x7fe5740a55f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe58002e0f0 tx=0x7fe580000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.070+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 0x55845e3a0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 0x55845e3a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe5947e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55845e305040 con 0x55845e3a6040 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe5947e6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55845e2ad9e0 con 0x55845e3a04e0 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe5947e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55845e300ff0 con 0x55845e3a0ed0 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe58affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.072 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 0x55845e3a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe58affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45550/0 (socket says 172.21.15.28:45550) 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe58affd640 1 -- 172.21.15.28:0/618918122 learned_addr learned my addr 172.21.15.28:0/618918122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.071+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 0x55845e3a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 -- 172.21.15.28:0/618918122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 msgr2=0x55845e3a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 -- 172.21.15.28:0/618918122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 msgr2=0x55845e3a84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 0x55845e3a84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 -- 172.21.15.28:0/618918122 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845e019210 con 0x55845e3a04e0 2024-03-22T02:28:31.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58bfff640 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 0x55845e3a84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:31.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 0x55845e3a0900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cc800 tx=0x7fe57c09bb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2d41a379af7684 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe571ffb640 1 -- 172.21.15.28:0/618918122 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c09f580 con 0x55845e3a04e0 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe571ffb640 1 -- 172.21.15.28:0/618918122 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a3e40 con 0x55845e3a04e0 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.072+0000 7fe571ffb640 1 -- 172.21.15.28:0/618918122 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0b4270 con 0x55845e3a04e0 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 -- 172.21.15.28:0/618918122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 msgr2=0x55845e3a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 0x55845e3a0900 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cc800 tx=0x7fe57c09bb30 comp rx=0 tx=0).stop 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 -- 172.21.15.28:0/618918122 shutdown_connections 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a6040 0x55845e3a84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 --2- 172.21.15.28:0/618918122 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a04e0 0x55845e3a0900 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 -- 172.21.15.28:0/618918122 >> 172.21.15.28:0/618918122 conn(0x55845e39ba50 msgr2=0x55845e39be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 -- 172.21.15.28:0/618918122 shutdown_connections 2024-03-22T02:28:31.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.073+0000 7fe5947e6740 1 -- 172.21.15.28:0/618918122 wait complete. 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 -- start start 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 Processor -- start 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3b3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e3ba390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.074+0000 7fe5947e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55845e3a8ff0 con 0x55845e3a04e0 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe5947e6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55845e2c0db0 con 0x55845e3a6040 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe5947e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55845e3a9170 con 0x55845e3a0ed0 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40112/0 (socket says 172.21.15.28:40112) 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 -- 172.21.15.28:0/3022970593 learned_addr learned my addr 172.21.15.28:0/3022970593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe5947e6740 10 client.0 _ll_register_callbacks cb 0x55845e018a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:31.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 -- 172.21.15.28:0/3022970593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 msgr2=0x55845e3b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 -- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 msgr2=0x55845e3ba390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:28:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e3ba390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 -- 172.21.15.28:0/3022970593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845e0e5620 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cf430 tx=0x7fe57c09f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba02821b0d5038cb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.075+0000 7fe56affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0b4480 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe57c0a3e40 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe57c09a880 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe57c09aa80 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe5947e6740 10 client.19716 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.076+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55845e3bd6e0 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a3e40 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fe57c09a450 con 0x55845e3a04e0 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 10 client.19716 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:31.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 10 client.19716 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 20 client.19716 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e3be860 0x55845e3c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 4 client.19716 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55845e312580 con 0x55845e3be860 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.077+0000 7fe58affd640 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e3be860 0x55845e3c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.078+0000 7fe58affd640 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e3be860 0x55845e3c0c70 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d48b9d0ae1bb70b0 server_cookie=83113f55b6676133 in_seq=0 out_seq=0 2024-03-22T02:28:31.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.078+0000 7fe56affd640 10 client.19716 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe56affd640 1 -- 172.21.15.28:0/3022970593 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55845e312580 con 0x55845e3be860 2024-03-22T02:28:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe56affd640 10 client.19716 handle_command_reply: tid=0 2024-03-22T02:28:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 client.19716 shutdown 2024-03-22T02:28:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 msgr2=0x55845e3b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cf430 tx=0x7fe57c09f580 comp rx=0 tx=0).stop 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 shutdown_connections 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e3be860 0x55845e3c0c70 crc :-1 s=READY pgs=368 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e3ba390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 --2- 172.21.15.28:0/3022970593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e3b2e10 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe56affd640 0 client.19716 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 >> 172.21.15.28:0/3022970593 conn(0x55845e39ba50 msgr2=0x55845e39ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 shutdown_connections 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.080+0000 7fe5947e6740 1 -- 172.21.15.28:0/3022970593 wait complete. 2024-03-22T02:28:31.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.081+0000 7fe5947e6740 20 client.19716 trim_cache size 0 max 16384 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.098+0000 7fe5947e6740 1 Processor -- start 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.098+0000 7fe5947e6740 1 -- start start 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3ba350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e39a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5780126f0 con 0x55845e3a04e0 2024-03-22T02:28:31.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe57800ff60 con 0x55845e3a6040 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5947e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe57800ddb0 con 0x55845e3a0ed0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40116/0 (socket says 172.21.15.28:40116) 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3ba350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 -- 172.21.15.28:0/326622936 learned_addr learned my addr 172.21.15.28:0/326622936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58bfff640 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e39a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 -- 172.21.15.28:0/326622936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 msgr2=0x55845e3ba350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3ba350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 -- 172.21.15.28:0/326622936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 msgr2=0x55845e39a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e39a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 -- 172.21.15.28:0/326622936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55845e39c7c0 con 0x55845e3a04e0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09b130 tx=0x7fe57c09bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f824dc06975b5c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.099+0000 7fe5717fa640 1 -- 172.21.15.28:0/326622936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0a7020 con 0x55845e3a04e0 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5717fa640 1 -- 172.21.15.28:0/326622936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a3e40 con 0x55845e3a04e0 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5717fa640 1 -- 172.21.15.28:0/326622936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0a0950 con 0x55845e3a04e0 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 -- 172.21.15.28:0/326622936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 msgr2=0x55845e39a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09b130 tx=0x7fe57c09bb30 comp rx=0 tx=0).stop 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 -- 172.21.15.28:0/326622936 shutdown_connections 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e39a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a0ed0 0x55845e3ba350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 --2- 172.21.15.28:0/326622936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a04e0 0x55845e39a3c0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 -- 172.21.15.28:0/326622936 >> 172.21.15.28:0/326622936 conn(0x55845e39ba50 msgr2=0x55845e39f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 -- 172.21.15.28:0/326622936 shutdown_connections 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.100+0000 7fe5947e6740 1 -- 172.21.15.28:0/326622936 wait complete. 2024-03-22T02:28:31.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 -- start start 2024-03-22T02:28:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 Processor -- start 2024-03-22T02:28:31.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a04e0 0x55845e3b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e422b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe57800e420 con 0x55845e3a0ed0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe57800f890 con 0x55845e3a6040 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe57800eaf0 con 0x55845e3a04e0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe5947e6740 10 client.0 _ll_register_callbacks cb 0x7fe584009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a04e0 0x55845e3b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40120/0 (socket says 172.21.15.28:40120) 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 -- 172.21.15.28:0/459761665 learned_addr learned my addr 172.21.15.28:0/459761665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58bfff640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e422b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 -- 172.21.15.28:0/459761665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a04e0 msgr2=0x55845e3b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a04e0 0x55845e3b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 -- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 msgr2=0x55845e422b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e422b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.101+0000 7fe58affd640 1 -- 172.21.15.28:0/459761665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55845e4230a0 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe58affd640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe58002e1c0 tx=0x7fe580004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f909b8fa767b797f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58004ed90 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe57c0a3e40 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fe5800052c0 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe58005d050 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 10 client.19722 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55845e4242c0 con 0x55845e3a0ed0 2024-03-22T02:28:31.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a3e40 con 0x55845e3a0ed0 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fe5800039c0 con 0x55845e3a0ed0 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 10 client.19722 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 10 client.19722 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 20 client.19722 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e424ca0 0x55845e4270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 4 client.19722 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.102+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55845e30c8d0 con 0x55845e424ca0 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.103+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e424ca0 0x55845e4270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.103+0000 7fe56affd640 10 client.19722 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:31.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.103+0000 7fe58b7fe640 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e424ca0 0x55845e4270c0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3adcf0a11300b137 server_cookie=bbd68195105381ef in_seq=0 out_seq=0 2024-03-22T02:28:31.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 21915 19.16 MB/sec warmup 71 sec latency 473.378 ms 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe56affd640 1 -- 172.21.15.28:0/459761665 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55845e30c8d0 con 0x55845e424ca0 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe56affd640 10 client.19722 handle_command_reply: tid=0 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 client.19722 shutdown 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 msgr2=0x55845e3b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fe58002e1c0 tx=0x7fe580004410 comp rx=0 tx=0).stop 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 shutdown_connections 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55845e424ca0 0x55845e4270c0 crc :-1 s=READY pgs=369 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe56affd640 0 client.19722 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55845e3a6040 0x55845e422b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.854+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55845e3a0ed0 0x55845e3b76a0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.855+0000 7fe5947e6740 1 --2- 172.21.15.28:0/459761665 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55845e3a04e0 0x55845e3b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.855+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 >> 172.21.15.28:0/459761665 conn(0x55845e39ba50 msgr2=0x55845e3a1970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.855+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 shutdown_connections 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.855+0000 7fe5947e6740 1 -- 172.21.15.28:0/459761665 wait complete. 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.855+0000 7fe5947e6740 20 client.19722 trim_cache size 0 max 16384 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:28:31.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:28:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 msgr2=0x7fe5740a55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 0x7fe5740a55f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fe58002e0f0 tx=0x7fe580000bc0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 msgr2=0x7fe58c195c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00b450 tx=0x7fe57c00b910 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 shutdown_connections 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe5740a3130 0x7fe5740a55f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe58c102660 0x7fe58c19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe58c101ca0 0x7fe58c1961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 --2- 172.21.15.28:0/854113126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe58c100ab0 0x7fe58c195c90 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.856+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 >> 172.21.15.28:0/854113126 conn(0x7fe58c0fc020 msgr2=0x7fe58c104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.857+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 shutdown_connections 2024-03-22T02:28:31.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:31.857+0000 7fe591bf7640 1 -- 172.21.15.28:0/854113126 wait complete. 2024-03-22T02:28:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:31 smithi120 ceph-mon[36803]: pgmap v809: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 396 op/s 2024-03-22T02:28:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:31 smithi083 ceph-mon[36340]: pgmap v809: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 396 op/s 2024-03-22T02:28:32.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:31 smithi028 ceph-mon[31916]: pgmap v809: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 3.9 MiB/s wr, 396 op/s 2024-03-22T02:28:32.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 22148 19.03 MB/sec warmup 72 sec latency 579.920 ms 2024-03-22T02:28:33.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 22326 18.87 MB/sec warmup 73 sec latency 387.907 ms 2024-03-22T02:28:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:33 smithi120 ceph-mon[36803]: pgmap v810: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 384 op/s 2024-03-22T02:28:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:33 smithi083 ceph-mon[36340]: pgmap v810: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 384 op/s 2024-03-22T02:28:34.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:33 smithi028 ceph-mon[31916]: pgmap v810: 129 pgs: 129 active+clean; 260 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 384 op/s 2024-03-22T02:28:34.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 22613 18.75 MB/sec warmup 74 sec latency 169.180 ms 2024-03-22T02:28:35.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 23104 18.76 MB/sec warmup 75 sec latency 90.102 ms 2024-03-22T02:28:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:35 smithi120 ceph-mon[36803]: pgmap v811: 129 pgs: 129 active+clean; 267 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 531 op/s 2024-03-22T02:28:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:35 smithi083 ceph-mon[36340]: pgmap v811: 129 pgs: 129 active+clean; 267 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 531 op/s 2024-03-22T02:28:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:35 smithi028 ceph-mon[31916]: pgmap v811: 129 pgs: 129 active+clean; 267 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 531 op/s 2024-03-22T02:28:36.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 23600 18.91 MB/sec warmup 76 sec latency 132.736 ms 2024-03-22T02:28:37.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 24133 19.07 MB/sec warmup 77 sec latency 50.592 ms 2024-03-22T02:28:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:37 smithi120 ceph-mon[36803]: pgmap v812: 129 pgs: 129 active+clean; 259 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:28:38.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:37 smithi083 ceph-mon[36340]: pgmap v812: 129 pgs: 129 active+clean; 259 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:28:38.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:37 smithi028 ceph-mon[31916]: pgmap v812: 129 pgs: 129 active+clean; 259 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:28:38.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 24486 19.15 MB/sec warmup 78 sec latency 147.329 ms 2024-03-22T02:28:39.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 24749 19.13 MB/sec warmup 79 sec latency 80.581 ms 2024-03-22T02:28:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:39 smithi120 ceph-mon[36803]: pgmap v813: 129 pgs: 129 active+clean; 256 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 543 op/s 2024-03-22T02:28:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:39 smithi083 ceph-mon[36340]: pgmap v813: 129 pgs: 129 active+clean; 256 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 543 op/s 2024-03-22T02:28:40.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:39 smithi028 ceph-mon[31916]: pgmap v813: 129 pgs: 129 active+clean; 256 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 543 op/s 2024-03-22T02:28:40.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25023 19.14 MB/sec warmup 80 sec latency 87.035 ms 2024-03-22T02:28:41.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25238 19.06 MB/sec warmup 81 sec latency 94.419 ms 2024-03-22T02:28:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:41 smithi120 ceph-mon[36803]: pgmap v814: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.9 MiB/s wr, 559 op/s 2024-03-22T02:28:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:41 smithi083 ceph-mon[36340]: pgmap v814: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.9 MiB/s wr, 559 op/s 2024-03-22T02:28:42.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:41 smithi028 ceph-mon[31916]: pgmap v814: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 4.9 MiB/s wr, 559 op/s 2024-03-22T02:28:42.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25425 18.89 MB/sec warmup 82 sec latency 129.986 ms 2024-03-22T02:28:43.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25592 18.77 MB/sec warmup 83 sec latency 270.053 ms 2024-03-22T02:28:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:43 smithi120 ceph-mon[36803]: pgmap v815: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-03-22T02:28:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:43 smithi083 ceph-mon[36340]: pgmap v815: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-03-22T02:28:44.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:43 smithi028 ceph-mon[31916]: pgmap v815: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.1 MiB/s wr, 547 op/s 2024-03-22T02:28:44.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25785 18.60 MB/sec warmup 84 sec latency 188.936 ms 2024-03-22T02:28:44.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 -- 172.21.15.28:0/3863368833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728101bf0 msgr2=0x7fd728102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 --2- 172.21.15.28:0/3863368833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728101bf0 0x7fd728102010 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd718009900 tx=0x7fd71802bc30 comp rx=0 tx=0).stop 2024-03-22T02:28:44.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 -- 172.21.15.28:0/3863368833 shutdown_connections 2024-03-22T02:28:44.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 --2- 172.21.15.28:0/3863368833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd7281037a0 0x7fd72810a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 --2- 172.21.15.28:0/3863368833 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd728102de0 0x7fd728103260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 --2- 172.21.15.28:0/3863368833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728101bf0 0x7fd728102010 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 -- 172.21.15.28:0/3863368833 >> 172.21.15.28:0/3863368833 conn(0x7fd7280fd160 msgr2=0x7fd7280ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.949+0000 7fd72de8d640 1 -- 172.21.15.28:0/3863368833 shutdown_connections 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 -- 172.21.15.28:0/3863368833 wait complete. 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 Processor -- start 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 -- start start 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7281037a0 0x7fd7280744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810d870 con 0x7fd728102de0 2024-03-22T02:28:44.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810d6f0 con 0x7fd7281037a0 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.950+0000 7fd72de8d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd72810d9f0 con 0x7fd728101bf0 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34438/0 (socket says 172.21.15.28:34438) 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 -- 172.21.15.28:0/861670365 learned_addr learned my addr 172.21.15.28:0/861670365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:32786/0 (socket says 172.21.15.28:32786) 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd727fff640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7281037a0 0x7fd7280744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 -- 172.21.15.28:0/861670365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 msgr2=0x7fd728075940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 -- 172.21.15.28:0/861670365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7281037a0 msgr2=0x7fd7280744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7281037a0 0x7fd7280744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 -- 172.21.15.28:0/861670365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd728074d50 con 0x7fd728102de0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd7277fe640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd726ffd640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00e9f0 tx=0x7fd71c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd724ff9640 1 -- 172.21.15.28:0/861670365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c00cd80 con 0x7fd728102de0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7281a1f70 con 0x7fd728102de0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd724ff9640 1 -- 172.21.15.28:0/861670365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd71c010910 con 0x7fd728102de0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd724ff9640 1 -- 172.21.15.28:0/861670365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71c018950 con 0x7fd728102de0 2024-03-22T02:28:44.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.951+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7281a23e0 con 0x7fd728102de0 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.953+0000 7fd724ff9640 1 -- 172.21.15.28:0/861670365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd71c010ac0 con 0x7fd728102de0 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 Processor -- start 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 -- start start 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6546900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd724ff9640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 0x7fd7100a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc654e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfc64ab040 con 0x55bfc654c040 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bfc64539e0 con 0x55bfc65464e0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd730a71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bfc64a6ff0 con 0x55bfc6546ed0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc654e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:32800/0 (socket says 172.21.15.28:32800) 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd727fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6546900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd7277fe640 1 -- 172.21.15.28:0/2130328275 learned_addr learned my addr 172.21.15.28:0/2130328275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.954+0000 7fd726ffd640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 0x7fd7100a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 -- 172.21.15.28:0/2130328275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 msgr2=0x55bfc6546900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd724ff9640 1 -- 172.21.15.28:0/861670365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd71c0cfae0 con 0x7fd728102de0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6546900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 -- 172.21.15.28:0/2130328275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 msgr2=0x55bfc654e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc654e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 -- 172.21.15.28:0/2130328275 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc61bf210 con 0x55bfc6546ed0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd7277fe640 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd718009900 tx=0x7fd71802cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a06a7f975e15d862 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd726ffd640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 0x7fd7100a5480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd71c098880 tx=0x7fd71c0c6ac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd70dffb640 1 -- 172.21.15.28:0/2130328275 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718005d50 con 0x55bfc6546ed0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd70dffb640 1 -- 172.21.15.28:0/2130328275 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718005520 con 0x55bfc6546ed0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.955+0000 7fd70dffb640 1 -- 172.21.15.28:0/2130328275 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718004c20 con 0x55bfc6546ed0 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- 172.21.15.28:0/2130328275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 msgr2=0x55bfc654b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd718009900 tx=0x7fd71802cd30 comp rx=0 tx=0).stop 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- 172.21.15.28:0/2130328275 shutdown_connections 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc654e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc6546ed0 0x55bfc654b9d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 --2- 172.21.15.28:0/2130328275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6546900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- 172.21.15.28:0/2130328275 >> 172.21.15.28:0/2130328275 conn(0x55bfc6541a50 msgr2=0x55bfc6541e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- 172.21.15.28:0/2130328275 shutdown_connections 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- 172.21.15.28:0/2130328275 wait complete. 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.956+0000 7fd730a71740 1 -- start start 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 Processor -- start 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc6550a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc6557a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bfc654ef90 con 0x55bfc654c040 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bfc6466db0 con 0x55bfc6546ed0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bfc654f110 con 0x55bfc65464e0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd730a71740 10 client.0 _ll_register_callbacks cb 0x55bfc61bea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc6557a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc6550a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd727fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd727fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:32816/0 (socket says 172.21.15.28:32816) 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc6550a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51726/0 (socket says 172.21.15.28:51726) 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.957+0000 7fd727fff640 1 -- 172.21.15.28:0/646190307 learned_addr learned my addr 172.21.15.28:0/646190307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 -- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 msgr2=0x55bfc6550a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc6550a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 -- 172.21.15.28:0/646190307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 msgr2=0x55bfc6557a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc6557a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 -- 172.21.15.28:0/646190307 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc628b620 con 0x55bfc65464e0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd727fff640 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd72000f8d0 tx=0x7fd720004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=885b44ce1e9cfa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720019040 con 0x55bfc65464e0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd718005520 con 0x55bfc65464e0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd720004d20 con 0x55bfc65464e0 2024-03-22T02:28:44.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd72002d050 con 0x55bfc65464e0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd730a71740 10 client.29530 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bfc655ade0 con 0x55bfc65464e0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.958+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd718005520 con 0x55bfc65464e0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fd72001f040 con 0x55bfc65464e0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 10 client.29530 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 10 client.29530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 20 client.29530 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc655c4d0 0x55bfc655e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 4 client.29530 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bfc64b8580 con 0x55bfc655c4d0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.959+0000 7fd7277fe640 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc655c4d0 0x55bfc655e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.960+0000 7fd706ffd640 10 client.29530 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:44.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.960+0000 7fd7277fe640 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc655c4d0 0x55bfc655e970 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=189f922f45eeeb24 server_cookie=ea3412315860592e in_seq=0 out_seq=0 2024-03-22T02:28:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.964+0000 7fd706ffd640 1 -- 172.21.15.28:0/646190307 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bfc64b8580 con 0x55bfc655c4d0 2024-03-22T02:28:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.964+0000 7fd706ffd640 10 client.29530 handle_command_reply: tid=0 2024-03-22T02:28:44.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.964+0000 7fd730a71740 1 client.29530 shutdown 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.964+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 msgr2=0x55bfc65504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.964+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd72000f8d0 tx=0x7fd720004ab0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 shutdown_connections 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc655c4d0 0x55bfc655e970 crc :-1 s=READY pgs=370 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc6557a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc6550a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd706ffd640 0 client.29530 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 --2- 172.21.15.28:0/646190307 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc65504f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 >> 172.21.15.28:0/646190307 conn(0x55bfc6541a50 msgr2=0x55bfc6540c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:44.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 shutdown_connections 2024-03-22T02:28:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 1 -- 172.21.15.28:0/646190307 wait complete. 2024-03-22T02:28:44.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.965+0000 7fd730a71740 20 client.29530 trim_cache size 0 max 16384 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 Processor -- start 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 -- start start 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc6550990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc65579c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281af4d0 con 0x55bfc654c040 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281acd40 con 0x55bfc6546ed0 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd730a71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281aab90 con 0x55bfc65464e0 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd727fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc6550990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc65579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd727fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.980+0000 7fd727fff640 1 -- 172.21.15.28:0/638174689 learned_addr learned my addr 172.21.15.28:0/638174689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 -- 172.21.15.28:0/638174689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 msgr2=0x55bfc6550990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc6550990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 -- 172.21.15.28:0/638174689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 msgr2=0x55bfc65579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc65579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 -- 172.21.15.28:0/638174689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bfc65427c0 con 0x55bfc654c040 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd727fff640 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fd72000f8a0 tx=0x7fd720004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b72f0e5007f9179 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd70d7fa640 1 -- 172.21.15.28:0/638174689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72000e870 con 0x55bfc654c040 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd70d7fa640 1 -- 172.21.15.28:0/638174689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd72000ea10 con 0x55bfc654c040 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.981+0000 7fd70d7fa640 1 -- 172.21.15.28:0/638174689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720003f70 con 0x55bfc654c040 2024-03-22T02:28:44.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- 172.21.15.28:0/638174689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 msgr2=0x55bfc65453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fd72000f8a0 tx=0x7fd720004a40 comp rx=0 tx=0).stop 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- 172.21.15.28:0/638174689 shutdown_connections 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc654c040 0x55bfc65453c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc6546ed0 0x55bfc65579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 --2- 172.21.15.28:0/638174689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc65464e0 0x55bfc6550990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- 172.21.15.28:0/638174689 >> 172.21.15.28:0/638174689 conn(0x55bfc6541a50 msgr2=0x55bfc6544cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- 172.21.15.28:0/638174689 shutdown_connections 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- 172.21.15.28:0/638174689 wait complete. 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 -- start start 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.982+0000 7fd730a71740 1 Processor -- start 2024-03-22T02:28:44.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6554a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc654c040 0x55bfc65c08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281ab200 con 0x55bfc6546ed0 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281ac670 con 0x55bfc65464e0 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281ab8d0 con 0x55bfc654c040 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 10 client.0 _ll_register_callbacks cb 0x7fd72000f4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd727fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6554a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34482/0 (socket says 172.21.15.28:34482) 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd727fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6554a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:51744/0 (socket says 172.21.15.28:51744) 2024-03-22T02:28:44.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 -- 172.21.15.28:0/4267800530 learned_addr learned my addr 172.21.15.28:0/4267800530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd727fff640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc654c040 0x55bfc65c08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 -- 172.21.15.28:0/4267800530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc654c040 msgr2=0x55bfc65c08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc654c040 0x55bfc65c08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 -- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 msgr2=0x55bfc6554a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6554a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 -- 172.21.15.28:0/4267800530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bfc65c0e90 con 0x55bfc6546ed0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd7277fe640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd7180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e410ce2086fc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd706ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718004450 con 0x55bfc6546ed0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7fd718005040 con 0x55bfc6546ed0 2024-03-22T02:28:44.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.983+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd72000ea10 con 0x55bfc6546ed0 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd71804eb10 con 0x55bfc6546ed0 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 10 client.19740 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55bfc65c2210 con 0x55bfc6546ed0 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd72000ea10 con 0x55bfc6546ed0 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7fd7180048f0 con 0x55bfc6546ed0 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 10 client.19740 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 10 client.19740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 20 client.19740 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:44.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc65c27f0 0x55bfc65c4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 4 client.19740 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:28:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bfc64b28d0 con 0x55bfc65c27f0 2024-03-22T02:28:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.984+0000 7fd727fff640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc65c27f0 0x55bfc65c4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.985+0000 7fd706ffd640 10 client.19740 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:44.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:44.985+0000 7fd727fff640 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc65c27f0 0x55bfc65c4be0 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c003722f44706408 server_cookie=58a884fbdcc4c662 in_seq=0 out_seq=0 2024-03-22T02:28:45.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 25956 18.50 MB/sec warmup 85 sec latency 477.876 ms 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd706ffd640 1 -- 172.21.15.28:0/4267800530 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bfc64b28d0 con 0x55bfc65c27f0 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd706ffd640 10 client.19740 handle_command_reply: tid=0 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 client.19740 shutdown 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 msgr2=0x55bfc6554fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd7180046c0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 shutdown_connections 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bfc65c27f0 0x55bfc65c4be0 crc :-1 s=READY pgs=371 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd706ffd640 0 client.19740 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bfc654c040 0x55bfc65c08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bfc6546ed0 0x55bfc6554fd0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 --2- 172.21.15.28:0/4267800530 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bfc65464e0 0x55bfc6554a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 >> 172.21.15.28:0/4267800530 conn(0x55bfc6541a50 msgr2=0x55bfc6545540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 shutdown_connections 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.832+0000 7fd730a71740 1 -- 172.21.15.28:0/4267800530 wait complete. 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd730a71740 20 client.19740 trim_cache size 0 max 16384 2024-03-22T02:28:45.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:28:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:28:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:28:45.837 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 msgr2=0x7fd7100a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 0x7fd7100a5480 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd71c098880 tx=0x7fd71c0c6ac0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 msgr2=0x7fd728073f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fd71c00e9f0 tx=0x7fd71c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 shutdown_connections 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd7100a2fc0 0x7fd7100a5480 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd7281037a0 0x7fd7280744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd728102de0 0x7fd728073f80 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 --2- 172.21.15.28:0/861670365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd728101bf0 0x7fd728075940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 >> 172.21.15.28:0/861670365 conn(0x7fd7280fd160 msgr2=0x7fd728106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 shutdown_connections 2024-03-22T02:28:45.839 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:45.833+0000 7fd72de8d640 1 -- 172.21.15.28:0/861670365 wait complete. 2024-03-22T02:28:46.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:45 smithi120 ceph-mon[36803]: pgmap v816: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-03-22T02:28:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:45 smithi083 ceph-mon[36340]: pgmap v816: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-03-22T02:28:46.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:45 smithi028 ceph-mon[31916]: pgmap v816: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.4 MiB/s wr, 535 op/s 2024-03-22T02:28:46.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 26316 18.41 MB/sec warmup 86 sec latency 495.993 ms 2024-03-22T02:28:47.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 26756 18.44 MB/sec warmup 87 sec latency 69.307 ms 2024-03-22T02:28:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:47 smithi120 ceph-mon[36803]: pgmap v817: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 395 op/s 2024-03-22T02:28:48.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:47 smithi083 ceph-mon[36340]: pgmap v817: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 395 op/s 2024-03-22T02:28:48.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:47 smithi028 ceph-mon[31916]: pgmap v817: 129 pgs: 129 active+clean; 263 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 395 op/s 2024-03-22T02:28:48.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 27160 18.51 MB/sec warmup 88 sec latency 156.491 ms 2024-03-22T02:28:49.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 27568 18.62 MB/sec warmup 89 sec latency 107.988 ms 2024-03-22T02:28:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:49 smithi120 ceph-mon[36803]: pgmap v818: 129 pgs: 129 active+clean; 265 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-03-22T02:28:50.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:49 smithi083 ceph-mon[36340]: pgmap v818: 129 pgs: 129 active+clean; 265 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-03-22T02:28:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:49 smithi028 ceph-mon[31916]: pgmap v818: 129 pgs: 129 active+clean; 265 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.2 MiB/s wr, 536 op/s 2024-03-22T02:28:50.675 INFO:tasks.workunit.client.0.smithi028.stdout: 10 28011 18.71 MB/sec warmup 90 sec latency 83.935 ms 2024-03-22T02:28:51.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 28330 18.75 MB/sec warmup 91 sec latency 60.205 ms 2024-03-22T02:28:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:51 smithi120 ceph-mon[36803]: pgmap v819: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:28:52.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:51 smithi083 ceph-mon[36340]: pgmap v819: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:28:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:51 smithi028 ceph-mon[31916]: pgmap v819: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:28:52.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 28593 18.73 MB/sec warmup 92 sec latency 62.822 ms 2024-03-22T02:28:53.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 28795 18.64 MB/sec warmup 93 sec latency 113.400 ms 2024-03-22T02:28:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:53 smithi120 ceph-mon[36803]: pgmap v820: 129 pgs: 129 active+clean; 260 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 500 op/s 2024-03-22T02:28:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:53 smithi083 ceph-mon[36340]: pgmap v820: 129 pgs: 129 active+clean; 260 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 500 op/s 2024-03-22T02:28:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:53 smithi028 ceph-mon[31916]: pgmap v820: 129 pgs: 129 active+clean; 260 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 500 op/s 2024-03-22T02:28:54.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 28984 18.54 MB/sec warmup 94 sec latency 120.563 ms 2024-03-22T02:28:55.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 29229 18.43 MB/sec warmup 95 sec latency 126.442 ms 2024-03-22T02:28:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:55 smithi120 ceph-mon[36803]: pgmap v821: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-03-22T02:28:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:55 smithi083 ceph-mon[36340]: pgmap v821: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-03-22T02:28:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:55 smithi028 ceph-mon[31916]: pgmap v821: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 571 op/s 2024-03-22T02:28:56.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 29519 18.39 MB/sec warmup 96 sec latency 107.076 ms 2024-03-22T02:28:57.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 29960 18.34 MB/sec warmup 97 sec latency 112.062 ms 2024-03-22T02:28:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:57 smithi120 ceph-mon[36803]: pgmap v822: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 517 op/s 2024-03-22T02:28:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:57 smithi083 ceph-mon[36340]: pgmap v822: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 517 op/s 2024-03-22T02:28:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:57 smithi028 ceph-mon[31916]: pgmap v822: 129 pgs: 129 active+clean; 259 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 517 op/s 2024-03-22T02:28:58.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 30296 18.33 MB/sec warmup 98 sec latency 184.238 ms 2024-03-22T02:28:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 -- 172.21.15.28:0/1622885839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 msgr2=0x7f4184101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 --2- 172.21.15.28:0/1622885839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f4184101c00 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f417c009ca0 tx=0x7f417c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 -- 172.21.15.28:0/1622885839 shutdown_connections 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 --2- 172.21.15.28:0/1622885839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f4184101c00 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 --2- 172.21.15.28:0/1622885839 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100de0 0x7f4184101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 --2- 172.21.15.28:0/1622885839 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100480 0x7f41841008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 -- 172.21.15.28:0/1622885839 >> 172.21.15.28:0/1622885839 conn(0x7f41840fc020 msgr2=0x7f41840fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 -- 172.21.15.28:0/1622885839 shutdown_connections 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.928+0000 7f418b0fb640 1 -- 172.21.15.28:0/1622885839 wait complete. 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 Processor -- start 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 -- start start 2024-03-22T02:28:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 0x7f418419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 0x7f418419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f41841a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f418410fa20 con 0x7f4184101780 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f418410f8a0 con 0x7f4184100de0 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f418b0fb640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f418410fba0 con 0x7f4184100480 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4183fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 0x7f418419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4189698640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f41841a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4188e97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 0x7f418419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4183fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 0x7f418419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:33366/0 (socket says 172.21.15.28:33366) 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4183fff640 1 -- 172.21.15.28:0/3714545187 learned_addr learned my addr 172.21.15.28:0/3714545187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4188e97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 0x7f418419a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53018/0 (socket says 172.21.15.28:53018) 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4189698640 1 -- 172.21.15.28:0/3714545187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 msgr2=0x7f418419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.929+0000 7f4189698640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 0x7f418419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4189698640 1 -- 172.21.15.28:0/3714545187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 msgr2=0x7f418419a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4189698640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 0x7f418419a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4189698640 1 -- 172.21.15.28:0/3714545187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41841a1c00 con 0x7f4184101780 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4189698640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f41841a1590 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f417c037040 tx=0x7f417c032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4181ffb640 1 -- 172.21.15.28:0/3714545187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c041020 con 0x7f4184101780 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41841a1e90 con 0x7f4184101780 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4181ffb640 1 -- 172.21.15.28:0/3714545187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f417c034cd0 con 0x7f4184101780 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.930+0000 7f4181ffb640 1 -- 172.21.15.28:0/3714545187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417c039040 con 0x7f4184101780 2024-03-22T02:28:58.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.931+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41841a22f0 con 0x7f4184101780 2024-03-22T02:28:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f418ddc0740 1 Processor -- start 2024-03-22T02:28:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f4181ffb640 1 -- 172.21.15.28:0/3714545187 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f417c044020 con 0x7f4184101780 2024-03-22T02:28:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f418ddc0740 1 -- start start 2024-03-22T02:28:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ed89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed9040 0x55e9a7edb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.934+0000 7f4183fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f418ddc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7e38040 con 0x55e9a7ed9040 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f418ddc0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7de09e0 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f418ddc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7e33ff0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:53024/0 (socket says 172.21.15.28:53024) 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 -- 172.21.15.28:0/2078689085 learned_addr learned my addr 172.21.15.28:0/2078689085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4188e97640 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ed89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4181ffb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 0x7f41740a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed9040 0x55e9a7edb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 -- 172.21.15.28:0/2078689085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed3ed0 msgr2=0x55e9a7ed89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ed89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4181ffb640 1 -- 172.21.15.28:0/3714545187 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f417c0f1600 con 0x7f4184101780 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 -- 172.21.15.28:0/2078689085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed9040 msgr2=0x55e9a7edb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed9040 0x55e9a7edb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 -- 172.21.15.28:0/2078689085 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a7b4c210 con 0x55e9a7ed34e0 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4189698640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 0x7f41740a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f4183fff640 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4170013720 tx=0x7f4170009930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e23dc6d4029daf7c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f416effd640 1 -- 172.21.15.28:0/2078689085 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417000f8c0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f416effd640 1 -- 172.21.15.28:0/2078689085 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4170010430 con 0x55e9a7ed34e0 2024-03-22T02:28:58.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.935+0000 7f416effd640 1 -- 172.21.15.28:0/2078689085 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417000e610 con 0x55e9a7ed34e0 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.936+0000 7f418ddc0740 1 -- 172.21.15.28:0/2078689085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 msgr2=0x55e9a7ed3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.936+0000 7f4189698640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 0x7f41740a5480 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f417c0b9e10 tx=0x7f417c003ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.936+0000 7f418ddc0740 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4170013720 tx=0x7f4170009930 comp rx=0 tx=0).stop 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 -- 172.21.15.28:0/2078689085 shutdown_connections 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed9040 0x55e9a7edb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ed89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 --2- 172.21.15.28:0/2078689085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed3900 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 -- 172.21.15.28:0/2078689085 >> 172.21.15.28:0/2078689085 conn(0x55e9a7ecea50 msgr2=0x55e9a7ecee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 -- 172.21.15.28:0/2078689085 shutdown_connections 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 -- 172.21.15.28:0/2078689085 wait complete. 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 -- start start 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.937+0000 7f418ddc0740 1 Processor -- start 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7edd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7eddae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7edbff0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7df3db0 con 0x55e9a7ed9040 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e9a7edc170 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f418ddc0740 10 client.0 _ll_register_callbacks cb 0x55e9a7b4ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7edd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7eddae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7edd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42504/0 (socket says 172.21.15.28:42504) 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4183fff640 1 -- 172.21.15.28:0/651789575 learned_addr learned my addr 172.21.15.28:0/651789575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4183fff640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 -- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 msgr2=0x55e9a7ee4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 -- 172.21.15.28:0/651789575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 msgr2=0x55e9a7edd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7edd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 -- 172.21.15.28:0/651789575 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a7c18620 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4188e97640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7eddae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55e9a7ed42d0 tx=0x7f4178012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed06d48402bcbf6c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.938+0000 7f4163fff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417800f890 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4170010430 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f417800fa30 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f417800fc30 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 10 client.29548 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e9a7ee7fb0 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4170010430 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f417801c420 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 10 client.29548 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 10 client.29548 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 20 client.29548 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7ee97b0 0x55e9a7eebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 4 client.29548 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.939+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9a7e45580 con 0x55e9a7ee97b0 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.940+0000 7f4183fff640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7ee97b0 0x55e9a7eebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.940+0000 7f4163fff640 10 client.29548 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.940+0000 7f4183fff640 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7ee97b0 0x55e9a7eebc20 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8d0cfd0c6b7d6c6 server_cookie=d87f113a9a8f58f8 in_seq=0 out_seq=0 2024-03-22T02:28:58.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.944+0000 7f4163fff640 1 -- 172.21.15.28:0/651789575 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9a7e45580 con 0x55e9a7ee97b0 2024-03-22T02:28:58.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.944+0000 7f4163fff640 10 client.29548 handle_command_reply: tid=0 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.944+0000 7f418ddc0740 1 client.29548 shutdown 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 msgr2=0x55e9a7eddae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7eddae0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x55e9a7ed42d0 tx=0x7f4178012b60 comp rx=0 tx=0).stop 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 shutdown_connections 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7ee97b0 0x55e9a7eebc20 crc :-1 s=READY pgs=372 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f4163fff640 0 client.29548 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7eddae0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 --2- 172.21.15.28:0/651789575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7edd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 >> 172.21.15.28:0/651789575 conn(0x55e9a7ecea50 msgr2=0x55e9a7ecdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 shutdown_connections 2024-03-22T02:28:58.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 1 -- 172.21.15.28:0/651789575 wait complete. 2024-03-22T02:28:58.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.945+0000 7f418ddc0740 20 client.29548 trim_cache size 0 max 16384 2024-03-22T02:28:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 Processor -- start 2024-03-22T02:28:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 -- start start 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed9040 0x55e9a7ed4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41841af3c0 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41841acc30 con 0x55e9a7ed34e0 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f418ddc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41841aaa80 con 0x55e9a7ed9040 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f4188e97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f4188e97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42518/0 (socket says 172.21.15.28:42518) 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f4188e97640 1 -- 172.21.15.28:0/320239827 learned_addr learned my addr 172.21.15.28:0/320239827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f4183fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.961+0000 7f4183fff640 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed9040 0x55e9a7ed4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 -- 172.21.15.28:0/320239827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed9040 msgr2=0x55e9a7ed4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed9040 0x55e9a7ed4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 -- 172.21.15.28:0/320239827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed34e0 msgr2=0x55e9a7ed4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 -- 172.21.15.28:0/320239827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9a7ed3120 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f4188e97640 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f4178009e00 tx=0x7f4178009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77252570732221a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f416e7fc640 1 -- 172.21.15.28:0/320239827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417801cb50 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f416e7fc640 1 -- 172.21.15.28:0/320239827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f417801ccf0 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.962+0000 7f416e7fc640 1 -- 172.21.15.28:0/320239827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417800fa80 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- 172.21.15.28:0/320239827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 msgr2=0x55e9a7ee4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f4178009e00 tx=0x7f4178009ea0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- 172.21.15.28:0/320239827 shutdown_connections 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed9040 0x55e9a7ed4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee4ae0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 --2- 172.21.15.28:0/320239827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ed4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- 172.21.15.28:0/320239827 >> 172.21.15.28:0/320239827 conn(0x55e9a7ecea50 msgr2=0x55e9a7edc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- 172.21.15.28:0/320239827 shutdown_connections 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- 172.21.15.28:0/320239827 wait complete. 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 -- start start 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.963+0000 7f418ddc0740 1 Processor -- start 2024-03-22T02:28:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ab0f0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ac560 con 0x55e9a7ed9040 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f41841ab7c0 con 0x55e9a7ed3ed0 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 10 client.0 _ll_register_callbacks cb 0x7f4178009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42524/0 (socket says 172.21.15.28:42524) 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 -- 172.21.15.28:0/1596734784 learned_addr learned my addr 172.21.15.28:0/1596734784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4188e97640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 -- 172.21.15.28:0/1596734784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 msgr2=0x55e9a7ee21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 -- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 msgr2=0x55e9a7ee31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 -- 172.21.15.28:0/1596734784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9a7ee37b0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4183fff640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4170005b90 tx=0x7f4170009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e72f422406093 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4163fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170010430 con 0x55e9a7ed34e0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f417801ccf0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.964+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f41700105d0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f41700107d0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 10 client.19758 fetch_fsmap learned FSMap version 42 2024-03-22T02:28:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55e9a7ee4b10 con 0x55e9a7ed34e0 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f417801ccf0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f41700354e0 con 0x55e9a7ed34e0 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 10 client.19758 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 10 client.19758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 20 client.19758 populate_metadata read hostname 'smithi028' 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7f4fda0 0x55e9a7f521c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 4 client.19758 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9a7e3f8d0 con 0x55e9a7f4fda0 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.965+0000 7f4188e97640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7f4fda0 0x55e9a7f521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.966+0000 7f4163fff640 10 client.19758 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:58.966+0000 7f4188e97640 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7f4fda0 0x55e9a7f521c0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=300d0d7591a9b90d server_cookie=6b46bfc5b69735df in_seq=0 out_seq=0 2024-03-22T02:28:59.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 30387 18.20 MB/sec warmup 99 sec latency 464.506 ms 2024-03-22T02:28:59.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.901+0000 7f4163fff640 1 -- 172.21.15.28:0/1596734784 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9a7e3f8d0 con 0x55e9a7f4fda0 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.901+0000 7f4163fff640 10 client.19758 handle_command_reply: tid=0 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.901+0000 7f418ddc0740 1 client.19758 shutdown 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 msgr2=0x55e9a7ee1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f4170005b90 tx=0x7f4170009930 comp rx=0 tx=0).stop 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 shutdown_connections 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e9a7f4fda0 0x55e9a7f521c0 crc :-1 s=READY pgs=373 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e9a7ed9040 0x55e9a7ee31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f4163fff640 0 client.19758 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:28:59.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e9a7ed3ed0 0x55e9a7ee21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 --2- 172.21.15.28:0/1596734784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e9a7ed34e0 0x55e9a7ee1c70 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 >> 172.21.15.28:0/1596734784 conn(0x55e9a7ecea50 msgr2=0x55e9a7ed1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 shutdown_connections 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.902+0000 7f418ddc0740 1 -- 172.21.15.28:0/1596734784 wait complete. 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.903+0000 7f418ddc0740 20 client.19758 trim_cache size 0 max 16384 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:28:59.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:28:59.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:28:59.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.903+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 msgr2=0x7f41740a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 0x7f41740a5480 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f417c0b9e10 tx=0x7f417c003ca0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 msgr2=0x7f41841a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f41841a1590 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f417c037040 tx=0x7f417c032ea0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 shutdown_connections 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41740a2fc0 0x7f41740a5480 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4184101780 0x7f41841a1590 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4184100de0 0x7f418419a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 --2- 172.21.15.28:0/3714545187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4184100480 0x7f418419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 >> 172.21.15.28:0/3714545187 conn(0x7f41840fc020 msgr2=0x7f41840fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 shutdown_connections 2024-03-22T02:28:59.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:28:59.904+0000 7f418b0fb640 1 -- 172.21.15.28:0/3714545187 wait complete. 2024-03-22T02:29:00.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:28:59 smithi120 ceph-mon[36803]: pgmap v823: 129 pgs: 129 active+clean; 269 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-03-22T02:29:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:28:59 smithi083 ceph-mon[36340]: pgmap v823: 129 pgs: 129 active+clean; 269 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-03-22T02:29:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:28:59 smithi028 ceph-mon[31916]: pgmap v823: 129 pgs: 129 active+clean; 269 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.8 MiB/s wr, 527 op/s 2024-03-22T02:29:00.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 30756 18.28 MB/sec warmup 100 sec latency 527.549 ms 2024-03-22T02:29:01.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 31235 18.39 MB/sec warmup 101 sec latency 58.868 ms 2024-03-22T02:29:02.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:01 smithi120 ceph-mon[36803]: pgmap v824: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-22T02:29:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:01 smithi083 ceph-mon[36340]: pgmap v824: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-22T02:29:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:01 smithi028 ceph-mon[31916]: pgmap v824: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.3 MiB/s wr, 406 op/s 2024-03-22T02:29:02.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 31598 18.40 MB/sec warmup 102 sec latency 103.245 ms 2024-03-22T02:29:03.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 31900 18.45 MB/sec warmup 103 sec latency 55.215 ms 2024-03-22T02:29:04.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:03 smithi120 ceph-mon[36803]: pgmap v825: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:29:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:03 smithi083 ceph-mon[36340]: pgmap v825: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:29:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:03 smithi028 ceph-mon[31916]: pgmap v825: 129 pgs: 129 active+clean; 276 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.3 MiB/s wr, 403 op/s 2024-03-22T02:29:04.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 32134 18.38 MB/sec warmup 104 sec latency 114.038 ms 2024-03-22T02:29:05.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 32363 18.33 MB/sec warmup 105 sec latency 94.154 ms 2024-03-22T02:29:06.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:05 smithi120 ceph-mon[36803]: pgmap v826: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.8 MiB/s wr, 582 op/s 2024-03-22T02:29:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:05 smithi083 ceph-mon[36340]: pgmap v826: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.8 MiB/s wr, 582 op/s 2024-03-22T02:29:06.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:05 smithi028 ceph-mon[31916]: pgmap v826: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.8 MiB/s wr, 582 op/s 2024-03-22T02:29:06.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 32607 18.27 MB/sec warmup 106 sec latency 86.560 ms 2024-03-22T02:29:07.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 32921 18.21 MB/sec warmup 107 sec latency 70.003 ms 2024-03-22T02:29:08.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:07 smithi120 ceph-mon[36803]: pgmap v827: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2024-03-22T02:29:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:07 smithi083 ceph-mon[36340]: pgmap v827: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2024-03-22T02:29:08.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:07 smithi028 ceph-mon[31916]: pgmap v827: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.0 MiB/s wr, 516 op/s 2024-03-22T02:29:08.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 33226 18.16 MB/sec warmup 108 sec latency 134.057 ms 2024-03-22T02:29:09.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 33431 18.07 MB/sec warmup 109 sec latency 276.104 ms 2024-03-22T02:29:10.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:09 smithi120 ceph-mon[36803]: pgmap v828: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 554 op/s 2024-03-22T02:29:10.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:09 smithi083 ceph-mon[36340]: pgmap v828: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 554 op/s 2024-03-22T02:29:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:09 smithi028 ceph-mon[31916]: pgmap v828: 129 pgs: 129 active+clean; 256 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 554 op/s 2024-03-22T02:29:10.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 33820 18.06 MB/sec warmup 110 sec latency 110.958 ms 2024-03-22T02:29:11.676 INFO:tasks.workunit.client.0.smithi028.stdout: 10 34253 18.15 MB/sec warmup 111 sec latency 78.851 ms 2024-03-22T02:29:12.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:11 smithi120 ceph-mon[36803]: pgmap v829: 129 pgs: 129 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 537 op/s 2024-03-22T02:29:12.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:11 smithi083 ceph-mon[36340]: pgmap v829: 129 pgs: 129 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 537 op/s 2024-03-22T02:29:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:11 smithi028 ceph-mon[31916]: pgmap v829: 129 pgs: 129 active+clean; 263 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.1 MiB/s wr, 537 op/s 2024-03-22T02:29:12.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 34771 18.26 MB/sec warmup 112 sec latency 51.330 ms 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:12.999+0000 7f575be81640 1 -- 172.21.15.28:0/1014487100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754107890 msgr2=0x7f5754109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:12.999+0000 7f575be81640 1 --2- 172.21.15.28:0/1014487100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754107890 0x7f5754109ce0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f574c009ca0 tx=0x7f574c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 -- 172.21.15.28:0/1014487100 shutdown_connections 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 --2- 172.21.15.28:0/1014487100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754107890 0x7f5754109ce0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 --2- 172.21.15.28:0/1014487100 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754104f60 0x7f5754107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 --2- 172.21.15.28:0/1014487100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754102630 0x7f5754104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 -- 172.21.15.28:0/1014487100 >> 172.21.15.28:0/1014487100 conn(0x7f57540fc020 msgr2=0x7f57540fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 -- 172.21.15.28:0/1014487100 shutdown_connections 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 -- 172.21.15.28:0/1014487100 wait complete. 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 Processor -- start 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.000+0000 7f575be81640 1 -- start start 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754104f60 0x7f5754198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 0x7f575419f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f575410d830 con 0x7f5754102630 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f575410d6b0 con 0x7f5754107890 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f575410d9b0 con 0x7f5754104f60 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56692/0 (socket says 172.21.15.28:56692) 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 -- 172.21.15.28:0/3225846164 learned_addr learned my addr 172.21.15.28:0/3225846164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575941c640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754104f60 0x7f5754198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575a41e640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 0x7f575419f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 -- 172.21.15.28:0/3225846164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754104f60 msgr2=0x7f5754198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754104f60 0x7f5754198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 -- 172.21.15.28:0/3225846164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 msgr2=0x7f575419f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 0x7f575419f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 -- 172.21.15.28:0/3225846164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575419fa10 con 0x7f5754102630 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575a41e640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 0x7f575419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f575000e9f0 tx=0x7f575000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f574affd640 1 -- 172.21.15.28:0/3225846164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575000cd80 con 0x7f5754102630 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f574affd640 1 -- 172.21.15.28:0/3225846164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5750010910 con 0x7f5754102630 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f574affd640 1 -- 172.21.15.28:0/3225846164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750018990 con 0x7f5754102630 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f575419fd00 con 0x7f5754102630 2024-03-22T02:29:13.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.001+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57541a01e0 con 0x7f5754102630 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f574affd640 1 -- 172.21.15.28:0/3225846164 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5750010ac0 con 0x7f5754102630 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 Processor -- start 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 -- start start 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae80404e0 0x564ae8040900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae80459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ae7fa5040 con 0x564ae8046040 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ae7f4d9e0 con 0x564ae80404e0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575ea63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ae7fa0ff0 con 0x564ae8040ed0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575941c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f574affd640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 0x7f57400a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae80459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575941c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56694/0 (socket says 172.21.15.28:56694) 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575941c640 1 -- 172.21.15.28:0/2449601305 learned_addr learned my addr 172.21.15.28:0/2449601305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.004+0000 7f575a41e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae80404e0 0x564ae8040900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f575a41e640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 0x7f57400a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.014 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f574affd640 1 -- 172.21.15.28:0/3225846164 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f57500cfb10 con 0x7f5754102630 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 -- 172.21.15.28:0/2449601305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae80404e0 msgr2=0x564ae8040900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae80404e0 0x564ae8040900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 -- 172.21.15.28:0/2449601305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 msgr2=0x564ae80484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 -- 172.21.15.28:0/2449601305 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ae7cb9210 con 0x564ae8040ed0 2024-03-22T02:29:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f575941c640 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae80459d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5750098800 tx=0x7f57500985d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1830797b4c2daaaa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f575a41e640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 0x7f57400a5500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f574c02c630 tx=0x7f574c02c0f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f573ffff640 1 -- 172.21.15.28:0/2449601305 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575009be30 con 0x564ae8040ed0 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f573ffff640 1 -- 172.21.15.28:0/2449601305 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575009c530 con 0x564ae8040ed0 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.005+0000 7f573ffff640 1 -- 172.21.15.28:0/2449601305 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500bae00 con 0x564ae8040ed0 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- 172.21.15.28:0/2449601305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 msgr2=0x564ae80459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae80459d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5750098800 tx=0x7f57500985d0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- 172.21.15.28:0/2449601305 shutdown_connections 2024-03-22T02:29:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae80484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae80459d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 --2- 172.21.15.28:0/2449601305 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae80404e0 0x564ae8040900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- 172.21.15.28:0/2449601305 >> 172.21.15.28:0/2449601305 conn(0x564ae803ba50 msgr2=0x564ae803be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- 172.21.15.28:0/2449601305 shutdown_connections 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- 172.21.15.28:0/2449601305 wait complete. 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 -- start start 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.006+0000 7f575ea63740 1 Processor -- start 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae804a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae80519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564ae8048f30 con 0x564ae80404e0 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564ae7f60db0 con 0x564ae8046040 2024-03-22T02:29:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564ae80490b0 con 0x564ae8040ed0 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575ea63740 10 client.0 _ll_register_callbacks cb 0x564ae7cb8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56706/0 (socket says 172.21.15.28:56706) 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 -- 172.21.15.28:0/3871243470 learned_addr learned my addr 172.21.15.28:0/3871243470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575a41e640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae80519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f575941c640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae804a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 -- 172.21.15.28:0/3871243470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 msgr2=0x564ae804a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae804a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 -- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 msgr2=0x564ae80519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae80519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 -- 172.21.15.28:0/3871243470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ae7d85620 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f575009bb30 tx=0x7f57500d0fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db86670a3b979a34 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.007+0000 7f573d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575009be30 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f57500ba730 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f575009c530 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f575009ff80 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 10 client.19770 fetch_fsmap learned FSMap version 42 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564ae8054d20 con 0x564ae80404e0 2024-03-22T02:29:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575009c530 con 0x564ae80404e0 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f5750018cc0 con 0x564ae80404e0 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 10 client.19770 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 10 client.19770 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 20 client.19770 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae8056070 0x564ae8058460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 4 client.19770 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ae7fb2580 con 0x564ae8056070 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.008+0000 7f575941c640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae8056070 0x564ae8058460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.009+0000 7f573d7fa640 10 client.19770 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.009+0000 7f575941c640 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae8056070 0x564ae8058460 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96227e230cae843 server_cookie=a0d9f05c8501826c in_seq=0 out_seq=0 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f573d7fa640 1 -- 172.21.15.28:0/3871243470 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564ae7fb2580 con 0x564ae8056070 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f573d7fa640 10 client.19770 handle_command_reply: tid=0 2024-03-22T02:29:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 client.19770 shutdown 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 msgr2=0x564ae804a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f575009bb30 tx=0x7f57500d0fd0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 shutdown_connections 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae8056070 0x564ae8058460 crc :-1 s=READY pgs=374 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae80519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae804a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 --2- 172.21.15.28:0/3871243470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae804a430 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f573d7fa640 0 client.19770 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 >> 172.21.15.28:0/3871243470 conn(0x564ae803ba50 msgr2=0x564ae803ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 shutdown_connections 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 1 -- 172.21.15.28:0/3871243470 wait complete. 2024-03-22T02:29:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.010+0000 7f575ea63740 20 client.19770 trim_cache size 0 max 16384 2024-03-22T02:29:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 Processor -- start 2024-03-22T02:29:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 -- start start 2024-03-22T02:29:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae8051970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae803a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57541ad270 con 0x564ae80404e0 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f57541aaae0 con 0x564ae8046040 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f575ea63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57541a8930 con 0x564ae8040ed0 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.025+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 -- 172.21.15.28:0/2150404630 learned_addr learned my addr 172.21.15.28:0/2150404630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f575941c640 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae8051970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f575a41e640 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae803a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 -- 172.21.15.28:0/2150404630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 msgr2=0x564ae8051970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae8051970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 -- 172.21.15.28:0/2150404630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 msgr2=0x564ae803a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae803a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 -- 172.21.15.28:0/2150404630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ae80587d0 con 0x564ae80404e0 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f5759c1d640 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5750098800 tx=0x7f5750018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8131df285d9e49a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f573f7fe640 1 -- 172.21.15.28:0/2150404630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500984a0 con 0x564ae80404e0 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f573f7fe640 1 -- 172.21.15.28:0/2150404630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575009c530 con 0x564ae80404e0 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.026+0000 7f573f7fe640 1 -- 172.21.15.28:0/2150404630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500ba730 con 0x564ae80404e0 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- 172.21.15.28:0/2150404630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 msgr2=0x564ae803a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f5750098800 tx=0x7f5750018420 comp rx=0 tx=0).stop 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- 172.21.15.28:0/2150404630 shutdown_connections 2024-03-22T02:29:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8046040 0x564ae803a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae8040ed0 0x564ae8051970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 --2- 172.21.15.28:0/2150404630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae80404e0 0x564ae803a3c0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- 172.21.15.28:0/2150404630 >> 172.21.15.28:0/2150404630 conn(0x564ae803ba50 msgr2=0x564ae803f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- 172.21.15.28:0/2150404630 shutdown_connections 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- 172.21.15.28:0/2150404630 wait complete. 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.027+0000 7f575ea63740 1 -- start start 2024-03-22T02:29:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 Processor -- start 2024-03-22T02:29:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 0x564ae804fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57541a8fa0 con 0x564ae8046040 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f57541aa410 con 0x564ae8040ed0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f57541a9670 con 0x564ae80404e0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575ea63740 10 client.0 _ll_register_callbacks cb 0x7f57440097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575a41e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575941c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 0x564ae804fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f5759c1d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58976/0 (socket says 172.21.15.28:58976) 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575a41e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56738/0 (socket says 172.21.15.28:56738) 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f5759c1d640 1 -- 172.21.15.28:0/3698754345 learned_addr learned my addr 172.21.15.28:0/3698754345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.028+0000 7f575941c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 0x564ae804fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:43368/0 (socket says 172.21.15.28:43368) 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 -- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 msgr2=0x564ae804fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 0x564ae804fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 -- 172.21.15.28:0/3698754345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 msgr2=0x564ae8050ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 -- 172.21.15.28:0/3698754345 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ae80510d0 con 0x564ae80404e0 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f575a41e640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f5759c1d640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f575000e9c0 tx=0x7f57500ba610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5038f6a456dcba server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f573d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500ba930 con 0x564ae80404e0 2024-03-22T02:29:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f5750018b30 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f575009c530 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f57500984a0 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f575ea63740 10 client.29572 fetch_fsmap learned FSMap version 42 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.029+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564ae80510d0 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f575009c530 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f57500ab020 con 0x564ae80404e0 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 10 client.29572 fetch_fsmap finished waiting for FSMap version 42 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 10 client.29572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 20 client.29572 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae80c5560 0x564ae80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 4 client.29572 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:29:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564ae7fac8d0 con 0x564ae80c5560 2024-03-22T02:29:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f575941c640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae80c5560 0x564ae80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.030+0000 7f573d7fa640 10 client.29572 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:13.038 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:13.031+0000 7f575941c640 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae80c5560 0x564ae80c7980 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0656cdc640a15f6 server_cookie=84c35cdcd90000f6 in_seq=0 out_seq=0 2024-03-22T02:29:13.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 34964 18.22 MB/sec warmup 113 sec latency 534.442 ms 2024-03-22T02:29:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f573d7fa640 1 -- 172.21.15.28:0/3698754345 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564ae7fac8d0 con 0x564ae80c5560 2024-03-22T02:29:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f573d7fa640 10 client.29572 handle_command_reply: tid=0 2024-03-22T02:29:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 client.29572 shutdown 2024-03-22T02:29:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 msgr2=0x564ae804f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:14.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f575000e9c0 tx=0x7f57500ba610 comp rx=0 tx=0).stop 2024-03-22T02:29:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 shutdown_connections 2024-03-22T02:29:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564ae80c5560 0x564ae80c7980 crc :-1 s=READY pgs=375 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564ae8046040 0x564ae8050ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564ae8040ed0 0x564ae804fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 --2- 172.21.15.28:0/3698754345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564ae80404e0 0x564ae804f5e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 >> 172.21.15.28:0/3698754345 conn(0x564ae803ba50 msgr2=0x564ae8041850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f573d7fa640 0 client.29572 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 shutdown_connections 2024-03-22T02:29:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.112+0000 7f575ea63740 1 -- 172.21.15.28:0/3698754345 wait complete. 2024-03-22T02:29:14.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575ea63740 20 client.29572 trim_cache size 0 max 16384 2024-03-22T02:29:14.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:29:14.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:29:14.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:29:14.118 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:29:14.120 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 msgr2=0x7f57400a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 0x7f57400a5500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f574c02c630 tx=0x7f574c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 msgr2=0x7f5754197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f575000e9f0 tx=0x7f575000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 shutdown_connections 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f57400a3040 0x7f57400a5500 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5754107890 0x7f575419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5754104f60 0x7f5754198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 --2- 172.21.15.28:0/3225846164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5754102630 0x7f5754197e40 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 >> 172.21.15.28:0/3225846164 conn(0x7f57540fc020 msgr2=0x7f5754103010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 shutdown_connections 2024-03-22T02:29:14.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:14.113+0000 7f575be81640 1 -- 172.21.15.28:0/3225846164 wait complete. 2024-03-22T02:29:14.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:13 smithi120 ceph-mon[36803]: pgmap v830: 129 pgs: 129 active+clean; 262 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 473 op/s 2024-03-22T02:29:14.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:13 smithi083 ceph-mon[36340]: pgmap v830: 129 pgs: 129 active+clean; 262 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 473 op/s 2024-03-22T02:29:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:13 smithi028 ceph-mon[31916]: pgmap v830: 129 pgs: 129 active+clean; 262 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 473 op/s 2024-03-22T02:29:14.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 35131 18.15 MB/sec warmup 114 sec latency 751.787 ms 2024-03-22T02:29:15.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 35476 18.20 MB/sec warmup 115 sec latency 66.366 ms 2024-03-22T02:29:16.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:15 smithi120 ceph-mon[36803]: pgmap v831: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:29:16.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:15 smithi083 ceph-mon[36340]: pgmap v831: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:29:16.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:15 smithi028 ceph-mon[31916]: pgmap v831: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:29:16.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 35733 18.17 MB/sec warmup 116 sec latency 80.657 ms 2024-03-22T02:29:17.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 35956 18.11 MB/sec warmup 117 sec latency 133.862 ms 2024-03-22T02:29:18.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:17 smithi120 ceph-mon[36803]: pgmap v832: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-03-22T02:29:18.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:17 smithi083 ceph-mon[36340]: pgmap v832: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-03-22T02:29:18.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:17 smithi028 ceph-mon[31916]: pgmap v832: 129 pgs: 129 active+clean; 253 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 465 op/s 2024-03-22T02:29:18.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 36254 18.05 MB/sec warmup 118 sec latency 88.791 ms 2024-03-22T02:29:19.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 36427 17.94 MB/sec warmup 119 sec latency 305.692 ms 2024-03-22T02:29:20.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:19 smithi120 ceph-mon[36803]: pgmap v833: 129 pgs: 129 active+clean; 255 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 610 op/s 2024-03-22T02:29:20.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:19 smithi083 ceph-mon[36340]: pgmap v833: 129 pgs: 129 active+clean; 255 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 610 op/s 2024-03-22T02:29:20.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:19 smithi028 ceph-mon[31916]: pgmap v833: 129 pgs: 129 active+clean; 255 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 610 op/s 2024-03-22T02:29:21.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 37101 14.13 MB/sec execute 1 sec latency 144.783 ms 2024-03-22T02:29:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:21 smithi120 ceph-mon[36803]: pgmap v834: 129 pgs: 129 active+clean; 257 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 567 op/s 2024-03-22T02:29:22.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:21 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:21 smithi083 ceph-mon[36340]: pgmap v834: 129 pgs: 129 active+clean; 257 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 567 op/s 2024-03-22T02:29:22.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:21 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:22.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:21 smithi028 ceph-mon[31916]: pgmap v834: 129 pgs: 129 active+clean; 257 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 567 op/s 2024-03-22T02:29:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:21 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:22.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 37446 16.07 MB/sec execute 2 sec latency 192.168 ms 2024-03-22T02:29:23.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 37827 18.35 MB/sec execute 3 sec latency 77.200 ms 2024-03-22T02:29:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:23 smithi120 ceph-mon[36803]: pgmap v835: 129 pgs: 129 active+clean; 258 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-03-22T02:29:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:29:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:29:24.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:23 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:29:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:23 smithi083 ceph-mon[36340]: pgmap v835: 129 pgs: 129 active+clean; 258 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-03-22T02:29:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:29:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:29:24.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:23 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:29:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:23 smithi028 ceph-mon[31916]: pgmap v835: 129 pgs: 129 active+clean; 258 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 554 op/s 2024-03-22T02:29:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:29:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:29:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:29:24.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:23 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:29:24.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 38147 18.95 MB/sec execute 4 sec latency 196.142 ms 2024-03-22T02:29:25.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 38484 19.14 MB/sec execute 5 sec latency 121.884 ms 2024-03-22T02:29:26.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:25 smithi120 ceph-mon[36803]: pgmap v836: 129 pgs: 129 active+clean; 259 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 585 op/s 2024-03-22T02:29:26.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:25 smithi083 ceph-mon[36340]: pgmap v836: 129 pgs: 129 active+clean; 259 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 585 op/s 2024-03-22T02:29:26.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:25 smithi028 ceph-mon[31916]: pgmap v836: 129 pgs: 129 active+clean; 259 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 585 op/s 2024-03-22T02:29:26.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 38837 19.46 MB/sec execute 6 sec latency 76.945 ms 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 -- 172.21.15.28:0/1986788779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 msgr2=0x7f2d84103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 --2- 172.21.15.28:0/1986788779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84103260 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009900 tx=0x7f2d7802bc30 comp rx=0 tx=0).stop 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 -- 172.21.15.28:0/1986788779 shutdown_connections 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 --2- 172.21.15.28:0/1986788779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d841037a0 0x7f2d8410a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 --2- 172.21.15.28:0/1986788779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84103260 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 --2- 172.21.15.28:0/1986788779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d84101bf0 0x7f2d84102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 -- 172.21.15.28:0/1986788779 >> 172.21.15.28:0/1986788779 conn(0x7f2d840fd160 msgr2=0x7f2d840ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:27.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 -- 172.21.15.28:0/1986788779 shutdown_connections 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.206+0000 7f2d889e9640 1 -- 172.21.15.28:0/1986788779 wait complete. 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 Processor -- start 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 -- start start 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d84101bf0 0x7f2d84197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d841037a0 0x7f2d8419f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d870 con 0x7f2d84102de0 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d6f0 con 0x7f2d84101bf0 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d889e9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8410d9f0 con 0x7f2d841037a0 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58570/0 (socket says 172.21.15.28:58570) 2024-03-22T02:29:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d81d9b640 1 -- 172.21.15.28:0/3544605340 learned_addr learned my addr 172.21.15.28:0/3544605340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d8259c640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d84101bf0 0x7f2d84197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.207+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d841037a0 0x7f2d8419f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 -- 172.21.15.28:0/3544605340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d841037a0 msgr2=0x7f2d8419f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d841037a0 0x7f2d8419f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 -- 172.21.15.28:0/3544605340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d84101bf0 msgr2=0x7f2d84197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d84101bf0 0x7f2d84197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 -- 172.21.15.28:0/3544605340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8419fa50 con 0x7f2d84102de0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009230 tx=0x7f2d7802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d737fe640 1 -- 172.21.15.28:0/3544605340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78041070 con 0x7f2d84102de0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8419fce0 con 0x7f2d84102de0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d737fe640 1 -- 172.21.15.28:0/3544605340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d78004450 con 0x7f2d84102de0 2024-03-22T02:29:27.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d737fe640 1 -- 172.21.15.28:0/3544605340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78003e20 con 0x7f2d84102de0 2024-03-22T02:29:27.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.208+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d841a01d0 con 0x7f2d84102de0 2024-03-22T02:29:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.211+0000 7f2d8b5cf740 1 Processor -- start 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.211+0000 7f2d737fe640 1 -- 172.21.15.28:0/3544605340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2d78044070 con 0x7f2d84102de0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.211+0000 7f2d8b5cf740 1 -- start start 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fdf6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fdfe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b2fd5b040 con 0x564b2fdf64e0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b2fd039e0 con 0x564b2fdf6ed0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2fd56ff0 con 0x564b2fdfc040 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fdf6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fdfe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:53958/0 (socket says 172.21.15.28:53958) 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8259c640 1 -- 172.21.15.28:0/1322967199 learned_addr learned my addr 172.21.15.28:0/1322967199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fdfe4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:36950/0 (socket says 172.21.15.28:36950) 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d737fe640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 0x7f2d6c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 -- 172.21.15.28:0/1322967199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 msgr2=0x564b2fdfe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fdfe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 -- 172.21.15.28:0/1322967199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 msgr2=0x564b2fdfb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 -- 172.21.15.28:0/1322967199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b2fa6f210 con 0x564b2fdf64e0 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d8259c640 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d737fe640 1 -- 172.21.15.28:0/3544605340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f2d780f1360 con 0x7f2d84102de0 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 0x7f2d6c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.212+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fdf6900 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009ca0 tx=0x7f2d7c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3691edbd72d76c8d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d70ff9640 1 -- 172.21.15.28:0/1322967199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c017050 con 0x564b2fdf64e0 2024-03-22T02:29:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 0x7f2d6c0a5480 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c023000 tx=0x7f2d7c024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d70ff9640 1 -- 172.21.15.28:0/1322967199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c003c40 con 0x564b2fdf64e0 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d70ff9640 1 -- 172.21.15.28:0/1322967199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c00f3c0 con 0x564b2fdf64e0 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/1322967199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 msgr2=0x564b2fdf6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.213+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fdf6900 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009ca0 tx=0x7f2d7c011810 comp rx=0 tx=0).stop 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/1322967199 shutdown_connections 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fdfe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fdfb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/1322967199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fdf6900 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/1322967199 >> 172.21.15.28:0/1322967199 conn(0x564b2fdf1a50 msgr2=0x564b2fdf1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/1322967199 shutdown_connections 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/1322967199 wait complete. 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- start start 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 Processor -- start 2024-03-22T02:29:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 0x564b2fe04980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf6ed0 0x564b2fe04ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564b2fdfeff0 con 0x564b2fdfc040 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564b2fd16db0 con 0x564b2fdf64e0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564b2fdff170 con 0x564b2fdf6ed0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d8b5cf740 10 client.0 _ll_register_callbacks cb 0x564b2fa6ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.214+0000 7f2d82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 0x564b2fe04980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf6ed0 0x564b2fe04ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58598/0 (socket says 172.21.15.28:58598) 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d82d9d640 1 -- 172.21.15.28:0/686713327 learned_addr learned my addr 172.21.15.28:0/686713327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 -- 172.21.15.28:0/686713327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf6ed0 msgr2=0x564b2fe04ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf6ed0 0x564b2fe04ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 -- 172.21.15.28:0/686713327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 msgr2=0x564b2fe0be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 -- 172.21.15.28:0/686713327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b2fb3b620 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8259c640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 0x564b2fe04980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x564b2fdf72d0 tx=0x7f2d7400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bec5203d047d5981 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d74004b20 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d7c003c40 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f2d74004cc0 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2d74025050 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8b5cf740 10 client.29594 fetch_fsmap learned FSMap version 43 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.215+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564b2fe0f2c0 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c003c40 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 43) ==== 8354+0+0 (secure 0 0 0) 0x7f2d74019070 con 0x564b2fdf64e0 2024-03-22T02:29:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 10 client.29594 fetch_fsmap finished waiting for FSMap version 43 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 10 client.29594 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 20 client.29594 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe10950 0x564b2fe12df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 4 client.29594 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b2fd68580 con 0x564b2fe10950 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.216+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe10950 0x564b2fe12df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.217+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe10950 0x564b2fe12df0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e0914e594538603 server_cookie=6034a4343f3fbdf5 in_seq=0 out_seq=0 2024-03-22T02:29:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.217+0000 7f2d69ffb640 10 client.29594 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.219+0000 7f2d69ffb640 1 -- 172.21.15.28:0/686713327 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b2fd68580 con 0x564b2fe10950 2024-03-22T02:29:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.219+0000 7f2d69ffb640 10 client.29594 handle_command_reply: tid=0 2024-03-22T02:29:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.219+0000 7f2d8b5cf740 1 client.29594 shutdown 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 msgr2=0x564b2fe04980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 0x564b2fe04980 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x564b2fdf72d0 tx=0x7f2d7400f790 comp rx=0 tx=0).stop 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 shutdown_connections 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe10950 0x564b2fe12df0 crc :-1 s=READY pgs=376 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fe0be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d69ffb640 0 client.29594 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf6ed0 0x564b2fe04ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/686713327 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf64e0 0x564b2fe04980 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 >> 172.21.15.28:0/686713327 conn(0x564b2fdf1a50 msgr2=0x564b2fdf0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 shutdown_connections 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/686713327 wait complete. 2024-03-22T02:29:27.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.220+0000 7f2d8b5cf740 20 client.29594 trim_cache size 0 max 16384 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 Processor -- start 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 -- start start 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 0x564b2fdf7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe0be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841ab040 con 0x564b2fdfc040 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841a88b0 con 0x564b2fdf6ed0 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841a6700 con 0x564b2fdf64e0 2024-03-22T02:29:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d82d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58610/0 (socket says 172.21.15.28:58610) 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d82d9d640 1 -- 172.21.15.28:0/316549812 learned_addr learned my addr 172.21.15.28:0/316549812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d8259c640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 0x564b2fdf7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.235+0000 7f2d81d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe0be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 -- 172.21.15.28:0/316549812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 msgr2=0x564b2fdf7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 0x564b2fdf7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 -- 172.21.15.28:0/316549812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 msgr2=0x564b2fe0be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe0be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 -- 172.21.15.28:0/316549812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b2fdf27c0 con 0x564b2fdfc040 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d8259c640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 0x564b2fdf7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe0be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0097b0 tx=0x7f2d7c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ac441d12a86d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d6bfff640 1 -- 172.21.15.28:0/316549812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c004000 con 0x564b2fdfc040 2024-03-22T02:29:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d6bfff640 1 -- 172.21.15.28:0/316549812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c01c750 con 0x564b2fdfc040 2024-03-22T02:29:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.236+0000 7f2d6bfff640 1 -- 172.21.15.28:0/316549812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c005350 con 0x564b2fdfc040 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/316549812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 msgr2=0x564b2fdf7c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0097b0 tx=0x7f2d7c009c20 comp rx=0 tx=0).stop 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/316549812 shutdown_connections 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdfc040 0x564b2fdf7c70 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe0be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/316549812 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdf64e0 0x564b2fdf7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/316549812 >> 172.21.15.28:0/316549812 conn(0x564b2fdf1a50 msgr2=0x564b2fdf5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/316549812 shutdown_connections 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/316549812 wait complete. 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- start start 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 Processor -- start 2024-03-22T02:29:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe09d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fe0ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841a6d70 con 0x564b2fdf64e0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841a81e0 con 0x564b2fdf6ed0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.237+0000 7f2d8b5cf740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2d841a7440 con 0x564b2fdfc040 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8b5cf740 10 client.0 _ll_register_callbacks cb 0x7f2d74009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58618/0 (socket says 172.21.15.28:58618) 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 -- 172.21.15.28:0/2428769602 learned_addr learned my addr 172.21.15.28:0/2428769602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe09d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d82d9d640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fe0ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 -- 172.21.15.28:0/2428769602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 msgr2=0x564b2fe0ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fe0ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 -- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 msgr2=0x564b2fe09d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe09d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 -- 172.21.15.28:0/2428769602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b2fe0b310 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8259c640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f2d74004900 tx=0x7f2d740055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3cd42544249d3ac server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7402d070 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f2d74005920 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d7c01c750 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f2d74039050 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8b5cf740 10 client.19794 fetch_fsmap learned FSMap version 43 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.238+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x564b2fe76f40 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c01c750 con 0x564b2fdf64e0 2024-03-22T02:29:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8354+0+0 (secure 0 0 0) 0x7f2d74019050 con 0x564b2fdf64e0 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 10 client.19794 fetch_fsmap finished waiting for FSMap version 43 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 10 client.19794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 20 client.19794 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe779d0 0x564b2fe79df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 4 client.19794 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b2fd628d0 con 0x564b2fe779d0 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe779d0 0x564b2fe79df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d81d9b640 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe779d0 0x564b2fe79df0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3086362c3484e1ca server_cookie=97ce2e79636402e1 in_seq=0 out_seq=0 2024-03-22T02:29:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:27.239+0000 7f2d69ffb640 10 client.19794 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:27.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 39056 18.97 MB/sec execute 7 sec latency 324.861 ms 2024-03-22T02:29:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d69ffb640 1 -- 172.21.15.28:0/2428769602 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b2fd628d0 con 0x564b2fe779d0 2024-03-22T02:29:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d69ffb640 10 client.19794 handle_command_reply: tid=0 2024-03-22T02:29:28.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 client.19794 shutdown 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 msgr2=0x564b2fe097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f2d74004900 tx=0x7f2d740055a0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 shutdown_connections 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564b2fe779d0 0x564b2fe79df0 crc :-1 s=READY pgs=377 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564b2fdfc040 0x564b2fe0ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564b2fdf6ed0 0x564b2fe09d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d69ffb640 0 client.19794 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 --2- 172.21.15.28:0/2428769602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564b2fdf64e0 0x564b2fe097d0 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 >> 172.21.15.28:0/2428769602 conn(0x564b2fdf1a50 msgr2=0x564b2fdf03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 shutdown_connections 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 1 -- 172.21.15.28:0/2428769602 wait complete. 2024-03-22T02:29:28.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.126+0000 7f2d8b5cf740 20 client.19794 trim_cache size 0 max 16384 2024-03-22T02:29:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:29:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:29:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:29:28.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 msgr2=0x7f2d6c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 0x7f2d6c0a5480 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c023000 tx=0x7f2d7c024000 comp rx=0 tx=0).stop 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 msgr2=0x7f2d84198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009230 tx=0x7f2d7802c650 comp rx=0 tx=0).stop 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 shutdown_connections 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2d6c0a2fc0 0x7f2d6c0a5480 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d841037a0 0x7f2d8419f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84102de0 0x7f2d84198370 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 --2- 172.21.15.28:0/3544605340 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2d84101bf0 0x7f2d84197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 >> 172.21.15.28:0/3544605340 conn(0x7f2d840fd160 msgr2=0x7f2d84106080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.127+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 shutdown_connections 2024-03-22T02:29:28.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:28.128+0000 7f2d889e9640 1 -- 172.21.15.28:0/3544605340 wait complete. 2024-03-22T02:29:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:27 smithi120 ceph-mon[36803]: pgmap v837: 129 pgs: 129 active+clean; 262 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 451 op/s 2024-03-22T02:29:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:27 smithi083 ceph-mon[36340]: pgmap v837: 129 pgs: 129 active+clean; 262 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 451 op/s 2024-03-22T02:29:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:27 smithi028 ceph-mon[31916]: pgmap v837: 129 pgs: 129 active+clean; 262 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.2 MiB/s wr, 451 op/s 2024-03-22T02:29:28.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 39224 17.80 MB/sec execute 8 sec latency 562.197 ms 2024-03-22T02:29:29.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 39397 16.99 MB/sec execute 9 sec latency 292.430 ms 2024-03-22T02:29:30.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:29 smithi120 ceph-mon[36803]: pgmap v838: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:29:30.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:29 smithi083 ceph-mon[36340]: pgmap v838: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:29:30.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:29 smithi028 ceph-mon[31916]: pgmap v838: 129 pgs: 129 active+clean; 257 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:29:30.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 39635 16.12 MB/sec execute 10 sec latency 180.461 ms 2024-03-22T02:29:31.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 39908 15.79 MB/sec execute 11 sec latency 75.354 ms 2024-03-22T02:29:32.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:31 smithi120 ceph-mon[36803]: pgmap v839: 129 pgs: 129 active+clean; 258 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 463 op/s 2024-03-22T02:29:32.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:31 smithi083 ceph-mon[36340]: pgmap v839: 129 pgs: 129 active+clean; 258 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 463 op/s 2024-03-22T02:29:32.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:31 smithi028 ceph-mon[31916]: pgmap v839: 129 pgs: 129 active+clean; 258 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 4.0 MiB/s wr, 463 op/s 2024-03-22T02:29:32.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 40215 16.01 MB/sec execute 12 sec latency 92.872 ms 2024-03-22T02:29:33.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 40611 15.77 MB/sec execute 13 sec latency 83.735 ms 2024-03-22T02:29:34.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:33 smithi120 ceph-mon[36803]: pgmap v840: 129 pgs: 129 active+clean; 253 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 514 op/s 2024-03-22T02:29:34.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:33 smithi083 ceph-mon[36340]: pgmap v840: 129 pgs: 129 active+clean; 253 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 514 op/s 2024-03-22T02:29:34.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:33 smithi028 ceph-mon[31916]: pgmap v840: 129 pgs: 129 active+clean; 253 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.9 MiB/s wr, 514 op/s 2024-03-22T02:29:34.677 INFO:tasks.workunit.client.0.smithi028.stdout: 10 40849 15.55 MB/sec execute 14 sec latency 259.985 ms 2024-03-22T02:29:35.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 41207 16.11 MB/sec execute 15 sec latency 149.692 ms 2024-03-22T02:29:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:35 smithi120 ceph-mon[36803]: pgmap v841: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 566 op/s 2024-03-22T02:29:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:35 smithi083 ceph-mon[36340]: pgmap v841: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 566 op/s 2024-03-22T02:29:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:35 smithi028 ceph-mon[31916]: pgmap v841: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 566 op/s 2024-03-22T02:29:36.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 41630 16.60 MB/sec execute 16 sec latency 126.871 ms 2024-03-22T02:29:37.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 41932 16.58 MB/sec execute 17 sec latency 129.653 ms 2024-03-22T02:29:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:37 smithi120 ceph-mon[36803]: pgmap v842: 129 pgs: 129 active+clean; 261 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.2 MiB/s wr, 549 op/s 2024-03-22T02:29:38.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:37 smithi083 ceph-mon[36340]: pgmap v842: 129 pgs: 129 active+clean; 261 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.2 MiB/s wr, 549 op/s 2024-03-22T02:29:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:37 smithi028 ceph-mon[31916]: pgmap v842: 129 pgs: 129 active+clean; 261 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.2 MiB/s wr, 549 op/s 2024-03-22T02:29:38.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 42262 16.83 MB/sec execute 18 sec latency 71.686 ms 2024-03-22T02:29:39.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 42528 16.88 MB/sec execute 19 sec latency 185.956 ms 2024-03-22T02:29:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:39 smithi120 ceph-mon[36803]: pgmap v843: 129 pgs: 129 active+clean; 262 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 5.2 MiB/s wr, 561 op/s 2024-03-22T02:29:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:39 smithi083 ceph-mon[36340]: pgmap v843: 129 pgs: 129 active+clean; 262 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 5.2 MiB/s wr, 561 op/s 2024-03-22T02:29:40.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:39 smithi028 ceph-mon[31916]: pgmap v843: 129 pgs: 129 active+clean; 262 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 5.2 MiB/s wr, 561 op/s 2024-03-22T02:29:40.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 42773 16.66 MB/sec execute 20 sec latency 194.185 ms 2024-03-22T02:29:41.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 -- 172.21.15.28:0/3751712855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 msgr2=0x7f4db0107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 --2- 172.21.15.28:0/3751712855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db0107350 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f4da0009900 tx=0x7f4da002bc30 comp rx=0 tx=0).stop 2024-03-22T02:29:41.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 -- 172.21.15.28:0/3751712855 shutdown_connections 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 --2- 172.21.15.28:0/3751712855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 --2- 172.21.15.28:0/3751712855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db0107350 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 --2- 172.21.15.28:0/3751712855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 -- 172.21.15.28:0/3751712855 >> 172.21.15.28:0/3751712855 conn(0x7f4db00fc020 msgr2=0x7f4db00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 -- 172.21.15.28:0/3751712855 shutdown_connections 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 -- 172.21.15.28:0/3751712855 wait complete. 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.218+0000 7f4db7756640 1 Processor -- start 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 -- start start 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4db010d820 con 0x7f4db0104f60 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4db010d6a0 con 0x7f4db0107890 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db7756640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4db010d9a0 con 0x7f4db0102630 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db4cf1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db4cf1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43872/0 (socket says 172.21.15.28:43872) 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db4cf1640 1 -- 172.21.15.28:0/1994723163 learned_addr learned my addr 172.21.15.28:0/1994723163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 -- 172.21.15.28:0/1994723163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 msgr2=0x7f4db019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db019f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 -- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 msgr2=0x7f4db01983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db54f2640 1 -- 172.21.15.28:0/1994723163 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db019fa80 con 0x7f4db0102630 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.219+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4db54f2640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0197e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00e9f0 tx=0x7f4dac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4da67fc640 1 -- 172.21.15.28:0/1994723163 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac00ccc0 con 0x7f4db0102630 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db019fd70 con 0x7f4db0102630 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4da67fc640 1 -- 172.21.15.28:0/1994723163 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dac010910 con 0x7f4db0102630 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4da67fc640 1 -- 172.21.15.28:0/1994723163 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac018960 con 0x7f4db0102630 2024-03-22T02:29:41.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.220+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db01a01d0 con 0x7f4db0102630 2024-03-22T02:29:41.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.222+0000 7f4da67fc640 1 -- 172.21.15.28:0/1994723163 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4dac010ac0 con 0x7f4db0102630 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.222+0000 7f4dba41d740 1 Processor -- start 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.222+0000 7f4dba41d740 1 -- start start 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.222+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e94301d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e94307450 0x562e943098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4da67fc640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 0x7f4d9c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4dba41d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e94262040 con 0x562e943022e0 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4dba41d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562e9420a9e0 con 0x562e94307450 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4dba41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e9425dff0 con 0x562e943018f0 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 0x7f4d9c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e94301d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e94301d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:41162/0 (socket says 172.21.15.28:41162) 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43884/0 (socket says 172.21.15.28:43884) 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db5cf3640 1 -- 172.21.15.28:0/3204125159 learned_addr learned my addr 172.21.15.28:0/3204125159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4da67fc640 1 -- 172.21.15.28:0/1994723163 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4dac0cfb60 con 0x7f4db0102630 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e94307450 0x562e943098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 -- 172.21.15.28:0/3204125159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 msgr2=0x562e94301d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e94301d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 -- 172.21.15.28:0/3204125159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e94307450 msgr2=0x562e943098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e94307450 0x562e943098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 -- 172.21.15.28:0/3204125159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e93f76210 con 0x562e943022e0 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 0x7f4d9c0a5480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4da002c8e0 tx=0x7f4da0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4db54f2640 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0cd2a0 tx=0x7f4dac091410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e757ac3cbc91fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4d9b7fe640 1 -- 172.21.15.28:0/3204125159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac026040 con 0x562e943022e0 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4d9b7fe640 1 -- 172.21.15.28:0/3204125159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dac09bf30 con 0x562e943022e0 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.223+0000 7f4d9b7fe640 1 -- 172.21.15.28:0/3204125159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dac0a4400 con 0x562e943022e0 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 -- 172.21.15.28:0/3204125159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 msgr2=0x562e94306de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f4dac0cd2a0 tx=0x7f4dac091410 comp rx=0 tx=0).stop 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 -- 172.21.15.28:0/3204125159 shutdown_connections 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e94307450 0x562e943098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e943022e0 0x562e94306de0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 --2- 172.21.15.28:0/3204125159 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e94301d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 -- 172.21.15.28:0/3204125159 >> 172.21.15.28:0/3204125159 conn(0x562e942f8a50 msgr2=0x562e942f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 -- 172.21.15.28:0/3204125159 shutdown_connections 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.224+0000 7f4dba41d740 1 -- 172.21.15.28:0/3204125159 wait complete. 2024-03-22T02:29:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 -- start start 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 Processor -- start 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e9430be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e9430a3a0 con 0x562e94307450 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562e9421ddb0 con 0x562e943018f0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562e9430a520 con 0x562e943022e0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4dba41d740 10 client.0 _ll_register_callbacks cb 0x562e93f75a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db4cf1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e9430be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43900/0 (socket says 172.21.15.28:43900) 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44992/0 (socket says 172.21.15.28:44992) 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db5cf3640 1 -- 172.21.15.28:0/2211310976 learned_addr learned my addr 172.21.15.28:0/2211310976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db4cf1640 1 -- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 msgr2=0x562e9430b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.225+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db4cf1640 1 -- 172.21.15.28:0/2211310976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 msgr2=0x562e94312e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db4cf1640 1 -- 172.21.15.28:0/2211310976 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e94042620 con 0x562e943022e0 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db54f2640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e9430be40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4da002c7b0 tx=0x7f4da002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=911bab1f1a92b459 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:29:41.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da004e070 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4da00050e0 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dac09bf30 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4da005a050 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 10 client.29602 fetch_fsmap learned FSMap version 43 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562e943162b0 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dac09bf30 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 43) ==== 8354+0+0 (secure 0 0 0) 0x7f4da0053440 con 0x562e943022e0 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 10 client.29602 fetch_fsmap finished waiting for FSMap version 43 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 10 client.29602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 20 client.29602 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.226+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e943179f0 0x562e94319e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.227+0000 7f4dba41d740 4 client.29602 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:29:41.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.227+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e9426f580 con 0x562e943179f0 2024-03-22T02:29:41.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.227+0000 7f4db54f2640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e943179f0 0x562e94319e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.227+0000 7f4d98ff9640 10 client.29602 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:41.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.227+0000 7f4db54f2640 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e943179f0 0x562e94319e30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70266d2494f6f555 server_cookie=8310dc9403c1d0c4 in_seq=0 out_seq=0 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4d98ff9640 1 -- 172.21.15.28:0/2211310976 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e9426f580 con 0x562e943179f0 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4d98ff9640 10 client.29602 handle_command_reply: tid=0 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 client.29602 shutdown 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 msgr2=0x562e9430be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e9430be40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4da002c7b0 tx=0x7f4da002bc30 comp rx=0 tx=0).stop 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 shutdown_connections 2024-03-22T02:29:41.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e943179f0 0x562e94319e30 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e94312e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e9430be40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4d98ff9640 0 client.29602 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2211310976 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 >> 172.21.15.28:0/2211310976 conn(0x562e942f8a50 msgr2=0x562e942f7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 shutdown_connections 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 1 -- 172.21.15.28:0/2211310976 wait complete. 2024-03-22T02:29:41.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.232+0000 7f4dba41d740 20 client.29602 trim_cache size 0 max 16384 2024-03-22T02:29:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 Processor -- start 2024-03-22T02:29:41.248 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 -- start start 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 0x562e94312e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01aafd0 con 0x562e94307450 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01a8840 con 0x562e943022e0 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4dba41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01a6690 con 0x562e943018f0 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43906/0 (socket says 172.21.15.28:43906) 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 -- 172.21.15.28:0/2586979359 learned_addr learned my addr 172.21.15.28:0/2586979359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 0x562e94312e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.249 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 -- 172.21.15.28:0/2586979359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 msgr2=0x562e9430b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 -- 172.21.15.28:0/2586979359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 msgr2=0x562e94312e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 0x562e94312e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db5cf3640 1 -- 172.21.15.28:0/2586979359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e943026c0 con 0x562e94307450 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.247+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 0x562e94312e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4da8003b00 tx=0x7f4da8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2486f7379fb058a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4d9affd640 1 -- 172.21.15.28:0/2586979359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8004700 con 0x562e94307450 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4d9affd640 1 -- 172.21.15.28:0/2586979359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da8005020 con 0x562e94307450 2024-03-22T02:29:41.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4d9affd640 1 -- 172.21.15.28:0/2586979359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da801f470 con 0x562e94307450 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 -- 172.21.15.28:0/2586979359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 msgr2=0x562e9430be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f4da8003b00 tx=0x7f4da8004a40 comp rx=0 tx=0).stop 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 -- 172.21.15.28:0/2586979359 shutdown_connections 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9430be40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943022e0 0x562e94312e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 --2- 172.21.15.28:0/2586979359 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943018f0 0x562e9430b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.248+0000 7f4dba41d740 1 -- 172.21.15.28:0/2586979359 >> 172.21.15.28:0/2586979359 conn(0x562e942f8a50 msgr2=0x562e942f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- 172.21.15.28:0/2586979359 shutdown_connections 2024-03-22T02:29:41.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- 172.21.15.28:0/2586979359 wait complete. 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- start start 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 Processor -- start 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e94310210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01a6d00 con 0x562e94307450 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01a8170 con 0x562e943018f0 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4db01a73d0 con 0x562e943022e0 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4dba41d740 10 client.0 _ll_register_callbacks cb 0x7f4da0042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4db5cf3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43914/0 (socket says 172.21.15.28:43914) 2024-03-22T02:29:41.255 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4db54f2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4db5cf3640 1 -- 172.21.15.28:0/1885475666 learned_addr learned my addr 172.21.15.28:0/1885475666 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.249+0000 7f4db4cf1640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e94310210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 -- 172.21.15.28:0/1885475666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 msgr2=0x562e94310210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e94310210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 -- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 msgr2=0x562e9430fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 -- 172.21.15.28:0/1885475666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e9437c2a0 con 0x562e94307450 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4db5cf3640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4da80044d0 tx=0x7f4da8003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d86180127307f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da8004ab0 con 0x562e94307450 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9274+0+0 (secure 0 0 0) 0x7f4da8024070 con 0x562e94307450 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da8005020 con 0x562e94307450 2024-03-22T02:29:41.256 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4da8004c50 con 0x562e94307450 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4dba41d740 10 client.19812 fetch_fsmap learned FSMap version 43 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562e9437d4c0 con 0x562e94307450 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da8005020 con 0x562e94307450 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.250+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 43) ==== 8354+0+0 (secure 0 0 0) 0x7f4da8030030 con 0x562e94307450 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 10 client.19812 fetch_fsmap finished waiting for FSMap version 43 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 10 client.19812 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 20 client.19812 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e9437def0 0x562e94380310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 4 client.19812 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e942698d0 con 0x562e9437def0 2024-03-22T02:29:41.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4db54f2640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e9437def0 0x562e94380310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4db54f2640 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e9437def0 0x562e94380310 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea4c9de06ef346dd server_cookie=90d5fef15f53a66b in_seq=0 out_seq=0 2024-03-22T02:29:41.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:41.251+0000 7f4d98ff9640 10 client.19812 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:41.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 42909 16.09 MB/sec execute 21 sec latency 315.351 ms 2024-03-22T02:29:42.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4d98ff9640 1 -- 172.21.15.28:0/1885475666 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e942698d0 con 0x562e9437def0 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4d98ff9640 10 client.19812 handle_command_reply: tid=0 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 client.19812 shutdown 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 msgr2=0x562e9437bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f4da80044d0 tx=0x7f4da8003970 comp rx=0 tx=0).stop 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 shutdown_connections 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562e9437def0 0x562e94380310 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e94307450 0x562e9437bd60 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562e943022e0 0x562e94310210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 --2- 172.21.15.28:0/1885475666 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562e943018f0 0x562e9430fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4d98ff9640 0 client.19812 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:42.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 >> 172.21.15.28:0/1885475666 conn(0x562e942f8a50 msgr2=0x562e94306560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 shutdown_connections 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 1 -- 172.21.15.28:0/1885475666 wait complete. 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.154+0000 7f4dba41d740 20 client.19812 trim_cache size 0 max 16384 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:29:42.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:29:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 msgr2=0x7f4d9c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 0x7f4d9c0a5480 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4da002c8e0 tx=0x7f4da0005b50 comp rx=0 tx=0).stop 2024-03-22T02:29:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 msgr2=0x7f4db0197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0197e60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00e9f0 tx=0x7f4dac00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 shutdown_connections 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4d9c0a2fc0 0x7f4d9c0a5480 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4db0107890 0x7f4db019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4db0104f60 0x7f4db01983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 --2- 172.21.15.28:0/1994723163 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4db0102630 0x7f4db0197e60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 >> 172.21.15.28:0/1994723163 conn(0x7f4db00fc020 msgr2=0x7f4db0102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 shutdown_connections 2024-03-22T02:29:42.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:42.155+0000 7f4db7756640 1 -- 172.21.15.28:0/1994723163 wait complete. 2024-03-22T02:29:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:41 smithi120 ceph-mon[36803]: pgmap v844: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 411 op/s 2024-03-22T02:29:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:41 smithi083 ceph-mon[36340]: pgmap v844: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 411 op/s 2024-03-22T02:29:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:41 smithi028 ceph-mon[31916]: pgmap v844: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 411 op/s 2024-03-22T02:29:42.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 43084 15.66 MB/sec execute 22 sec latency 700.831 ms 2024-03-22T02:29:42.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:29:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setuser ceph since I am not root 2024-03-22T02:29:42.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:29:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setgroup ceph since I am not root 2024-03-22T02:29:42.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:29:42 smithi083 ceph-mds[66500]: -- 172.21.15.83:0/762101926 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d35f9f9a40 con 0x55d35f9ccc00 2024-03-22T02:29:42.765 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:29:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: starting mds.h at 2024-03-22T02:29:43.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:42 smithi120 ceph-mon[36803]: daemon mds.h restarted 2024-03-22T02:29:43.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:42 smithi083 ceph-mon[36340]: daemon mds.h restarted 2024-03-22T02:29:43.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:42 smithi028 ceph-mon[31916]: daemon mds.h restarted 2024-03-22T02:29:43.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 43373 15.62 MB/sec execute 23 sec latency 104.581 ms 2024-03-22T02:29:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: pgmap v845: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2024-03-22T02:29:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T02:29:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/2257967634,v1:172.21.15.83:6837/2257967634] up:boot 2024-03-22T02:29:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:29:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:29:44.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:43 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: pgmap v845: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/2257967634,v1:172.21.15.83:6837/2257967634] up:boot 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:29:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:43 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: pgmap v845: 129 pgs: 129 active+clean; 265 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.0 MiB/s wr, 396 op/s 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: osdmap e98: 12 total, 12 up, 12 in 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/2257967634,v1:172.21.15.83:6837/2257967634] up:boot 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:29:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:43 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:44.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 43649 15.71 MB/sec execute 24 sec latency 117.689 ms 2024-03-22T02:29:45.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 43907 15.43 MB/sec execute 25 sec latency 344.713 ms 2024-03-22T02:29:46.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:45 smithi120 ceph-mon[36803]: pgmap v847: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 540 op/s 2024-03-22T02:29:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:45 smithi083 ceph-mon[36340]: pgmap v847: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 540 op/s 2024-03-22T02:29:46.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:45 smithi028 ceph-mon[31916]: pgmap v847: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.3 MiB/s wr, 540 op/s 2024-03-22T02:29:46.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 44252 15.50 MB/sec execute 26 sec latency 105.071 ms 2024-03-22T02:29:47.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 44662 15.91 MB/sec execute 27 sec latency 74.892 ms 2024-03-22T02:29:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:47 smithi120 ceph-mon[36803]: pgmap v848: 129 pgs: 129 active+clean; 255 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 583 op/s 2024-03-22T02:29:48.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:47 smithi083 ceph-mon[36340]: pgmap v848: 129 pgs: 129 active+clean; 255 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 583 op/s 2024-03-22T02:29:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:47 smithi028 ceph-mon[31916]: pgmap v848: 129 pgs: 129 active+clean; 255 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 583 op/s 2024-03-22T02:29:48.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 45110 15.86 MB/sec execute 28 sec latency 53.978 ms 2024-03-22T02:29:49.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 45396 16.12 MB/sec execute 29 sec latency 109.440 ms 2024-03-22T02:29:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:49 smithi120 ceph-mon[36803]: pgmap v849: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 608 op/s 2024-03-22T02:29:50.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:49 smithi083 ceph-mon[36340]: pgmap v849: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 608 op/s 2024-03-22T02:29:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:49 smithi028 ceph-mon[31916]: pgmap v849: 129 pgs: 129 active+clean; 257 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.2 MiB/s wr, 608 op/s 2024-03-22T02:29:50.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 45652 16.19 MB/sec execute 30 sec latency 251.214 ms 2024-03-22T02:29:51.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 45973 16.23 MB/sec execute 31 sec latency 95.295 ms 2024-03-22T02:29:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:51 smithi120 ceph-mon[36803]: pgmap v850: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 620 op/s 2024-03-22T02:29:52.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:51 smithi083 ceph-mon[36340]: pgmap v850: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 620 op/s 2024-03-22T02:29:52.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:51 smithi028 ceph-mon[31916]: pgmap v850: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 620 op/s 2024-03-22T02:29:52.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 46243 16.24 MB/sec execute 32 sec latency 60.001 ms 2024-03-22T02:29:53.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 46537 16.12 MB/sec execute 33 sec latency 109.091 ms 2024-03-22T02:29:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:53 smithi120 ceph-mon[36803]: pgmap v851: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 561 op/s 2024-03-22T02:29:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:53 smithi083 ceph-mon[36340]: pgmap v851: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 561 op/s 2024-03-22T02:29:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:53 smithi028 ceph-mon[31916]: pgmap v851: 129 pgs: 129 active+clean; 259 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 561 op/s 2024-03-22T02:29:54.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 46802 16.09 MB/sec execute 34 sec latency 112.189 ms 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 -- 172.21.15.28:0/3064326643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac0ffcf0 msgr2=0x7f18ac100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/3064326643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac100110 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f189c009930 tx=0x7f189c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 -- 172.21.15.28:0/3064326643 shutdown_connections 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/3064326643 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac103680 0x7f18ac109f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/3064326643 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/3064326643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac100110 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 -- 172.21.15.28:0/3064326643 >> 172.21.15.28:0/3064326643 conn(0x7f18ac0fb620 msgr2=0x7f18ac0fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 -- 172.21.15.28:0/3064326643 shutdown_connections 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.245+0000 7f18b0c4d640 1 -- 172.21.15.28:0/3064326643 wait complete. 2024-03-22T02:29:55.250 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 Processor -- start 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 -- start start 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 0x7f18ac1011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac10d6c0 con 0x7f18ac103680 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac10d540 con 0x7f18ac102cc0 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18b0c4d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac10d840 con 0x7f18ac0ffcf0 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.246+0000 7f18abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 0x7f18ac1011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.251 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55316/0 (socket says 172.21.15.28:55316) 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:42300/0 (socket says 172.21.15.28:42300) 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 -- 172.21.15.28:0/1897578650 learned_addr learned my addr 172.21.15.28:0/1897578650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 -- 172.21.15.28:0/1897578650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 msgr2=0x7f18ac102630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 -- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 msgr2=0x7f18ac1011b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 0x7f18ac1011b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 -- 172.21.15.28:0/1897578650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac101a40 con 0x7f18ac102cc0 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18abfff640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 0x7f18ac1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18a000e9f0 tx=0x7f18a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18a8ff9640 1 -- 172.21.15.28:0/1897578650 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a000cd80 con 0x7f18ac102cc0 2024-03-22T02:29:55.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18ac1a1e80 con 0x7f18ac102cc0 2024-03-22T02:29:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18a8ff9640 1 -- 172.21.15.28:0/1897578650 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a0010910 con 0x7f18ac102cc0 2024-03-22T02:29:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18a8ff9640 1 -- 172.21.15.28:0/1897578650 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0018950 con 0x7f18ac102cc0 2024-03-22T02:29:55.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.247+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18ac1a22f0 con 0x7f18ac102cc0 2024-03-22T02:29:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18a8ff9640 1 -- 172.21.15.28:0/1897578650 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f18a0010ac0 con 0x7f18ac102cc0 2024-03-22T02:29:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 Processor -- start 2024-03-22T02:29:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 -- start start 2024-03-22T02:29:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5618ed851040 con 0x5618ed8f2210 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618ed7f99e0 con 0x5618ed8ed0a0 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18b4b72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618ed84cff0 con 0x5618ed8ec6b0 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18a8ff9640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 0x7f18940a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18abfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:42314/0 (socket says 172.21.15.28:42314) 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18abfff640 1 -- 172.21.15.28:0/3234897507 learned_addr learned my addr 172.21.15.28:0/3234897507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.250+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55322/0 (socket says 172.21.15.28:55322) 2024-03-22T02:29:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 0x7f18940a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18a8ff9640 1 -- 172.21.15.28:0/1897578650 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f18a00cfb20 con 0x7f18ac102cc0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 -- 172.21.15.28:0/3234897507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 msgr2=0x5618ed8f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 -- 172.21.15.28:0/3234897507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 msgr2=0x5618ed8f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 -- 172.21.15.28:0/3234897507 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618ed565210 con 0x5618ed8ec6b0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f18a4011d40 tx=0x7f18a400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c7670c61609c0f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f18abfff640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 0x7f18940a53d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f18a401d000 tx=0x7f18a401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f1891ffb640 1 -- 172.21.15.28:0/3234897507 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a401ece0 con 0x5618ed8ec6b0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f1891ffb640 1 -- 172.21.15.28:0/3234897507 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a4022070 con 0x5618ed8ec6b0 2024-03-22T02:29:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.251+0000 7f1891ffb640 1 -- 172.21.15.28:0/3234897507 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4027e60 con 0x5618ed8ec6b0 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- 172.21.15.28:0/3234897507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 msgr2=0x5618ed8ecad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f18a4011d40 tx=0x7f18a400f790 comp rx=0 tx=0).stop 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- 172.21.15.28:0/3234897507 shutdown_connections 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3234897507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8ecad0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- 172.21.15.28:0/3234897507 >> 172.21.15.28:0/3234897507 conn(0x5618ed8e7c20 msgr2=0x5618ed8e8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- 172.21.15.28:0/3234897507 shutdown_connections 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- 172.21.15.28:0/3234897507 wait complete. 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 -- start start 2024-03-22T02:29:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.252+0000 7f18b4b72740 1 Processor -- start 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8f2210 0x5618ed8fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5618ed8f5160 con 0x5618ed8ec6b0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5618ed80cdb0 con 0x5618ed8ed0a0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5618ed8f52e0 con 0x5618ed8f2210 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 10 client.0 _ll_register_callbacks cb 0x5618ed564a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50152/0 (socket says 172.21.15.28:50152) 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18ab7fe640 1 -- 172.21.15.28:0/1145541031 learned_addr learned my addr 172.21.15.28:0/1145541031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18abfff640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8f2210 0x5618ed8fdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 -- 172.21.15.28:0/1145541031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8f2210 msgr2=0x5618ed8fdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8f2210 0x5618ed8fdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 -- 172.21.15.28:0/1145541031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 msgr2=0x5618ed8f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 -- 172.21.15.28:0/1145541031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618ed631620 con 0x5618ed8ed0a0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18aaffd640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f6c50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f18a00cd260 tx=0x7f18a006aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edf2643051598dbb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f188affd640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00a4850 con 0x5618ed8ed0a0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a4022070 con 0x5618ed8ed0a0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.253+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f18a00badb0 con 0x5618ed8ed0a0 2024-03-22T02:29:55.265 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f18a009e070 con 0x5618ed8ed0a0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 10 client.29636 fetch_fsmap learned FSMap version 46 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5618ed901000 con 0x5618ed8ed0a0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a4022070 con 0x5618ed8ed0a0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f18a006bb40 con 0x5618ed8ed0a0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 10 client.29636 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 10 client.29636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 20 client.29636 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed9025c0 0x5618ed9049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 4 client.29636 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618ed85e580 con 0x5618ed9025c0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.254+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed9025c0 0x5618ed9049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.266 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.255+0000 7f188affd640 10 client.29636 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.255+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed9025c0 0x5618ed9049d0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e401dd13dcae9e26 server_cookie=5b450e6a38580e81 in_seq=0 out_seq=0 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.256+0000 7f188affd640 1 -- 172.21.15.28:0/1145541031 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618ed85e580 con 0x5618ed9025c0 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.256+0000 7f188affd640 10 client.29636 handle_command_reply: tid=0 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.256+0000 7f18b4b72740 1 client.29636 shutdown 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 msgr2=0x5618ed8f6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f6c50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f18a00cd260 tx=0x7f18a006aba0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 shutdown_connections 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed9025c0 0x5618ed9049d0 crc :-1 s=READY pgs=380 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8f2210 0x5618ed8fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8f6c50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f188affd640 0 client.29636 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 --2- 172.21.15.28:0/1145541031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ec6b0 0x5618ed8f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 >> 172.21.15.28:0/1145541031 conn(0x5618ed8e7c20 msgr2=0x5618ed8e6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 shutdown_connections 2024-03-22T02:29:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 1 -- 172.21.15.28:0/1145541031 wait complete. 2024-03-22T02:29:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.257+0000 7f18b4b72740 20 client.29636 trim_cache size 0 max 16384 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 Processor -- start 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 -- start start 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8f2210 0x5618ed8edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1af360 con 0x5618ed8ed0a0 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1acc30 con 0x5618ed8f2210 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18b4b72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1aaa80 con 0x5618ed8ec6b0 2024-03-22T02:29:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50154/0 (socket says 172.21.15.28:50154) 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.272+0000 7f18aaffd640 1 -- 172.21.15.28:0/3929586683 learned_addr learned my addr 172.21.15.28:0/3929586683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18abfff640 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8f2210 0x5618ed8edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 -- 172.21.15.28:0/3929586683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 msgr2=0x5618ed8edaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8edaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 -- 172.21.15.28:0/3929586683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8f2210 msgr2=0x5618ed8edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8f2210 0x5618ed8edfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 -- 172.21.15.28:0/3929586683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618ed8e8230 con 0x5618ed8ed0a0 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18aaffd640 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f18a009bbf0 tx=0x7f18a00bc1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84120a29686c3ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18917fa640 1 -- 172.21.15.28:0/3929586683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00a4850 con 0x5618ed8ed0a0 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18917fa640 1 -- 172.21.15.28:0/3929586683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a009b410 con 0x5618ed8ed0a0 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.273+0000 7f18917fa640 1 -- 172.21.15.28:0/3929586683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0098190 con 0x5618ed8ed0a0 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- 172.21.15.28:0/3929586683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 msgr2=0x5618ed8fdbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f18a009bbf0 tx=0x7f18a00bc1a0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- 172.21.15.28:0/3929586683 shutdown_connections 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8f2210 0x5618ed8edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fdbe0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3929586683 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8edaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- 172.21.15.28:0/3929586683 >> 172.21.15.28:0/3929586683 conn(0x5618ed8e7c20 msgr2=0x5618ed8eb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- 172.21.15.28:0/3929586683 shutdown_connections 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- 172.21.15.28:0/3929586683 wait complete. 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 -- start start 2024-03-22T02:29:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 Processor -- start 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.274+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8fb130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1ab0f0 con 0x5618ed8f2210 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1ac560 con 0x5618ed8ed0a0 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac1ab7c0 con 0x5618ed8ec6b0 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 10 client.0 _ll_register_callbacks cb 0x7f18a000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:29:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8fb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18aaffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18abfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50156/0 (socket says 172.21.15.28:50156) 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18abfff640 1 -- 172.21.15.28:0/3509553147 learned_addr learned my addr 172.21.15.28:0/3509553147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 -- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 msgr2=0x5618ed8fb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 -- 172.21.15.28:0/3509553147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 msgr2=0x5618ed8fc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 -- 172.21.15.28:0/3509553147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618ed8fcc20 con 0x5618ed8ec6b0 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18abfff640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:29:55.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18ab7fe640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8fb130 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f189c009900 tx=0x7f189c004540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b1e4cfd98400fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f188affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c046360 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18a009b410 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.275+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f189c046500 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f189c057050 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 10 client.29632 fetch_fsmap learned FSMap version 46 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5618ed968910 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18a009b410 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f189c046040 con 0x5618ed8ec6b0 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 10 client.29632 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 10 client.29632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 20 client.29632 populate_metadata read hostname 'smithi028' 2024-03-22T02:29:55.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed969140 0x5618ed96b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:29:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 4 client.29632 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:29:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5618ed8588d0 con 0x5618ed969140 2024-03-22T02:29:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.276+0000 7f18aaffd640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed969140 0x5618ed96b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:29:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.277+0000 7f188affd640 10 client.29632 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:55.277+0000 7f18aaffd640 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed969140 0x5618ed96b530 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7dc935f053fee server_cookie=fbc585845a6f3a31 in_seq=0 out_seq=0 2024-03-22T02:29:55.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 46912 15.80 MB/sec execute 35 sec latency 374.391 ms 2024-03-22T02:29:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:55 smithi120 ceph-mon[36803]: pgmap v852: 129 pgs: 129 active+clean; 264 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-03-22T02:29:56.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:55 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:55 smithi083 ceph-mon[36340]: pgmap v852: 129 pgs: 129 active+clean; 264 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-03-22T02:29:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:55 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:55 smithi028 ceph-mon[31916]: pgmap v852: 129 pgs: 129 active+clean; 264 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 5.4 MiB/s wr, 561 op/s 2024-03-22T02:29:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:55 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f188affd640 1 -- 172.21.15.28:0/3509553147 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5618ed8588d0 con 0x5618ed969140 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f188affd640 10 client.29632 handle_command_reply: tid=0 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 client.29632 shutdown 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 msgr2=0x5618ed8fb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8fb130 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f189c009900 tx=0x7f189c004540 comp rx=0 tx=0).stop 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 shutdown_connections 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5618ed969140 0x5618ed96b530 crc :-1 s=READY pgs=381 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5618ed8f2210 0x5618ed8fc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f188affd640 0 client.29632 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5618ed8ed0a0 0x5618ed8fb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 --2- 172.21.15.28:0/3509553147 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5618ed8ec6b0 0x5618ed8fb130 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 >> 172.21.15.28:0/3509553147 conn(0x5618ed8e7c20 msgr2=0x5618ed8e6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 shutdown_connections 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 1 -- 172.21.15.28:0/3509553147 wait complete. 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.360+0000 7f18b4b72740 20 client.29632 trim_cache size 0 max 16384 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:29:56.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:29:56.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:29:56.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 msgr2=0x7f18940a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:56.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 0x7f18940a53d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f18a401d000 tx=0x7f18a401f000 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 msgr2=0x7f18ac100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f18a000e9f0 tx=0x7f18a000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 shutdown_connections 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f18940a2f10 0x7f18940a53d0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18ac103680 0x7f18ac1011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f18ac102cc0 0x7f18ac100c70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 --2- 172.21.15.28:0/1897578650 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f18ac0ffcf0 0x7f18ac102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 >> 172.21.15.28:0/1897578650 conn(0x7f18ac0fb620 msgr2=0x7f18ac1060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 shutdown_connections 2024-03-22T02:29:56.365 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:29:56.361+0000 7f18b0c4d640 1 -- 172.21.15.28:0/1897578650 wait complete. 2024-03-22T02:29:56.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 47019 15.49 MB/sec execute 36 sec latency 703.094 ms 2024-03-22T02:29:57.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 47287 15.40 MB/sec execute 37 sec latency 96.138 ms 2024-03-22T02:29:58.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:57 smithi120 ceph-mon[36803]: pgmap v853: 129 pgs: 129 active+clean; 265 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 389 op/s 2024-03-22T02:29:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:57 smithi083 ceph-mon[36340]: pgmap v853: 129 pgs: 129 active+clean; 265 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 389 op/s 2024-03-22T02:29:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:57 smithi028 ceph-mon[31916]: pgmap v853: 129 pgs: 129 active+clean; 265 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.9 MiB/s wr, 389 op/s 2024-03-22T02:29:58.678 INFO:tasks.workunit.client.0.smithi028.stdout: 10 47712 15.45 MB/sec execute 38 sec latency 93.034 ms 2024-03-22T02:29:59.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 48104 15.69 MB/sec execute 39 sec latency 48.055 ms 2024-03-22T02:30:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:29:59 smithi083 ceph-mon[36340]: pgmap v854: 129 pgs: 129 active+clean; 261 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2024-03-22T02:30:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:29:59 smithi028 ceph-mon[31916]: pgmap v854: 129 pgs: 129 active+clean; 261 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2024-03-22T02:30:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:30:00.001+0000 7fa3d2c81700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:30:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:30:00.001+0000 7fa3d2c81700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-22T02:30:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:30:00.001+0000 7fa3d2c81700 -1 log_channel(cluster) log [ERR] : mds.c(mds.2): Metadata damage detected 2024-03-22T02:30:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:29:59 smithi120 ceph-mon[36803]: pgmap v854: 129 pgs: 129 active+clean; 261 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2024-03-22T02:30:00.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 48436 15.72 MB/sec execute 40 sec latency 177.209 ms 2024-03-22T02:30:01.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:01 smithi083 ceph-mon[36340]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:30:01.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:01 smithi083 ceph-mon[36340]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-22T02:30:01.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:01 smithi083 ceph-mon[36340]: mds.c(mds.2): Metadata damage detected 2024-03-22T02:30:01.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:01 smithi028 ceph-mon[31916]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:30:01.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:01 smithi028 ceph-mon[31916]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-22T02:30:01.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:01 smithi028 ceph-mon[31916]: mds.c(mds.2): Metadata damage detected 2024-03-22T02:30:01.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:01 smithi120 ceph-mon[36803]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:30:01.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:01 smithi120 ceph-mon[36803]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-03-22T02:30:01.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:01 smithi120 ceph-mon[36803]: mds.c(mds.2): Metadata damage detected 2024-03-22T02:30:01.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 48814 15.94 MB/sec execute 41 sec latency 81.518 ms 2024-03-22T02:30:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:02 smithi083 ceph-mon[36340]: pgmap v855: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-03-22T02:30:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:02 smithi028 ceph-mon[31916]: pgmap v855: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-03-22T02:30:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:02 smithi120 ceph-mon[36803]: pgmap v855: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 562 op/s 2024-03-22T02:30:02.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 49128 15.93 MB/sec execute 42 sec latency 63.657 ms 2024-03-22T02:30:03.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 49471 16.14 MB/sec execute 43 sec latency 67.920 ms 2024-03-22T02:30:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:04 smithi083 ceph-mon[36340]: pgmap v856: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 575 op/s 2024-03-22T02:30:04.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:04 smithi028 ceph-mon[31916]: pgmap v856: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 575 op/s 2024-03-22T02:30:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:04 smithi120 ceph-mon[36803]: pgmap v856: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 575 op/s 2024-03-22T02:30:04.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 49806 16.17 MB/sec execute 44 sec latency 88.954 ms 2024-03-22T02:30:05.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 50042 16.02 MB/sec execute 45 sec latency 222.507 ms 2024-03-22T02:30:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:06 smithi028 ceph-mon[31916]: pgmap v857: 129 pgs: 129 active+clean; 261 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 599 op/s 2024-03-22T02:30:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:06 smithi120 ceph-mon[36803]: pgmap v857: 129 pgs: 129 active+clean; 261 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 599 op/s 2024-03-22T02:30:06.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:06 smithi083 ceph-mon[36340]: pgmap v857: 129 pgs: 129 active+clean; 261 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 599 op/s 2024-03-22T02:30:06.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 50235 15.81 MB/sec execute 46 sec latency 133.306 ms 2024-03-22T02:30:07.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 50565 16.01 MB/sec execute 47 sec latency 73.487 ms 2024-03-22T02:30:08.300 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:08 smithi120 ceph-mon[36803]: pgmap v858: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 529 op/s 2024-03-22T02:30:08.316 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:08 smithi083 ceph-mon[36340]: pgmap v858: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 529 op/s 2024-03-22T02:30:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:08 smithi028 ceph-mon[31916]: pgmap v858: 129 pgs: 129 active+clean; 260 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.9 MiB/s wr, 529 op/s 2024-03-22T02:30:08.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 50880 15.95 MB/sec execute 48 sec latency 73.153 ms 2024-03-22T02:30:09.464 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 -- 172.21.15.28:0/2157437033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 msgr2=0x7f83bc10a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/2157437033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc10a0e0 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009cd0 tx=0x7f83b402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 -- 172.21.15.28:0/2157437033 shutdown_connections 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/2157437033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc10a0e0 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/2157437033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc103310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/2157437033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc1002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 -- 172.21.15.28:0/2157437033 >> 172.21.15.28:0/2157437033 conn(0x7f83bc0fb7f0 msgr2=0x7f83bc0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 -- 172.21.15.28:0/2157437033 shutdown_connections 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.461+0000 7f83c1f0c640 1 -- 172.21.15.28:0/2157437033 wait complete. 2024-03-22T02:30:09.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 Processor -- start 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 -- start start 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10d8a0 con 0x7f83bc103850 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10d720 con 0x7f83bc0ffec0 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83c1f0c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc10da20 con 0x7f83bc102e90 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41580/0 (socket says 172.21.15.28:41580) 2024-03-22T02:30:09.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83bbfff640 1 -- 172.21.15.28:0/4132297324 learned_addr learned my addr 172.21.15.28:0/4132297324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.462+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 -- 172.21.15.28:0/4132297324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 msgr2=0x7f83bc197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 -- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 msgr2=0x7f83bc19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 -- 172.21.15.28:0/4132297324 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83bc19fa50 con 0x7f83bc102e90 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83bbfff640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc198370 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f83a400e9f0 tx=0x7f83a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b97fa640 1 -- 172.21.15.28:0/4132297324 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a400ccc0 con 0x7f83bc102e90 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83bc19fd40 con 0x7f83bc102e90 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b97fa640 1 -- 172.21.15.28:0/4132297324 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83a4005020 con 0x7f83bc102e90 2024-03-22T02:30:09.467 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83b97fa640 1 -- 172.21.15.28:0/4132297324 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a40049e0 con 0x7f83bc102e90 2024-03-22T02:30:09.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.463+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83bc1a01e0 con 0x7f83bc102e90 2024-03-22T02:30:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.465+0000 7f83b97fa640 1 -- 172.21.15.28:0/4132297324 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f83a4005560 con 0x7f83bc102e90 2024-03-22T02:30:09.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.465+0000 7f83c4bd4740 1 Processor -- start 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 -- start start 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83b97fa640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 0x7f83a80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580df9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580e0040 0x55bb580e24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb5803f040 con 0x55bb580e0040 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb57fe79e0 con 0x55bb580da4e0 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb5803aff0 con 0x55bb580daed0 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bbfff640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 0x7f83a80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:57036/0 (socket says 172.21.15.28:57036) 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 -- 172.21.15.28:0/796449097 learned_addr learned my addr 172.21.15.28:0/796449097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83b97fa640 1 -- 172.21.15.28:0/4132297324 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f83a40cf760 con 0x7f83bc102e90 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580df9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bbfff640 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580e0040 0x55bb580e24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 -- 172.21.15.28:0/796449097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 msgr2=0x55bb580df9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580df9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 -- 172.21.15.28:0/796449097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580e0040 msgr2=0x55bb580e24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580e0040 0x55bb580e24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 -- 172.21.15.28:0/796449097 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb57d53210 con 0x55bb580da4e0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bbfff640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 0x7f83a80a5480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009a60 tx=0x7f83b4005fa0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.466+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0114c0 tx=0x7f83ac011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff7c6bf88551c8a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83b1ffb640 1 -- 172.21.15.28:0/796449097 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac011030 con 0x55bb580da4e0 2024-03-22T02:30:09.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83b1ffb640 1 -- 172.21.15.28:0/796449097 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83ac0111d0 con 0x55bb580da4e0 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83b1ffb640 1 -- 172.21.15.28:0/796449097 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac00f5c0 con 0x55bb580da4e0 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 -- 172.21.15.28:0/796449097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 msgr2=0x55bb580da900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f83ac0114c0 tx=0x7f83ac011980 comp rx=0 tx=0).stop 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 -- 172.21.15.28:0/796449097 shutdown_connections 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580e0040 0x55bb580e24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580df9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/796449097 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580da900 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 -- 172.21.15.28:0/796449097 >> 172.21.15.28:0/796449097 conn(0x55bb580d5a50 msgr2=0x55bb580d5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 -- 172.21.15.28:0/796449097 shutdown_connections 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.467+0000 7f83c4bd4740 1 -- 172.21.15.28:0/796449097 wait complete. 2024-03-22T02:30:09.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 -- start start 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 Processor -- start 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580e49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb580eba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb580e2ff0 con 0x55bb580da4e0 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb57ffadb0 con 0x55bb580e0040 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb580e3170 con 0x55bb580daed0 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83c4bd4740 10 client.0 _ll_register_callbacks cb 0x55bb57d52a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:09.479 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41584/0 (socket says 172.21.15.28:41584) 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83bb7fe640 1 -- 172.21.15.28:0/3758483731 learned_addr learned my addr 172.21.15.28:0/3758483731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb580eba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.468+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580e49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 -- 172.21.15.28:0/3758483731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 msgr2=0x55bb580e49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580e49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 -- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 msgr2=0x55bb580eba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb580eba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 -- 172.21.15.28:0/3758483731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb57e1f620 con 0x55bb580da4e0 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bbfff640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb580eba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f83ac011e80 tx=0x7f83ac00fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae63d2aa7daae4db server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.480 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f839affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac028030 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f83ac016040 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83ac0111d0 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f83ac032050 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83c4bd4740 10 client.19848 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.469+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bb580eeda0 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83ac0111d0 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f83ac023c20 con 0x55bb580da4e0 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 10 client.19848 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 10 client.19848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 20 client.19848 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb580f0270 0x55bb580f2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.481 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 4 client.19848 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb5804c580 con 0x55bb580f0270 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb580f0270 0x55bb580f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f839affd640 10 client.19848 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.470+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb580f0270 0x55bb580f2660 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fb455edb21b6c09 server_cookie=d40a567d9542a2c4 in_seq=0 out_seq=0 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f839affd640 1 -- 172.21.15.28:0/3758483731 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb5804c580 con 0x55bb580f0270 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f839affd640 10 client.19848 handle_command_reply: tid=0 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 client.19848 shutdown 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 msgr2=0x55bb580e44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f83ac011e80 tx=0x7f83ac00fca0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 shutdown_connections 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb580f0270 0x55bb580f2660 crc :-1 s=READY pgs=382 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.482 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb580eba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580daed0 0x55bb580e49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f839affd640 0 client.19848 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/3758483731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580da4e0 0x55bb580e44b0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 >> 172.21.15.28:0/3758483731 conn(0x55bb580d5a50 msgr2=0x55bb580d4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 shutdown_connections 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.473+0000 7f83c4bd4740 1 -- 172.21.15.28:0/3758483731 wait complete. 2024-03-22T02:30:09.483 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.474+0000 7f83c4bd4740 20 client.19848 trim_cache size 0 max 16384 2024-03-22T02:30:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.489+0000 7f83c4bd4740 1 Processor -- start 2024-03-22T02:30:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 -- start start 2024-03-22T02:30:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580e0040 0x55bb580e3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1ad2d0 con 0x55bb580daed0 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1aaba0 con 0x55bb580da4e0 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1a89f0 con 0x55bb580e0040 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41592/0 (socket says 172.21.15.28:41592) 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 -- 172.21.15.28:0/1342501226 learned_addr learned my addr 172.21.15.28:0/1342501226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83bbfff640 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580e0040 0x55bb580e3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 -- 172.21.15.28:0/1342501226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580e0040 msgr2=0x55bb580e3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580e0040 0x55bb580e3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 -- 172.21.15.28:0/1342501226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 msgr2=0x55bb580dc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580dc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 -- 172.21.15.28:0/1342501226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb580e2ed0 con 0x55bb580daed0 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f83a4097ee0 tx=0x7f83a409c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ab3a672e63e960 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b17fa640 1 -- 172.21.15.28:0/1342501226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a40b2070 con 0x55bb580daed0 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b17fa640 1 -- 172.21.15.28:0/1342501226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83a409ba80 con 0x55bb580daed0 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.490+0000 7f83b17fa640 1 -- 172.21.15.28:0/1342501226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83a401c040 con 0x55bb580daed0 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 -- 172.21.15.28:0/1342501226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 msgr2=0x55bb580eb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f83a4097ee0 tx=0x7f83a409c180 comp rx=0 tx=0).stop 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 -- 172.21.15.28:0/1342501226 shutdown_connections 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580e0040 0x55bb580e3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580eb950 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/1342501226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580da4e0 0x55bb580dc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 -- 172.21.15.28:0/1342501226 >> 172.21.15.28:0/1342501226 conn(0x55bb580d5a50 msgr2=0x55bb580db940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 -- 172.21.15.28:0/1342501226 shutdown_connections 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.491+0000 7f83c4bd4740 1 -- 172.21.15.28:0/1342501226 wait complete. 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 -- start start 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 Processor -- start 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb58154590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1a9060 con 0x55bb580daed0 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1aa4d0 con 0x55bb580e0040 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f83bc1a9730 con 0x55bb580da4e0 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83c4bd4740 10 client.0 _ll_register_callbacks cb 0x7f83ac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:09.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:52644/0 (socket says 172.21.15.28:52644) 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb58154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83b2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41594/0 (socket says 172.21.15.28:41594) 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 -- 172.21.15.28:0/4000555661 learned_addr learned my addr 172.21.15.28:0/4000555661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 -- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 msgr2=0x55bb58154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb58154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 -- 172.21.15.28:0/4000555661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 msgr2=0x55bb580e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83bb7fe640 1 -- 172.21.15.28:0/4000555661 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb580ebc90 con 0x55bb580da4e0 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.492+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83bb7fe640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009730 tx=0x7f83ac009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb574abfb2b951f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:09.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83ac015070 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83a409ba80 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f83ac011040 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f83ac028040 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 10 client.29650 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bb58155a50 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83a409ba80 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f83ac0037b0 con 0x55bb580da4e0 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 10 client.29650 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 10 client.29650 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 20 client.29650 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.493+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb58156470 0x55bb58158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:09.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.494+0000 7f83c4bd4740 4 client.29650 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:30:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.494+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb580468d0 con 0x55bb58156470 2024-03-22T02:30:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.494+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb58156470 0x55bb58158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.494+0000 7f83b2ffd640 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb58156470 0x55bb58158890 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed7d60f3cedde36 server_cookie=1664889deaf8b04b in_seq=0 out_seq=0 2024-03-22T02:30:09.502 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:09.494+0000 7f839affd640 10 client.29650 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:09.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 51255 15.88 MB/sec execute 49 sec latency 202.152 ms 2024-03-22T02:30:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.666+0000 7f839affd640 1 -- 172.21.15.28:0/4000555661 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb580468d0 con 0x55bb58156470 2024-03-22T02:30:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.666+0000 7f839affd640 10 client.29650 handle_command_reply: tid=0 2024-03-22T02:30:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.666+0000 7f83c4bd4740 1 client.29650 shutdown 2024-03-22T02:30:10.668 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 msgr2=0x55bb580e87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f83ac009730 tx=0x7f83ac009ee0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 shutdown_connections 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb58156470 0x55bb58158890 crc :-1 s=READY pgs=383 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb580e0040 0x55bb58154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb580daed0 0x55bb580e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 --2- 172.21.15.28:0/4000555661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb580da4e0 0x55bb580e87c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 >> 172.21.15.28:0/4000555661 conn(0x55bb580d5a50 msgr2=0x55bb580d4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f839affd640 0 client.29650 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 shutdown_connections 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 1 -- 172.21.15.28:0/4000555661 wait complete. 2024-03-22T02:30:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.667+0000 7f83c4bd4740 20 client.29650 trim_cache size 0 max 16384 2024-03-22T02:30:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:30:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:30:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:30:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 msgr2=0x7f83a80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 0x7f83a80a5480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009a60 tx=0x7f83b4005fa0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 msgr2=0x7f83bc198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc198370 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f83a400e9f0 tx=0x7f83a400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 shutdown_connections 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f83a80a2fc0 0x7f83a80a5480 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83bc103850 0x7f83bc19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f83bc102e90 0x7f83bc198370 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 --2- 172.21.15.28:0/4132297324 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f83bc0ffec0 0x7f83bc197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 >> 172.21.15.28:0/4132297324 conn(0x7f83bc0fb7f0 msgr2=0x7f83bc1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 shutdown_connections 2024-03-22T02:30:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:10.668+0000 7f83c1f0c640 1 -- 172.21.15.28:0/4132297324 wait complete. 2024-03-22T02:30:10.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 51314 15.72 MB/sec execute 50 sec latency 705.774 ms 2024-03-22T02:30:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:10 smithi120 ceph-mon[36803]: pgmap v859: 129 pgs: 129 active+clean; 266 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 591 op/s 2024-03-22T02:30:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:10 smithi083 ceph-mon[36340]: pgmap v859: 129 pgs: 129 active+clean; 266 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 591 op/s 2024-03-22T02:30:10.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:10 smithi028 ceph-mon[31916]: pgmap v859: 129 pgs: 129 active+clean; 266 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 591 op/s 2024-03-22T02:30:11.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 51690 15.89 MB/sec execute 51 sec latency 88.731 ms 2024-03-22T02:30:12.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 52082 15.84 MB/sec execute 52 sec latency 101.336 ms 2024-03-22T02:30:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:12 smithi120 ceph-mon[36803]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-03-22T02:30:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:12 smithi083 ceph-mon[36340]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-03-22T02:30:12.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:12 smithi028 ceph-mon[31916]: pgmap v860: 129 pgs: 129 active+clean; 265 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.0 MiB/s wr, 432 op/s 2024-03-22T02:30:13.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 52442 15.99 MB/sec execute 53 sec latency 68.443 ms 2024-03-22T02:30:14.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 52816 16.21 MB/sec execute 54 sec latency 86.131 ms 2024-03-22T02:30:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:14 smithi120 ceph-mon[36803]: pgmap v861: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-03-22T02:30:14.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:14 smithi083 ceph-mon[36340]: pgmap v861: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-03-22T02:30:14.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:14 smithi028 ceph-mon[31916]: pgmap v861: 129 pgs: 129 active+clean; 257 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.5 MiB/s wr, 401 op/s 2024-03-22T02:30:15.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 53093 16.16 MB/sec execute 55 sec latency 201.977 ms 2024-03-22T02:30:16.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 53371 16.12 MB/sec execute 56 sec latency 99.034 ms 2024-03-22T02:30:16.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:16 smithi120 ceph-mon[36803]: pgmap v862: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 535 op/s 2024-03-22T02:30:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:16 smithi083 ceph-mon[36340]: pgmap v862: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 535 op/s 2024-03-22T02:30:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:16 smithi028 ceph-mon[31916]: pgmap v862: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 535 op/s 2024-03-22T02:30:17.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 53641 16.05 MB/sec execute 57 sec latency 69.121 ms 2024-03-22T02:30:18.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 53934 16.04 MB/sec execute 58 sec latency 62.296 ms 2024-03-22T02:30:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:18 smithi120 ceph-mon[36803]: pgmap v863: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 559 op/s 2024-03-22T02:30:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:18 smithi083 ceph-mon[36340]: pgmap v863: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 559 op/s 2024-03-22T02:30:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:18 smithi028 ceph-mon[31916]: pgmap v863: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.1 MiB/s wr, 559 op/s 2024-03-22T02:30:19.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 54232 16.09 MB/sec execute 59 sec latency 159.114 ms 2024-03-22T02:30:20.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 54509 16.04 MB/sec execute 60 sec latency 242.112 ms 2024-03-22T02:30:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:20 smithi120 ceph-mon[36803]: pgmap v864: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 586 op/s 2024-03-22T02:30:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:20 smithi083 ceph-mon[36340]: pgmap v864: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 586 op/s 2024-03-22T02:30:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:20 smithi028 ceph-mon[31916]: pgmap v864: 129 pgs: 129 active+clean; 262 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.6 MiB/s wr, 586 op/s 2024-03-22T02:30:21.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 54872 16.03 MB/sec execute 61 sec latency 415.424 ms 2024-03-22T02:30:22.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 55168 15.98 MB/sec execute 62 sec latency 86.610 ms 2024-03-22T02:30:22.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:22 smithi120 ceph-mon[36803]: pgmap v865: 129 pgs: 129 active+clean; 264 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 507 op/s 2024-03-22T02:30:22.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:22 smithi083 ceph-mon[36340]: pgmap v865: 129 pgs: 129 active+clean; 264 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 507 op/s 2024-03-22T02:30:22.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:22 smithi028 ceph-mon[31916]: pgmap v865: 129 pgs: 129 active+clean; 264 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 507 op/s 2024-03-22T02:30:23.679 INFO:tasks.workunit.client.0.smithi028.stdout: 10 55585 16.06 MB/sec execute 63 sec latency 67.018 ms 2024-03-22T02:30:23.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.790+0000 7f769a5c9640 1 -- 172.21.15.28:0/852384907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410b7c0 msgr2=0x7f769410dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 --2- 172.21.15.28:0/852384907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410b7c0 0x7f769410dbd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f768800c370 tx=0x7f768802b7a0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 -- 172.21.15.28:0/852384907 shutdown_connections 2024-03-22T02:30:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 --2- 172.21.15.28:0/852384907 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410e260 0x7f7694110720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 --2- 172.21.15.28:0/852384907 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410b7c0 0x7f769410dbd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 --2- 172.21.15.28:0/852384907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7694108e00 0x7f769410b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.791+0000 7f769a5c9640 1 -- 172.21.15.28:0/852384907 >> 172.21.15.28:0/852384907 conn(0x7f769406ebd0 msgr2=0x7f7694071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.793+0000 7f769a5c9640 1 -- 172.21.15.28:0/852384907 shutdown_connections 2024-03-22T02:30:23.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.793+0000 7f769a5c9640 1 -- 172.21.15.28:0/852384907 wait complete. 2024-03-22T02:30:23.794 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.794+0000 7f769a5c9640 1 Processor -- start 2024-03-22T02:30:23.795 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 -- start start 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7694108e00 0x7f769419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410e260 0x7f76941a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7694113e80 con 0x7f7694108e00 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7694113d00 con 0x7f769410b7c0 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f769a5c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7694114000 con 0x7f769410e260 2024-03-22T02:30:23.796 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f7698b66640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410e260 0x7f76941a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f76937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.795+0000 7f76937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:41394/0 (socket says 172.21.15.28:41394) 2024-03-22T02:30:23.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 -- 172.21.15.28:0/3242468008 learned_addr learned my addr 172.21.15.28:0/3242468008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 -- 172.21.15.28:0/3242468008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410e260 msgr2=0x7f76941a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410e260 0x7f76941a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 -- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7694108e00 msgr2=0x7f769419e410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7694108e00 0x7f769419e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 -- 172.21.15.28:0/3242468008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76941a6030 con 0x7f769410b7c0 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76937fe640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7688005ab0 tx=0x7f768800aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.796+0000 7f76917fa640 1 -- 172.21.15.28:0/3242468008 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7688041070 con 0x7f769410b7c0 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.797+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76941a62c0 con 0x7f769410b7c0 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.797+0000 7f76917fa640 1 -- 172.21.15.28:0/3242468008 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7688003ad0 con 0x7f769410b7c0 2024-03-22T02:30:23.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.797+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76941a68e0 con 0x7f769410b7c0 2024-03-22T02:30:23.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.798+0000 7f76917fa640 1 -- 172.21.15.28:0/3242468008 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768803dc60 con 0x7f769410b7c0 2024-03-22T02:30:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.800+0000 7f76917fa640 1 -- 172.21.15.28:0/3242468008 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7688003c80 con 0x7f769410b7c0 2024-03-22T02:30:23.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.800+0000 7f769d1b5740 1 Processor -- start 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 -- start start 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac1b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac1bed0 0x561aeac209d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561aeab80040 con 0x561aeac1bed0 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561aeab289e0 con 0x561aeac1b4e0 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f769d1b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561aeab7bff0 con 0x561aeac21040 2024-03-22T02:30:23.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f76937fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f76917fa640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 0x7f767c0a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f76937fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35872/0 (socket says 172.21.15.28:35872) 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f76937fe640 1 -- 172.21.15.28:0/2035332536 learned_addr learned my addr 172.21.15.28:0/2035332536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.801+0000 7f76917fa640 1 -- 172.21.15.28:0/3242468008 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f76880f0a70 con 0x7f769410b7c0 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 -- 172.21.15.28:0/2035332536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 msgr2=0x561aeac1b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac1b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 -- 172.21.15.28:0/2035332536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac1bed0 msgr2=0x561aeac209d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac1bed0 0x561aeac209d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 -- 172.21.15.28:0/2035332536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aea894210 con 0x561aeac21040 2024-03-22T02:30:23.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f76937fe640 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f76880b9700 tx=0x7f768802b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce6612f9e53b51d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.802+0000 7f7698b66640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 0x7f767c0a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.803+0000 7f767a7fc640 1 -- 172.21.15.28:0/2035332536 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768804a070 con 0x561aeac21040 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.803+0000 7f767a7fc640 1 -- 172.21.15.28:0/2035332536 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768800c030 con 0x561aeac21040 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.803+0000 7f7698b66640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 0x7f767c0a5470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x561aeac1c730 tx=0x7f768c00b710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.803+0000 7f767a7fc640 1 -- 172.21.15.28:0/2035332536 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76880bae60 con 0x561aeac21040 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- 172.21.15.28:0/2035332536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 msgr2=0x561aeac234d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f76880b9700 tx=0x7f768802b7a0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- 172.21.15.28:0/2035332536 shutdown_connections 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac21040 0x561aeac234d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac1bed0 0x561aeac209d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 --2- 172.21.15.28:0/2035332536 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac1b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- 172.21.15.28:0/2035332536 >> 172.21.15.28:0/2035332536 conn(0x561aeac16a50 msgr2=0x561aeac16e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- 172.21.15.28:0/2035332536 shutdown_connections 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- 172.21.15.28:0/2035332536 wait complete. 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.804+0000 7f769d1b5740 1 -- start start 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 Processor -- start 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac295d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac29b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561aeac281c0 con 0x561aeac21040 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561aeab3bdb0 con 0x561aeac1b4e0 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.805+0000 7f769d1b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561aeac28340 con 0x561aeac1bed0 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f769d1b5740 10 client.0 _ll_register_callbacks cb 0x561aea893a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:23.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38110/0 (socket says 172.21.15.28:38110) 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 -- 172.21.15.28:0/1751154540 learned_addr learned my addr 172.21.15.28:0/1751154540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f76937fe640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac29b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 -- 172.21.15.28:0/1751154540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 msgr2=0x561aeac29b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac29b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 -- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 msgr2=0x561aeac295d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac295d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.808 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 -- 172.21.15.28:0/1751154540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aea960620 con 0x561aeac21040 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.806+0000 7f7698b66640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f768c00b450 tx=0x7f768c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3162e05bdf976b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.808+0000 7f76737fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.808+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c027060 con 0x561aeac21040 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.808+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f768800c030 con 0x561aeac21040 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.808+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f768c017d70 con 0x561aeac21040 2024-03-22T02:30:23.809 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.808+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f768c02e050 con 0x561aeac21040 2024-03-22T02:30:23.810 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.809+0000 7f769d1b5740 10 client.19857 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.810+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561aeac33f90 con 0x561aeac21040 2024-03-22T02:30:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.811+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768c0228e0 con 0x561aeac21040 2024-03-22T02:30:23.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.811+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f768c021680 con 0x561aeac21040 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 10 client.19857 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 10 client.19857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 20 client.19857 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac356b0 0x561aeac37b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 4 client.19857 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:30:23.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.812+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f768800c030 con 0x561aeac356b0 2024-03-22T02:30:23.815 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.815+0000 7f7693fff640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac356b0 0x561aeac37b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.817+0000 7f76737fe640 10 client.19857 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:23.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.817+0000 7f7693fff640 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac356b0 0x561aeac37b40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff287527f5a92d6d server_cookie=c8fb4921af12a6d4 in_seq=0 out_seq=0 2024-03-22T02:30:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.830+0000 7f76737fe640 1 -- 172.21.15.28:0/1751154540 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f768800c030 con 0x561aeac356b0 2024-03-22T02:30:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.830+0000 7f76737fe640 10 client.19857 handle_command_reply: tid=0 2024-03-22T02:30:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.831+0000 7f769d1b5740 1 client.19857 shutdown 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 msgr2=0x561aeac30b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f768c00b450 tx=0x7f768c004ab0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 shutdown_connections 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac356b0 0x561aeac37b40 crc :-1 s=READY pgs=384 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac30b50 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac29b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1751154540 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac295d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 >> 172.21.15.28:0/1751154540 conn(0x561aeac16a50 msgr2=0x561aeac15c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.834+0000 7f76737fe640 0 client.19857 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.837+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 shutdown_connections 2024-03-22T02:30:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.837+0000 7f769d1b5740 1 -- 172.21.15.28:0/1751154540 wait complete. 2024-03-22T02:30:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.837+0000 7f769d1b5740 20 client.19857 trim_cache size 0 max 16384 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 Processor -- start 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 -- start start 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1b4e0 0x561aeac37b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1bed0 0x561aeac30b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76941b3850 con 0x561aeac21040 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f76941b1120 con 0x561aeac1bed0 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f769d1b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76941aef70 con 0x561aeac1b4e0 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.857+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38112/0 (socket says 172.21.15.28:38112) 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f76937fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1bed0 0x561aeac30b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 -- 172.21.15.28:0/307475072 learned_addr learned my addr 172.21.15.28:0/307475072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 -- 172.21.15.28:0/307475072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1b4e0 msgr2=0x561aeac37b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1b4e0 0x561aeac37b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 -- 172.21.15.28:0/307475072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1bed0 msgr2=0x561aeac30b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1bed0 0x561aeac30b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 -- 172.21.15.28:0/307475072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aeac1c760 con 0x561aeac21040 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7698b66640 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f768c006010 tx=0x7f768c021ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17f35bbe7488632 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7679ffb640 1 -- 172.21.15.28:0/307475072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c0213a0 con 0x561aeac21040 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7679ffb640 1 -- 172.21.15.28:0/307475072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768c003e50 con 0x561aeac21040 2024-03-22T02:30:23.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.858+0000 7f7679ffb640 1 -- 172.21.15.28:0/307475072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c004370 con 0x561aeac21040 2024-03-22T02:30:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- 172.21.15.28:0/307475072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 msgr2=0x561aeac1d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f768c006010 tx=0x7f768c021ec0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- 172.21.15.28:0/307475072 shutdown_connections 2024-03-22T02:30:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac1d080 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1bed0 0x561aeac30b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 --2- 172.21.15.28:0/307475072 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1b4e0 0x561aeac37b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- 172.21.15.28:0/307475072 >> 172.21.15.28:0/307475072 conn(0x561aeac16a50 msgr2=0x561aeac1c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- 172.21.15.28:0/307475072 shutdown_connections 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- 172.21.15.28:0/307475072 wait complete. 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.859+0000 7f769d1b5740 1 -- start start 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 Processor -- start 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac2e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac2ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac2fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76941af5e0 con 0x561aeac21040 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f76941b0a50 con 0x561aeac1b4e0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f76941afcb0 con 0x561aeac1bed0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f769d1b5740 10 client.0 _ll_register_callbacks cb 0x7f768c00b380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac2fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac2ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f7693fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac2e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f7698b66640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac2fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38116/0 (socket says 172.21.15.28:38116) 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f7698b66640 1 -- 172.21.15.28:0/1194567394 learned_addr learned my addr 172.21.15.28:0/1194567394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 -- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 msgr2=0x561aeac2e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac2e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 -- 172.21.15.28:0/1194567394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 msgr2=0x561aeac2fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:23.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac2fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.860+0000 7f76937fe640 1 -- 172.21.15.28:0/1194567394 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aeac300d0 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76937fe640 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac2ead0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f768803e000 tx=0x7f76880bc970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0b89b50ccf777a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76737fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768804a040 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f768c003e50 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f76880bb030 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f76880bd430 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f769d1b5740 10 client.29665 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561aeac9bde0 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.861+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f768c003e50 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f768804d060 con 0x561aeac1bed0 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 10 client.29665 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 10 client.29665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 20 client.29665 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac9c5a0 0x561aeac9e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 4 client.29665 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561aeab8d580 con 0x561aeac9c5a0 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f7693fff640 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac9c5a0 0x561aeac9e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f7693fff640 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac9c5a0 0x561aeac9e9c0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c53ce31d2bdf1a2 server_cookie=4893d2909f7ef28 in_seq=0 out_seq=0 2024-03-22T02:30:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:23.862+0000 7f76737fe640 10 client.29665 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:24.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 55701 15.90 MB/sec execute 64 sec latency 684.538 ms 2024-03-22T02:30:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:24 smithi120 ceph-mon[36803]: pgmap v866: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 500 op/s 2024-03-22T02:30:24.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:24 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f76737fe640 1 -- 172.21.15.28:0/1194567394 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561aeab8d580 con 0x561aeac9c5a0 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f76737fe640 10 client.29665 handle_command_reply: tid=0 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 client.29665 shutdown 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 msgr2=0x561aeac2ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac2ead0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f768803e000 tx=0x7f76880bc970 comp rx=0 tx=0).stop 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 shutdown_connections 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561aeac9c5a0 0x561aeac9e9c0 crc :-1 s=READY pgs=385 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561aeac21040 0x561aeac2fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561aeac1bed0 0x561aeac2ead0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 --2- 172.21.15.28:0/1194567394 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561aeac1b4e0 0x561aeac2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.759 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 >> 172.21.15.28:0/1194567394 conn(0x561aeac16a50 msgr2=0x561aeac1d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f76737fe640 0 client.29665 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.758+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 shutdown_connections 2024-03-22T02:30:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.759+0000 7f769d1b5740 1 -- 172.21.15.28:0/1194567394 wait complete. 2024-03-22T02:30:24.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.759+0000 7f769d1b5740 20 client.29665 trim_cache size 0 max 16384 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.759+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 msgr2=0x7f767c0a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.759+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 0x7f767c0a5470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x561aeac1c730 tx=0x7f768c00b710 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 msgr2=0x7f769419e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7688005ab0 tx=0x7f768800aae0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 shutdown_connections 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f767c0a2fb0 0x7f767c0a5470 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f769410e260 0x7f76941a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f769410b7c0 0x7f769419e950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 --2- 172.21.15.28:0/3242468008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7694108e00 0x7f769419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 >> 172.21.15.28:0/3242468008 conn(0x7f769406ebd0 msgr2=0x7f76941091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 shutdown_connections 2024-03-22T02:30:24.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:24.760+0000 7f769a5c9640 1 -- 172.21.15.28:0/3242468008 wait complete. 2024-03-22T02:30:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:24 smithi083 ceph-mon[36340]: pgmap v866: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 500 op/s 2024-03-22T02:30:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:24 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:24 smithi028 ceph-mon[31916]: pgmap v866: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 500 op/s 2024-03-22T02:30:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:24 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:30:25.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 56027 16.02 MB/sec execute 65 sec latency 592.537 ms 2024-03-22T02:30:25.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:25.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:30:25.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:30:26.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 56338 16.14 MB/sec execute 66 sec latency 253.481 ms 2024-03-22T02:30:26.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:26 smithi120 ceph-mon[36803]: pgmap v867: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-22T02:30:26.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:26 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:30:26.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:26 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:30:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:26 smithi083 ceph-mon[36340]: pgmap v867: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-22T02:30:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:26 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:30:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:26 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:30:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:26 smithi028 ceph-mon[31916]: pgmap v867: 129 pgs: 129 active+clean; 268 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.2 MiB/s wr, 481 op/s 2024-03-22T02:30:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:26 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi028 to 177.2M 2024-03-22T02:30:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:26 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi028 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-03-22T02:30:27.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 56630 16.10 MB/sec execute 67 sec latency 84.920 ms 2024-03-22T02:30:28.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 56973 16.13 MB/sec execute 68 sec latency 61.007 ms 2024-03-22T02:30:28.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:28 smithi120 ceph-mon[36803]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 375 op/s 2024-03-22T02:30:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:28 smithi083 ceph-mon[36340]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 375 op/s 2024-03-22T02:30:28.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:28 smithi028 ceph-mon[31916]: pgmap v868: 129 pgs: 129 active+clean; 257 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 375 op/s 2024-03-22T02:30:29.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 57237 16.08 MB/sec execute 69 sec latency 97.920 ms 2024-03-22T02:30:30.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 57538 16.07 MB/sec execute 70 sec latency 54.900 ms 2024-03-22T02:30:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:30 smithi120 ceph-mon[36803]: pgmap v869: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2024-03-22T02:30:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:30 smithi083 ceph-mon[36340]: pgmap v869: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2024-03-22T02:30:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:30 smithi028 ceph-mon[31916]: pgmap v869: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 521 op/s 2024-03-22T02:30:31.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 57732 16.01 MB/sec execute 71 sec latency 326.621 ms 2024-03-22T02:30:32.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 57996 15.95 MB/sec execute 72 sec latency 183.794 ms 2024-03-22T02:30:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:32 smithi120 ceph-mon[36803]: pgmap v870: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-03-22T02:30:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:32 smithi083 ceph-mon[36340]: pgmap v870: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-03-22T02:30:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:32 smithi028 ceph-mon[31916]: pgmap v870: 129 pgs: 129 active+clean; 256 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 513 op/s 2024-03-22T02:30:33.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 58435 16.03 MB/sec execute 73 sec latency 61.909 ms 2024-03-22T02:30:34.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 58778 16.10 MB/sec execute 74 sec latency 80.196 ms 2024-03-22T02:30:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:34 smithi120 ceph-mon[36803]: pgmap v871: 129 pgs: 129 active+clean; 262 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-03-22T02:30:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:34 smithi083 ceph-mon[36340]: pgmap v871: 129 pgs: 129 active+clean; 262 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-03-22T02:30:34.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:34 smithi028 ceph-mon[31916]: pgmap v871: 129 pgs: 129 active+clean; 262 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 5.1 MiB/s wr, 542 op/s 2024-03-22T02:30:35.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 59198 16.07 MB/sec execute 75 sec latency 60.226 ms 2024-03-22T02:30:36.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 59442 16.13 MB/sec execute 76 sec latency 353.258 ms 2024-03-22T02:30:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:36 smithi120 ceph-mon[36803]: pgmap v872: 129 pgs: 129 active+clean; 263 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-03-22T02:30:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:36 smithi083 ceph-mon[36340]: pgmap v872: 129 pgs: 129 active+clean; 263 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-03-22T02:30:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:36 smithi028 ceph-mon[31916]: pgmap v872: 129 pgs: 129 active+clean; 263 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 5.0 MiB/s wr, 544 op/s 2024-03-22T02:30:37.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 59705 16.10 MB/sec execute 77 sec latency 88.131 ms 2024-03-22T02:30:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 -- 172.21.15.28:0/777612961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530102680 msgr2=0x7fb530108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 --2- 172.21.15.28:0/777612961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530102680 0x7fb530108f10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb528009ca0 tx=0x7fb52802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 -- 172.21.15.28:0/777612961 shutdown_connections 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 --2- 172.21.15.28:0/777612961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530102680 0x7fb530108f10 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 --2- 172.21.15.28:0/777612961 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 0x7fb530102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 --2- 172.21.15.28:0/777612961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530100ad0 0x7fb530100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 -- 172.21.15.28:0/777612961 >> 172.21.15.28:0/777612961 conn(0x7fb5300fc040 msgr2=0x7fb5300fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 -- 172.21.15.28:0/777612961 shutdown_connections 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.861+0000 7fb536f1e640 1 -- 172.21.15.28:0/777612961 wait complete. 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 Processor -- start 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 -- start start 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 0x7fb5301971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 0x7fb53019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c750 con 0x7fb530100ad0 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c5d0 con 0x7fb530101cc0 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb536f1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb53010c8d0 con 0x7fb530102680 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47158/0 (socket says 172.21.15.28:47158) 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 -- 172.21.15.28:0/871463640 learned_addr learned my addr 172.21.15.28:0/871463640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb52ffff640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 0x7fb5301971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb5354bb640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 0x7fb53019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 -- 172.21.15.28:0/871463640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 msgr2=0x7fb53019e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 0x7fb53019e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 -- 172.21.15.28:0/871463640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 msgr2=0x7fb5301971e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 0x7fb5301971e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 -- 172.21.15.28:0/871463640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53019e8e0 con 0x7fb530100ad0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb5354bb640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 0x7fb53019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb534cba640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb52400e9c0 tx=0x7fb52400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.862+0000 7fb52dffb640 1 -- 172.21.15.28:0/871463640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52400cd80 con 0x7fb530100ad0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.863+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53019ebd0 con 0x7fb530100ad0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.863+0000 7fb52dffb640 1 -- 172.21.15.28:0/871463640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5240035f0 con 0x7fb530100ad0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.863+0000 7fb52dffb640 1 -- 172.21.15.28:0/871463640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb524010690 con 0x7fb530100ad0 2024-03-22T02:30:37.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.863+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53019f120 con 0x7fb530100ad0 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb52dffb640 1 -- 172.21.15.28:0/871463640 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb5240037a0 con 0x7fb530100ad0 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb52dffb640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 0x7fb5200a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb539be6740 1 Processor -- start 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb52dffb640 1 -- 172.21.15.28:0/871463640 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb5240cfc70 con 0x7fb530100ad0 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb52ffff640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 0x7fb5200a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb539be6740 1 -- start start 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.865+0000 7fb539be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7ca6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb539be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cabbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb539be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb539be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567a7c09040 con 0x5567a7cac250 2024-03-22T02:30:37.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb539be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5567a7bb19e0 con 0x5567a7ca66f0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb539be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5567a7c04ff0 con 0x5567a7ca70e0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47164/0 (socket says 172.21.15.28:47164) 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 -- 172.21.15.28:0/2273010387 learned_addr learned my addr 172.21.15.28:0/2273010387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb534cba640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7ca6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb52ffff640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cabbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb52ffff640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 0x7fb5200a5500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb5301981a0 tx=0x7fb51c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 -- 172.21.15.28:0/2273010387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 msgr2=0x5567a7cabbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cabbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 -- 172.21.15.28:0/2273010387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 msgr2=0x5567a7ca6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7ca6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 -- 172.21.15.28:0/2273010387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a791d210 con 0x5567a7cac250 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb534cba640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7ca6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb5354bb640 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fb5280324c0 tx=0x7fb528005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245a2a6825c422a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb51affd640 1 -- 172.21.15.28:0/2273010387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528043020 con 0x5567a7cac250 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb51affd640 1 -- 172.21.15.28:0/2273010387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb528032c90 con 0x5567a7cac250 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.866+0000 7fb51affd640 1 -- 172.21.15.28:0/2273010387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528039040 con 0x5567a7cac250 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 -- 172.21.15.28:0/2273010387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 msgr2=0x5567a7cae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fb5280324c0 tx=0x7fb528005f70 comp rx=0 tx=0).stop 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 -- 172.21.15.28:0/2273010387 shutdown_connections 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7cac250 0x5567a7cae6e0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cabbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 --2- 172.21.15.28:0/2273010387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7ca6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 -- 172.21.15.28:0/2273010387 >> 172.21.15.28:0/2273010387 conn(0x5567a7c9fa50 msgr2=0x5567a7c9fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 -- 172.21.15.28:0/2273010387 shutdown_connections 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.867+0000 7fb539be6740 1 -- 172.21.15.28:0/2273010387 wait complete. 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 -- start start 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 Processor -- start 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cb0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7cb7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5567a7caf1a0 con 0x5567a7ca66f0 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5567a7bc4db0 con 0x5567a7cac250 2024-03-22T02:30:37.876 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5567a7caf320 con 0x5567a7ca70e0 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb539be6740 10 client.0 _ll_register_callbacks cb 0x5567a791ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47180/0 (socket says 172.21.15.28:47180) 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 -- 172.21.15.28:0/570670041 learned_addr learned my addr 172.21.15.28:0/570670041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb52ffff640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cb0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb5354bb640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7cb7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 -- 172.21.15.28:0/570670041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 msgr2=0x5567a7cb0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cb0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.877 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 -- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 msgr2=0x5567a7cb7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7cb7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.868+0000 7fb534cba640 1 -- 172.21.15.28:0/570670041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a79e9620 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb534cba640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fb52409bec0 tx=0x7fb52409c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6893fcc97523e243 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5240b4070 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb524003b10 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb528032c90 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb5240a4940 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 10 client.19875 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5567a7cbb010 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb528032c90 con 0x5567a7ca66f0 2024-03-22T02:30:37.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb524097c80 con 0x5567a7ca66f0 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 10 client.19875 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 10 client.19875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 20 client.19875 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7cbc7a0 0x5567a7cbebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 4 client.19875 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567a7c16580 con 0x5567a7cbc7a0 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.869+0000 7fb52ffff640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7cbc7a0 0x5567a7cbebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.870+0000 7fb50ffff640 10 client.19875 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.870+0000 7fb52ffff640 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7cbc7a0 0x5567a7cbebf0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c751295e25a46c1c server_cookie=5984aa5f43898087 in_seq=0 out_seq=0 2024-03-22T02:30:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.873+0000 7fb50ffff640 1 -- 172.21.15.28:0/570670041 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567a7c16580 con 0x5567a7cbc7a0 2024-03-22T02:30:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.873+0000 7fb50ffff640 10 client.19875 handle_command_reply: tid=0 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.873+0000 7fb539be6740 1 client.19875 shutdown 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 msgr2=0x5567a7cb06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fb52409bec0 tx=0x7fb52409c6f0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 shutdown_connections 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7cbc7a0 0x5567a7cbebf0 crc :-1 s=READY pgs=386 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7cb7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7cb0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 --2- 172.21.15.28:0/570670041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7cb06c0 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb50ffff640 0 client.19875 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 >> 172.21.15.28:0/570670041 conn(0x5567a7c9fa50 msgr2=0x5567a7c9ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 shutdown_connections 2024-03-22T02:30:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 1 -- 172.21.15.28:0/570670041 wait complete. 2024-03-22T02:30:37.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.874+0000 7fb539be6740 20 client.19875 trim_cache size 0 max 16384 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 Processor -- start 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 -- start start 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7cb09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7cac250 0x5567a7c9e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301aaf70 con 0x5567a7ca70e0 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a8840 con 0x5567a7ca66f0 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb539be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a6690 con 0x5567a7cac250 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47186/0 (socket says 172.21.15.28:47186) 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb52ffff640 1 -- 172.21.15.28:0/3571750400 learned_addr learned my addr 172.21.15.28:0/3571750400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb5354bb640 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7cac250 0x5567a7c9e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.890+0000 7fb534cba640 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7cb09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 -- 172.21.15.28:0/3571750400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7cac250 msgr2=0x5567a7c9e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7cac250 0x5567a7c9e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 -- 172.21.15.28:0/3571750400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 msgr2=0x5567a7cb09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7cb09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 -- 172.21.15.28:0/3571750400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a7cbf270 con 0x5567a7ca70e0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb52ffff640 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0118d0 tx=0x7fb51c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9d10dd050a52a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb51a7fc640 1 -- 172.21.15.28:0/3571750400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c005220 con 0x5567a7ca70e0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb51a7fc640 1 -- 172.21.15.28:0/3571750400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51c004c80 con 0x5567a7ca70e0 2024-03-22T02:30:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.891+0000 7fb51a7fc640 1 -- 172.21.15.28:0/3571750400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c02c670 con 0x5567a7ca70e0 2024-03-22T02:30:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- 172.21.15.28:0/3571750400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 msgr2=0x5567a7cb7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fb51c0118d0 tx=0x7fb51c016000 comp rx=0 tx=0).stop 2024-03-22T02:30:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- 172.21.15.28:0/3571750400 shutdown_connections 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7cac250 0x5567a7c9e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca70e0 0x5567a7cb7b60 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 --2- 172.21.15.28:0/3571750400 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7ca66f0 0x5567a7cb09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- 172.21.15.28:0/3571750400 >> 172.21.15.28:0/3571750400 conn(0x5567a7c9fa50 msgr2=0x5567a7cb02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- 172.21.15.28:0/3571750400 shutdown_connections 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- 172.21.15.28:0/3571750400 wait complete. 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 -- start start 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.892+0000 7fb539be6740 1 Processor -- start 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7ca5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7ca47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a6d00 con 0x5567a7ca66f0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a8170 con 0x5567a7cac250 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb5301a73d0 con 0x5567a7ca70e0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb539be6740 10 client.0 _ll_register_callbacks cb 0x7fb524003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:34614/0 (socket says 172.21.15.28:34614) 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 -- 172.21.15.28:0/1579492844 learned_addr learned my addr 172.21.15.28:0/1579492844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb5354bb640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7ca47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb534cba640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7ca5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 -- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 msgr2=0x5567a7ca47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7ca47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 -- 172.21.15.28:0/1579492844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 msgr2=0x5567a7ca5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7ca5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 -- 172.21.15.28:0/1579492844 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a7ca4f10 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb52ffff640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb51c01c590 tx=0x7fb51c005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d213fc60f40796b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.893+0000 7fb50ffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51c004840 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb51c0049e0 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb51c004c80 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb51c045050 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 10 client.29683 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5567a7cb7f60 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb51c004c80 con 0x5567a7ca70e0 2024-03-22T02:30:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb51c004fd0 con 0x5567a7ca70e0 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 10 client.29683 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 10 client.29683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 20 client.29683 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7d25730 0x5567a7d27b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 4 client.29683 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.894+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5567a7c108d0 con 0x5567a7d25730 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.895+0000 7fb534cba640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7d25730 0x5567a7d27b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.896+0000 7fb534cba640 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7d25730 0x5567a7d27b50 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ced2bac61e80b3df server_cookie=a34462f7b06bf6b4 in_seq=0 out_seq=0 2024-03-22T02:30:37.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:37.896+0000 7fb50ffff640 10 client.29683 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:38.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 59830 16.06 MB/sec execute 78 sec latency 661.417 ms 2024-03-22T02:30:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:38 smithi120 ceph-mon[36803]: pgmap v873: 129 pgs: 129 active+clean; 265 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 523 op/s 2024-03-22T02:30:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:38 smithi083 ceph-mon[36340]: pgmap v873: 129 pgs: 129 active+clean; 265 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 523 op/s 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb50ffff640 1 -- 172.21.15.28:0/1579492844 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5567a7c108d0 con 0x5567a7d25730 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb50ffff640 10 client.29683 handle_command_reply: tid=0 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 client.29683 shutdown 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 msgr2=0x5567a7ca4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb51c01c590 tx=0x7fb51c005dc0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 shutdown_connections 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5567a7d25730 0x5567a7d27b50 crc :-1 s=READY pgs=387 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5567a7cac250 0x5567a7ca47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5567a7ca70e0 0x5567a7ca4260 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 --2- 172.21.15.28:0/1579492844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5567a7ca66f0 0x5567a7ca5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 >> 172.21.15.28:0/1579492844 conn(0x5567a7c9fa50 msgr2=0x5567a7ca7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.795+0000 7fb50ffff640 0 client.29683 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.796+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 shutdown_connections 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.796+0000 7fb539be6740 1 -- 172.21.15.28:0/1579492844 wait complete. 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.796+0000 7fb539be6740 20 client.29683 trim_cache size 0 max 16384 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:30:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:30:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:30:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:30:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.796+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 msgr2=0x7fb5200a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 0x7fb5200a5500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb5301981a0 tx=0x7fb51c005f90 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 msgr2=0x7fb530196ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fb52400e9c0 tx=0x7fb52400ee80 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 shutdown_connections 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb5200a3040 0x7fb5200a5500 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb530102680 0x7fb53019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb530101cc0 0x7fb5301971e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 --2- 172.21.15.28:0/871463640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb530100ad0 0x7fb530196ca0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 >> 172.21.15.28:0/871463640 conn(0x7fb5300fc040 msgr2=0x7fb530104ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 shutdown_connections 2024-03-22T02:30:38.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:38.797+0000 7fb536f1e640 1 -- 172.21.15.28:0/871463640 wait complete. 2024-03-22T02:30:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:38 smithi028 ceph-mon[31916]: pgmap v873: 129 pgs: 129 active+clean; 265 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 523 op/s 2024-03-22T02:30:39.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 60109 16.06 MB/sec execute 79 sec latency 704.143 ms 2024-03-22T02:30:40.635 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:40 smithi083 ceph-mon[36340]: pgmap v874: 129 pgs: 129 active+clean; 267 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-22T02:30:40.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 60388 16.06 MB/sec execute 80 sec latency 90.177 ms 2024-03-22T02:30:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:40 smithi120 ceph-mon[36803]: pgmap v874: 129 pgs: 129 active+clean; 267 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-22T02:30:40.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:40 smithi028 ceph-mon[31916]: pgmap v874: 129 pgs: 129 active+clean; 267 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-22T02:30:41.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 60545 15.91 MB/sec execute 81 sec latency 286.276 ms 2024-03-22T02:30:42.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 60806 15.83 MB/sec execute 82 sec latency 144.254 ms 2024-03-22T02:30:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:42 smithi120 ceph-mon[36803]: pgmap v875: 129 pgs: 129 active+clean; 264 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 471 op/s 2024-03-22T02:30:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:42 smithi083 ceph-mon[36340]: pgmap v875: 129 pgs: 129 active+clean; 264 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 471 op/s 2024-03-22T02:30:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:42 smithi028 ceph-mon[31916]: pgmap v875: 129 pgs: 129 active+clean; 264 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 471 op/s 2024-03-22T02:30:43.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 61079 15.81 MB/sec execute 83 sec latency 85.319 ms 2024-03-22T02:30:44.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 61373 15.84 MB/sec execute 84 sec latency 195.556 ms 2024-03-22T02:30:44.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:44 smithi120 ceph-mon[36803]: pgmap v876: 129 pgs: 129 active+clean; 256 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 521 op/s 2024-03-22T02:30:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:44 smithi083 ceph-mon[36340]: pgmap v876: 129 pgs: 129 active+clean; 256 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 521 op/s 2024-03-22T02:30:44.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:44 smithi028 ceph-mon[31916]: pgmap v876: 129 pgs: 129 active+clean; 256 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.9 MiB/s wr, 521 op/s 2024-03-22T02:30:45.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 61782 15.85 MB/sec execute 85 sec latency 102.033 ms 2024-03-22T02:30:46.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 61969 15.84 MB/sec execute 86 sec latency 266.593 ms 2024-03-22T02:30:46.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:46 smithi120 ceph-mon[36803]: pgmap v877: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2024-03-22T02:30:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:46 smithi083 ceph-mon[36340]: pgmap v877: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2024-03-22T02:30:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:46 smithi028 ceph-mon[31916]: pgmap v877: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.2 MiB/s wr, 551 op/s 2024-03-22T02:30:47.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 62258 15.87 MB/sec execute 87 sec latency 102.053 ms 2024-03-22T02:30:48.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 62720 15.90 MB/sec execute 88 sec latency 56.155 ms 2024-03-22T02:30:48.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:48 smithi120 ceph-mon[36803]: pgmap v878: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-03-22T02:30:48.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:48 smithi083 ceph-mon[36340]: pgmap v878: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-03-22T02:30:48.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:48 smithi028 ceph-mon[31916]: pgmap v878: 129 pgs: 129 active+clean; 261 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-03-22T02:30:49.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 63053 15.96 MB/sec execute 89 sec latency 72.658 ms 2024-03-22T02:30:50.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 63386 16.04 MB/sec execute 90 sec latency 74.764 ms 2024-03-22T02:30:50.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:50 smithi120 ceph-mon[36803]: pgmap v879: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.8 MiB/s wr, 574 op/s 2024-03-22T02:30:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:50 smithi083 ceph-mon[36340]: pgmap v879: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.8 MiB/s wr, 574 op/s 2024-03-22T02:30:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:50 smithi028 ceph-mon[31916]: pgmap v879: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.8 MiB/s wr, 574 op/s 2024-03-22T02:30:51.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 63625 16.00 MB/sec execute 91 sec latency 227.218 ms 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- 172.21.15.28:0/1913821172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84104fd0 msgr2=0x7f9b841073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/1913821172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84104fd0 0x7f9b841073c0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f9b74009900 tx=0x7f9b7402bc30 comp rx=0 tx=0).stop 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- 172.21.15.28:0/1913821172 shutdown_connections 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/1913821172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84107900 0x7f9b84109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/1913821172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84104fd0 0x7f9b841073c0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/1913821172 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b841026a0 0x7f9b84104a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- 172.21.15.28:0/1913821172 >> 172.21.15.28:0/1913821172 conn(0x7f9b840fc030 msgr2=0x7f9b840fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- 172.21.15.28:0/1913821172 shutdown_connections 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- 172.21.15.28:0/1913821172 wait complete. 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 Processor -- start 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.901+0000 7f9b8bf93640 1 -- start start 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 0x7f9b84102010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b84107900 0x7f9b84100b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8410d810 con 0x7f9b841026a0 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8410d690 con 0x7f9b84107900 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8bf93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8410d990 con 0x7f9b84104fd0 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 0x7f9b84102010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55654/0 (socket says 172.21.15.28:55654) 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 0x7f9b84102010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38092/0 (socket says 172.21.15.28:38092) 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 -- 172.21.15.28:0/234200784 learned_addr learned my addr 172.21.15.28:0/234200784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8a530640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b84107900 0x7f9b84100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 -- 172.21.15.28:0/234200784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b84107900 msgr2=0x7f9b84100b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b84107900 0x7f9b84100b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 -- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 msgr2=0x7f9b84102010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 0x7f9b84102010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 -- 172.21.15.28:0/234200784 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b84101420 con 0x7f9b84104fd0 2024-03-22T02:30:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.902+0000 7f9b8952e640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9b74000c00 tx=0x7f9b7402c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.903+0000 7f9b7affd640 1 -- 172.21.15.28:0/234200784 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74041070 con 0x7f9b84104fd0 2024-03-22T02:30:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.903+0000 7f9b7affd640 1 -- 172.21.15.28:0/234200784 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b74004450 con 0x7f9b84104fd0 2024-03-22T02:30:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.903+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b841a1eb0 con 0x7f9b84104fd0 2024-03-22T02:30:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.903+0000 7f9b7affd640 1 -- 172.21.15.28:0/234200784 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74034440 con 0x7f9b84104fd0 2024-03-22T02:30:51.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.903+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b841a2320 con 0x7f9b84104fd0 2024-03-22T02:30:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b8eb80740 1 Processor -- start 2024-03-22T02:30:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b7affd640 1 -- 172.21.15.28:0/234200784 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f9b74044070 con 0x7f9b84104fd0 2024-03-22T02:30:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b8eb80740 1 -- start start 2024-03-22T02:30:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41c8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.906+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624d412d040 con 0x5624d41c84e0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5624d40d59e0 con 0x5624d41ce040 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624d4128ff0 con 0x5624d41c8ed0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55202/0 (socket says 172.21.15.28:55202) 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8952e640 1 -- 172.21.15.28:0/305155187 learned_addr learned my addr 172.21.15.28:0/305155187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b7affd640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 0x7f9b700a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8a530640 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41c8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b8a530640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 0x7f9b700a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b7affd640 1 -- 172.21.15.28:0/234200784 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f9b740f11a0 con 0x7f9b84104fd0 2024-03-22T02:30:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 -- 172.21.15.28:0/305155187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 msgr2=0x5624d41d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 -- 172.21.15.28:0/305155187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 msgr2=0x5624d41c8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41c8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 -- 172.21.15.28:0/305155187 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d3e41210 con 0x5624d41c8ed0 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.907+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41cd9d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c20 tx=0x7f9b80011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33648405ca9df3c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.908+0000 7f9b6ffff640 1 -- 172.21.15.28:0/305155187 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80004050 con 0x5624d41c8ed0 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.908+0000 7f9b6ffff640 1 -- 172.21.15.28:0/305155187 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b80005500 con 0x5624d41c8ed0 2024-03-22T02:30:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.908+0000 7f9b6ffff640 1 -- 172.21.15.28:0/305155187 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80004dd0 con 0x5624d41c8ed0 2024-03-22T02:30:51.913 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- 172.21.15.28:0/305155187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 msgr2=0x5624d41cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8a530640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 0x7f9b700a5500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x5624d41c9730 tx=0x7f9b7c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41cd9d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009c20 tx=0x7f9b80011810 comp rx=0 tx=0).stop 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- 172.21.15.28:0/305155187 shutdown_connections 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41cd9d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/305155187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41c8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- 172.21.15.28:0/305155187 >> 172.21.15.28:0/305155187 conn(0x5624d41c3a50 msgr2=0x5624d41c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- 172.21.15.28:0/305155187 shutdown_connections 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- 172.21.15.28:0/305155187 wait complete. 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 -- start start 2024-03-22T02:30:51.914 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.909+0000 7f9b8eb80740 1 Processor -- start 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5624d41d0ff0 con 0x5624d41c84e0 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5624d40e8db0 con 0x5624d41ce040 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5624d41d1170 con 0x5624d41c8ed0 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38100/0 (socket says 172.21.15.28:38100) 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 -- 172.21.15.28:0/3786830734 learned_addr learned my addr 172.21.15.28:0/3786830734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8eb80740 10 client.0 _ll_register_callbacks cb 0x5624d3e40a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 -- 172.21.15.28:0/3786830734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 msgr2=0x5624d41d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.915 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 -- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 msgr2=0x5624d41d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8a530640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b89d2f640 1 -- 172.21.15.28:0/3786830734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d3f0d620 con 0x5624d41c84e0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8952e640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.910+0000 7f9b8a530640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009700 tx=0x7f9b800098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d85bc79781d14be server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80029070 con 0x5624d41c84e0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f9b80004d60 con 0x5624d41c84e0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b80005500 con 0x5624d41c84e0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9b80035050 con 0x5624d41c84e0 2024-03-22T02:30:51.916 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b8eb80740 10 client.19893 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.911+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5624d41dcd80 con 0x5624d41c84e0 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b800035f0 con 0x5624d41c84e0 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f9b80023430 con 0x5624d41c84e0 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 10 client.19893 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 10 client.19893 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 20 client.19893 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d41e0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 4 client.19893 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9b80005500 con 0x5624d41de250 2024-03-22T02:30:51.917 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.912+0000 7f9b8952e640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d41e0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.913+0000 7f9b8952e640 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d41e0640 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f28bc57fa27b2048 server_cookie=c32d0b6877e066b in_seq=0 out_seq=0 2024-03-22T02:30:51.918 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.913+0000 7f9b6d7fa640 10 client.19893 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.914+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/3786830734 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f9b80005500 con 0x5624d41de250 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.914+0000 7f9b6d7fa640 10 client.19893 handle_command_reply: tid=0 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 client.19893 shutdown 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 msgr2=0x5624d41d2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009700 tx=0x7f9b800098d0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 shutdown_connections 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d41e0640 crc :-1 s=READY pgs=388 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b6d7fa640 0 client.19893 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/3786830734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c84e0 0x5624d41d2520 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 >> 172.21.15.28:0/3786830734 conn(0x5624d41c3a50 msgr2=0x5624d41c2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 shutdown_connections 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 1 -- 172.21.15.28:0/3786830734 wait complete. 2024-03-22T02:30:51.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.915+0000 7f9b8eb80740 20 client.19893 trim_cache size 0 max 16384 2024-03-22T02:30:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 Processor -- start 2024-03-22T02:30:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 -- start start 2024-03-22T02:30:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 0x5624d41d1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841af480 con 0x5624d41ce040 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841acd50 con 0x5624d41c84e0 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.931+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841aaba0 con 0x5624d41c8ed0 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 0x5624d41d1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:55688/0 (socket says 172.21.15.28:55688) 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55224/0 (socket says 172.21.15.28:55224) 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8952e640 1 -- 172.21.15.28:0/1182115932 learned_addr learned my addr 172.21.15.28:0/1182115932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 -- 172.21.15.28:0/1182115932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 msgr2=0x5624d41d99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 -- 172.21.15.28:0/1182115932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 msgr2=0x5624d41c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 -- 172.21.15.28:0/1182115932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5624d40e3f40 con 0x5624d41ce040 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b8a530640 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 0x5624d41d1b40 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005c40 tx=0x7f9b7c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1219336020934df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b6f7fe640 1 -- 172.21.15.28:0/1182115932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c026070 con 0x5624d41ce040 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b6f7fe640 1 -- 172.21.15.28:0/1182115932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b7c013df0 con 0x5624d41ce040 2024-03-22T02:30:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.932+0000 7f9b6f7fe640 1 -- 172.21.15.28:0/1182115932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c0107b0 con 0x5624d41ce040 2024-03-22T02:30:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1182115932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 msgr2=0x5624d41d1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:51.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 0x5624d41d1b40 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005c40 tx=0x7f9b7c005bd0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1182115932 shutdown_connections 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41ce040 0x5624d41d1b40 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c8ed0 0x5624d41d99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1182115932 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41c84e0 0x5624d41c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1182115932 >> 172.21.15.28:0/1182115932 conn(0x5624d41c3a50 msgr2=0x5624d41ca1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.933+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1182115932 shutdown_connections 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1182115932 wait complete. 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 -- start start 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 Processor -- start 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 0x5624d41d6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841ab210 con 0x5624d41c8ed0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841ac680 con 0x5624d41ce040 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f9b841ab8e0 con 0x5624d41c84e0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8eb80740 10 client.0 _ll_register_callbacks cb 0x7f9b7c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b89d2f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 0x5624d41d6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8952e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38116/0 (socket says 172.21.15.28:38116) 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8a530640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.934+0000 7f9b8952e640 1 -- 172.21.15.28:0/1868942339 learned_addr learned my addr 172.21.15.28:0/1868942339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 -- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 msgr2=0x5624d41d81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 -- 172.21.15.28:0/1868942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 msgr2=0x5624d41d71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 -- 172.21.15.28:0/1868942339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624d41d87e0 con 0x5624d41c84e0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b8952e640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b89d2f640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 0x5624d41d6ca0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009730 tx=0x7f9b80004ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9fc5c09460df127 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80029040 con 0x5624d41c84e0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b7c013df0 con 0x5624d41c84e0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f9b80023430 con 0x5624d41c84e0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f9b80023c70 con 0x5624d41c84e0 2024-03-22T02:30:51.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b8eb80740 10 client.29701 fetch_fsmap learned FSMap version 46 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5624d41d9af0 con 0x5624d41c84e0 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.935+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b7c013df0 con 0x5624d41c84e0 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f9b80004d60 con 0x5624d41c84e0 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 10 client.29701 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 10 client.29701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 20 client.29701 populate_metadata read hostname 'smithi028' 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d4246b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 4 client.29701 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5624d413a580 con 0x5624d41de250 2024-03-22T02:30:51.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8952e640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d4246b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:30:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b6d7fa640 10 client.29701 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:51.943 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:51.936+0000 7f9b8952e640 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d4246b90 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8bb691bd4f5c327c server_cookie=d7567cc100da4029 in_seq=0 out_seq=0 2024-03-22T02:30:52.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 63771 15.93 MB/sec execute 92 sec latency 464.414 ms 2024-03-22T02:30:52.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:52 smithi120 ceph-mon[36803]: pgmap v880: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 434 op/s 2024-03-22T02:30:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:52 smithi083 ceph-mon[36340]: pgmap v880: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 434 op/s 2024-03-22T02:30:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:52 smithi028 ceph-mon[31916]: pgmap v880: 129 pgs: 129 active+clean; 266 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.1 MiB/s wr, 434 op/s 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.123+0000 7f9b6d7fa640 1 -- 172.21.15.28:0/1868942339 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5624d413a580 con 0x5624d41de250 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.123+0000 7f9b6d7fa640 10 client.29701 handle_command_reply: tid=0 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.123+0000 7f9b8eb80740 1 client.29701 shutdown 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.123+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 msgr2=0x5624d41d6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.123+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 0x5624d41d6ca0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009730 tx=0x7f9b80004ae0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 shutdown_connections 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5624d41de250 0x5624d4246b90 crc :-1 s=READY pgs=389 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5624d41ce040 0x5624d41d81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b6d7fa640 0 client.29701 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624d41c8ed0 0x5624d41d71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 --2- 172.21.15.28:0/1868942339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5624d41c84e0 0x5624d41d6ca0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 >> 172.21.15.28:0/1868942339 conn(0x5624d41c3a50 msgr2=0x5624d41ca160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 shutdown_connections 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 1 -- 172.21.15.28:0/1868942339 wait complete. 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.124+0000 7f9b8eb80740 20 client.29701 trim_cache size 0 max 16384 2024-03-22T02:30:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:30:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 msgr2=0x7f9b700a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 0x7f9b700a5500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x5624d41c9730 tx=0x7f9b7c004ab0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 msgr2=0x7f9b84100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9b74000c00 tx=0x7f9b7402c640 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 shutdown_connections 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f9b700a3040 0x7f9b700a5500 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f9b84107900 0x7f9b84100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b84104fd0 0x7f9b84100650 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 --2- 172.21.15.28:0/234200784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b841026a0 0x7f9b84102010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.125+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 >> 172.21.15.28:0/234200784 conn(0x7f9b840fc030 msgr2=0x7f9b84102fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.126+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 shutdown_connections 2024-03-22T02:30:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:30:53.126+0000 7f9b8bf93640 1 -- 172.21.15.28:0/234200784 wait complete. 2024-03-22T02:30:53.680 INFO:tasks.workunit.client.0.smithi028.stdout: 10 63973 15.85 MB/sec execute 93 sec latency 654.620 ms 2024-03-22T02:30:54.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 64276 15.83 MB/sec execute 94 sec latency 82.036 ms 2024-03-22T02:30:54.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:54 smithi120 ceph-mon[36803]: pgmap v881: 129 pgs: 129 active+clean; 270 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-22T02:30:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:54 smithi083 ceph-mon[36340]: pgmap v881: 129 pgs: 129 active+clean; 270 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-22T02:30:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:54 smithi028 ceph-mon[31916]: pgmap v881: 129 pgs: 129 active+clean; 270 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.2 MiB/s wr, 422 op/s 2024-03-22T02:30:55.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 64655 15.91 MB/sec execute 95 sec latency 49.433 ms 2024-03-22T02:30:56.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 64832 15.80 MB/sec execute 96 sec latency 206.521 ms 2024-03-22T02:30:56.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:56 smithi120 ceph-mon[36803]: pgmap v882: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.7 MiB/s wr, 513 op/s 2024-03-22T02:30:56.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:56 smithi083 ceph-mon[36340]: pgmap v882: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.7 MiB/s wr, 513 op/s 2024-03-22T02:30:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:56 smithi028 ceph-mon[31916]: pgmap v882: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 4.7 MiB/s wr, 513 op/s 2024-03-22T02:30:57.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 65170 15.78 MB/sec execute 97 sec latency 163.326 ms 2024-03-22T02:30:58.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 65521 15.79 MB/sec execute 98 sec latency 58.577 ms 2024-03-22T02:30:58.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:30:58 smithi120 ceph-mon[36803]: pgmap v883: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 498 op/s 2024-03-22T02:30:58.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:30:58 smithi083 ceph-mon[36340]: pgmap v883: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 498 op/s 2024-03-22T02:30:58.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:30:58 smithi028 ceph-mon[31916]: pgmap v883: 129 pgs: 129 active+clean; 259 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.0 MiB/s wr, 498 op/s 2024-03-22T02:30:59.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 65968 15.94 MB/sec execute 99 sec latency 66.850 ms 2024-03-22T02:31:00.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 66413 15.99 MB/sec execute 100 sec latency 90.604 ms 2024-03-22T02:31:00.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:00 smithi120 ceph-mon[36803]: pgmap v884: 129 pgs: 129 active+clean; 258 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:00.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:00 smithi083 ceph-mon[36340]: pgmap v884: 129 pgs: 129 active+clean; 258 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:00 smithi028 ceph-mon[31916]: pgmap v884: 129 pgs: 129 active+clean; 258 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:01.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 66669 16.02 MB/sec execute 101 sec latency 142.792 ms 2024-03-22T02:31:02.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 66936 16.02 MB/sec execute 102 sec latency 98.618 ms 2024-03-22T02:31:02.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:02 smithi120 ceph-mon[36803]: pgmap v885: 129 pgs: 129 active+clean; 264 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-03-22T02:31:02.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:02 smithi083 ceph-mon[36340]: pgmap v885: 129 pgs: 129 active+clean; 264 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-03-22T02:31:02.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:02 smithi028 ceph-mon[31916]: pgmap v885: 129 pgs: 129 active+clean; 264 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 512 op/s 2024-03-22T02:31:03.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 67277 16.05 MB/sec execute 103 sec latency 49.103 ms 2024-03-22T02:31:04.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 67519 16.03 MB/sec execute 104 sec latency 155.532 ms 2024-03-22T02:31:04.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:04 smithi120 ceph-mon[36803]: pgmap v886: 129 pgs: 129 active+clean; 266 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 516 op/s 2024-03-22T02:31:04.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:04 smithi083 ceph-mon[36340]: pgmap v886: 129 pgs: 129 active+clean; 266 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 516 op/s 2024-03-22T02:31:04.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:04 smithi028 ceph-mon[31916]: pgmap v886: 129 pgs: 129 active+clean; 266 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 516 op/s 2024-03-22T02:31:05.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 67837 16.02 MB/sec execute 105 sec latency 96.832 ms 2024-03-22T02:31:06.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.222+0000 7f713f239640 1 -- 172.21.15.28:0/1498577121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138101ca0 msgr2=0x7f7138102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.222+0000 7f713f239640 1 --2- 172.21.15.28:0/1498577121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138101ca0 0x7f7138102120 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f7124009900 tx=0x7f712402bc30 comp rx=0 tx=0).stop 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 -- 172.21.15.28:0/1498577121 shutdown_connections 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 --2- 172.21.15.28:0/1498577121 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138102660 0x7f7138108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 --2- 172.21.15.28:0/1498577121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138101ca0 0x7f7138102120 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 --2- 172.21.15.28:0/1498577121 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 -- 172.21.15.28:0/1498577121 >> 172.21.15.28:0/1498577121 conn(0x7f71380fc020 msgr2=0x7f71380fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 -- 172.21.15.28:0/1498577121 shutdown_connections 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 -- 172.21.15.28:0/1498577121 wait complete. 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 Processor -- start 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.223+0000 7f713f239640 1 -- start start 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138101ca0 0x7f71381972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138102660 0x7f713819e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f713810c780 con 0x7f7138102660 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f713810c600 con 0x7f7138101ca0 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713f239640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f713810c900 con 0x7f7138100ab0 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f7137fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138101ca0 0x7f71381972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35020/0 (socket says 172.21.15.28:35020) 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 -- 172.21.15.28:0/4033571203 learned_addr learned my addr 172.21.15.28:0/4033571203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713d7d6640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138102660 0x7f713819e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 -- 172.21.15.28:0/4033571203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138101ca0 msgr2=0x7f71381972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138101ca0 0x7f71381972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 -- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138102660 msgr2=0x7f713819e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138102660 0x7f713819e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 -- 172.21.15.28:0/4033571203 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713819e990 con 0x7f7138100ab0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.224+0000 7f713cfd5640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f712c00ead0 tx=0x7f712c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.225+0000 7f7135ffb640 1 -- 172.21.15.28:0/4033571203 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c01c070 con 0x7f7138100ab0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.225+0000 7f7135ffb640 1 -- 172.21.15.28:0/4033571203 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f712c00bbd0 con 0x7f7138100ab0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.225+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7138069270 con 0x7f7138100ab0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.225+0000 7f7135ffb640 1 -- 172.21.15.28:0/4033571203 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f712c003de0 con 0x7f7138100ab0 2024-03-22T02:31:06.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.225+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7138069740 con 0x7f7138100ab0 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.227+0000 7f7141e29740 1 Processor -- start 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.227+0000 7f7141e29740 1 -- start start 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7135ffb640 1 -- 172.21.15.28:0/4033571203 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f712c01f070 con 0x7f7138100ab0 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705c2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560270527040 con 0x5602705c2ed0 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5602704cf9e0 con 0x5602705c24e0 2024-03-22T02:31:06.235 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7141e29740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560270522ff0 con 0x5602705c8040 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40628/0 (socket says 172.21.15.28:40628) 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35030/0 (socket says 172.21.15.28:35030) 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f713cfd5640 1 -- 172.21.15.28:0/2903105485 learned_addr learned my addr 172.21.15.28:0/2903105485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f713d7d6640 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705c2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 -- 172.21.15.28:0/2903105485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 msgr2=0x5602705c2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705c2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 -- 172.21.15.28:0/2903105485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 msgr2=0x5602705c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7137fff640 1 -- 172.21.15.28:0/2903105485 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56027023b210 con 0x5602705c8040 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f713cfd5640 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.236 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.228+0000 7f7135ffb640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 0x7f71280a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f713d7d6640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 0x7f71280a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f7137fff640 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7124036040 tx=0x7f712402c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40473a438768aa8b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f7135ffb640 1 -- 172.21.15.28:0/4033571203 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f712c0d0ef0 con 0x7f7138100ab0 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f7122ffd640 1 -- 172.21.15.28:0/2903105485 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124045070 con 0x5602705c8040 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f7122ffd640 1 -- 172.21.15.28:0/2903105485 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7124004040 con 0x5602705c8040 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.229+0000 7f7122ffd640 1 -- 172.21.15.28:0/2903105485 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124004eb0 con 0x5602705c8040 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- 172.21.15.28:0/2903105485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 msgr2=0x5602705ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7124036040 tx=0x7f712402c680 comp rx=0 tx=0).stop 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f713d7d6640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 0x7f71280a5480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5602705c3730 tx=0x7f7130009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- 172.21.15.28:0/2903105485 shutdown_connections 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c8040 0x5602705ca4d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 --2- 172.21.15.28:0/2903105485 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705c2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.237 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- 172.21.15.28:0/2903105485 >> 172.21.15.28:0/2903105485 conn(0x5602705bda50 msgr2=0x5602705bde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- 172.21.15.28:0/2903105485 shutdown_connections 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- 172.21.15.28:0/2903105485 wait complete. 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 -- start start 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.230+0000 7f7141e29740 1 Processor -- start 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 0x5602705cc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 0x5602705cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5602705caf90 con 0x5602705c8040 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5602704e2db0 con 0x5602705c2ed0 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5602705cb110 con 0x5602705c24e0 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7141e29740 10 client.0 _ll_register_callbacks cb 0x56027023aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:06.238 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 0x5602705cc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713d7d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 0x5602705cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 0x5602705cc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35036/0 (socket says 172.21.15.28:35036) 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713d7d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40642/0 (socket says 172.21.15.28:40642) 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713cfd5640 1 -- 172.21.15.28:0/102938894 learned_addr learned my addr 172.21.15.28:0/102938894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 -- 172.21.15.28:0/102938894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 msgr2=0x5602705cc510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 0x5602705cc510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 -- 172.21.15.28:0/102938894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 msgr2=0x5602705d3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 -- 172.21.15.28:0/102938894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560270307620 con 0x5602705c2ed0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f713d7d6640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7137fff640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 0x5602705cca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f712402c240 tx=0x7f71240041f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c86a20dc3d8715f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.231+0000 7f7117fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:31:06.239 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124045040 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7124004040 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f712402cb80 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7124059050 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 10 client.29717 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5602705d6e00 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7124004040 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7124003860 con 0x5602705c2ed0 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 10 client.29717 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 10 client.29717 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 20 client.29717 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705d8150 0x5602705da540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 4 client.29717 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:31:06.240 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560270534580 con 0x5602705d8150 2024-03-22T02:31:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.232+0000 7f713cfd5640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705d8150 0x5602705da540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.233+0000 7f713cfd5640 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705d8150 0x5602705da540 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e18321b94a3309e server_cookie=b864a9d5d0046431 in_seq=0 out_seq=0 2024-03-22T02:31:06.241 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.233+0000 7f7117fff640 10 client.29717 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7117fff640 1 -- 172.21.15.28:0/102938894 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560270534580 con 0x5602705d8150 2024-03-22T02:31:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7117fff640 10 client.29717 handle_command_reply: tid=0 2024-03-22T02:31:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 client.29717 shutdown 2024-03-22T02:31:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 msgr2=0x5602705cca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.242 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 0x5602705cca50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f712402c240 tx=0x7f71240041f0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 shutdown_connections 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705d8150 0x5602705da540 crc :-1 s=READY pgs=390 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c8040 0x5602705d3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c2ed0 0x5602705cca50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 --2- 172.21.15.28:0/102938894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c24e0 0x5602705cc510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7117fff640 0 client.29717 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.234+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 >> 172.21.15.28:0/102938894 conn(0x5602705bda50 msgr2=0x5602705bcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.235+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 shutdown_connections 2024-03-22T02:31:06.243 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.235+0000 7f7141e29740 1 -- 172.21.15.28:0/102938894 wait complete. 2024-03-22T02:31:06.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.235+0000 7f7141e29740 20 client.29717 trim_cache size 0 max 16384 2024-03-22T02:31:06.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.249+0000 7f7141e29740 1 Processor -- start 2024-03-22T02:31:06.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.249+0000 7f7141e29740 1 -- start start 2024-03-22T02:31:06.252 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 0x5602705c7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71381af320 con 0x5602705c2ed0 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f71381acbf0 con 0x5602705c24e0 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7141e29740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71381aaa40 con 0x5602705bda50 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40648/0 (socket says 172.21.15.28:40648) 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 -- 172.21.15.28:0/1387010061 learned_addr learned my addr 172.21.15.28:0/1387010061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713cfd5640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 0x5602705c7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7137fff640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 -- 172.21.15.28:0/1387010061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 msgr2=0x5602705c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 0x5602705c7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.253 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 -- 172.21.15.28:0/1387010061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 msgr2=0x5602705d3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 -- 172.21.15.28:0/1387010061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602705cb240 con 0x5602705c2ed0 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713cfd5640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 0x5602705c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f7137fff640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.250+0000 7f713d7d6640 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f7130004170 tx=0x7f713001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e637011eb185e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f71227fc640 1 -- 172.21.15.28:0/1387010061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713000f810 con 0x5602705c2ed0 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f71227fc640 1 -- 172.21.15.28:0/1387010061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7130029430 con 0x5602705c2ed0 2024-03-22T02:31:06.254 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f71227fc640 1 -- 172.21.15.28:0/1387010061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713001e670 con 0x5602705c2ed0 2024-03-22T02:31:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 -- 172.21.15.28:0/1387010061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 msgr2=0x5602705bcad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:06.257 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f7130004170 tx=0x7f713001a000 comp rx=0 tx=0).stop 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 -- 172.21.15.28:0/1387010061 shutdown_connections 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705c2ed0 0x5602705bcad0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 --2- 172.21.15.28:0/1387010061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705bda50 0x5602705c7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 -- 172.21.15.28:0/1387010061 >> 172.21.15.28:0/1387010061 conn(0x5602705cc0d0 msgr2=0x5602705bc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 -- 172.21.15.28:0/1387010061 shutdown_connections 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.251+0000 7f7141e29740 1 -- 172.21.15.28:0/1387010061 wait complete. 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 -- start start 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 Processor -- start 2024-03-22T02:31:06.258 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 0x5602705d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71381ab0b0 con 0x5602705bda50 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f71381ac520 con 0x5602705c24e0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f71381ab780 con 0x5602705c2ed0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7141e29740 10 client.0 _ll_register_callbacks cb 0x7f712c0d48a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f713cfd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40650/0 (socket says 172.21.15.28:40650) 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f713cfd5640 1 -- 172.21.15.28:0/6685107 learned_addr learned my addr 172.21.15.28:0/6685107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f713d7d6640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 0x5602705d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7137fff640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7137fff640 1 -- 172.21.15.28:0/6685107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 msgr2=0x5602705d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.259 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7137fff640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 0x5602705d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7137fff640 1 -- 172.21.15.28:0/6685107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 msgr2=0x5602705d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.252+0000 7f7137fff640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7137fff640 1 -- 172.21.15.28:0/6685107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602705d2da0 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f713d7d6640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 0x5602705d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f713cfd5640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7137fff640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d17a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f712402c7b0 tx=0x7f7124004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afffa030aedfe13d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124062060 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7130029430 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f712402cb90 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7124069050 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 10 client.29723 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56027063e4f0 con 0x5602705c24e0 2024-03-22T02:31:06.260 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7130029430 con 0x5602705c24e0 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7124005420 con 0x5602705c24e0 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 10 client.29723 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 10 client.29723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 20 client.29723 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705c8040 0x560270640f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 4 client.29723 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.253+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56027052e8d0 con 0x5602705c8040 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.254+0000 7f713cfd5640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705c8040 0x560270640f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.254+0000 7f713cfd5640 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705c8040 0x560270640f00 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=431dd4042a1ea099 server_cookie=a59c8a13f9eadcd9 in_seq=0 out_seq=0 2024-03-22T02:31:06.261 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:06.254+0000 7f7117fff640 10 client.29723 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:06.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 67992 15.94 MB/sec execute 106 sec latency 300.669 ms 2024-03-22T02:31:06.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:06 smithi120 ceph-mon[36803]: pgmap v887: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 507 op/s 2024-03-22T02:31:06.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:06 smithi083 ceph-mon[36340]: pgmap v887: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 507 op/s 2024-03-22T02:31:06.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:06 smithi028 ceph-mon[31916]: pgmap v887: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.1 MiB/s wr, 507 op/s 2024-03-22T02:31:07.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.494+0000 7f7117fff640 1 -- 172.21.15.28:0/6685107 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56027052e8d0 con 0x5602705c8040 2024-03-22T02:31:07.497 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.494+0000 7f7117fff640 10 client.29723 handle_command_reply: tid=0 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.494+0000 7f7141e29740 1 client.29723 shutdown 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 msgr2=0x5602705d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d17a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f712402c7b0 tx=0x7f7124004170 comp rx=0 tx=0).stop 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 shutdown_connections 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5602705c8040 0x560270640f00 crc :-1 s=READY pgs=391 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7117fff640 0 client.29723 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5602705c2ed0 0x5602705d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5602705c24e0 0x5602705d17a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 --2- 172.21.15.28:0/6685107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5602705bda50 0x5602705d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 >> 172.21.15.28:0/6685107 conn(0x5602705cc0d0 msgr2=0x5602705bc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:07.498 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 shutdown_connections 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 1 -- 172.21.15.28:0/6685107 wait complete. 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.495+0000 7f7141e29740 20 client.29723 trim_cache size 0 max 16384 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:31:07.499 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:31:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 msgr2=0x7f71280a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 0x7f71280a5480 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5602705c3730 tx=0x7f7130009500 comp rx=0 tx=0).stop 2024-03-22T02:31:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 msgr2=0x7f7138196d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:07.500 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f712c00ead0 tx=0x7f712c00ef90 comp rx=0 tx=0).stop 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 shutdown_connections 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f71280a2fc0 0x7f71280a5480 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7138102660 0x7f713819e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7138101ca0 0x7f71381972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 --2- 172.21.15.28:0/4033571203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7138100ab0 0x7f7138196d70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 >> 172.21.15.28:0/4033571203 conn(0x7f71380fc020 msgr2=0x7f7138104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 shutdown_connections 2024-03-22T02:31:07.501 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:07.496+0000 7f713f239640 1 -- 172.21.15.28:0/4033571203 wait complete. 2024-03-22T02:31:07.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 68111 15.83 MB/sec execute 107 sec latency 714.968 ms 2024-03-22T02:31:08.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 68359 15.81 MB/sec execute 108 sec latency 131.461 ms 2024-03-22T02:31:08.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:08 smithi120 ceph-mon[36803]: pgmap v888: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 353 op/s 2024-03-22T02:31:08.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:08 smithi083 ceph-mon[36340]: pgmap v888: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 353 op/s 2024-03-22T02:31:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:08 smithi028 ceph-mon[31916]: pgmap v888: 129 pgs: 129 active+clean; 268 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 353 op/s 2024-03-22T02:31:09.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 68686 15.74 MB/sec execute 109 sec latency 127.496 ms 2024-03-22T02:31:10.681 INFO:tasks.workunit.client.0.smithi028.stdout: 10 69046 15.79 MB/sec execute 110 sec latency 56.068 ms 2024-03-22T02:31:10.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:10 smithi120 ceph-mon[36803]: pgmap v889: 129 pgs: 129 active+clean; 272 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.5 MiB/s wr, 490 op/s 2024-03-22T02:31:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:10 smithi083 ceph-mon[36340]: pgmap v889: 129 pgs: 129 active+clean; 272 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.5 MiB/s wr, 490 op/s 2024-03-22T02:31:10.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:10 smithi028 ceph-mon[31916]: pgmap v889: 129 pgs: 129 active+clean; 272 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.5 MiB/s wr, 490 op/s 2024-03-22T02:31:11.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 69459 15.93 MB/sec execute 111 sec latency 89.450 ms 2024-03-22T02:31:12.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 69859 15.96 MB/sec execute 112 sec latency 169.337 ms 2024-03-22T02:31:12.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:12 smithi120 ceph-mon[36803]: pgmap v890: 129 pgs: 129 active+clean; 267 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 539 op/s 2024-03-22T02:31:12.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:12 smithi083 ceph-mon[36340]: pgmap v890: 129 pgs: 129 active+clean; 267 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 539 op/s 2024-03-22T02:31:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:12 smithi028 ceph-mon[31916]: pgmap v890: 129 pgs: 129 active+clean; 267 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 539 op/s 2024-03-22T02:31:13.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 70177 15.97 MB/sec execute 113 sec latency 70.978 ms 2024-03-22T02:31:14.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 70545 16.02 MB/sec execute 114 sec latency 78.709 ms 2024-03-22T02:31:14.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:14 smithi120 ceph-mon[36803]: pgmap v891: 129 pgs: 129 active+clean; 260 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:31:14.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:14 smithi083 ceph-mon[36340]: pgmap v891: 129 pgs: 129 active+clean; 260 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:31:14.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:14 smithi028 ceph-mon[31916]: pgmap v891: 129 pgs: 129 active+clean; 260 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:31:15.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 70822 16.04 MB/sec execute 115 sec latency 160.855 ms 2024-03-22T02:31:16.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 71074 16.04 MB/sec execute 116 sec latency 72.424 ms 2024-03-22T02:31:16.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:16 smithi120 ceph-mon[36803]: pgmap v892: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-03-22T02:31:16.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:16 smithi083 ceph-mon[36340]: pgmap v892: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-03-22T02:31:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:16 smithi028 ceph-mon[31916]: pgmap v892: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 572 op/s 2024-03-22T02:31:17.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 71284 15.99 MB/sec execute 117 sec latency 348.120 ms 2024-03-22T02:31:18.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 71553 15.96 MB/sec execute 118 sec latency 102.708 ms 2024-03-22T02:31:18.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:18 smithi120 ceph-mon[36803]: pgmap v893: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-03-22T02:31:18.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:18 smithi083 ceph-mon[36340]: pgmap v893: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-03-22T02:31:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:18 smithi028 ceph-mon[31916]: pgmap v893: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.8 MiB/s wr, 521 op/s 2024-03-22T02:31:19.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 71873 15.94 MB/sec execute 119 sec latency 137.369 ms 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 -- 172.21.15.28:0/187092574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c103770 msgr2=0x7f634c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 --2- 172.21.15.28:0/187092574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c103770 0x7f634c10a000 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f6344009ca0 tx=0x7f634402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 -- 172.21.15.28:0/187092574 shutdown_connections 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 --2- 172.21.15.28:0/187092574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c103770 0x7f634c10a000 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 --2- 172.21.15.28:0/187092574 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c102db0 0x7f634c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 --2- 172.21.15.28:0/187092574 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c101bc0 0x7f634c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 -- 172.21.15.28:0/187092574 >> 172.21.15.28:0/187092574 conn(0x7f634c0fd150 msgr2=0x7f634c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 -- 172.21.15.28:0/187092574 shutdown_connections 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 -- 172.21.15.28:0/187092574 wait complete. 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.605+0000 7f63515c9640 1 Processor -- start 2024-03-22T02:31:20.608 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 -- start start 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c101bc0 0x7f634c197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c103770 0x7f634c19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10d820 con 0x7f634c102db0 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10d6a0 con 0x7f634c101bc0 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f63515c9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f634c10d9a0 con 0x7f634c103770 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59812/0 (socket says 172.21.15.28:59812) 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 -- 172.21.15.28:0/712042268 learned_addr learned my addr 172.21.15.28:0/712042268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634affd640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c101bc0 0x7f634c197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634b7fe640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c103770 0x7f634c19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.609 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 -- 172.21.15.28:0/712042268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c103770 msgr2=0x7f634c19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c103770 0x7f634c19f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 -- 172.21.15.28:0/712042268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c101bc0 msgr2=0x7f634c197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c101bc0 0x7f634c197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 -- 172.21.15.28:0/712042268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c19f9d0 con 0x7f634c102db0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f634a7fc640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9f0 tx=0x7f634000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f6333fff640 1 -- 172.21.15.28:0/712042268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634000cd80 con 0x7f634c102db0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.606+0000 7f6333fff640 1 -- 172.21.15.28:0/712042268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6340010910 con 0x7f634c102db0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.607+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c19fcc0 con 0x7f634c102db0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.607+0000 7f6333fff640 1 -- 172.21.15.28:0/712042268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340018990 con 0x7f634c102db0 2024-03-22T02:31:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.607+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c1a01c0 con 0x7f634c102db0 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.609+0000 7f6333fff640 1 -- 172.21.15.28:0/712042268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6340010ac0 con 0x7f634c102db0 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.609+0000 7f63541b5740 1 Processor -- start 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.609+0000 7f6333fff640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 0x7f63340a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.609+0000 7f63541b5740 1 -- start start 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f6333fff640 1 -- 172.21.15.28:0/712042268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f63400cfb80 con 0x7f634c102db0 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186dfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 0x55e6186e28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e61863b040 con 0x55e6186e0450 2024-03-22T02:31:20.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e6185e39e0 con 0x55e6186db2e0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f63541b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e618636ff0 con 0x55e6186da8f0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634b7fe640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 0x7f63340a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56374/0 (socket says 172.21.15.28:56374) 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 -- 172.21.15.28:0/2478332375 learned_addr learned my addr 172.21.15.28:0/2478332375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634a7fc640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186dfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 -- 172.21.15.28:0/2478332375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 msgr2=0x55e6186dfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186dfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 -- 172.21.15.28:0/2478332375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 msgr2=0x55e6186e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634b7fe640 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 0x55e6186e28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 0x55e6186e28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634affd640 1 -- 172.21.15.28:0/2478332375 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e61834f210 con 0x55e6186da8f0 2024-03-22T02:31:20.617 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.610+0000 7f634b7fe640 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 0x55e6186e28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f634affd640 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f633c009c50 tx=0x7f633c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1faa3dcd4508ab3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f634b7fe640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 0x7f63340a5480 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f634402c630 tx=0x7f6344032360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f63317fa640 1 -- 172.21.15.28:0/2478332375 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c004230 con 0x55e6186da8f0 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f63317fa640 1 -- 172.21.15.28:0/2478332375 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c0043d0 con 0x55e6186da8f0 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f63317fa640 1 -- 172.21.15.28:0/2478332375 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c019760 con 0x55e6186da8f0 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f63541b5740 1 -- 172.21.15.28:0/2478332375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 msgr2=0x55e6186dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.611+0000 7f63541b5740 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f633c009c50 tx=0x7f633c011810 comp rx=0 tx=0).stop 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- 172.21.15.28:0/2478332375 shutdown_connections 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186e0450 0x55e6186e28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186dfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- 172.21.15.28:0/2478332375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186dad10 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.618 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- 172.21.15.28:0/2478332375 >> 172.21.15.28:0/2478332375 conn(0x55e6186d1a50 msgr2=0x55e6186d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- 172.21.15.28:0/2478332375 shutdown_connections 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- 172.21.15.28:0/2478332375 wait complete. 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- start start 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 Processor -- start 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186db2e0 0x55e6186e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186e0450 0x55e6186ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e6186e3400 con 0x55e6186db2e0 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e6185f6db0 con 0x55e6186e0450 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e6186e3580 con 0x55e6186da8f0 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f634affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.619 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f63541b5740 10 client.0 _ll_register_callbacks cb 0x55e61834ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f634affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56388/0 (socket says 172.21.15.28:56388) 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f634b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186e0450 0x55e6186ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f634affd640 1 -- 172.21.15.28:0/1446146280 learned_addr learned my addr 172.21.15.28:0/1446146280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.612+0000 7f634a7fc640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186db2e0 0x55e6186e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 -- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186e0450 msgr2=0x55e6186ebdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186e0450 0x55e6186ebdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 -- 172.21.15.28:0/1446146280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186db2e0 msgr2=0x55e6186e4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186db2e0 0x55e6186e4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 -- 172.21.15.28:0/1446146280 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e61841b620 con 0x55e6186da8f0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f634affd640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f633c011d10 tx=0x7f633c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=672046505350f526 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f632eff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c028020 con 0x55e6186da8f0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f633c005040 con 0x55e6186da8f0 2024-03-22T02:31:20.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f633c0043d0 con 0x55e6186da8f0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f633c033050 con 0x55e6186da8f0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f63541b5740 10 client.29731 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.613+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e6186ef090 con 0x55e6186da8f0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c0043d0 con 0x55e6186da8f0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f633c018070 con 0x55e6186da8f0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 10 client.29731 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 10 client.29731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 20 client.29731 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186f05c0 0x55e6186f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 4 client.29731 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e618648580 con 0x55e6186f05c0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f634a7fc640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186f05c0 0x55e6186f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f632eff5640 10 client.29731 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:20.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.614+0000 7f634a7fc640 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186f05c0 0x55e6186f29b0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b2036b2534e8b2b server_cookie=82c43b573718fdd3 in_seq=0 out_seq=0 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f632eff5640 1 -- 172.21.15.28:0/1446146280 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e618648580 con 0x55e6186f05c0 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f632eff5640 10 client.29731 handle_command_reply: tid=0 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 client.29731 shutdown 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 msgr2=0x55e6186e48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f633c011d10 tx=0x7f633c003820 comp rx=0 tx=0).stop 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 shutdown_connections 2024-03-22T02:31:20.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186f05c0 0x55e6186f29b0 crc :-1 s=READY pgs=392 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186e0450 0x55e6186ebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f632eff5640 0 client.29731 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186db2e0 0x55e6186e4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 --2- 172.21.15.28:0/1446146280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186e48c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 >> 172.21.15.28:0/1446146280 conn(0x55e6186d1a50 msgr2=0x55e6186d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 shutdown_connections 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 1 -- 172.21.15.28:0/1446146280 wait complete. 2024-03-22T02:31:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.619+0000 7f63541b5740 20 client.29731 trim_cache size 0 max 16384 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 Processor -- start 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 -- start start 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186ebd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186e3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.642 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ab070 con 0x55e6186d1a50 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a8940 con 0x55e6186db2e0 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f63541b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a6790 con 0x55e6186da8f0 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59834/0 (socket says 172.21.15.28:59834) 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 -- 172.21.15.28:0/3283266689 learned_addr learned my addr 172.21.15.28:0/3283266689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634a7fc640 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186ebd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634b7fe640 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186e3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 -- 172.21.15.28:0/3283266689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 msgr2=0x55e6186ebd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.643 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186ebd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 -- 172.21.15.28:0/3283266689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 msgr2=0x55e6186e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186e3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.639+0000 7f634affd640 1 -- 172.21.15.28:0/3283266689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6186f2be0 con 0x55e6186d1a50 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f634affd640 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f633c004d60 tx=0x7f633c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cc69d3cf4120c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f6330ff9640 1 -- 172.21.15.28:0/3283266689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c03b020 con 0x55e6186d1a50 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f6330ff9640 1 -- 172.21.15.28:0/3283266689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c009850 con 0x55e6186d1a50 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f6330ff9640 1 -- 172.21.15.28:0/3283266689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c028030 con 0x55e6186d1a50 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f63541b5740 1 -- 172.21.15.28:0/3283266689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 msgr2=0x55e6186d5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f63541b5740 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f633c004d60 tx=0x7f633c001e50 comp rx=0 tx=0).stop 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f63541b5740 1 -- 172.21.15.28:0/3283266689 shutdown_connections 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.640+0000 7f63541b5740 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186db2e0 0x55e6186e3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186da8f0 0x55e6186ebd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.644 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 --2- 172.21.15.28:0/3283266689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186d1a50 0x55e6186d5320 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- 172.21.15.28:0/3283266689 >> 172.21.15.28:0/3283266689 conn(0x55e6186dff90 msgr2=0x55e6186dc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- 172.21.15.28:0/3283266689 shutdown_connections 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- 172.21.15.28:0/3283266689 wait complete. 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- start start 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 Processor -- start 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 0x55e6186e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 0x55e6186eae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a6e00 con 0x55e6186da8f0 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a8270 con 0x55e6186d1a50 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1a74d0 con 0x55e6186db2e0 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f63541b5740 10 client.0 _ll_register_callbacks cb 0x7f6340009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:20.645 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f634a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f634a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59846/0 (socket says 172.21.15.28:59846) 2024-03-22T02:31:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f634a7fc640 1 -- 172.21.15.28:0/798098703 learned_addr learned my addr 172.21.15.28:0/798098703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f634affd640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 0x55e6186e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.646 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.641+0000 7f634b7fe640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 0x55e6186eae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 -- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 msgr2=0x55e6186e9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 0x55e6186e9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 -- 172.21.15.28:0/798098703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 msgr2=0x55e6186e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:20.648 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 -- 172.21.15.28:0/798098703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6186eb3f0 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634affd640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 0x55e6186e9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634a7fc640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f634b7fe640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 0x55e6186eae00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6344037040 tx=0x7f63440095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0a636fa5e317a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f632eff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344039040 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f633c009850 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.642+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f6344004040 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6344004240 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 10 client.29737 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e618756e20 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633c009850 con 0x55e6186db2e0 2024-03-22T02:31:20.649 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f634403f040 con 0x55e6186db2e0 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 10 client.29737 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 10 client.29737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 20 client.29737 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186e0450 0x55e618759860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 4 client.29737 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e6186428d0 con 0x55e6186e0450 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.643+0000 7f634affd640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186e0450 0x55e618759860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.644+0000 7f632eff5640 10 client.29737 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:20.650 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:20.644+0000 7f634affd640 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186e0450 0x55e618759860 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b96b82879f436c server_cookie=f0b76f55f302f9e9 in_seq=0 out_seq=0 2024-03-22T02:31:20.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 72195 15.87 MB/sec execute 120 sec latency 120.876 ms 2024-03-22T02:31:20.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:20 smithi120 ceph-mon[36803]: pgmap v894: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 565 op/s 2024-03-22T02:31:20.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:20 smithi083 ceph-mon[36340]: pgmap v894: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 565 op/s 2024-03-22T02:31:20.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:20 smithi028 ceph-mon[31916]: pgmap v894: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.5 MiB/s wr, 565 op/s 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f632eff5640 1 -- 172.21.15.28:0/798098703 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e6186428d0 con 0x55e6186e0450 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f632eff5640 10 client.29737 handle_command_reply: tid=0 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f63541b5740 1 client.29737 shutdown 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 msgr2=0x55e6186eae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 0x55e6186eae00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6344037040 tx=0x7f63440095c0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 shutdown_connections 2024-03-22T02:31:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.573+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e6186e0450 0x55e618759860 crc :-1 s=READY pgs=393 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f632eff5640 0 client.29737 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e6186db2e0 0x55e6186eae00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e6186da8f0 0x55e6186e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 --2- 172.21.15.28:0/798098703 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e6186d1a50 0x55e6186e9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 >> 172.21.15.28:0/798098703 conn(0x55e6186dff90 msgr2=0x55e6186dbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 shutdown_connections 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 1 -- 172.21.15.28:0/798098703 wait complete. 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.574+0000 7f63541b5740 20 client.29737 trim_cache size 0 max 16384 2024-03-22T02:31:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:31:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:31:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:31:21.577 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.575+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 msgr2=0x7f63340a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.575+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 0x7f63340a5480 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f634402c630 tx=0x7f6344032360 comp rx=0 tx=0).stop 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.575+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 msgr2=0x7f634c198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.575+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9f0 tx=0x7f634000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 shutdown_connections 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f63340a2fc0 0x7f63340a5480 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f634c103770 0x7f634c19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102db0 0x7f634c198300 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 --2- 172.21.15.28:0/712042268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f634c101bc0 0x7f634c197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 >> 172.21.15.28:0/712042268 conn(0x7f634c0fd150 msgr2=0x7f634c105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 shutdown_connections 2024-03-22T02:31:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:21.576+0000 7f63515c9640 1 -- 172.21.15.28:0/712042268 wait complete. 2024-03-22T02:31:21.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 72328 15.81 MB/sec execute 121 sec latency 577.268 ms 2024-03-22T02:31:22.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 72547 15.79 MB/sec execute 122 sec latency 336.369 ms 2024-03-22T02:31:22.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:22 smithi120 ceph-mon[36803]: pgmap v895: 129 pgs: 129 active+clean; 270 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-03-22T02:31:22.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:22 smithi083 ceph-mon[36340]: pgmap v895: 129 pgs: 129 active+clean; 270 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-03-22T02:31:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:22 smithi028 ceph-mon[31916]: pgmap v895: 129 pgs: 129 active+clean; 270 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.1 MiB/s wr, 421 op/s 2024-03-22T02:31:23.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 72891 15.89 MB/sec execute 123 sec latency 104.447 ms 2024-03-22T02:31:24.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 73394 15.94 MB/sec execute 124 sec latency 81.307 ms 2024-03-22T02:31:24.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:24 smithi120 ceph-mon[36803]: pgmap v896: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 376 op/s 2024-03-22T02:31:24.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:24 smithi083 ceph-mon[36340]: pgmap v896: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 376 op/s 2024-03-22T02:31:24.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:24 smithi028 ceph-mon[31916]: pgmap v896: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 376 op/s 2024-03-22T02:31:25.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 73732 15.96 MB/sec execute 125 sec latency 82.520 ms 2024-03-22T02:31:25.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:25 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:25.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:25 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:25.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:25 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:31:26.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 74045 16.00 MB/sec execute 126 sec latency 76.453 ms 2024-03-22T02:31:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: pgmap v897: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 506 op/s 2024-03-22T02:31:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:26.766 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: pgmap v897: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 506 op/s 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:26.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: pgmap v897: 129 pgs: 129 active+clean; 265 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.7 MiB/s wr, 506 op/s 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:31:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:31:27.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:31:27.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 74283 15.98 MB/sec execute 127 sec latency 239.500 ms 2024-03-22T02:31:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:27 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:31:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:27 smithi083 ceph-mon[36340]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:31:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:27 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:31:27.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:27 smithi083 ceph-mon[36340]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:31:27.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:27 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:31:27.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:27 smithi028 ceph-mon[31916]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:31:27.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:27 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:31:27.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:27 smithi028 ceph-mon[31916]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:31:27.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:27 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-22T02:31:27.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:27 smithi120 ceph-mon[36803]: Adjusting osd_memory_target on smithi083 to 174.1M 2024-03-22T02:31:27.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:27 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-22T02:31:27.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:27 smithi120 ceph-mon[36803]: Unable to set osd_memory_target on smithi083 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-22T02:31:28.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 74528 15.98 MB/sec execute 128 sec latency 119.603 ms 2024-03-22T02:31:28.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:28 smithi083 ceph-mon[36340]: pgmap v898: 129 pgs: 129 active+clean; 266 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 518 op/s 2024-03-22T02:31:28.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:28 smithi028 ceph-mon[31916]: pgmap v898: 129 pgs: 129 active+clean; 266 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 518 op/s 2024-03-22T02:31:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:28 smithi120 ceph-mon[36803]: pgmap v898: 129 pgs: 129 active+clean; 266 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 518 op/s 2024-03-22T02:31:29.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 74791 15.96 MB/sec execute 129 sec latency 106.704 ms 2024-03-22T02:31:30.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 75176 15.94 MB/sec execute 130 sec latency 68.708 ms 2024-03-22T02:31:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:30 smithi083 ceph-mon[36340]: pgmap v899: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2024-03-22T02:31:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:30 smithi028 ceph-mon[31916]: pgmap v899: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2024-03-22T02:31:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:30 smithi120 ceph-mon[36803]: pgmap v899: 129 pgs: 129 active+clean; 263 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2024-03-22T02:31:31.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 75425 15.93 MB/sec execute 131 sec latency 136.868 ms 2024-03-22T02:31:32.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 75685 15.89 MB/sec execute 132 sec latency 290.505 ms 2024-03-22T02:31:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:32 smithi083 ceph-mon[36340]: pgmap v900: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:31:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:32 smithi028 ceph-mon[31916]: pgmap v900: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:31:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:32 smithi120 ceph-mon[36803]: pgmap v900: 129 pgs: 129 active+clean; 269 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 552 op/s 2024-03-22T02:31:33.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 76025 15.87 MB/sec execute 133 sec latency 85.271 ms 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 -- 172.21.15.28:0/1401928355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 msgr2=0x7f16cc1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 --2- 172.21.15.28:0/1401928355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc1018a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009900 tx=0x7f16c002bc30 comp rx=0 tx=0).stop 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 -- 172.21.15.28:0/1401928355 shutdown_connections 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 --2- 172.21.15.28:0/1401928355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 --2- 172.21.15.28:0/1401928355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc1018a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 --2- 172.21.15.28:0/1401928355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 -- 172.21.15.28:0/1401928355 >> 172.21.15.28:0/1401928355 conn(0x7f16cc0fb7e0 msgr2=0x7f16cc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 -- 172.21.15.28:0/1401928355 shutdown_connections 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.674+0000 7f16d0847640 1 -- 172.21.15.28:0/1401928355 wait complete. 2024-03-22T02:31:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.675+0000 7f16d0847640 1 Processor -- start 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.675+0000 7f16d0847640 1 -- start start 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.675+0000 7f16d0847640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc1041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.675+0000 7f16d0847640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16d0847640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16d0847640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10c790 con 0x7f16cc100230 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16d0847640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10c610 con 0x7f16cc101de0 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16d0847640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10c910 con 0x7f16cc101420 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc1041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc1041c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45124/0 (socket says 172.21.15.28:45124) 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51186/0 (socket says 172.21.15.28:51186) 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16ca59c640 1 -- 172.21.15.28:0/3904285690 learned_addr learned my addr 172.21.15.28:0/3904285690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 -- 172.21.15.28:0/3904285690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 msgr2=0x7f16cc102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 -- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 msgr2=0x7f16cc1041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc1041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 -- 172.21.15.28:0/3904285690 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc1035d0 con 0x7f16cc101420 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.676+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f16c0036040 tx=0x7f16c002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.677+0000 7f16bb7fe640 1 -- 172.21.15.28:0/3904285690 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0043020 con 0x7f16cc101420 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.677+0000 7f16bb7fe640 1 -- 172.21.15.28:0/3904285690 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c00054e0 con 0x7f16cc101420 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.677+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16cc1a1f30 con 0x7f16cc101420 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.677+0000 7f16bb7fe640 1 -- 172.21.15.28:0/3904285690 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0003c30 con 0x7f16cc101420 2024-03-22T02:31:34.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.677+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a2390 con 0x7f16cc101420 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 Processor -- start 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16bb7fe640 1 -- 172.21.15.28:0/3904285690 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f16c0046020 con 0x7f16cc101420 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 -- start start 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1d9e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1da14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628e1cfe040 con 0x5628e1d994e0 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5628e1ca69e0 con 0x5628e1d99ed0 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16d342c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628e1cf9ff0 con 0x5628e1d9f040 2024-03-22T02:31:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16cad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16cad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45134/0 (socket says 172.21.15.28:45134) 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16c9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1da14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16cad9d640 1 -- 172.21.15.28:0/508332863 learned_addr learned my addr 172.21.15.28:0/508332863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.680+0000 7f16ca59c640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1d9e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 -- 172.21.15.28:0/508332863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 msgr2=0x5628e1d9e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1d9e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 -- 172.21.15.28:0/508332863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 msgr2=0x5628e1d99900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16bb7fe640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 0x7f16b40a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 -- 172.21.15.28:0/508332863 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e1a12210 con 0x5628e1d9f040 2024-03-22T02:31:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16ca59c640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1d9e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16cad9d640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 0x7f16b40a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1da14d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f16c00eedb0 tx=0x7f16c00b9d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32407aad856af57f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16bb7fe640 1 -- 172.21.15.28:0/3904285690 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f16c00f16f0 con 0x7f16cc101420 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16b8ff9640 1 -- 172.21.15.28:0/508332863 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c00bd7e0 con 0x5628e1d9f040 2024-03-22T02:31:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.681+0000 7f16b8ff9640 1 -- 172.21.15.28:0/508332863 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c00f7e50 con 0x5628e1d9f040 2024-03-22T02:31:34.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 76359 15.97 MB/sec execute 134 sec latency 72.505 ms 2024-03-22T02:31:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16b8ff9640 1 -- 172.21.15.28:0/508332863 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c00bd370 con 0x5628e1d9f040 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 -- 172.21.15.28:0/508332863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 msgr2=0x5628e1da14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 0x7f16b40a5480 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5628e1d9a730 tx=0x7f16c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1da14d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f16c00eedb0 tx=0x7f16c00b9d70 comp rx=0 tx=0).stop 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 -- 172.21.15.28:0/508332863 shutdown_connections 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1da14d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1d9e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 --2- 172.21.15.28:0/508332863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1d99900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.682+0000 7f16d342c740 1 -- 172.21.15.28:0/508332863 >> 172.21.15.28:0/508332863 conn(0x5628e1d94a50 msgr2=0x5628e1d94e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.683+0000 7f16d342c740 1 -- 172.21.15.28:0/508332863 shutdown_connections 2024-03-22T02:31:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.683+0000 7f16d342c740 1 -- 172.21.15.28:0/508332863 wait complete. 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.683+0000 7f16d342c740 1 -- start start 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.683+0000 7f16d342c740 1 Processor -- start 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 0x5628e1dabf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1db2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628e1da1db0 con 0x5628e1d99ed0 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5628e1cb9db0 con 0x5628e1d994e0 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5628e1da1f30 con 0x5628e1d9f040 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16d342c740 10 client.0 _ll_register_callbacks cb 0x5628e1a11a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:34730/0 (socket says 172.21.15.28:34730) 2024-03-22T02:31:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 -- 172.21.15.28:0/2616124809 learned_addr learned my addr 172.21.15.28:0/2616124809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 0x5628e1dabf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16cad9d640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1db2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 -- 172.21.15.28:0/2616124809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 msgr2=0x5628e1db2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1db2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 -- 172.21.15.28:0/2616124809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 msgr2=0x5628e1dabf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 0x5628e1dabf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16ca59c640 1 -- 172.21.15.28:0/2616124809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e1ade620 con 0x5628e1d994e0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.684+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 0x5628e1dabf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.685+0000 7f16ca59c640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f16bc012700 tx=0x7f16bc012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d258048ad7c7d96e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.685+0000 7f16b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.685+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc004020 con 0x5628e1d994e0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.685+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c00f7e50 con 0x5628e1d994e0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.685+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f16bc0041c0 con 0x5628e1d994e0 2024-03-22T02:31:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f16bc025050 con 0x5628e1d994e0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 10 client.29753 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5628e1db63d0 con 0x5628e1d994e0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c00f7e50 con 0x5628e1d994e0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f16bc003a60 con 0x5628e1d994e0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 10 client.29753 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 10 client.29753 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 20 client.29753 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1db79f0 0x5628e1db9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 4 client.29753 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.686+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628e1d0b580 con 0x5628e1db79f0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.687+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1db79f0 0x5628e1db9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.687+0000 7f16b1ffb640 10 client.29753 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:34.693 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.687+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1db79f0 0x5628e1db9e00 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bf03d1322b310dc server_cookie=aa9a5d604e69d9ef in_seq=0 out_seq=0 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16b1ffb640 1 -- 172.21.15.28:0/2616124809 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628e1d0b580 con 0x5628e1db79f0 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16b1ffb640 10 client.29753 handle_command_reply: tid=0 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 client.29753 shutdown 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 msgr2=0x5628e1daba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f16bc012700 tx=0x7f16bc012bc0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 shutdown_connections 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1db79f0 0x5628e1db9e00 crc :-1 s=READY pgs=394 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1db2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d99ed0 0x5628e1dabf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 --2- 172.21.15.28:0/2616124809 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d994e0 0x5628e1daba20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 >> 172.21.15.28:0/2616124809 conn(0x5628e1d94a50 msgr2=0x5628e1d93c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.689+0000 7f16b1ffb640 0 client.29753 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.690+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 shutdown_connections 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.690+0000 7f16d342c740 1 -- 172.21.15.28:0/2616124809 wait complete. 2024-03-22T02:31:34.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.690+0000 7f16d342c740 20 client.29753 trim_cache size 0 max 16384 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.707+0000 7f16d342c740 1 Processor -- start 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.707+0000 7f16d342c740 1 -- start start 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d994e0 0x5628e1d933c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1ab010 con 0x5628e1d9f040 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a88e0 con 0x5628e1d99ed0 2024-03-22T02:31:34.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16d342c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a6730 con 0x5628e1d994e0 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45144/0 (socket says 172.21.15.28:45144) 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 -- 172.21.15.28:0/3496941451 learned_addr learned my addr 172.21.15.28:0/3496941451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16ca59c640 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d994e0 0x5628e1d933c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 -- 172.21.15.28:0/3496941451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d994e0 msgr2=0x5628e1d933c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d994e0 0x5628e1d933c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 -- 172.21.15.28:0/3496941451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 msgr2=0x5628e1db2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 -- 172.21.15.28:0/3496941451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628e1d904b0 con 0x5628e1d9f040 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f16c4011510 tx=0x7f16c401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591f5262ca37dba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.708+0000 7f16b3fff640 1 -- 172.21.15.28:0/3496941451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4004b20 con 0x5628e1d9f040 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16b3fff640 1 -- 172.21.15.28:0/3496941451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c4004cc0 con 0x5628e1d9f040 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16b3fff640 1 -- 172.21.15.28:0/3496941451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c40055f0 con 0x5628e1d9f040 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 -- 172.21.15.28:0/3496941451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 msgr2=0x5628e1d93900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f16c4011510 tx=0x7f16c401a000 comp rx=0 tx=0).stop 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 -- 172.21.15.28:0/3496941451 shutdown_connections 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d9f040 0x5628e1d93900 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 --2- 172.21.15.28:0/3496941451 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d994e0 0x5628e1d933c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 -- 172.21.15.28:0/3496941451 >> 172.21.15.28:0/3496941451 conn(0x5628e1d94a50 msgr2=0x5628e1d98540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 -- 172.21.15.28:0/3496941451 shutdown_connections 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.709+0000 7f16d342c740 1 -- 172.21.15.28:0/3496941451 wait complete. 2024-03-22T02:31:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 -- start start 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 Processor -- start 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1e1bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a6da0 con 0x5628e1d994e0 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a8210 con 0x5628e1d99ed0 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc1a7470 con 0x5628e1d9f040 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16d342c740 10 client.0 _ll_register_callbacks cb 0x7f16bc009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45148/0 (socket says 172.21.15.28:45148) 2024-03-22T02:31:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 -- 172.21.15.28:0/3105137170 learned_addr learned my addr 172.21.15.28:0/3105137170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16cad9d640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1e1bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 -- 172.21.15.28:0/3105137170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 msgr2=0x5628e1e1bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1e1bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 -- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 msgr2=0x5628e1db01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 -- 172.21.15.28:0/3105137170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628e1db3270 con 0x5628e1d994e0 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16ca59c640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0059f0 tx=0x7f16bc0042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79726581250460c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.710+0000 7f16b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:31:34.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc02e060 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f16bc00fb30 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f16c4004cc0 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f16bc035050 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 10 client.19950 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5628e1e1d350 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c4004cc0 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f16bc003710 con 0x5628e1d994e0 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 10 client.19950 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 10 client.19950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 20 client.19950 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1e1dd20 0x5628e1e20140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 4 client.19950 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:31:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628e1d058d0 con 0x5628e1e1dd20 2024-03-22T02:31:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.711+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1e1dd20 0x5628e1e20140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.712+0000 7f16c9d9b640 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1e1dd20 0x5628e1e20140 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2aeb8b9170827be2 server_cookie=506e92afd50ae3d in_seq=0 out_seq=0 2024-03-22T02:31:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:34.712+0000 7f16b1ffb640 10 client.19950 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:34 smithi083 ceph-mon[36340]: pgmap v901: 129 pgs: 129 active+clean; 272 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 516 op/s 2024-03-22T02:31:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:34 smithi028 ceph-mon[31916]: pgmap v901: 129 pgs: 129 active+clean; 272 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 516 op/s 2024-03-22T02:31:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:34 smithi120 ceph-mon[36803]: pgmap v901: 129 pgs: 129 active+clean; 272 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.3 MiB/s wr, 516 op/s 2024-03-22T02:31:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16b1ffb640 1 -- 172.21.15.28:0/3105137170 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628e1d058d0 con 0x5628e1e1dd20 2024-03-22T02:31:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16b1ffb640 10 client.19950 handle_command_reply: tid=0 2024-03-22T02:31:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 client.19950 shutdown 2024-03-22T02:31:35.652 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 msgr2=0x5628e1dafc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f16bc0059f0 tx=0x7f16bc0042d0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 shutdown_connections 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5628e1e1dd20 0x5628e1e20140 crc :-1 s=READY pgs=395 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5628e1d9f040 0x5628e1e1bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5628e1d99ed0 0x5628e1db01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 --2- 172.21.15.28:0/3105137170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628e1d994e0 0x5628e1dafc60 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16b1ffb640 0 client.19950 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 >> 172.21.15.28:0/3105137170 conn(0x5628e1d94a50 msgr2=0x5628e1d9a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 shutdown_connections 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 1 -- 172.21.15.28:0/3105137170 wait complete. 2024-03-22T02:31:35.653 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.651+0000 7f16d342c740 20 client.19950 trim_cache size 0 max 16384 2024-03-22T02:31:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:31:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:31:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:31:35.654 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 msgr2=0x7f16b40a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 0x7f16b40a5480 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x5628e1d9a730 tx=0x7f16c4009500 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 msgr2=0x7f16cc102800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f16c0036040 tx=0x7f16c002c650 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 shutdown_connections 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f16b40a2fc0 0x7f16b40a5480 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f16cc101de0 0x7f16cc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f16cc101420 0x7f16cc102800 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 --2- 172.21.15.28:0/3904285690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16cc100230 0x7f16cc1041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 >> 172.21.15.28:0/3904285690 conn(0x7f16cc0fb7e0 msgr2=0x7f16cc0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:35.655 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 shutdown_connections 2024-03-22T02:31:35.656 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:35.652+0000 7f16d0847640 1 -- 172.21.15.28:0/3904285690 wait complete. 2024-03-22T02:31:35.682 INFO:tasks.workunit.client.0.smithi028.stdout: 10 76423 15.86 MB/sec execute 135 sec latency 699.632 ms 2024-03-22T02:31:36.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 76898 15.91 MB/sec execute 136 sec latency 58.319 ms 2024-03-22T02:31:36.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:36 smithi028 ceph-mon[31916]: pgmap v902: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 508 op/s 2024-03-22T02:31:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:36 smithi120 ceph-mon[36803]: pgmap v902: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 508 op/s 2024-03-22T02:31:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:36 smithi083 ceph-mon[36340]: pgmap v902: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 508 op/s 2024-03-22T02:31:37.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 77172 15.91 MB/sec execute 137 sec latency 159.563 ms 2024-03-22T02:31:38.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 77399 15.90 MB/sec execute 138 sec latency 117.547 ms 2024-03-22T02:31:38.800 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:38 smithi120 ceph-mon[36803]: pgmap v903: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-03-22T02:31:38.816 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:38 smithi083 ceph-mon[36340]: pgmap v903: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-03-22T02:31:38.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:38 smithi028 ceph-mon[31916]: pgmap v903: 129 pgs: 129 active+clean; 276 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 377 op/s 2024-03-22T02:31:39.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 77748 15.93 MB/sec execute 139 sec latency 67.155 ms 2024-03-22T02:31:40.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 77997 15.92 MB/sec execute 140 sec latency 137.373 ms 2024-03-22T02:31:40.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:40 smithi028 ceph-mon[31916]: pgmap v904: 129 pgs: 129 active+clean; 270 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 499 op/s 2024-03-22T02:31:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:40 smithi120 ceph-mon[36803]: pgmap v904: 129 pgs: 129 active+clean; 270 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 499 op/s 2024-03-22T02:31:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:40 smithi083 ceph-mon[36340]: pgmap v904: 129 pgs: 129 active+clean; 270 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 499 op/s 2024-03-22T02:31:41.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 78246 15.93 MB/sec execute 141 sec latency 99.058 ms 2024-03-22T02:31:42.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 78479 15.90 MB/sec execute 142 sec latency 258.531 ms 2024-03-22T02:31:42.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:42 smithi028 ceph-mon[31916]: pgmap v905: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 532 op/s 2024-03-22T02:31:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:42 smithi120 ceph-mon[36803]: pgmap v905: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 532 op/s 2024-03-22T02:31:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:42 smithi083 ceph-mon[36340]: pgmap v905: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.8 MiB/s wr, 532 op/s 2024-03-22T02:31:43.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 78719 15.84 MB/sec execute 143 sec latency 166.063 ms 2024-03-22T02:31:44.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 79023 15.84 MB/sec execute 144 sec latency 63.116 ms 2024-03-22T02:31:44.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:44 smithi028 ceph-mon[31916]: pgmap v906: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 498 op/s 2024-03-22T02:31:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:44 smithi120 ceph-mon[36803]: pgmap v906: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 498 op/s 2024-03-22T02:31:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:44 smithi083 ceph-mon[36340]: pgmap v906: 129 pgs: 129 active+clean; 262 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 498 op/s 2024-03-22T02:31:45.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 79369 15.81 MB/sec execute 145 sec latency 108.648 ms 2024-03-22T02:31:46.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 79734 15.85 MB/sec execute 146 sec latency 59.998 ms 2024-03-22T02:31:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:46 smithi028 ceph-mon[31916]: pgmap v907: 129 pgs: 129 active+clean; 266 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:46 smithi120 ceph-mon[36803]: pgmap v907: 129 pgs: 129 active+clean; 266 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:46 smithi083 ceph-mon[36340]: pgmap v907: 129 pgs: 129 active+clean; 266 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.2 MiB/s wr, 555 op/s 2024-03-22T02:31:47.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 80081 15.91 MB/sec execute 147 sec latency 195.509 ms 2024-03-22T02:31:48.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 80371 15.91 MB/sec execute 148 sec latency 232.569 ms 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- 172.21.15.28:0/2339793222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 msgr2=0x7fc078104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 --2- 172.21.15.28:0/2339793222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc078104a20 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fc074009860 tx=0x7fc07402bb70 comp rx=0 tx=0).stop 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- 172.21.15.28:0/2339793222 shutdown_connections 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 --2- 172.21.15.28:0/2339793222 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 0x7fc078109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 --2- 172.21.15.28:0/2339793222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc078107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 --2- 172.21.15.28:0/2339793222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc078104a20 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- 172.21.15.28:0/2339793222 >> 172.21.15.28:0/2339793222 conn(0x7fc0780fc020 msgr2=0x7fc0780fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- 172.21.15.28:0/2339793222 shutdown_connections 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- 172.21.15.28:0/2339793222 wait complete. 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 Processor -- start 2024-03-22T02:31:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.783+0000 7fc0801bf640 1 -- start start 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc07819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 0x7fc0781a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc078101ca0 con 0x7fc078102630 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc078101b20 con 0x7fc078107890 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc078101e20 con 0x7fc078104f60 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39962/0 (socket says 172.21.15.28:39962) 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 -- 172.21.15.28:0/3866738447 learned_addr learned my addr 172.21.15.28:0/3866738447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07d75a640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc07819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07e75c640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 0x7fc0781a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 -- 172.21.15.28:0/3866738447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 msgr2=0x7fc07819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc07819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 -- 172.21.15.28:0/3866738447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 msgr2=0x7fc0781a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 0x7fc0781a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 -- 172.21.15.28:0/3866738447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0781a1c70 con 0x7fc078102630 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07d75a640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc07819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc07df5b640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc074005940 tx=0x7fc0740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc06effd640 1 -- 172.21.15.28:0/3866738447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074041020 con 0x7fc078102630 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc06effd640 1 -- 172.21.15.28:0/3866738447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740323f0 con 0x7fc078102630 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc06effd640 1 -- 172.21.15.28:0/3866738447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074039040 con 0x7fc078102630 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.784+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0781a1f00 con 0x7fc078102630 2024-03-22T02:31:48.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.785+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0781a2310 con 0x7fc078102630 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc06effd640 1 -- 172.21.15.28:0/3866738447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc074044020 con 0x7fc078102630 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 Processor -- start 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 -- start start 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.797 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67ad3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556f67a28040 con 0x556f67acbcf0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556f679d09e0 con 0x556f67acc6e0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc082d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556f67a23ff0 con 0x556f67ad1850 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc06effd640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 0x7fc0640a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc07e75c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc07e75c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39964/0 (socket says 172.21.15.28:39964) 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.787+0000 7fc07e75c640 1 -- 172.21.15.28:0/518153176 learned_addr learned my addr 172.21.15.28:0/518153176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07df5b640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07d75a640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67ad3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 0x7fc0640a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc06effd640 1 -- 172.21.15.28:0/3866738447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc0740f1200 con 0x7fc078102630 2024-03-22T02:31:48.798 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 -- 172.21.15.28:0/518153176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 msgr2=0x556f67ad3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67ad3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 -- 172.21.15.28:0/518153176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 msgr2=0x556f67ad11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 -- 172.21.15.28:0/518153176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f6773c210 con 0x556f67acbcf0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07df5b640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc070012d80 tx=0x7fc070010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd7186cb1af43f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc07e75c640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 0x7fc0640a5580 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc070010c40 tx=0x7fc070017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc063fff640 1 -- 172.21.15.28:0/518153176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070023070 con 0x556f67acbcf0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc063fff640 1 -- 172.21.15.28:0/518153176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0700125a0 con 0x556f67acbcf0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.788+0000 7fc063fff640 1 -- 172.21.15.28:0/518153176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070003bb0 con 0x556f67acbcf0 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- 172.21.15.28:0/518153176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 msgr2=0x556f67acc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fc070012d80 tx=0x7fc070010740 comp rx=0 tx=0).stop 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- 172.21.15.28:0/518153176 shutdown_connections 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67ad3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 --2- 172.21.15.28:0/518153176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67acc110 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- 172.21.15.28:0/518153176 >> 172.21.15.28:0/518153176 conn(0x556f67ac7260 msgr2=0x556f67ac7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- 172.21.15.28:0/518153176 shutdown_connections 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- 172.21.15.28:0/518153176 wait complete. 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.789+0000 7fc082d9c740 1 -- start start 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 Processor -- start 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67add2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556f67ad47a0 con 0x556f67ad1850 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556f679e3db0 con 0x556f67acc6e0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556f67ad4920 con 0x556f67acbcf0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc082d9c740 10 client.0 _ll_register_callbacks cb 0x556f6773ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07e75c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67add2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59608/0 (socket says 172.21.15.28:59608) 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07df5b640 1 -- 172.21.15.28:0/2118812852 learned_addr learned my addr 172.21.15.28:0/2118812852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 -- 172.21.15.28:0/2118812852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 msgr2=0x556f67ad5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 -- 172.21.15.28:0/2118812852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 msgr2=0x556f67add2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67add2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07d75a640 1 -- 172.21.15.28:0/2118812852 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f67808620 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07df5b640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.790+0000 7fc07e75c640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67add2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc07d75a640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad6290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc07819b560 tx=0x7fc06800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65632a4f0d55ad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc068004050 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc0680041f0 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0700125a0 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc068025050 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 10 client.29771 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556f67ae06b0 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0700125a0 con 0x556f67acc6e0 2024-03-22T02:31:48.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fc068011ce0 con 0x556f67acc6e0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 10 client.29771 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 10 client.29771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 20 client.29771 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67ae1de0 0x556f67ae4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.791+0000 7fc082d9c740 4 client.29771 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.792+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f67a35580 con 0x556f67ae1de0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.792+0000 7fc07df5b640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67ae1de0 0x556f67ae4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.792+0000 7fc0617fa640 10 client.29771 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.792+0000 7fc07df5b640 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67ae1de0 0x556f67ae4210 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abac1c71b2510a97 server_cookie=e969c5641898fc77 in_seq=0 out_seq=0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc0617fa640 1 -- 172.21.15.28:0/2118812852 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556f67a35580 con 0x556f67ae1de0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc0617fa640 10 client.29771 handle_command_reply: tid=0 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 client.29771 shutdown 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 msgr2=0x556f67ad6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad6290 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc07819b560 tx=0x7fc06800f790 comp rx=0 tx=0).stop 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 shutdown_connections 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67ae1de0 0x556f67ae4210 crc :-1 s=READY pgs=396 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc0617fa640 0 client.29771 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67add2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ad6290 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 --2- 172.21.15.28:0/2118812852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ad5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 >> 172.21.15.28:0/2118812852 conn(0x556f67ac7260 msgr2=0x556f67abdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 shutdown_connections 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 1 -- 172.21.15.28:0/2118812852 wait complete. 2024-03-22T02:31:48.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.796+0000 7fc082d9c740 20 client.29771 trim_cache size 0 max 16384 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 Processor -- start 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 -- start start 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67add290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67acaa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781aafb0 con 0x556f67acbcf0 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781a8880 con 0x556f67acc6e0 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc082d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781a66f0 con 0x556f67ad1850 2024-03-22T02:31:48.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39990/0 (socket says 172.21.15.28:39990) 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc07df5b640 1 -- 172.21.15.28:0/799631483 learned_addr learned my addr 172.21.15.28:0/799631483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc07e75c640 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67acaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.812+0000 7fc07d75a640 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67add290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 -- 172.21.15.28:0/799631483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 msgr2=0x556f67acaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67acaa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 -- 172.21.15.28:0/799631483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 msgr2=0x556f67add290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67add290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 -- 172.21.15.28:0/799631483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f67acaf40 con 0x556f67acbcf0 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc07df5b640 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fc074009830 tx=0x7fc0740901f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46ab0805f2bdef6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc0637fe640 1 -- 172.21.15.28:0/799631483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0740b9760 con 0x556f67acbcf0 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc0637fe640 1 -- 172.21.15.28:0/799631483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740bdbc0 con 0x556f67acbcf0 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.813+0000 7fc0637fe640 1 -- 172.21.15.28:0/799631483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0740bce20 con 0x556f67acbcf0 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- 172.21.15.28:0/799631483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 msgr2=0x556f67aca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7fc074009830 tx=0x7fc0740901f0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- 172.21.15.28:0/799631483 shutdown_connections 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67ad1850 0x556f67acaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67add290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- 172.21.15.28:0/799631483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67acbcf0 0x556f67aca4c0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- 172.21.15.28:0/799631483 >> 172.21.15.28:0/799631483 conn(0x556f67ac7260 msgr2=0x556f67ad5d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- 172.21.15.28:0/799631483 shutdown_connections 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- 172.21.15.28:0/799631483 wait complete. 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 -- start start 2024-03-22T02:31:48.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 Processor -- start 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ada250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ada790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.814+0000 7fc082d9c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67b46140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781a6d40 con 0x556f67ad1850 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781a81b0 con 0x556f67acc6e0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc0781a7410 con 0x556f67acbcf0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 10 client.0 _ll_register_callbacks cb 0x7fc068009d20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ada250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67b46140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07d75a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ada790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07df5b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ada250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:59624/0 (socket says 172.21.15.28:59624) 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07df5b640 1 -- 172.21.15.28:0/1655780179 learned_addr learned my addr 172.21.15.28:0/1655780179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:31:48.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 -- 172.21.15.28:0/1655780179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 msgr2=0x556f67ada250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ada250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 -- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 msgr2=0x556f67ada790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ada790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 -- 172.21.15.28:0/1655780179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f67b466b0 con 0x556f67ad1850 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07d75a640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ada790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc07e75c640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67b46140 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc0700096e0 tx=0x7fc0700124a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b6c62e9c42ddfd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc0617fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc070023040 con 0x556f67ad1850 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc070009e40 con 0x556f67ad1850 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0740bdbc0 con 0x556f67ad1850 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc07003a050 con 0x556f67ad1850 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 10 client.19968 fetch_fsmap learned FSMap version 46 2024-03-22T02:31:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.815+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556f67b47a30 con 0x556f67ad1850 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0740bdbc0 con 0x556f67ad1850 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fc070005710 con 0x556f67ad1850 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 10 client.19968 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 10 client.19968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 20 client.19968 populate_metadata read hostname 'smithi028' 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67b48100 0x556f67b4a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 4 client.19968 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f67a2f8d0 con 0x556f67b48100 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc07df5b640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67b48100 0x556f67b4a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc0617fa640 10 client.19968 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:48.816+0000 7fc07df5b640 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67b48100 0x556f67b4a520 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27cb053ce45e66c0 server_cookie=ca13fce1a431dd74 in_seq=0 out_seq=0 2024-03-22T02:31:48.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:48 smithi028 ceph-mon[31916]: pgmap v908: 129 pgs: 129 active+clean; 268 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 528 op/s 2024-03-22T02:31:48.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:48 smithi120 ceph-mon[36803]: pgmap v908: 129 pgs: 129 active+clean; 268 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 528 op/s 2024-03-22T02:31:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:48 smithi083 ceph-mon[36340]: pgmap v908: 129 pgs: 129 active+clean; 268 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 528 op/s 2024-03-22T02:31:49.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 80496 15.83 MB/sec execute 149 sec latency 621.504 ms 2024-03-22T02:31:49.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc0617fa640 1 -- 172.21.15.28:0/1655780179 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556f67a2f8d0 con 0x556f67b48100 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc0617fa640 10 client.19968 handle_command_reply: tid=0 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 client.19968 shutdown 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 msgr2=0x556f67b46140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67b46140 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fc0700096e0 tx=0x7fc0700124a0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 shutdown_connections 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556f67b48100 0x556f67b4a520 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556f67ad1850 0x556f67b46140 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556f67acc6e0 0x556f67ada790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc0617fa640 0 client.19968 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:31:49.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 --2- 172.21.15.28:0/1655780179 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556f67acbcf0 0x556f67ada250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 >> 172.21.15.28:0/1655780179 conn(0x556f67ac7260 msgr2=0x556f67ad59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 shutdown_connections 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 1 -- 172.21.15.28:0/1655780179 wait complete. 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.705+0000 7fc082d9c740 20 client.19968 trim_cache size 0 max 16384 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:31:49.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:31:49.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 msgr2=0x7fc0640a5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 0x7fc0640a5580 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc070010c40 tx=0x7fc070017000 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 msgr2=0x7fc07819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fc074005940 tx=0x7fc0740042f0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 shutdown_connections 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc0640a30c0 0x7fc0640a5580 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc078107890 0x7fc0781a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc078104f60 0x7fc07819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 --2- 172.21.15.28:0/3866738447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc078102630 0x7fc07819a050 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 >> 172.21.15.28:0/3866738447 conn(0x7fc0780fc020 msgr2=0x7fc078102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 shutdown_connections 2024-03-22T02:31:49.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:31:49.706+0000 7fc0801bf640 1 -- 172.21.15.28:0/3866738447 wait complete. 2024-03-22T02:31:50.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 80823 15.86 MB/sec execute 150 sec latency 129.504 ms 2024-03-22T02:31:50.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:50 smithi028 ceph-mon[31916]: pgmap v909: 129 pgs: 129 active+clean; 274 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:31:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:50 smithi120 ceph-mon[36803]: pgmap v909: 129 pgs: 129 active+clean; 274 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:31:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:50 smithi083 ceph-mon[36340]: pgmap v909: 129 pgs: 129 active+clean; 274 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:31:51.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 81159 15.91 MB/sec execute 151 sec latency 110.350 ms 2024-03-22T02:31:52.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 81400 15.86 MB/sec execute 152 sec latency 139.137 ms 2024-03-22T02:31:52.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:52 smithi028 ceph-mon[31916]: pgmap v910: 129 pgs: 129 active+clean; 275 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 415 op/s 2024-03-22T02:31:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:52 smithi120 ceph-mon[36803]: pgmap v910: 129 pgs: 129 active+clean; 275 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 415 op/s 2024-03-22T02:31:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:52 smithi083 ceph-mon[36340]: pgmap v910: 129 pgs: 129 active+clean; 275 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 415 op/s 2024-03-22T02:31:53.683 INFO:tasks.workunit.client.0.smithi028.stdout: 10 81627 15.88 MB/sec execute 153 sec latency 223.412 ms 2024-03-22T02:31:54.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 81877 15.85 MB/sec execute 154 sec latency 153.891 ms 2024-03-22T02:31:54.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:54 smithi028 ceph-mon[31916]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:31:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:54 smithi120 ceph-mon[36803]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:31:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:54 smithi083 ceph-mon[36340]: pgmap v911: 129 pgs: 129 active+clean; 270 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:31:55.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 82178 15.84 MB/sec execute 155 sec latency 87.775 ms 2024-03-22T02:31:56.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 82451 15.78 MB/sec execute 156 sec latency 90.655 ms 2024-03-22T02:31:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:56 smithi028 ceph-mon[31916]: pgmap v912: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 580 op/s 2024-03-22T02:31:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:56 smithi120 ceph-mon[36803]: pgmap v912: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 580 op/s 2024-03-22T02:31:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:56 smithi083 ceph-mon[36340]: pgmap v912: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 580 op/s 2024-03-22T02:31:57.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 82760 15.79 MB/sec execute 157 sec latency 94.768 ms 2024-03-22T02:31:58.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 83016 15.76 MB/sec execute 158 sec latency 223.016 ms 2024-03-22T02:31:58.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:31:58 smithi028 ceph-mon[31916]: pgmap v913: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:31:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:31:58 smithi120 ceph-mon[36803]: pgmap v913: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:31:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:31:58 smithi083 ceph-mon[36340]: pgmap v913: 129 pgs: 129 active+clean; 264 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:31:59.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 83379 15.78 MB/sec execute 159 sec latency 57.026 ms 2024-03-22T02:32:00.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 83799 15.88 MB/sec execute 160 sec latency 83.960 ms 2024-03-22T02:32:00.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:00 smithi028 ceph-mon[31916]: pgmap v914: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:32:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:00 smithi120 ceph-mon[36803]: pgmap v914: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:32:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:00 smithi083 ceph-mon[36340]: pgmap v914: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 564 op/s 2024-03-22T02:32:01.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 84255 15.92 MB/sec execute 161 sec latency 61.711 ms 2024-03-22T02:32:02.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 84586 15.94 MB/sec execute 162 sec latency 86.684 ms 2024-03-22T02:32:02.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.802+0000 7f7409e88640 1 -- 172.21.15.28:0/3718815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404104f60 msgr2=0x7f7404107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.802+0000 7f7409e88640 1 --2- 172.21.15.28:0/3718815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404104f60 0x7f7404107350 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f73f8009900 tx=0x7f73f802bc30 comp rx=0 tx=0).stop 2024-03-22T02:32:02.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.803+0000 7f7409e88640 1 -- 172.21.15.28:0/3718815913 shutdown_connections 2024-03-22T02:32:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.804+0000 7f7409e88640 1 --2- 172.21.15.28:0/3718815913 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 0x7f7404109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.804+0000 7f7409e88640 1 --2- 172.21.15.28:0/3718815913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404104f60 0x7f7404107350 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.804+0000 7f7409e88640 1 --2- 172.21.15.28:0/3718815913 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404102630 0x7f7404104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.804+0000 7f7409e88640 1 -- 172.21.15.28:0/3718815913 >> 172.21.15.28:0/3718815913 conn(0x7f74040fc020 msgr2=0x7f74040fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:02.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.812+0000 7f7409e88640 1 -- 172.21.15.28:0/3718815913 shutdown_connections 2024-03-22T02:32:02.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.813+0000 7f7409e88640 1 -- 172.21.15.28:0/3718815913 wait complete. 2024-03-22T02:32:02.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.814+0000 7f7409e88640 1 Processor -- start 2024-03-22T02:32:02.823 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:02 smithi028 ceph-mon[31916]: pgmap v915: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 519 op/s 2024-03-22T02:32:02.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.828+0000 7f7409e88640 1 -- start start 2024-03-22T02:32:02.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.828+0000 7f7409e88640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 0x7f7404197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.828+0000 7f7409e88640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 0x7f7404198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.829 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.829+0000 7f7409e88640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 0x7f740419f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.829+0000 7f7409e88640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f740410d840 con 0x7f7404102630 2024-03-22T02:32:02.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.829+0000 7f7409e88640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f740410d6c0 con 0x7f7404107890 2024-03-22T02:32:02.830 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.829+0000 7f7409e88640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f740410d9c0 con 0x7f7404104f60 2024-03-22T02:32:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.830+0000 7f74037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 0x7f7404197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.830+0000 7f74037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 0x7f7404197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44008/0 (socket says 172.21.15.28:44008) 2024-03-22T02:32:02.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.830+0000 7f74037fe640 1 -- 172.21.15.28:0/2472367728 learned_addr learned my addr 172.21.15.28:0/2472367728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:02.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.831+0000 7f7403fff640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 0x7f740419f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.832+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 0x7f7404198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.835+0000 7f7402ffd640 1 -- 172.21.15.28:0/2472367728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 msgr2=0x7f740419f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.835+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 0x7f740419f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.835+0000 7f7402ffd640 1 -- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 msgr2=0x7f7404197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.835+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 0x7f7404197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.840 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.835+0000 7f7402ffd640 1 -- 172.21.15.28:0/2472367728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f740419fa70 con 0x7f7404104f60 2024-03-22T02:32:02.844 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.842+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 0x7f7404198390 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c170 tx=0x7f73f802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.845+0000 7f7400ff9640 1 -- 172.21.15.28:0/2472367728 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f804d070 con 0x7f7404104f60 2024-03-22T02:32:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.845+0000 7f7400ff9640 1 -- 172.21.15.28:0/2472367728 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f8040de0 con 0x7f7404104f60 2024-03-22T02:32:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.845+0000 7f7400ff9640 1 -- 172.21.15.28:0/2472367728 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f80038e0 con 0x7f7404104f60 2024-03-22T02:32:02.847 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.847+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7404100420 con 0x7f7404104f60 2024-03-22T02:32:02.848 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.847+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7404100890 con 0x7f7404104f60 2024-03-22T02:32:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 Processor -- start 2024-03-22T02:32:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 -- start start 2024-03-22T02:32:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1276910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c127b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569c11ca040 con 0x5569c12764f0 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5569c11729e0 con 0x5569c127c050 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.859+0000 7f740ca77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569c11c5ff0 con 0x5569c1276ee0 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.861+0000 7f7400ff9640 1 -- 172.21.15.28:0/2472367728 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f73f8005190 con 0x7f7404104f60 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.861+0000 7f7402ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.861+0000 7f7402ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49540/0 (socket says 172.21.15.28:49540) 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.861+0000 7f7402ffd640 1 -- 172.21.15.28:0/1876227237 learned_addr learned my addr 172.21.15.28:0/1876227237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 -- 172.21.15.28:0/1876227237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 msgr2=0x5569c127b9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c127b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 -- 172.21.15.28:0/1876227237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 msgr2=0x5569c1276910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1276910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 -- 172.21.15.28:0/1876227237 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c0ede210 con 0x5569c127c050 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7402ffd640 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f73f81231a0 tx=0x7f73f8123290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae2b9a4a2891373c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.862+0000 7f7400ff9640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 0x7f73ec0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.863+0000 7f73e9ffb640 1 -- 172.21.15.28:0/1876227237 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f812e070 con 0x5569c127c050 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.863+0000 7f73e9ffb640 1 -- 172.21.15.28:0/1876227237 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f80fb2e0 con 0x5569c127c050 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.863+0000 7f7400ff9640 1 -- 172.21.15.28:0/2472367728 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f73f80f9160 con 0x7f7404104f60 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.864+0000 7f740ca77740 1 -- 172.21.15.28:0/1876227237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 msgr2=0x5569c127e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.864+0000 7f740ca77740 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f73f81231a0 tx=0x7f73f8123290 comp rx=0 tx=0).stop 2024-03-22T02:32:02.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.866+0000 7f7403fff640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 0x7f73ec0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.867+0000 7f7403fff640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 0x7f73ec0a5670 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x5569c1277740 tx=0x7f73fc009b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.868+0000 7f740ca77740 1 -- 172.21.15.28:0/1876227237 shutdown_connections 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.868+0000 7f740ca77740 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c127c050 0x5569c127e4e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.868+0000 7f740ca77740 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c127b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.868+0000 7f740ca77740 1 --2- 172.21.15.28:0/1876227237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1276910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.868+0000 7f740ca77740 1 -- 172.21.15.28:0/1876227237 >> 172.21.15.28:0/1876227237 conn(0x5569c1260a50 msgr2=0x5569c1260e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.876+0000 7f740ca77740 1 -- 172.21.15.28:0/1876227237 shutdown_connections 2024-03-22T02:32:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.880+0000 7f740ca77740 1 -- 172.21.15.28:0/1876227237 wait complete. 2024-03-22T02:32:02.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.880+0000 7f740ca77740 1 -- start start 2024-03-22T02:32:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 Processor -- start 2024-03-22T02:32:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1266b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5569c1265180 con 0x5569c12764f0 2024-03-22T02:32:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5569c1185db0 con 0x5569c1276ee0 2024-03-22T02:32:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5569c1265300 con 0x5569c127c050 2024-03-22T02:32:02.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.881+0000 7f740ca77740 10 client.0 _ll_register_callbacks cb 0x5569c0edda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44028/0 (socket says 172.21.15.28:44028) 2024-03-22T02:32:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 -- 172.21.15.28:0/2986827713 learned_addr learned my addr 172.21.15.28:0/2986827713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:02.886 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 -- 172.21.15.28:0/2986827713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 msgr2=0x5569c126dbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:32:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 -- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 msgr2=0x5569c1266b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:32:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1266b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.885+0000 7f74037fe640 1 -- 172.21.15.28:0/2986827713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c0faa620 con 0x5569c12764f0 2024-03-22T02:32:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.886+0000 7f74037fe640 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f73f40116f0 tx=0x7f73f4011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940d87263814280a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.889+0000 7f73e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.889+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73f401e060 con 0x5569c12764f0 2024-03-22T02:32:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.889+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73f80fb2e0 con 0x5569c12764f0 2024-03-22T02:32:02.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.889+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f73f400fcd0 con 0x5569c12764f0 2024-03-22T02:32:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.897+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f73f4025050 con 0x5569c12764f0 2024-03-22T02:32:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.898+0000 7f740ca77740 10 client.19977 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.898+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5569c1271060 con 0x5569c12764f0 2024-03-22T02:32:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.907+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73f80fb2e0 con 0x5569c12764f0 2024-03-22T02:32:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f73f400d520 con 0x5569c12764f0 2024-03-22T02:32:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 10 client.19977 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:02.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 10 client.19977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 20 client.19977 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c1272bc0 0x5569c1275040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 4 client.19977 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:32:02.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.908+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569c11d7580 con 0x5569c1272bc0 2024-03-22T02:32:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.912+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c1272bc0 0x5569c1275040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.920+0000 7f7402ffd640 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c1272bc0 0x5569c1275040 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ffed56066dad456 server_cookie=ca52b47f4e4889bb in_seq=0 out_seq=0 2024-03-22T02:32:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.920+0000 7f73e2ffd640 10 client.19977 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.928+0000 7f73e2ffd640 1 -- 172.21.15.28:0/2986827713 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569c11d7580 con 0x5569c1272bc0 2024-03-22T02:32:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.928+0000 7f73e2ffd640 10 client.19977 handle_command_reply: tid=0 2024-03-22T02:32:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.928+0000 7f740ca77740 1 client.19977 shutdown 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 msgr2=0x5569c1266650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f73f40116f0 tx=0x7f73f4011bb0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 shutdown_connections 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c1272bc0 0x5569c1275040 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1266b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 --2- 172.21.15.28:0/2986827713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12764f0 0x5569c1266650 secure :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f73f40116f0 tx=0x7f73f4011bb0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 >> 172.21.15.28:0/2986827713 conn(0x5569c1260a50 msgr2=0x5569c125fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.932+0000 7f73e2ffd640 0 client.19977 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.936+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 shutdown_connections 2024-03-22T02:32:02.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.940+0000 7f740ca77740 1 -- 172.21.15.28:0/2986827713 wait complete. 2024-03-22T02:32:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.941+0000 7f740ca77740 20 client.19977 trim_cache size 0 max 16384 2024-03-22T02:32:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 Processor -- start 2024-03-22T02:32:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 -- start start 2024-03-22T02:32:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1274bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f74041a2820 con 0x5569c12643c0 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f74041a00f0 con 0x5569c1276ee0 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f740ca77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f74040720c0 con 0x5569c127c050 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f7403fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f7403fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44032/0 (socket says 172.21.15.28:44032) 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.974+0000 7f7403fff640 1 -- 172.21.15.28:0/2047507684 learned_addr learned my addr 172.21.15.28:0/2047507684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:02.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 -- 172.21.15.28:0/2047507684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 msgr2=0x5569c126db90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:32:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 -- 172.21.15.28:0/2047507684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 msgr2=0x5569c1274bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1274bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 -- 172.21.15.28:0/2047507684 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569c125c4b0 con 0x5569c12643c0 2024-03-22T02:32:02.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.975+0000 7f7403fff640 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f73fc004a80 tx=0x7f73fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259c016db3d41c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.978+0000 7f73e97fa640 1 -- 172.21.15.28:0/2047507684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc01d7a0 con 0x5569c12643c0 2024-03-22T02:32:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.978+0000 7f73e97fa640 1 -- 172.21.15.28:0/2047507684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73fc021070 con 0x5569c12643c0 2024-03-22T02:32:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.978+0000 7f73e97fa640 1 -- 172.21.15.28:0/2047507684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc01a410 con 0x5569c12643c0 2024-03-22T02:32:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.981+0000 7f740ca77740 1 -- 172.21.15.28:0/2047507684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 msgr2=0x5569c125f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:02.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.981+0000 7f740ca77740 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f73fc004a80 tx=0x7f73fc013000 comp rx=0 tx=0).stop 2024-03-22T02:32:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 -- 172.21.15.28:0/2047507684 shutdown_connections 2024-03-22T02:32:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c12643c0 0x5569c125f3c0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c127c050 0x5569c126db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 --2- 172.21.15.28:0/2047507684 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c1276ee0 0x5569c1274bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 -- 172.21.15.28:0/2047507684 >> 172.21.15.28:0/2047507684 conn(0x5569c1260a50 msgr2=0x5569c1263cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 -- 172.21.15.28:0/2047507684 shutdown_connections 2024-03-22T02:32:02.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.984+0000 7f740ca77740 1 -- 172.21.15.28:0/2047507684 wait complete. 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 -- start start 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 Processor -- start 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c12643c0 0x5569c126a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c126adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7404072730 con 0x5569c127c050 2024-03-22T02:32:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.985+0000 7f740ca77740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7404073ba0 con 0x5569c12643c0 2024-03-22T02:32:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f740ca77740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7404072e00 con 0x5569c1276ee0 2024-03-22T02:32:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f740ca77740 10 client.0 _ll_register_callbacks cb 0x7f73f4009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f7403fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f7403fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44048/0 (socket says 172.21.15.28:44048) 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f7403fff640 1 -- 172.21.15.28:0/844734587 learned_addr learned my addr 172.21.15.28:0/844734587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.986+0000 7f74037fe640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c12643c0 0x5569c126a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 -- 172.21.15.28:0/844734587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 msgr2=0x5569c126adc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c126adc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 -- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c12643c0 msgr2=0x5569c126a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c12643c0 0x5569c126a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:02.988 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 -- 172.21.15.28:0/844734587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569c126b3b0 con 0x5569c127c050 2024-03-22T02:32:02.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f7403fff640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0266a0 tx=0x7f73fc0266d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61ec5b39c10277cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.987+0000 7f73e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73fc004cc0 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f73fc01d5a0 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73fc021070 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f73fc004e60 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f740ca77740 10 client.19983 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.988+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5569c12f29e0 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f73fc021070 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f73fc005630 con 0x5569c127c050 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 10 client.19983 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:02.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 10 client.19983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 20 client.19983 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c12f3300 0x5569c12f5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 4 client.19983 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.989+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569c11d18d0 con 0x5569c12f3300 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.990+0000 7f74037fe640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c12f3300 0x5569c12f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.990+0000 7f73e2ffd640 10 client.19983 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:02.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:02.990+0000 7f74037fe640 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c12f3300 0x5569c12f5720 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d312400b5a0fced2 server_cookie=7e0230d68ba1eca6 in_seq=0 out_seq=0 2024-03-22T02:32:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:02 smithi120 ceph-mon[36803]: pgmap v915: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 519 op/s 2024-03-22T02:32:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:02 smithi083 ceph-mon[36340]: pgmap v915: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 519 op/s 2024-03-22T02:32:03.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 84692 15.90 MB/sec execute 163 sec latency 400.859 ms 2024-03-22T02:32:04.217 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.216+0000 7f73e2ffd640 1 -- 172.21.15.28:0/844734587 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569c11d18d0 con 0x5569c12f3300 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.216+0000 7f73e2ffd640 10 client.19983 handle_command_reply: tid=0 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.216+0000 7f740ca77740 1 client.19983 shutdown 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.216+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 msgr2=0x5569c12e0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.216+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0266a0 tx=0x7f73fc0266d0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 shutdown_connections 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5569c12f3300 0x5569c12f5720 crc :-1 s=READY pgs=399 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5569c127c050 0x5569c12e0410 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5569c1276ee0 0x5569c126adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 --2- 172.21.15.28:0/844734587 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5569c12643c0 0x5569c126a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 >> 172.21.15.28:0/844734587 conn(0x5569c1260a50 msgr2=0x5569c12778e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f73e2ffd640 0 client.19983 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:04.218 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 shutdown_connections 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 1 -- 172.21.15.28:0/844734587 wait complete. 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.217+0000 7f740ca77740 20 client.19983 trim_cache size 0 max 16384 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:32:04.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 msgr2=0x7f73ec0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 0x7f73ec0a5670 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x5569c1277740 tx=0x7f73fc009b00 comp rx=0 tx=0).stop 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 msgr2=0x7f7404198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 0x7f7404198390 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f73f802c170 tx=0x7f73f802c5d0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 shutdown_connections 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73ec0a31b0 0x7f73ec0a5670 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7404107890 0x7f740419f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7404104f60 0x7f7404198390 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 --2- 172.21.15.28:0/2472367728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7404102630 0x7f7404197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 >> 172.21.15.28:0/2472367728 conn(0x7f74040fc020 msgr2=0x7f7404102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 shutdown_connections 2024-03-22T02:32:04.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:04.218+0000 7f7409e88640 1 -- 172.21.15.28:0/2472367728 wait complete. 2024-03-22T02:32:04.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 84793 15.83 MB/sec execute 164 sec latency 689.246 ms 2024-03-22T02:32:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:04 smithi028 ceph-mon[31916]: pgmap v916: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 459 op/s 2024-03-22T02:32:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:04 smithi120 ceph-mon[36803]: pgmap v916: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 459 op/s 2024-03-22T02:32:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:04 smithi083 ceph-mon[36340]: pgmap v916: 129 pgs: 129 active+clean; 272 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 459 op/s 2024-03-22T02:32:05.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 85091 15.85 MB/sec execute 165 sec latency 83.932 ms 2024-03-22T02:32:06.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 85366 15.84 MB/sec execute 166 sec latency 82.949 ms 2024-03-22T02:32:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:06 smithi028 ceph-mon[31916]: pgmap v917: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 546 op/s 2024-03-22T02:32:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:06 smithi120 ceph-mon[36803]: pgmap v917: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 546 op/s 2024-03-22T02:32:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:06 smithi083 ceph-mon[36340]: pgmap v917: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 546 op/s 2024-03-22T02:32:07.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 85634 15.82 MB/sec execute 167 sec latency 108.102 ms 2024-03-22T02:32:08.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 85826 15.77 MB/sec execute 168 sec latency 227.756 ms 2024-03-22T02:32:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:08 smithi028 ceph-mon[31916]: pgmap v918: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 454 op/s 2024-03-22T02:32:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:08 smithi120 ceph-mon[36803]: pgmap v918: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 454 op/s 2024-03-22T02:32:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:08 smithi083 ceph-mon[36340]: pgmap v918: 129 pgs: 129 active+clean; 277 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.2 MiB/s wr, 454 op/s 2024-03-22T02:32:09.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 86171 15.76 MB/sec execute 169 sec latency 66.733 ms 2024-03-22T02:32:10.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 86438 15.73 MB/sec execute 170 sec latency 94.557 ms 2024-03-22T02:32:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:10 smithi120 ceph-mon[36803]: pgmap v919: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 586 op/s 2024-03-22T02:32:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:10 smithi083 ceph-mon[36340]: pgmap v919: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 586 op/s 2024-03-22T02:32:11.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:10 smithi028 ceph-mon[31916]: pgmap v919: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 586 op/s 2024-03-22T02:32:11.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 86884 15.77 MB/sec execute 171 sec latency 63.738 ms 2024-03-22T02:32:12.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 87243 15.82 MB/sec execute 172 sec latency 62.816 ms 2024-03-22T02:32:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:12 smithi120 ceph-mon[36803]: pgmap v920: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:12 smithi083 ceph-mon[36340]: pgmap v920: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:12 smithi028 ceph-mon[31916]: pgmap v920: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:13.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 87591 15.86 MB/sec execute 173 sec latency 211.740 ms 2024-03-22T02:32:14.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 87965 15.87 MB/sec execute 174 sec latency 156.548 ms 2024-03-22T02:32:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:14 smithi120 ceph-mon[36803]: pgmap v921: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:14 smithi083 ceph-mon[36340]: pgmap v921: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:14 smithi028 ceph-mon[31916]: pgmap v921: 129 pgs: 129 active+clean; 269 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:32:15.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 88253 15.90 MB/sec execute 175 sec latency 87.822 ms 2024-03-22T02:32:16.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 88509 15.89 MB/sec execute 176 sec latency 92.607 ms 2024-03-22T02:32:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:16 smithi120 ceph-mon[36803]: pgmap v922: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 602 op/s 2024-03-22T02:32:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:16 smithi083 ceph-mon[36340]: pgmap v922: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 602 op/s 2024-03-22T02:32:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:16 smithi028 ceph-mon[31916]: pgmap v922: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.0 MiB/s wr, 602 op/s 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 -- 172.21.15.28:0/2082150259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94100ab0 msgr2=0x7fbf94100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2082150259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94100ab0 0x7fbf94100ed0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009900 tx=0x7fbf8402bc30 comp rx=0 tx=0).stop 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 -- 172.21.15.28:0/2082150259 shutdown_connections 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2082150259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf94102660 0x7fbf94108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2082150259 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.313 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2082150259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94100ab0 0x7fbf94100ed0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 -- 172.21.15.28:0/2082150259 >> 172.21.15.28:0/2082150259 conn(0x7fbf940fc020 msgr2=0x7fbf940fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 -- 172.21.15.28:0/2082150259 shutdown_connections 2024-03-22T02:32:17.314 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.311+0000 7fbf9a781640 1 -- 172.21.15.28:0/2082150259 wait complete. 2024-03-22T02:32:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 Processor -- start 2024-03-22T02:32:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 -- start start 2024-03-22T02:32:17.316 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9419d610 0x7fbf9419fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410c780 con 0x7fbf94102660 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410c600 con 0x7fbf94101ca0 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf9a781640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9410c900 con 0x7fbf9419d610 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.312+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41714/0 (socket says 172.21.15.28:41714) 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 -- 172.21.15.28:0/2240579040 learned_addr learned my addr 172.21.15.28:0/2240579040 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:17.317 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9419d610 0x7fbf9419fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 -- 172.21.15.28:0/2240579040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9419d610 msgr2=0x7fbf9419fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9419d610 0x7fbf9419fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 -- 172.21.15.28:0/2240579040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 msgr2=0x7fbf94198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 -- 172.21.15.28:0/2240579040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf941a0070 con 0x7fbf94102660 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf93fff640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9f0 tx=0x7fbf7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf917fa640 1 -- 172.21.15.28:0/2240579040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c00cd80 con 0x7fbf94102660 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf941a0360 con 0x7fbf94102660 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf917fa640 1 -- 172.21.15.28:0/2240579040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7c010910 con 0x7fbf94102660 2024-03-22T02:32:17.318 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf917fa640 1 -- 172.21.15.28:0/2240579040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c018990 con 0x7fbf94102660 2024-03-22T02:32:17.319 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.313+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf941a0900 con 0x7fbf94102660 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf917fa640 1 -- 172.21.15.28:0/2240579040 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fbf7c010ac0 con 0x7fbf94102660 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 Processor -- start 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 -- start start 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4afde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cdc40b040 con 0x563cdc4aa8f0 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563cdc3b39e0 con 0x563cdc4b0450 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563cdc406ff0 con 0x563cdc4ab2e0 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf917fa640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 0x7fbf800a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf98d1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf98d1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41718/0 (socket says 172.21.15.28:41718) 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf98d1e640 1 -- 172.21.15.28:0/2908076784 learned_addr learned my addr 172.21.15.28:0/2908076784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf93fff640 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.316+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 0x7fbf800a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf917fa640 1 -- 172.21.15.28:0/2240579040 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fbf7c0cfac0 con 0x7fbf94102660 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 -- 172.21.15.28:0/2908076784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 msgr2=0x563cdc4afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4afde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 -- 172.21.15.28:0/2908076784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 msgr2=0x563cdc4b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 -- 172.21.15.28:0/2908076784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cdc11f210 con 0x563cdc4aa8f0 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005c40 tx=0x7fbf8c00f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3bd64ce5f61d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 0x7fbf800a5500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00f8c0 tx=0x7fbf8c01f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf8a7fc640 1 -- 172.21.15.28:0/2908076784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c00fd30 con 0x563cdc4aa8f0 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf8a7fc640 1 -- 172.21.15.28:0/2908076784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c022030 con 0x563cdc4aa8f0 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.317+0000 7fbf8a7fc640 1 -- 172.21.15.28:0/2908076784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c010930 con 0x563cdc4aa8f0 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2908076784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 msgr2=0x563cdc4aad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c005c40 tx=0x7fbf8c00f3c0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2908076784 shutdown_connections 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2908076784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4aad10 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2908076784 >> 172.21.15.28:0/2908076784 conn(0x563cdc4a5e60 msgr2=0x563cdc4a6260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.328 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2908076784 shutdown_connections 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.318+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2908076784 wait complete. 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 -- start start 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 Processor -- start 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4bbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563cdc4b33a0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x563cdc3c6db0 con 0x563cdc4b0450 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x563cdc4b3520 con 0x563cdc4ab2e0 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf9d36a740 10 client.0 _ll_register_callbacks cb 0x563cdc11ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41732/0 (socket says 172.21.15.28:41732) 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf93fff640 1 -- 172.21.15.28:0/2261254740 learned_addr learned my addr 172.21.15.28:0/2261254740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4bbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.319+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 -- 172.21.15.28:0/2261254740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 msgr2=0x563cdc4b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 -- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 msgr2=0x563cdc4bbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4bbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 -- 172.21.15.28:0/2261254740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cdc1eb620 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf93fff640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009230 tx=0x7fbf840048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85695b637a6c698 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf737fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84005d50 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fbf84004e70 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf8c022030 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fbf84054050 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf9d36a740 10 client.19995 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x563cdc4bf1c0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.320+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf8c022030 con 0x563cdc4aa8f0 2024-03-22T02:32:17.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fbf840038c0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 10 client.19995 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 10 client.19995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 20 client.19995 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc4c0d10 0x563cdc4c3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 4 client.19995 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cdc418580 con 0x563cdc4c0d10 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc4c0d10 0x563cdc4c3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf937fe640 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc4c0d10 0x563cdc4c3180 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74316ce661052d8f server_cookie=a35e804694f7dca5 in_seq=0 out_seq=0 2024-03-22T02:32:17.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.321+0000 7fbf737fe640 10 client.19995 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.326+0000 7fbf737fe640 1 -- 172.21.15.28:0/2261254740 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563cdc418580 con 0x563cdc4c0d10 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.326+0000 7fbf737fe640 10 client.19995 handle_command_reply: tid=0 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.326+0000 7fbf9d36a740 1 client.19995 shutdown 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 msgr2=0x563cdc4b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009230 tx=0x7fbf840048f0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 shutdown_connections 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc4c0d10 0x563cdc4c3180 crc :-1 s=READY pgs=400 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4bbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/2261254740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b4860 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf737fe640 0 client.19995 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 >> 172.21.15.28:0/2261254740 conn(0x563cdc4a5e60 msgr2=0x563cdc4a0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 shutdown_connections 2024-03-22T02:32:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 1 -- 172.21.15.28:0/2261254740 wait complete. 2024-03-22T02:32:17.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.327+0000 7fbf9d36a740 20 client.19995 trim_cache size 0 max 16384 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 Processor -- start 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 -- start start 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4aa8f0 0x563cdc4abb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941ab5c0 con 0x563cdc4ab2e0 2024-03-22T02:32:17.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941a8e90 con 0x563cdc4b0450 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941a6ce0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.343+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4aa8f0 0x563cdc4abb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41744/0 (socket says 172.21.15.28:41744) 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 -- 172.21.15.28:0/4019091767 learned_addr learned my addr 172.21.15.28:0/4019091767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 -- 172.21.15.28:0/4019091767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4aa8f0 msgr2=0x563cdc4abb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4aa8f0 0x563cdc4abb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 -- 172.21.15.28:0/4019091767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 msgr2=0x563cdc4ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 -- 172.21.15.28:0/4019091767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cdc4ac5f0 con 0x563cdc4ab2e0 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf937fe640 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c098800 tx=0x7fbf7c0cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472c5ed7c75d0065 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf89ffb640 1 -- 172.21.15.28:0/4019091767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c026040 con 0x563cdc4ab2e0 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf89ffb640 1 -- 172.21.15.28:0/4019091767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7c09fed0 con 0x563cdc4ab2e0 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.344+0000 7fbf89ffb640 1 -- 172.21.15.28:0/4019091767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c018d90 con 0x563cdc4ab2e0 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- 172.21.15.28:0/4019091767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 msgr2=0x563cdc4bbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c098800 tx=0x7fbf7c0cf040 comp rx=0 tx=0).stop 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- 172.21.15.28:0/4019091767 shutdown_connections 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4b0450 0x563cdc4ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4ab2e0 0x563cdc4bbda0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/4019091767 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4aa8f0 0x563cdc4abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- 172.21.15.28:0/4019091767 >> 172.21.15.28:0/4019091767 conn(0x563cdc4a5e60 msgr2=0x563cdc4a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- 172.21.15.28:0/4019091767 shutdown_connections 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- 172.21.15.28:0/4019091767 wait complete. 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.345+0000 7fbf9d36a740 1 -- start start 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 Processor -- start 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4b0450 0x563cdc4ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941a7350 con 0x563cdc4aa8f0 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941a87c0 con 0x563cdc4ab2e0 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fbf941a7a20 con 0x563cdc4b0450 2024-03-22T02:32:17.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf9d36a740 10 client.0 _ll_register_callbacks cb 0x7fbf84009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41760/0 (socket says 172.21.15.28:41760) 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 -- 172.21.15.28:0/3249183836 learned_addr learned my addr 172.21.15.28:0/3249183836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf937fe640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf98d1e640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4b0450 0x563cdc4ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 -- 172.21.15.28:0/3249183836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4b0450 msgr2=0x563cdc4ba6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4b0450 0x563cdc4ba6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 -- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4ab2e0 msgr2=0x563cdc4b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:17.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 -- 172.21.15.28:0/3249183836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cdc4bace0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf93fff640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009850 tx=0x7fbf84009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebe4fe025e514b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.346+0000 7fbf737fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84051040 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fbf84003a00 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf7c09fed0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fbf84069050 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 10 client.20001 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x563cdc4bc060 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf7c09fed0 con 0x563cdc4aa8f0 2024-03-22T02:32:17.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fbf8404ba80 con 0x563cdc4aa8f0 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 10 client.20001 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 10 client.20001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 20 client.20001 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc527780 0x563cdc529ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 4 client.20001 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cdc4128d0 con 0x563cdc527780 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.347+0000 7fbf937fe640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc527780 0x563cdc529ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.348+0000 7fbf937fe640 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc527780 0x563cdc529ba0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3687b72bb5f731f4 server_cookie=d9ff9b0fd03f72ba in_seq=0 out_seq=0 2024-03-22T02:32:17.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:17.348+0000 7fbf737fe640 10 client.20001 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:17.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 88746 15.89 MB/sec execute 177 sec latency 261.724 ms 2024-03-22T02:32:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.676+0000 7fbf737fe640 1 -- 172.21.15.28:0/3249183836 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563cdc4128d0 con 0x563cdc527780 2024-03-22T02:32:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.676+0000 7fbf737fe640 10 client.20001 handle_command_reply: tid=0 2024-03-22T02:32:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.676+0000 7fbf9d36a740 1 client.20001 shutdown 2024-03-22T02:32:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 msgr2=0x563cdc4b91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009850 tx=0x7fbf84009260 comp rx=0 tx=0).stop 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 shutdown_connections 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x563cdc527780 0x563cdc529ba0 crc :-1 s=READY pgs=401 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf737fe640 0 client.20001 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x563cdc4b0450 0x563cdc4ba6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x563cdc4ab2e0 0x563cdc4b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 --2- 172.21.15.28:0/3249183836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563cdc4aa8f0 0x563cdc4b91a0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 >> 172.21.15.28:0/3249183836 conn(0x563cdc4a5e60 msgr2=0x563cdc4a9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 shutdown_connections 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 1 -- 172.21.15.28:0/3249183836 wait complete. 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.677+0000 7fbf9d36a740 20 client.20001 trim_cache size 0 max 16384 2024-03-22T02:32:18.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:32:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:32:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:32:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:32:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.678+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 msgr2=0x7fbf800a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.678+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 0x7fbf800a5500 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c00f8c0 tx=0x7fbf8c01f000 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.678+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 msgr2=0x7fbf941985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.678+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e9f0 tx=0x7fbf7c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 shutdown_connections 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fbf800a3040 0x7fbf800a5500 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbf9419d610 0x7fbf9419fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94102660 0x7fbf941985b0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 --2- 172.21.15.28:0/2240579040 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fbf94101ca0 0x7fbf94198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 >> 172.21.15.28:0/2240579040 conn(0x7fbf940fc020 msgr2=0x7fbf940fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 shutdown_connections 2024-03-22T02:32:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:18.679+0000 7fbf9a781640 1 -- 172.21.15.28:0/2240579040 wait complete. 2024-03-22T02:32:18.684 INFO:tasks.workunit.client.0.smithi028.stdout: 10 88766 15.80 MB/sec execute 178 sec latency 989.371 ms 2024-03-22T02:32:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:18 smithi120 ceph-mon[36803]: pgmap v923: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 431 op/s 2024-03-22T02:32:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:18 smithi083 ceph-mon[36340]: pgmap v923: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 431 op/s 2024-03-22T02:32:19.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:18 smithi028 ceph-mon[31916]: pgmap v923: 129 pgs: 129 active+clean; 276 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.3 MiB/s wr, 431 op/s 2024-03-22T02:32:19.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 89010 15.79 MB/sec execute 179 sec latency 149.741 ms 2024-03-22T02:32:20.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 89216 15.76 MB/sec execute 180 sec latency 185.005 ms 2024-03-22T02:32:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:20 smithi120 ceph-mon[36803]: pgmap v924: 129 pgs: 129 active+clean; 271 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 480 op/s 2024-03-22T02:32:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:20 smithi083 ceph-mon[36340]: pgmap v924: 129 pgs: 129 active+clean; 271 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 480 op/s 2024-03-22T02:32:21.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:20 smithi028 ceph-mon[31916]: pgmap v924: 129 pgs: 129 active+clean; 271 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 480 op/s 2024-03-22T02:32:21.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 89599 15.72 MB/sec execute 181 sec latency 56.423 ms 2024-03-22T02:32:22.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 89891 15.73 MB/sec execute 182 sec latency 108.654 ms 2024-03-22T02:32:22.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:22 smithi028 ceph-mon[31916]: pgmap v925: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 480 op/s 2024-03-22T02:32:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:22 smithi120 ceph-mon[36803]: pgmap v925: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 480 op/s 2024-03-22T02:32:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:22 smithi083 ceph-mon[36340]: pgmap v925: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.7 MiB/s wr, 480 op/s 2024-03-22T02:32:23.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 90202 15.71 MB/sec execute 183 sec latency 249.058 ms 2024-03-22T02:32:24.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 90581 15.73 MB/sec execute 184 sec latency 86.807 ms 2024-03-22T02:32:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:24 smithi120 ceph-mon[36803]: pgmap v926: 129 pgs: 129 active+clean; 276 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 499 op/s 2024-03-22T02:32:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:24 smithi083 ceph-mon[36340]: pgmap v926: 129 pgs: 129 active+clean; 276 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 499 op/s 2024-03-22T02:32:25.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:24 smithi028 ceph-mon[31916]: pgmap v926: 129 pgs: 129 active+clean; 276 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.4 MiB/s wr, 499 op/s 2024-03-22T02:32:25.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 90969 15.84 MB/sec execute 185 sec latency 108.679 ms 2024-03-22T02:32:26.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 91444 15.87 MB/sec execute 186 sec latency 82.427 ms 2024-03-22T02:32:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:26 smithi120 ceph-mon[36803]: pgmap v927: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 581 op/s 2024-03-22T02:32:26.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:26 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:26 smithi083 ceph-mon[36340]: pgmap v927: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 581 op/s 2024-03-22T02:32:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:26 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:27.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:26 smithi028 ceph-mon[31916]: pgmap v927: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.5 MiB/s wr, 581 op/s 2024-03-22T02:32:27.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:26 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:32:27.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 91709 15.86 MB/sec execute 187 sec latency 76.065 ms 2024-03-22T02:32:27.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:27.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:27.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:32:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:32:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:32:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:32:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:32:28.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 91925 15.87 MB/sec execute 188 sec latency 285.189 ms 2024-03-22T02:32:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:28 smithi120 ceph-mon[36803]: pgmap v928: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 501 op/s 2024-03-22T02:32:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:28 smithi083 ceph-mon[36340]: pgmap v928: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 501 op/s 2024-03-22T02:32:29.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:28 smithi028 ceph-mon[31916]: pgmap v928: 129 pgs: 129 active+clean; 278 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.3 MiB/s wr, 501 op/s 2024-03-22T02:32:29.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 92148 15.84 MB/sec execute 189 sec latency 378.925 ms 2024-03-22T02:32:30.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 92385 15.84 MB/sec execute 190 sec latency 106.589 ms 2024-03-22T02:32:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:30 smithi120 ceph-mon[36803]: pgmap v929: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 521 op/s 2024-03-22T02:32:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:30 smithi083 ceph-mon[36340]: pgmap v929: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 521 op/s 2024-03-22T02:32:31.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:30 smithi028 ceph-mon[31916]: pgmap v929: 129 pgs: 129 active+clean; 277 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 521 op/s 2024-03-22T02:32:31.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 92617 15.82 MB/sec execute 191 sec latency 96.122 ms 2024-03-22T02:32:31.775 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 -- 172.21.15.28:0/1823966804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac075bd0 msgr2=0x7fe9ac074020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/1823966804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac074020 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8009860 tx=0x7fe9a802bb70 comp rx=0 tx=0).stop 2024-03-22T02:32:31.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 -- 172.21.15.28:0/1823966804 shutdown_connections 2024-03-22T02:32:31.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/1823966804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 0x7fe9ac10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/1823966804 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac074a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.776 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/1823966804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac074020 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 -- 172.21.15.28:0/1823966804 >> 172.21.15.28:0/1823966804 conn(0x7fe9ac0fd150 msgr2=0x7fe9ac0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 -- 172.21.15.28:0/1823966804 shutdown_connections 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.773+0000 7fe9b33c7640 1 -- 172.21.15.28:0/1823966804 wait complete. 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 Processor -- start 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 -- start start 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 0x7fe9ac1a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10fa20 con 0x7fe9ac109aa0 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10f8a0 con 0x7fe9ac075bd0 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b33c7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac10fba0 con 0x7fe9ac0745f0 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.777 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 0x7fe9ac1a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b0962640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40592/0 (socket says 172.21.15.28:40592) 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 -- 172.21.15.28:0/639534542 learned_addr learned my addr 172.21.15.28:0/639534542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 -- 172.21.15.28:0/639534542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac075bd0 msgr2=0x7fe9ac19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 -- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 msgr2=0x7fe9ac1a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 0x7fe9ac1a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 -- 172.21.15.28:0/639534542 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ac1a1c50 con 0x7fe9ac0745f0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.774+0000 7fe9b1163640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8005940 tx=0x7fe9a8032470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.775+0000 7fe9a27fc640 1 -- 172.21.15.28:0/639534542 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8041020 con 0x7fe9ac0745f0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.775+0000 7fe9a27fc640 1 -- 172.21.15.28:0/639534542 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a802c9f0 con 0x7fe9ac0745f0 2024-03-22T02:32:31.778 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.775+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ac1a1ee0 con 0x7fe9ac0745f0 2024-03-22T02:32:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.775+0000 7fe9a27fc640 1 -- 172.21.15.28:0/639534542 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8039040 con 0x7fe9ac0745f0 2024-03-22T02:32:31.779 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.775+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ac1a22f0 con 0x7fe9ac0745f0 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.777+0000 7fe9a27fc640 1 -- 172.21.15.28:0/639534542 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe9a8044020 con 0x7fe9ac0745f0 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 Processor -- start 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9a27fc640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 0x7fe9980a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 -- start start 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b0962640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 0x7fe9980a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9a27fc640 1 -- 172.21.15.28:0/639534542 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe9a80f1170 con 0x7fe9ac0745f0 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213179d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e21277040 con 0x560e21318040 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560e2121f9e0 con 0x560e213124e0 2024-03-22T02:32:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.778+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e21272ff0 con 0x560e21312ed0 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:59280/0 (socket says 172.21.15.28:59280) 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51908/0 (socket says 172.21.15.28:51908) 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1163640 1 -- 172.21.15.28:0/3184264380 learned_addr learned my addr 172.21.15.28:0/3184264380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b0962640 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213179d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b0962640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 0x7fe9980a5480 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac19b540 tx=0x7fe99c011440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 -- 172.21.15.28:0/3184264380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 msgr2=0x560e213179d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213179d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 -- 172.21.15.28:0/3184264380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 msgr2=0x560e21312900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 -- 172.21.15.28:0/3184264380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e20f8b210 con 0x560e21318040 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1163640 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9b1964640 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40099b0 tx=0x7fe9a40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d38a72f700a373 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9977fe640 1 -- 172.21.15.28:0/3184264380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4003970 con 0x560e21318040 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9977fe640 1 -- 172.21.15.28:0/3184264380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a400d6b0 con 0x560e21318040 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.779+0000 7fe9977fe640 1 -- 172.21.15.28:0/3184264380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4004310 con 0x560e21318040 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- 172.21.15.28:0/3184264380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 msgr2=0x560e2131a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40099b0 tx=0x7fe9a40095c0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- 172.21.15.28:0/3184264380 shutdown_connections 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131a4d0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213179d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/3184264380 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e213124e0 0x560e21312900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- 172.21.15.28:0/3184264380 >> 172.21.15.28:0/3184264380 conn(0x560e2130da50 msgr2=0x560e2130de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- 172.21.15.28:0/3184264380 shutdown_connections 2024-03-22T02:32:31.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- 172.21.15.28:0/3184264380 wait complete. 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 -- start start 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.780+0000 7fe9b5fad740 1 Processor -- start 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e2131c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213239f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2131aff0 con 0x560e21312ed0 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560e21232db0 con 0x560e21318040 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560e2131b170 con 0x560e213124e0 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b5fad740 10 client.0 _ll_register_callbacks cb 0x560e20f8aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51918/0 (socket says 172.21.15.28:51918) 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 -- 172.21.15.28:0/2631941377 learned_addr learned my addr 172.21.15.28:0/2631941377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:31.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b1964640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b1163640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e2131c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 -- 172.21.15.28:0/2631941377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 msgr2=0x560e2131c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e2131c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 -- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 msgr2=0x560e213239f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213239f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 -- 172.21.15.28:0/2631941377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e21057620 con 0x560e21312ed0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe9b0962640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00fa50 tx=0x7fe99c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2715e0ad7f3811 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.781+0000 7fe994ff9640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c017040 con 0x560e21312ed0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe99c01ea60 con 0x560e21312ed0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9a400d6b0 con 0x560e21312ed0 2024-03-22T02:32:31.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe99c031050 con 0x560e21312ed0 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 10 client.20013 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560e21326dd0 con 0x560e21312ed0 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a400d6b0 con 0x560e21312ed0 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fe99c025030 con 0x560e21312ed0 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 10 client.20013 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 10 client.20013 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 20 client.20013 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e21328580 0x560e2132a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 4 client.20013 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e21284580 con 0x560e21328580 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.782+0000 7fe9b1163640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e21328580 0x560e2132a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.783+0000 7fe994ff9640 10 client.20013 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:31.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.783+0000 7fe9b1163640 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e21328580 0x560e2132a9b0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d4a3818f22961ec server_cookie=e69816e77bb2d416 in_seq=0 out_seq=0 2024-03-22T02:32:31.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe994ff9640 1 -- 172.21.15.28:0/2631941377 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e21284580 con 0x560e21328580 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe994ff9640 10 client.20013 handle_command_reply: tid=0 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 client.20013 shutdown 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 msgr2=0x560e2131ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00fa50 tx=0x7fe99c004730 comp rx=0 tx=0).stop 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 shutdown_connections 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e21328580 0x560e2132a9b0 crc :-1 s=READY pgs=402 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213239f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21312ed0 0x560e2131ca20 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/2631941377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e2131c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 >> 172.21.15.28:0/2631941377 conn(0x560e2130da50 msgr2=0x560e2130cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe994ff9640 0 client.20013 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 shutdown_connections 2024-03-22T02:32:31.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 1 -- 172.21.15.28:0/2631941377 wait complete. 2024-03-22T02:32:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.788+0000 7fe9b5fad740 20 client.20013 trim_cache size 0 max 16384 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 Processor -- start 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 -- start start 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e213113c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 0x560e213239b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1af3f0 con 0x560e21318040 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1accc0 con 0x560e21312ed0 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1aab10 con 0x560e213124e0 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b1964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e213113c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b1964640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51920/0 (socket says 172.21.15.28:51920) 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.811+0000 7fe9b1964640 1 -- 172.21.15.28:0/1359126264 learned_addr learned my addr 172.21.15.28:0/1359126264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b0962640 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 0x560e213239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 -- 172.21.15.28:0/1359126264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 msgr2=0x560e213239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 0x560e213239b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 -- 172.21.15.28:0/1359126264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 msgr2=0x560e2131bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 -- 172.21.15.28:0/1359126264 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e213132b0 con 0x560e213124e0 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b0962640 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 0x560e213239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe9b1163640 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e213113c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9a80b98f0 tx=0x7fe9a80ee730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c458bace2a1288 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe996ffd640 1 -- 172.21.15.28:0/1359126264 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a80bd490 con 0x560e213124e0 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe996ffd640 1 -- 172.21.15.28:0/1359126264 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a80c15e0 con 0x560e213124e0 2024-03-22T02:32:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.812+0000 7fe996ffd640 1 -- 172.21.15.28:0/1359126264 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8032640 con 0x560e213124e0 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 -- 172.21.15.28:0/1359126264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 msgr2=0x560e213113c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e213113c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe9a80b98f0 tx=0x7fe9a80ee730 comp rx=0 tx=0).stop 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 -- 172.21.15.28:0/1359126264 shutdown_connections 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e21318040 0x560e2131bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21312ed0 0x560e213239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/1359126264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e213124e0 0x560e213113c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 -- 172.21.15.28:0/1359126264 >> 172.21.15.28:0/1359126264 conn(0x560e2130da50 msgr2=0x560e21310cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 -- 172.21.15.28:0/1359126264 shutdown_connections 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.813+0000 7fe9b5fad740 1 -- 172.21.15.28:0/1359126264 wait complete. 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 -- start start 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 Processor -- start 2024-03-22T02:32:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 0x560e21320d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213222d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1ab180 con 0x560e213124e0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1ac5f0 con 0x560e21318040 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ac1ab850 con 0x560e21312ed0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b5fad740 10 client.0 _ll_register_callbacks cb 0x7fe99c00f4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.814+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 0x560e21320d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b1163640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 0x560e21320d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51934/0 (socket says 172.21.15.28:51934) 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b1163640 1 -- 172.21.15.28:0/123262499 learned_addr learned my addr 172.21.15.28:0/123262499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b1964640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213222d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 -- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 msgr2=0x560e213222d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213222d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 -- 172.21.15.28:0/123262499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 msgr2=0x560e21320d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 0x560e21320d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 -- 172.21.15.28:0/123262499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e213228c0 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b0962640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213212c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe99c004aa0 tx=0x7fe99c00ff10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57db5fb1a763202c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe994ff9640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c017070 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe99c004040 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.815+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe9a80c15e0 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe99c03d050 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 10 client.29815 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560e21323be0 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9a80c15e0 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fe99c025040 con 0x560e21312ed0 2024-03-22T02:32:31.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 10 client.29815 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 10 client.29815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 20 client.29815 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e2138ef90 0x560e213913b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 4 client.29815 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e2131c2d0 con 0x560e2138ef90 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.816+0000 7fe9b1163640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e2138ef90 0x560e213913b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.817+0000 7fe994ff9640 10 client.29815 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:31.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:31.817+0000 7fe9b1163640 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e2138ef90 0x560e213913b0 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b4891b2a58924cb server_cookie=97faa0e72ecb60db in_seq=0 out_seq=0 2024-03-22T02:32:32.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 92683 15.75 MB/sec execute 192 sec latency 659.595 ms 2024-03-22T02:32:32.753 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe994ff9640 1 -- 172.21.15.28:0/123262499 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e2131c2d0 con 0x560e2138ef90 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe994ff9640 10 client.29815 handle_command_reply: tid=0 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 client.29815 shutdown 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 msgr2=0x560e213212c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213212c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe99c004aa0 tx=0x7fe99c00ff10 comp rx=0 tx=0).stop 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 shutdown_connections 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560e2138ef90 0x560e213913b0 crc :-1 s=READY pgs=403 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560e21318040 0x560e213222d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560e21312ed0 0x560e213212c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 --2- 172.21.15.28:0/123262499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e213124e0 0x560e21320d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe994ff9640 0 client.29815 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:32.754 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.752+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 >> 172.21.15.28:0/123262499 conn(0x560e2130da50 msgr2=0x560e21310cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 shutdown_connections 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b5fad740 1 -- 172.21.15.28:0/123262499 wait complete. 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b5fad740 20 client.29815 trim_cache size 0 max 16384 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:32:32.755 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:32:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 msgr2=0x7fe9980a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 0x7fe9980a5480 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac19b540 tx=0x7fe99c011440 comp rx=0 tx=0).stop 2024-03-22T02:32:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 msgr2=0x7fe9ac19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:32.756 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe9a8005940 tx=0x7fe9a8032470 comp rx=0 tx=0).stop 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 shutdown_connections 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe9980a2fc0 0x7fe9980a5480 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9ac109aa0 0x7fe9ac1a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe9ac075bd0 0x7fe9ac19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 --2- 172.21.15.28:0/639534542 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9ac0745f0 0x7fe9ac19a030 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 >> 172.21.15.28:0/639534542 conn(0x7fe9ac0fd150 msgr2=0x7fe9ac1059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 shutdown_connections 2024-03-22T02:32:32.757 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:32.753+0000 7fe9b33c7640 1 -- 172.21.15.28:0/639534542 wait complete. 2024-03-22T02:32:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:32 smithi120 ceph-mon[36803]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 465 op/s 2024-03-22T02:32:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:32 smithi083 ceph-mon[36340]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 465 op/s 2024-03-22T02:32:33.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:32 smithi028 ceph-mon[31916]: pgmap v930: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 465 op/s 2024-03-22T02:32:33.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 92984 15.71 MB/sec execute 193 sec latency 173.192 ms 2024-03-22T02:32:34.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 93208 15.67 MB/sec execute 194 sec latency 327.866 ms 2024-03-22T02:32:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:34 smithi120 ceph-mon[36803]: pgmap v931: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 352 op/s 2024-03-22T02:32:35.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:34 smithi083 ceph-mon[36340]: pgmap v931: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 352 op/s 2024-03-22T02:32:35.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:34 smithi028 ceph-mon[31916]: pgmap v931: 129 pgs: 129 active+clean; 281 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.6 MiB/s wr, 352 op/s 2024-03-22T02:32:35.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 93453 15.68 MB/sec execute 195 sec latency 148.157 ms 2024-03-22T02:32:36.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 93872 15.68 MB/sec execute 196 sec latency 65.731 ms 2024-03-22T02:32:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:36 smithi120 ceph-mon[36803]: pgmap v932: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-03-22T02:32:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:36 smithi083 ceph-mon[36340]: pgmap v932: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-03-22T02:32:37.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:36 smithi028 ceph-mon[31916]: pgmap v932: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.7 MiB/s wr, 524 op/s 2024-03-22T02:32:37.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 94244 15.71 MB/sec execute 197 sec latency 62.289 ms 2024-03-22T02:32:38.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 94696 15.79 MB/sec execute 198 sec latency 75.594 ms 2024-03-22T02:32:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:38 smithi120 ceph-mon[36803]: pgmap v933: 129 pgs: 129 active+clean; 272 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 503 op/s 2024-03-22T02:32:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:38 smithi083 ceph-mon[36340]: pgmap v933: 129 pgs: 129 active+clean; 272 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 503 op/s 2024-03-22T02:32:39.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:38 smithi028 ceph-mon[31916]: pgmap v933: 129 pgs: 129 active+clean; 272 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.7 MiB/s wr, 503 op/s 2024-03-22T02:32:39.685 INFO:tasks.workunit.client.0.smithi028.stdout: 10 95058 15.80 MB/sec execute 199 sec latency 213.599 ms 2024-03-22T02:32:40.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 95287 15.80 MB/sec execute 200 sec latency 73.324 ms 2024-03-22T02:32:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:40 smithi120 ceph-mon[36803]: pgmap v934: 129 pgs: 129 active+clean; 273 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 544 op/s 2024-03-22T02:32:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:40 smithi083 ceph-mon[36340]: pgmap v934: 129 pgs: 129 active+clean; 273 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 544 op/s 2024-03-22T02:32:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:40 smithi028 ceph-mon[31916]: pgmap v934: 129 pgs: 129 active+clean; 273 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 544 op/s 2024-03-22T02:32:41.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 95570 15.83 MB/sec execute 201 sec latency 137.753 ms 2024-03-22T02:32:42.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 95839 15.82 MB/sec execute 202 sec latency 81.384 ms 2024-03-22T02:32:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:42 smithi120 ceph-mon[36803]: pgmap v935: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 539 op/s 2024-03-22T02:32:43.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:42 smithi083 ceph-mon[36340]: pgmap v935: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 539 op/s 2024-03-22T02:32:43.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:42 smithi028 ceph-mon[31916]: pgmap v935: 129 pgs: 129 active+clean; 271 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 539 op/s 2024-03-22T02:32:43.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 96100 15.80 MB/sec execute 203 sec latency 90.557 ms 2024-03-22T02:32:44.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 96262 15.78 MB/sec execute 204 sec latency 218.105 ms 2024-03-22T02:32:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:44 smithi120 ceph-mon[36803]: pgmap v936: 129 pgs: 129 active+clean; 275 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 513 op/s 2024-03-22T02:32:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:44 smithi083 ceph-mon[36340]: pgmap v936: 129 pgs: 129 active+clean; 275 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 513 op/s 2024-03-22T02:32:45.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:44 smithi028 ceph-mon[31916]: pgmap v936: 129 pgs: 129 active+clean; 275 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.8 MiB/s wr, 513 op/s 2024-03-22T02:32:45.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 96517 15.72 MB/sec execute 205 sec latency 92.088 ms 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- 172.21.15.28:0/658777677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101470 msgr2=0x7f20581018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 --2- 172.21.15.28:0/658777677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101470 0x7f20581018f0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f2044009900 tx=0x7f204402bc30 comp rx=0 tx=0).stop 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- 172.21.15.28:0/658777677 shutdown_connections 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 --2- 172.21.15.28:0/658777677 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101e30 0x7f20581086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 --2- 172.21.15.28:0/658777677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101470 0x7f20581018f0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 --2- 172.21.15.28:0/658777677 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f20581006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- 172.21.15.28:0/658777677 >> 172.21.15.28:0/658777677 conn(0x7f20580fb7f0 msgr2=0x7f20580fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:45.850 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- 172.21.15.28:0/658777677 shutdown_connections 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- 172.21.15.28:0/658777677 wait complete. 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 Processor -- start 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.847+0000 7f205ed7a640 1 -- start start 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f2058197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101470 0x7f2058198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f205810b690 con 0x7f2058101e30 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f205810b510 con 0x7f2058101470 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f205810b810 con 0x7f2058100280 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205d317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f2058197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.851 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f2057fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101470 0x7f2058198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205d317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46878/0 (socket says 172.21.15.28:46878) 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205d317640 1 -- 172.21.15.28:0/2900925276 learned_addr learned my addr 172.21.15.28:0/2900925276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 -- 172.21.15.28:0/2900925276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101470 msgr2=0x7f2058198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101470 0x7f2058198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 -- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 msgr2=0x7f205819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 -- 172.21.15.28:0/2900925276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205819fa30 con 0x7f2058100280 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205d317640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205cb16640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f2058197e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f204c00ea60 tx=0x7f204c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f2055ffb640 1 -- 172.21.15.28:0/2900925276 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c00cc80 con 0x7f2058100280 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205819fd20 con 0x7f2058100280 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f2055ffb640 1 -- 172.21.15.28:0/2900925276 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f204c00ce20 con 0x7f2058100280 2024-03-22T02:32:45.852 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.848+0000 7f2055ffb640 1 -- 172.21.15.28:0/2900925276 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c016670 con 0x7f2058100280 2024-03-22T02:32:45.853 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.849+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20581a01d0 con 0x7f2058100280 2024-03-22T02:32:45.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2055ffb640 1 -- 172.21.15.28:0/2900925276 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f204c003780 con 0x7f2058100280 2024-03-22T02:32:45.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 Processor -- start 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 -- start start 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d80109d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d8011040 0x5585d80134d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585d7f70040 con 0x5585d800bed0 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585d7f189e0 con 0x5585d800b4e0 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2061964740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5585d7f6bff0 con 0x5585d8011040 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205cb16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d80109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205d317640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2055ffb640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 0x7f20480a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205d317640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:48148/0 (socket says 172.21.15.28:48148) 2024-03-22T02:32:45.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205cb16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d80109d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46880/0 (socket says 172.21.15.28:46880) 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205d317640 1 -- 172.21.15.28:0/3754253221 learned_addr learned my addr 172.21.15.28:0/3754253221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f2057fff640 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d8011040 0x5585d80134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.851+0000 7f205d317640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 0x7f20480a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 -- 172.21.15.28:0/3754253221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d8011040 msgr2=0x5585d80134d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f2055ffb640 1 -- 172.21.15.28:0/2900925276 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f204c0cf540 con 0x7f2058100280 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d8011040 0x5585d80134d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 -- 172.21.15.28:0/3754253221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 msgr2=0x5585d80109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d80109d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 -- 172.21.15.28:0/3754253221 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585d7c84210 con 0x5585d800b4e0 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f205000ff20 tx=0x7f20500138f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4dcbbdfdbf7d9b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f205d317640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 0x7f20480a5480 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f205001e000 tx=0x7f2050020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.861 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f2042ffd640 1 -- 172.21.15.28:0/3754253221 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000f470 con 0x5585d800b4e0 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f2042ffd640 1 -- 172.21.15.28:0/3754253221 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2050023030 con 0x5585d800b4e0 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.852+0000 7f2042ffd640 1 -- 172.21.15.28:0/3754253221 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2050010690 con 0x5585d800b4e0 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- 172.21.15.28:0/3754253221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 msgr2=0x5585d800b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f205000ff20 tx=0x7f20500138f0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- 172.21.15.28:0/3754253221 shutdown_connections 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d8011040 0x5585d80134d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d80109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- 172.21.15.28:0/3754253221 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d800b900 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- 172.21.15.28:0/3754253221 >> 172.21.15.28:0/3754253221 conn(0x5585d8006a50 msgr2=0x5585d8006e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- 172.21.15.28:0/3754253221 shutdown_connections 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- 172.21.15.28:0/3754253221 wait complete. 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- start start 2024-03-22T02:32:45.862 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 Processor -- start 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d8015480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d80159c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585d8013f90 con 0x5585d8011040 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5585d7f2bdb0 con 0x5585d800b4e0 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.853+0000 7f2061964740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5585d8014110 con 0x5585d800bed0 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2061964740 10 client.0 _ll_register_callbacks cb 0x5585d7c83a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46896/0 (socket says 172.21.15.28:46896) 2024-03-22T02:32:45.863 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 -- 172.21.15.28:0/261594339 learned_addr learned my addr 172.21.15.28:0/261594339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2057fff640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d80159c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205cb16640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d8015480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 -- 172.21.15.28:0/261594339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 msgr2=0x5585d80159c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d80159c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 -- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 msgr2=0x5585d8015480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d8015480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 -- 172.21.15.28:0/261594339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585d7d50620 con 0x5585d8011040 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f205d317640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f2050013df0 tx=0x7f205000fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef0ee667ba62e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2037fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205000f470 con 0x5585d8011040 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2050023030 con 0x5585d8011040 2024-03-22T02:32:45.864 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f205001f480 con 0x5585d8011040 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f205003b050 con 0x5585d8011040 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2061964740 10 client.20031 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.854+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5585d801fe30 con 0x5585d8011040 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2050023030 con 0x5585d8011040 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f2050028b00 con 0x5585d8011040 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 10 client.20031 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 10 client.20031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 20 client.20031 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d80212b0 0x5585d80236a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 4 client.20031 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:32:45.865 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585d7f7d580 con 0x5585d80212b0 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f205cb16640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d80212b0 0x5585d80236a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f205cb16640 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d80212b0 0x5585d80236a0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3630540e67c07a server_cookie=9263596c277c7eeb in_seq=0 out_seq=0 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.855+0000 7f2037fff640 10 client.20031 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2037fff640 1 -- 172.21.15.28:0/261594339 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5585d7f7d580 con 0x5585d80212b0 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2037fff640 10 client.20031 handle_command_reply: tid=0 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 client.20031 shutdown 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 msgr2=0x5585d801ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f2050013df0 tx=0x7f205000fb00 comp rx=0 tx=0).stop 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 shutdown_connections 2024-03-22T02:32:45.866 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d80212b0 0x5585d80236a0 crc :-1 s=READY pgs=404 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d8011040 0x5585d801ca00 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2037fff640 0 client.20031 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d80159c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 --2- 172.21.15.28:0/261594339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d800b4e0 0x5585d8015480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 >> 172.21.15.28:0/261594339 conn(0x5585d8006a50 msgr2=0x5585d8005c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 shutdown_connections 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 1 -- 172.21.15.28:0/261594339 wait complete. 2024-03-22T02:32:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.857+0000 7f2061964740 20 client.20031 trim_cache size 0 max 16384 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 Processor -- start 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 -- start start 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d801c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d8027f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20581ad1c0 con 0x5585d800b4e0 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f20581aaa90 con 0x5585d8011040 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2061964740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20581a88e0 con 0x5585d800bed0 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.878 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46906/0 (socket says 172.21.15.28:46906) 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 -- 172.21.15.28:0/485368320 learned_addr learned my addr 172.21.15.28:0/485368320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2057fff640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d801c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205d317640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d8027f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 -- 172.21.15.28:0/485368320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 msgr2=0x5585d801c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d801c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 -- 172.21.15.28:0/485368320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 msgr2=0x5585d8027f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d8027f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f205cb16640 1 -- 172.21.15.28:0/485368320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585d8010190 con 0x5585d800b4e0 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.872+0000 7f2057fff640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d801c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f205cb16640 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f204c0ccfa0 tx=0x7f204c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a568822c4ae5452 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f20427fc640 1 -- 172.21.15.28:0/485368320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c09b860 con 0x5585d800b4e0 2024-03-22T02:32:45.879 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f20427fc640 1 -- 172.21.15.28:0/485368320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f204c0a4270 con 0x5585d800b4e0 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f20427fc640 1 -- 172.21.15.28:0/485368320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204c0860d0 con 0x5585d800b4e0 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 -- 172.21.15.28:0/485368320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 msgr2=0x5585d800cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f204c0ccfa0 tx=0x7f204c003b20 comp rx=0 tx=0).stop 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 -- 172.21.15.28:0/485368320 shutdown_connections 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d8027f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800bed0 0x5585d801c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 --2- 172.21.15.28:0/485368320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800b4e0 0x5585d800cfe0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 -- 172.21.15.28:0/485368320 >> 172.21.15.28:0/485368320 conn(0x5585d8006a50 msgr2=0x5585d800c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 -- 172.21.15.28:0/485368320 shutdown_connections 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.873+0000 7f2061964740 1 -- 172.21.15.28:0/485368320 wait complete. 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 -- start start 2024-03-22T02:32:45.880 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 Processor -- start 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 0x5585d801c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d801c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d801a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20581a8f50 con 0x5585d800bed0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f20581aa3c0 con 0x5585d8011040 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f20581a9620 con 0x5585d800b4e0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2061964740 10 client.0 _ll_register_callbacks cb 0x7f204c09b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d801c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f205cb16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 0x5585d801c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f205d317640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d801a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f205cb16640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 0x5585d801c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:40622/0 (socket says 172.21.15.28:40622) 2024-03-22T02:32:45.881 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f205cb16640 1 -- 172.21.15.28:0/3328698489 learned_addr learned my addr 172.21.15.28:0/3328698489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 -- 172.21.15.28:0/3328698489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 msgr2=0x5585d801c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 0x5585d801c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 -- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 msgr2=0x5585d801a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d801a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.874+0000 7f2057fff640 1 -- 172.21.15.28:0/3328698489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585d801b0b0 con 0x5585d800bed0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2057fff640 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d801c870 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f2044009230 tx=0x7f2044004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511b4b2f695e6406 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204404b060 con 0x5585d800bed0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f204c0a4270 con 0x5585d800bed0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f2044005250 con 0x5585d800bed0 2024-03-22T02:32:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f2044052050 con 0x5585d800bed0 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 10 client.20037 fetch_fsmap learned FSMap version 46 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5585d80880a0 con 0x5585d800bed0 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f204c0a4270 con 0x5585d800bed0 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f2044003680 con 0x5585d800bed0 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 10 client.20037 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 10 client.20037 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 20 client.20037 populate_metadata read hostname 'smithi028' 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d8088a80 0x5585d808aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 4 client.20037 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.875+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5585d7f778d0 con 0x5585d8088a80 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.876+0000 7f205cb16640 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d8088a80 0x5585d808aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.876+0000 7f2037fff640 10 client.20037 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:45.876+0000 7f205cb16640 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d8088a80 0x5585d808aea0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8538e976a29ee9 server_cookie=839ecc4b4e73c13b in_seq=0 out_seq=0 2024-03-22T02:32:46.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 96676 15.66 MB/sec execute 206 sec latency 509.854 ms 2024-03-22T02:32:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2037fff640 1 -- 172.21.15.28:0/3328698489 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5585d7f778d0 con 0x5585d8088a80 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2037fff640 10 client.20037 handle_command_reply: tid=0 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 client.20037 shutdown 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 msgr2=0x5585d801c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d801c870 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f2044009230 tx=0x7f2044004250 comp rx=0 tx=0).stop 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 shutdown_connections 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5585d8088a80 0x5585d808aea0 crc :-1 s=READY pgs=405 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5585d8011040 0x5585d801a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585d800bed0 0x5585d801c870 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 --2- 172.21.15.28:0/3328698489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5585d800b4e0 0x5585d801c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 >> 172.21.15.28:0/3328698489 conn(0x5585d8006a50 msgr2=0x5585d800d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2037fff640 0 client.20037 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:32:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 shutdown_connections 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 1 -- 172.21.15.28:0/3328698489 wait complete. 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.943+0000 7f2061964740 20 client.20037 trim_cache size 0 max 16384 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:32:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 msgr2=0x7f20480a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 0x7f20480a5480 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f205001e000 tx=0x7f2050020000 comp rx=0 tx=0).stop 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 msgr2=0x7f2058197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f2058197e10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f204c00ea60 tx=0x7f204c00ef20 comp rx=0 tx=0).stop 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 shutdown_connections 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f20480a2fc0 0x7f20480a5480 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2058101e30 0x7f205819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2058101470 0x7f2058198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 --2- 172.21.15.28:0/2900925276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2058100280 0x7f2058197e10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:32:46.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 >> 172.21.15.28:0/2900925276 conn(0x7f20580fb7f0 msgr2=0x7f20581068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:32:46.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 shutdown_connections 2024-03-22T02:32:46.948 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:32:46.944+0000 7f205ed7a640 1 -- 172.21.15.28:0/2900925276 wait complete. 2024-03-22T02:32:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:46 smithi120 ceph-mon[36803]: pgmap v937: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 520 op/s 2024-03-22T02:32:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:46 smithi083 ceph-mon[36340]: pgmap v937: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 520 op/s 2024-03-22T02:32:47.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:46 smithi028 ceph-mon[31916]: pgmap v937: 129 pgs: 129 active+clean; 279 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 520 op/s 2024-03-22T02:32:47.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 96951 15.66 MB/sec execute 207 sec latency 100.701 ms 2024-03-22T02:32:48.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 97385 15.68 MB/sec execute 208 sec latency 99.507 ms 2024-03-22T02:32:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:48 smithi120 ceph-mon[36803]: pgmap v938: 129 pgs: 129 active+clean; 282 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-22T02:32:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:48 smithi083 ceph-mon[36340]: pgmap v938: 129 pgs: 129 active+clean; 282 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-22T02:32:49.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:48 smithi028 ceph-mon[31916]: pgmap v938: 129 pgs: 129 active+clean; 282 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 353 op/s 2024-03-22T02:32:49.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 97693 15.68 MB/sec execute 209 sec latency 161.285 ms 2024-03-22T02:32:50.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 98114 15.77 MB/sec execute 210 sec latency 71.417 ms 2024-03-22T02:32:51.002 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:50 smithi120 ceph-mon[36803]: pgmap v939: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 390 op/s 2024-03-22T02:32:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:50 smithi083 ceph-mon[36340]: pgmap v939: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 390 op/s 2024-03-22T02:32:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:50 smithi028 ceph-mon[31916]: pgmap v939: 129 pgs: 129 active+clean; 286 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.4 MiB/s wr, 390 op/s 2024-03-22T02:32:51.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 98482 15.76 MB/sec execute 211 sec latency 214.322 ms 2024-03-22T02:32:52.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 98780 15.80 MB/sec execute 212 sec latency 89.964 ms 2024-03-22T02:32:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:52 smithi120 ceph-mon[36803]: pgmap v940: 129 pgs: 129 active+clean; 276 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-03-22T02:32:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:52 smithi083 ceph-mon[36340]: pgmap v940: 129 pgs: 129 active+clean; 276 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-03-22T02:32:53.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:52 smithi028 ceph-mon[31916]: pgmap v940: 129 pgs: 129 active+clean; 276 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.7 MiB/s wr, 549 op/s 2024-03-22T02:32:53.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 99112 15.83 MB/sec execute 213 sec latency 81.888 ms 2024-03-22T02:32:54.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 99291 15.80 MB/sec execute 214 sec latency 218.586 ms 2024-03-22T02:32:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:54 smithi120 ceph-mon[36803]: pgmap v941: 129 pgs: 129 active+clean; 268 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 550 op/s 2024-03-22T02:32:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:54 smithi083 ceph-mon[36340]: pgmap v941: 129 pgs: 129 active+clean; 268 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 550 op/s 2024-03-22T02:32:55.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:54 smithi028 ceph-mon[31916]: pgmap v941: 129 pgs: 129 active+clean; 268 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.6 MiB/s wr, 550 op/s 2024-03-22T02:32:55.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 99477 15.77 MB/sec execute 215 sec latency 132.364 ms 2024-03-22T02:32:56.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 99674 15.74 MB/sec execute 216 sec latency 92.233 ms 2024-03-22T02:32:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:56 smithi120 ceph-mon[36803]: pgmap v942: 129 pgs: 129 active+clean; 269 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 573 op/s 2024-03-22T02:32:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:56 smithi083 ceph-mon[36340]: pgmap v942: 129 pgs: 129 active+clean; 269 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 573 op/s 2024-03-22T02:32:57.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:56 smithi028 ceph-mon[31916]: pgmap v942: 129 pgs: 129 active+clean; 269 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.0 MiB/s wr, 573 op/s 2024-03-22T02:32:57.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 99904 15.72 MB/sec execute 217 sec latency 132.513 ms 2024-03-22T02:32:58.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 100233 15.68 MB/sec execute 218 sec latency 70.778 ms 2024-03-22T02:32:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:32:58 smithi120 ceph-mon[36803]: pgmap v943: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:32:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:32:58 smithi083 ceph-mon[36340]: pgmap v943: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:32:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:32:58 smithi028 ceph-mon[31916]: pgmap v943: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.2 MiB/s wr, 545 op/s 2024-03-22T02:32:59.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 100473 15.67 MB/sec execute 219 sec latency 189.568 ms 2024-03-22T02:33:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 -- 172.21.15.28:0/474001152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102690 msgr2=0x7fd0a8108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/474001152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102690 0x7fd0a8108f20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd09c01a910 tx=0x7fd09c03cc60 comp rx=0 tx=0).stop 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 -- 172.21.15.28:0/474001152 shutdown_connections 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/474001152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8102690 0x7fd0a8108f20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/474001152 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8101cd0 0x7fd0a8102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/474001152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8100ae0 0x7fd0a8100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.085+0000 7fd0ae10f640 1 -- 172.21.15.28:0/474001152 >> 172.21.15.28:0/474001152 conn(0x7fd0a80fc030 msgr2=0x7fd0a80fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:00.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.086+0000 7fd0ae10f640 1 -- 172.21.15.28:0/474001152 shutdown_connections 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.086+0000 7fd0ae10f640 1 -- 172.21.15.28:0/474001152 wait complete. 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.086+0000 7fd0ae10f640 1 Processor -- start 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 -- start start 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ae0 0x7fd0a8107ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8101cd0 0x7fd0a8108450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a806e1b0 con 0x7fd0a8102690 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a806e030 con 0x7fd0a8101cd0 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0ae10f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a806e330 con 0x7fd0a8100ae0 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0a7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55766/0 (socket says 172.21.15.28:55766) 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0a7fff640 1 -- 172.21.15.28:0/906606608 learned_addr learned my addr 172.21.15.28:0/906606608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.087+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ae0 0x7fd0a8107ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 -- 172.21.15.28:0/906606608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ae0 msgr2=0x7fd0a8107ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ae0 0x7fd0a8107ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 -- 172.21.15.28:0/906606608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8101cd0 msgr2=0x7fd0a8108450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8101cd0 0x7fd0a8108450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 -- 172.21.15.28:0/906606608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a8194fe0 con 0x7fd0a8102690 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.088+0000 7fd0a7fff640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd09c004880 tx=0x7fd09c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.089+0000 7fd0a4ff9640 1 -- 172.21.15.28:0/906606608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c052020 con 0x7fd0a8102690 2024-03-22T02:33:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.089+0000 7fd0a4ff9640 1 -- 172.21.15.28:0/906606608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd09c003da0 con 0x7fd0a8102690 2024-03-22T02:33:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.089+0000 7fd0a4ff9640 1 -- 172.21.15.28:0/906606608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c04e480 con 0x7fd0a8102690 2024-03-22T02:33:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.089+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a8195270 con 0x7fd0a8102690 2024-03-22T02:33:00.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.089+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a8195770 con 0x7fd0a8102690 2024-03-22T02:33:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.092+0000 7fd0b0dcb740 1 Processor -- start 2024-03-22T02:33:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.092+0000 7fd0a4ff9640 1 -- 172.21.15.28:0/906606608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd09c055020 con 0x7fd0a8102690 2024-03-22T02:33:00.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 -- start start 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c81900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d380be6040 con 0x55d380c81ed0 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d380b8e9e0 con 0x55d380c87040 2024-03-22T02:33:00.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d380be1ff0 con 0x55d380c814e0 2024-03-22T02:33:00.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.093+0000 7fd0a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55778/0 (socket says 172.21.15.28:55778) 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a4ff9640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 0x7fd0900a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 -- 172.21.15.28:0/3047707528 learned_addr learned my addr 172.21.15.28:0/3047707528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a7fff640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 0x7fd0900a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a4ff9640 1 -- 172.21.15.28:0/906606608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd09c102660 con 0x7fd0a8102690 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 -- 172.21.15.28:0/3047707528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 msgr2=0x55d380c81900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c81900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 -- 172.21.15.28:0/3047707528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 msgr2=0x55d380c894d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.094+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c894d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.095+0000 7fd0a77fe640 1 -- 172.21.15.28:0/3047707528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3808fa210 con 0x55d380c81ed0 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.095+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cc40 tx=0x7fd09800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08126c51c2e42f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.095+0000 7fd0a7fff640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 0x7fd0900a56c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0cad30 tx=0x7fd09c0cac40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.095+0000 7fd08dffb640 1 -- 172.21.15.28:0/3047707528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098014070 con 0x55d380c81ed0 2024-03-22T02:33:00.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.095+0000 7fd08dffb640 1 -- 172.21.15.28:0/3047707528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd098005500 con 0x55d380c81ed0 2024-03-22T02:33:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.096+0000 7fd08dffb640 1 -- 172.21.15.28:0/3047707528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098004d90 con 0x55d380c81ed0 2024-03-22T02:33:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.096+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3047707528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 msgr2=0x55d380c869d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.096+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd09800cc40 tx=0x7fd09800e720 comp rx=0 tx=0).stop 2024-03-22T02:33:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3047707528 shutdown_connections 2024-03-22T02:33:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c869d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3047707528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c81900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3047707528 >> 172.21.15.28:0/3047707528 conn(0x55d380c7ca50 msgr2=0x55d380c7ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3047707528 shutdown_connections 2024-03-22T02:33:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3047707528 wait complete. 2024-03-22T02:33:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 -- start start 2024-03-22T02:33:00.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.099+0000 7fd0b0dcb740 1 Processor -- start 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 0x55d380c8b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c8bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c87040 0x55d380c92a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d380c8a040 con 0x55d380c87040 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55d380ba1db0 con 0x55d380c814e0 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55d380c8a1c0 con 0x55d380c81ed0 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0b0dcb740 10 client.0 _ll_register_callbacks cb 0x55d3808f9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 0x55d380c8b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.102 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 0x55d380c8b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46980/0 (socket says 172.21.15.28:46980) 2024-03-22T02:33:00.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/155034883 learned_addr learned my addr 172.21.15.28:0/155034883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:00.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c8bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.100+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c87040 0x55d380c92a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 -- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 msgr2=0x55d380c8b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 0x55d380c8b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 -- 172.21.15.28:0/155034883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c87040 msgr2=0x55d380c92a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c87040 0x55d380c92a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 -- 172.21.15.28:0/155034883 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3809c6620 con 0x55d380c81ed0 2024-03-22T02:33:00.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c8bad0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55d380c7d9c0 tx=0x7fd098005360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f033c1cdeb263812 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd086ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098014040 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd098005500 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.101+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd098010430 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.102+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd0980046c0 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.102+0000 7fd0b0dcb740 10 client.29842 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.102+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55d380c95df0 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.102+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd098005500 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fd0980186e0 con 0x55d380c81ed0 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 10 client.29842 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 10 client.29842 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 20 client.29842 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:00.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380c97940 0x55d380c99d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 4 client.29842 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:33:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.103+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d380bf3580 con 0x55d380c97940 2024-03-22T02:33:00.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.104+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380c97940 0x55d380c99d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.108+0000 7fd086ffd640 10 client.29842 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:00.110 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.108+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380c97940 0x55d380c99d30 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82135fdb83a958eb server_cookie=a27164857b3a12e8 in_seq=0 out_seq=0 2024-03-22T02:33:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.113+0000 7fd086ffd640 1 -- 172.21.15.28:0/155034883 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d380bf3580 con 0x55d380c97940 2024-03-22T02:33:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.113+0000 7fd086ffd640 10 client.29842 handle_command_reply: tid=0 2024-03-22T02:33:00.114 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.113+0000 7fd0b0dcb740 1 client.29842 shutdown 2024-03-22T02:33:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.113+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 msgr2=0x55d380c8bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.113+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c8bad0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x55d380c7d9c0 tx=0x7fd098005360 comp rx=0 tx=0).stop 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 shutdown_connections 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380c97940 0x55d380c99d30 crc :-1 s=READY pgs=406 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c87040 0x55d380c92a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c8bad0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/155034883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c814e0 0x55d380c8b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 >> 172.21.15.28:0/155034883 conn(0x55d380c7ca50 msgr2=0x55d380c7bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:00.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.114+0000 7fd086ffd640 0 client.29842 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.115+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 shutdown_connections 2024-03-22T02:33:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.115+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/155034883 wait complete. 2024-03-22T02:33:00.117 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.115+0000 7fd0b0dcb740 20 client.29842 trim_cache size 0 max 16384 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 Processor -- start 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 -- start start 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c929f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c82970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd094012660 con 0x55d380c814e0 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400ff30 con 0x55d380c87040 2024-03-22T02:33:00.145 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.143+0000 7fd0b0dcb740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400dd80 con 0x55d380c81ed0 2024-03-22T02:33:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55796/0 (socket says 172.21.15.28:55796) 2024-03-22T02:33:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/3074009087 learned_addr learned my addr 172.21.15.28:0/3074009087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:00.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c82970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/3074009087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 msgr2=0x55d380c929f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c929f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/3074009087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 msgr2=0x55d380c82970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c82970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.144+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/3074009087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d380c81110 con 0x55d380c814e0 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.145+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0005fd0 tx=0x7fd0a0005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8260bb792fe76d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.145+0000 7fd08d7fa640 1 -- 172.21.15.28:0/3074009087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0023030 con 0x55d380c814e0 2024-03-22T02:33:00.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.145+0000 7fd08d7fa640 1 -- 172.21.15.28:0/3074009087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a001d380 con 0x55d380c814e0 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.145+0000 7fd08d7fa640 1 -- 172.21.15.28:0/3074009087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a000fa60 con 0x55d380c814e0 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3074009087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 msgr2=0x55d380c8af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0005fd0 tx=0x7fd0a0005bd0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3074009087 shutdown_connections 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c82970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c81ed0 0x55d380c929f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/3074009087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c814e0 0x55d380c8af90 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.146+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3074009087 >> 172.21.15.28:0/3074009087 conn(0x55d380c7ca50 msgr2=0x55d380c8a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:00.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.147+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3074009087 shutdown_connections 2024-03-22T02:33:00.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.147+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/3074009087 wait complete. 2024-03-22T02:33:00.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.147+0000 7fd0b0dcb740 1 -- start start 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.147+0000 7fd0b0dcb740 1 Processor -- start 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.147+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c8f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:35094/0 (socket says 172.21.15.28:35094) 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c8f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/2794813363 learned_addr learned my addr 172.21.15.28:0/2794813363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c8f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400e3f0 con 0x55d380c81ed0 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400f860 con 0x55d380c87040 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd09400eac0 con 0x55d380c814e0 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0b0dcb740 10 client.0 _ll_register_callbacks cb 0x7fd09800ca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c8f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 msgr2=0x55d380c8f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c8f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/2794813363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 msgr2=0x55d380c8f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c8f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a6ffd640 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d380c92ba0 con 0x55d380c814e0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.148+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c8f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0a6ffd640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000a010 tx=0x7fd0a00125a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24722f3af72d80bd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0032060 con 0x55d380c814e0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd0a0004b20 con 0x55d380c814e0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0a001d380 con 0x55d380c814e0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd0a0039050 con 0x55d380c814e0 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 10 client.29848 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:00.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55d380cfcb40 con 0x55d380c814e0 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0a001d380 con 0x55d380c814e0 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fd0a001b030 con 0x55d380c814e0 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 10 client.29848 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 10 client.29848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 20 client.29848 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380cfd340 0x55d380cff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 4 client.29848 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.149+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d380bed8d0 con 0x55d380cfd340 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.150+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380cfd340 0x55d380cff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.150+0000 7fd0a77fe640 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380cfd340 0x55d380cff760 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45de3974e9a54c7 server_cookie=4ab33f6cd1462e15 in_seq=0 out_seq=0 2024-03-22T02:33:00.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:00.150+0000 7fd086ffd640 10 client.29848 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:00.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 100598 15.61 MB/sec execute 220 sec latency 341.121 ms 2024-03-22T02:33:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:00 smithi120 ceph-mon[36803]: pgmap v944: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 510 op/s 2024-03-22T02:33:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:00 smithi083 ceph-mon[36340]: pgmap v944: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 510 op/s 2024-03-22T02:33:01.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:00 smithi028 ceph-mon[31916]: pgmap v944: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 510 op/s 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.242+0000 7fd086ffd640 1 -- 172.21.15.28:0/2794813363 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d380bed8d0 con 0x55d380cfd340 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.242+0000 7fd086ffd640 10 client.29848 handle_command_reply: tid=0 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.242+0000 7fd0b0dcb740 1 client.29848 shutdown 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.242+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 msgr2=0x55d380c92570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.242+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000a010 tx=0x7fd0a00125a0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 shutdown_connections 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55d380cfd340 0x55d380cff760 crc :-1 s=READY pgs=407 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55d380c87040 0x55d380c8f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd086ffd640 0 client.29848 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d380c81ed0 0x55d380c8f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 --2- 172.21.15.28:0/2794813363 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d380c814e0 0x55d380c92570 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 >> 172.21.15.28:0/2794813363 conn(0x55d380c7ca50 msgr2=0x55d380c7fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 shutdown_connections 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 1 -- 172.21.15.28:0/2794813363 wait complete. 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.243+0000 7fd0b0dcb740 20 client.29848 trim_cache size 0 max 16384 2024-03-22T02:33:01.245 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:33:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:33:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:33:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.244+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 msgr2=0x7fd0900a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.244+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 0x7fd0900a56c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0cad30 tx=0x7fd09c0cac40 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.244+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 msgr2=0x7fd0a81948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.244+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd09c004880 tx=0x7fd09c0045e0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 shutdown_connections 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0900a3200 0x7fd0900a56c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a8102690 0x7fd0a81948b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a8101cd0 0x7fd0a8108450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 --2- 172.21.15.28:0/906606608 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a8100ae0 0x7fd0a8107ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 >> 172.21.15.28:0/906606608 conn(0x7fd0a80fc030 msgr2=0x7fd0a80fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 shutdown_connections 2024-03-22T02:33:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:01.245+0000 7fd0ae10f640 1 -- 172.21.15.28:0/906606608 wait complete. 2024-03-22T02:33:01.686 INFO:tasks.workunit.client.0.smithi028.stdout: 10 100829 15.58 MB/sec execute 221 sec latency 776.605 ms 2024-03-22T02:33:02.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 101332 15.67 MB/sec execute 222 sec latency 80.777 ms 2024-03-22T02:33:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:02 smithi120 ceph-mon[36803]: pgmap v945: 129 pgs: 129 active+clean; 277 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 500 op/s 2024-03-22T02:33:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:02 smithi083 ceph-mon[36340]: pgmap v945: 129 pgs: 129 active+clean; 277 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 500 op/s 2024-03-22T02:33:03.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:02 smithi028 ceph-mon[31916]: pgmap v945: 129 pgs: 129 active+clean; 277 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.5 MiB/s wr, 500 op/s 2024-03-22T02:33:03.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 101817 15.73 MB/sec execute 223 sec latency 67.051 ms 2024-03-22T02:33:04.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 102179 15.77 MB/sec execute 224 sec latency 201.643 ms 2024-03-22T02:33:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:04 smithi120 ceph-mon[36803]: pgmap v946: 129 pgs: 129 active+clean; 273 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.5 MiB/s wr, 340 op/s 2024-03-22T02:33:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:04 smithi083 ceph-mon[36340]: pgmap v946: 129 pgs: 129 active+clean; 273 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.5 MiB/s wr, 340 op/s 2024-03-22T02:33:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:04 smithi028 ceph-mon[31916]: pgmap v946: 129 pgs: 129 active+clean; 273 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 3.5 MiB/s wr, 340 op/s 2024-03-22T02:33:05.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 102409 15.77 MB/sec execute 225 sec latency 120.564 ms 2024-03-22T02:33:06.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 102703 15.78 MB/sec execute 226 sec latency 66.262 ms 2024-03-22T02:33:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:06 smithi120 ceph-mon[36803]: pgmap v947: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 542 op/s 2024-03-22T02:33:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:06 smithi083 ceph-mon[36340]: pgmap v947: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 542 op/s 2024-03-22T02:33:07.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:06 smithi028 ceph-mon[31916]: pgmap v947: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.7 MiB/s wr, 542 op/s 2024-03-22T02:33:07.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 102936 15.77 MB/sec execute 227 sec latency 64.901 ms 2024-03-22T02:33:08.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 103145 15.74 MB/sec execute 228 sec latency 111.166 ms 2024-03-22T02:33:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:08 smithi120 ceph-mon[36803]: pgmap v948: 129 pgs: 129 active+clean; 267 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 512 op/s 2024-03-22T02:33:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:08 smithi083 ceph-mon[36340]: pgmap v948: 129 pgs: 129 active+clean; 267 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 512 op/s 2024-03-22T02:33:09.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:08 smithi028 ceph-mon[31916]: pgmap v948: 129 pgs: 129 active+clean; 267 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 512 op/s 2024-03-22T02:33:09.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 103317 15.70 MB/sec execute 229 sec latency 293.466 ms 2024-03-22T02:33:10.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 103522 15.68 MB/sec execute 230 sec latency 512.977 ms 2024-03-22T02:33:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:10 smithi120 ceph-mon[36803]: pgmap v949: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:33:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:10 smithi083 ceph-mon[36340]: pgmap v949: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:33:11.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:10 smithi028 ceph-mon[31916]: pgmap v949: 129 pgs: 129 active+clean; 270 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:33:11.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 103786 15.64 MB/sec execute 231 sec latency 138.583 ms 2024-03-22T02:33:12.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 104131 15.63 MB/sec execute 232 sec latency 80.186 ms 2024-03-22T02:33:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:12 smithi120 ceph-mon[36803]: pgmap v950: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 554 op/s 2024-03-22T02:33:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:12 smithi083 ceph-mon[36340]: pgmap v950: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 554 op/s 2024-03-22T02:33:13.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:12 smithi028 ceph-mon[31916]: pgmap v950: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.4 MiB/s wr, 554 op/s 2024-03-22T02:33:13.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 104611 15.67 MB/sec execute 233 sec latency 60.479 ms 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 -- 172.21.15.28:0/1537114541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc103780 msgr2=0x7f67bc10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 --2- 172.21.15.28:0/1537114541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc103780 0x7f67bc10a010 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009ca0 tx=0x7f67b402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 -- 172.21.15.28:0/1537114541 shutdown_connections 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 --2- 172.21.15.28:0/1537114541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc103780 0x7f67bc10a010 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 --2- 172.21.15.28:0/1537114541 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc102dc0 0x7f67bc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 --2- 172.21.15.28:0/1537114541 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc101bd0 0x7f67bc101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.342 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 -- 172.21.15.28:0/1537114541 >> 172.21.15.28:0/1537114541 conn(0x7f67bc0fd120 msgr2=0x7f67bc0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 -- 172.21.15.28:0/1537114541 shutdown_connections 2024-03-22T02:33:14.343 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.341+0000 7f67c11f7640 1 -- 172.21.15.28:0/1537114541 wait complete. 2024-03-22T02:33:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.342+0000 7f67c11f7640 1 Processor -- start 2024-03-22T02:33:14.344 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.342+0000 7f67c11f7640 1 -- start start 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.342+0000 7f67c11f7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc101bd0 0x7f67bc197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.342+0000 7f67c11f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc102dc0 0x7f67bc1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.342+0000 7f67c11f7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67c11f7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10d830 con 0x7f67bc101bd0 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67c11f7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10d6b0 con 0x7f67bc103780 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67c11f7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc10d9b0 con 0x7f67bc102dc0 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50740/0 (socket says 172.21.15.28:50740) 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 -- 172.21.15.28:0/3155391524 learned_addr learned my addr 172.21.15.28:0/3155391524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67ba59c640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc102dc0 0x7f67bc1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bad9d640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc101bd0 0x7f67bc197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 -- 172.21.15.28:0/3155391524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc102dc0 msgr2=0x7f67bc1982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.345 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc102dc0 0x7f67bc1982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 -- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc101bd0 msgr2=0x7f67bc197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc101bd0 0x7f67bc197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 -- 172.21.15.28:0/3155391524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67bc19f9b0 con 0x7f67bc103780 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.343+0000 7f67bb59e640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f67b40325f0 tx=0x7f67b4032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.344+0000 7f67a3fff640 1 -- 172.21.15.28:0/3155391524 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4044070 con 0x7f67bc103780 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.344+0000 7f67a3fff640 1 -- 172.21.15.28:0/3155391524 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4034cd0 con 0x7f67bc103780 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.344+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67bc073dc0 con 0x7f67bc103780 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.344+0000 7f67a3fff640 1 -- 172.21.15.28:0/3155391524 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4039040 con 0x7f67bc103780 2024-03-22T02:33:14.346 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.344+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67bc0742b0 con 0x7f67bc103780 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.346+0000 7f67a3fff640 1 -- 172.21.15.28:0/3155391524 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f67b4047070 con 0x7f67bc103780 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 Processor -- start 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 -- start start 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee044e0 0x56534ee04900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.348 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee0a040 0x56534ee0c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56534ed69040 con 0x56534ee04ed0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56534ed119e0 con 0x56534ee0a040 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67c3de4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56534ed64ff0 con 0x56534ee044e0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67a3fff640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 0x7f67a40a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34400/0 (socket says 172.21.15.28:34400) 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 -- 172.21.15.28:0/3828696008 learned_addr learned my addr 172.21.15.28:0/3828696008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67ba59c640 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee044e0 0x56534ee04900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67ba59c640 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee0a040 0x56534ee0c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67a3fff640 1 -- 172.21.15.28:0/3155391524 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f67b40f0ec0 con 0x7f67bc103780 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bb59e640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 0x7f67a40a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 -- 172.21.15.28:0/3828696008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee044e0 msgr2=0x56534ee04900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee044e0 0x56534ee04900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.349 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 -- 172.21.15.28:0/3828696008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee0a040 msgr2=0x56534ee0c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee0a040 0x56534ee0c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.347+0000 7f67bad9d640 1 -- 172.21.15.28:0/3828696008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56534ea7d210 con 0x56534ee04ed0 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67bad9d640 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009bf0 tx=0x7f67ac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f85d90ffac91f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67bb59e640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 0x7f67a40a5480 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f67b40b95b0 tx=0x7f67b40095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67a17fa640 1 -- 172.21.15.28:0/3828696008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac012450 con 0x56534ee04ed0 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67a17fa640 1 -- 172.21.15.28:0/3828696008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac012ad0 con 0x56534ee04ed0 2024-03-22T02:33:14.350 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67a17fa640 1 -- 172.21.15.28:0/3828696008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00fb40 con 0x56534ee04ed0 2024-03-22T02:33:14.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67c3de4740 1 -- 172.21.15.28:0/3828696008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 msgr2=0x56534ee099d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.348+0000 7f67c3de4740 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009bf0 tx=0x7f67ac0117b0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- 172.21.15.28:0/3828696008 shutdown_connections 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee0a040 0x56534ee0c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee04ed0 0x56534ee099d0 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- 172.21.15.28:0/3828696008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee044e0 0x56534ee04900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- 172.21.15.28:0/3828696008 >> 172.21.15.28:0/3828696008 conn(0x56534edffa50 msgr2=0x56534edffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- 172.21.15.28:0/3828696008 shutdown_connections 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- 172.21.15.28:0/3828696008 wait complete. 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- start start 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 Processor -- start 2024-03-22T02:33:14.354 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee0e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee0a040 0x56534ee159f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56534ee0cff0 con 0x56534ee0a040 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56534ed24db0 con 0x56534ee044e0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.349+0000 7f67c3de4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56534ee0d170 con 0x56534ee04ed0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67c3de4740 10 client.0 _ll_register_callbacks cb 0x56534ea7ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37144/0 (socket says 172.21.15.28:37144) 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 -- 172.21.15.28:0/955332900 learned_addr learned my addr 172.21.15.28:0/955332900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67ba59c640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee0a040 0x56534ee159f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67ba59c640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee0e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 -- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 msgr2=0x56534ee0e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee0e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.355 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 -- 172.21.15.28:0/955332900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee0a040 msgr2=0x56534ee159f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee0a040 0x56534ee159f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 -- 172.21.15.28:0/955332900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56534eb49620 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67bad9d640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00f480 tx=0x7f67ac012c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d73dc8a2d61a71 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f679eff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac028070 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ac012ad0 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.350+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f67ac004450 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f67ac033050 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 10 client.29860 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56534ee18d60 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac012ad0 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f67ac00f8c0 con 0x56534ee04ed0 2024-03-22T02:33:14.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 10 client.29860 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 10 client.29860 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 20 client.29860 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee1a170 0x56534ee1c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 4 client.29860 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56534ed76580 con 0x56534ee1a170 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.351+0000 7f67ba59c640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee1a170 0x56534ee1c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.352+0000 7f67ba59c640 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee1a170 0x56534ee1c560 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ed98dc31e6f1ad8 server_cookie=c0fe235bd5916166 in_seq=0 out_seq=0 2024-03-22T02:33:14.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.352+0000 7f679eff5640 10 client.29860 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:14.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f679eff5640 1 -- 172.21.15.28:0/955332900 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56534ed76580 con 0x56534ee1a170 2024-03-22T02:33:14.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f679eff5640 10 client.29860 handle_command_reply: tid=0 2024-03-22T02:33:14.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 client.29860 shutdown 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 msgr2=0x56534ee0ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00f480 tx=0x7f67ac012c80 comp rx=0 tx=0).stop 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 shutdown_connections 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee1a170 0x56534ee1c560 crc :-1 s=READY pgs=408 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee0a040 0x56534ee159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f679eff5640 0 client.29860 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee04ed0 0x56534ee0ea20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.355+0000 7f67c3de4740 1 --2- 172.21.15.28:0/955332900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee0e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.356+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 >> 172.21.15.28:0/955332900 conn(0x56534edffa50 msgr2=0x56534edfec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.356+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 shutdown_connections 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.356+0000 7f67c3de4740 1 -- 172.21.15.28:0/955332900 wait complete. 2024-03-22T02:33:14.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.356+0000 7f67c3de4740 20 client.29860 trim_cache size 0 max 16384 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 Processor -- start 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 -- start start 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee05760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1aafb0 con 0x56534ee044e0 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1a8880 con 0x56534ee04ed0 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67c3de4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1a66d0 con 0x56534ee0a040 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.376 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50762/0 (socket says 172.21.15.28:50762) 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67bad9d640 1 -- 172.21.15.28:0/4192768852 learned_addr learned my addr 172.21.15.28:0/4192768852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34414/0 (socket says 172.21.15.28:34414) 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.374+0000 7f67ba59c640 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee05760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 -- 172.21.15.28:0/4192768852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 msgr2=0x56534ee05760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee05760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 -- 172.21.15.28:0/4192768852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 msgr2=0x56534edfead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 -- 172.21.15.28:0/4192768852 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56534edff3a0 con 0x56534ee04ed0 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67ba59c640 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67bad9d640 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0092e0 tx=0x7f67ac00ffa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33464493c76846ad server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67a0ff9640 1 -- 172.21.15.28:0/4192768852 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac028070 con 0x56534ee04ed0 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67a0ff9640 1 -- 172.21.15.28:0/4192768852 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac009a40 con 0x56534ee04ed0 2024-03-22T02:33:14.377 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.375+0000 7f67a0ff9640 1 -- 172.21.15.28:0/4192768852 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac012880 con 0x56534ee04ed0 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- 172.21.15.28:0/4192768852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 msgr2=0x56534ee159b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0092e0 tx=0x7f67ac00ffa0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- 172.21.15.28:0/4192768852 shutdown_connections 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee05760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee04ed0 0x56534ee159b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- 172.21.15.28:0/4192768852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee044e0 0x56534edfead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- 172.21.15.28:0/4192768852 >> 172.21.15.28:0/4192768852 conn(0x56534edffa50 msgr2=0x56534edfe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- 172.21.15.28:0/4192768852 shutdown_connections 2024-03-22T02:33:14.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- 172.21.15.28:0/4192768852 wait complete. 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- start start 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 Processor -- start 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee13b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee1a170 0x56534ee14b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1a6d40 con 0x56534ee1a170 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1a81b0 con 0x56534ee044e0 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f67bc1a7410 con 0x56534ee0a040 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.376+0000 7f67c3de4740 10 client.0 _ll_register_callbacks cb 0x7f67b0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee1a170 0x56534ee14b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee13b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50770/0 (socket says 172.21.15.28:50770) 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67bad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee13b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:37158/0 (socket says 172.21.15.28:37158) 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 -- 172.21.15.28:0/2657501823 learned_addr learned my addr 172.21.15.28:0/2657501823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 -- 172.21.15.28:0/2657501823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 msgr2=0x56534ee13b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee13b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 -- 172.21.15.28:0/2657501823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee1a170 msgr2=0x56534ee14b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee1a170 0x56534ee14b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 -- 172.21.15.28:0/2657501823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56534ee15110 con 0x56534ee044e0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67ba59c640 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009820 tx=0x7f67b002f040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f8d374a2241431 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f679eff5640 10 client.0 ms_handle_connect on v2:172.21.15.83:3300/0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b002d5f0 con 0x56534ee044e0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67ac009a40 con 0x56534ee044e0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mon.2 v2:172.21.15.83:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f67b002d790 con 0x56534ee044e0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f67b0040050 con 0x56534ee044e0 2024-03-22T02:33:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.377+0000 7f67c3de4740 10 client.29870 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56534ee80a00 con 0x56534ee044e0 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mon.2 v2:172.21.15.83:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ac009a40 con 0x56534ee044e0 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mon.2 v2:172.21.15.83:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f67b0004b40 con 0x56534ee044e0 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 10 client.29870 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 10 client.29870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 20 client.29870 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee813d0 0x56534ee837f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 4 client.29870 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56534ed708d0 con 0x56534ee813d0 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67bad9d640 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee813d0 0x56534ee837f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f679eff5640 10 client.29870 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:14.378+0000 7f67bad9d640 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee813d0 0x56534ee837f0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58049454c93bc074 server_cookie=3aa26b448fd7cebe in_seq=0 out_seq=0 2024-03-22T02:33:14.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 104970 15.70 MB/sec execute 234 sec latency 212.979 ms 2024-03-22T02:33:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:14 smithi120 ceph-mon[36803]: pgmap v951: 129 pgs: 129 active+clean; 274 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 464 op/s 2024-03-22T02:33:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:14 smithi083 ceph-mon[36340]: pgmap v951: 129 pgs: 129 active+clean; 274 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 464 op/s 2024-03-22T02:33:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:14 smithi028 ceph-mon[31916]: pgmap v951: 129 pgs: 129 active+clean; 274 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.2 MiB/s wr, 464 op/s 2024-03-22T02:33:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.568+0000 7f679eff5640 1 -- 172.21.15.28:0/2657501823 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56534ed708d0 con 0x56534ee813d0 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.568+0000 7f679eff5640 10 client.29870 handle_command_reply: tid=0 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.568+0000 7f67c3de4740 1 client.29870 shutdown 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 msgr2=0x56534ee13620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009820 tx=0x7f67b002f040 comp rx=0 tx=0).stop 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 shutdown_connections 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56534ee813d0 0x56534ee837f0 crc :-1 s=READY pgs=409 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56534ee1a170 0x56534ee14b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f679eff5640 0 client.29870 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56534ee0a040 0x56534ee13b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 --2- 172.21.15.28:0/2657501823 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56534ee044e0 0x56534ee13620 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 >> 172.21.15.28:0/2657501823 conn(0x56534edffa50 msgr2=0x56534edfec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 shutdown_connections 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.569+0000 7f67c3de4740 1 -- 172.21.15.28:0/2657501823 wait complete. 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.570+0000 7f67c3de4740 20 client.29870 trim_cache size 0 max 16384 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:33:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.570+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 msgr2=0x7f67a40a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.570+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 0x7f67a40a5480 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f67b40b95b0 tx=0x7f67b40095c0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.570+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 msgr2=0x7f67bc19f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.570+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f67b40325f0 tx=0x7f67b4032f20 comp rx=0 tx=0).stop 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 shutdown_connections 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f67a40a2fc0 0x7f67a40a5480 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f67bc103780 0x7f67bc19f310 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f67bc102dc0 0x7f67bc1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 --2- 172.21.15.28:0/3155391524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67bc101bd0 0x7f67bc197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 >> 172.21.15.28:0/3155391524 conn(0x7f67bc0fd120 msgr2=0x7f67bc105ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 shutdown_connections 2024-03-22T02:33:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:15.571+0000 7f67c11f7640 1 -- 172.21.15.28:0/3155391524 wait complete. 2024-03-22T02:33:15.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 105144 15.70 MB/sec execute 235 sec latency 983.771 ms 2024-03-22T02:33:16.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 105576 15.71 MB/sec execute 236 sec latency 111.937 ms 2024-03-22T02:33:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:16 smithi120 ceph-mon[36803]: pgmap v952: 129 pgs: 129 active+clean; 276 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 482 op/s 2024-03-22T02:33:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:16 smithi083 ceph-mon[36340]: pgmap v952: 129 pgs: 129 active+clean; 276 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 482 op/s 2024-03-22T02:33:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:16 smithi028 ceph-mon[31916]: pgmap v952: 129 pgs: 129 active+clean; 276 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 482 op/s 2024-03-22T02:33:17.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 105933 15.78 MB/sec execute 237 sec latency 58.891 ms 2024-03-22T02:33:18.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 106200 15.78 MB/sec execute 238 sec latency 86.329 ms 2024-03-22T02:33:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:18 smithi120 ceph-mon[36803]: pgmap v953: 129 pgs: 129 active+clean; 280 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 389 op/s 2024-03-22T02:33:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:18 smithi083 ceph-mon[36340]: pgmap v953: 129 pgs: 129 active+clean; 280 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 389 op/s 2024-03-22T02:33:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:18 smithi028 ceph-mon[31916]: pgmap v953: 129 pgs: 129 active+clean; 280 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 389 op/s 2024-03-22T02:33:19.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 106431 15.78 MB/sec execute 239 sec latency 97.823 ms 2024-03-22T02:33:20.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 106551 15.73 MB/sec execute 240 sec latency 379.107 ms 2024-03-22T02:33:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:20 smithi120 ceph-mon[36803]: pgmap v954: 129 pgs: 129 active+clean; 276 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 420 op/s 2024-03-22T02:33:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:20 smithi083 ceph-mon[36340]: pgmap v954: 129 pgs: 129 active+clean; 276 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 420 op/s 2024-03-22T02:33:21.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:20 smithi028 ceph-mon[31916]: pgmap v954: 129 pgs: 129 active+clean; 276 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 420 op/s 2024-03-22T02:33:21.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 106717 15.69 MB/sec execute 241 sec latency 107.198 ms 2024-03-22T02:33:22.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 106925 15.66 MB/sec execute 242 sec latency 103.084 ms 2024-03-22T02:33:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:22 smithi120 ceph-mon[36803]: pgmap v955: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 571 op/s 2024-03-22T02:33:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:22 smithi083 ceph-mon[36340]: pgmap v955: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 571 op/s 2024-03-22T02:33:23.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:22 smithi028 ceph-mon[31916]: pgmap v955: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 571 op/s 2024-03-22T02:33:23.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 107279 15.64 MB/sec execute 243 sec latency 72.528 ms 2024-03-22T02:33:24.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 107665 15.64 MB/sec execute 244 sec latency 69.409 ms 2024-03-22T02:33:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:24 smithi120 ceph-mon[36803]: pgmap v956: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:33:25.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:24 smithi083 ceph-mon[36340]: pgmap v956: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:33:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:24 smithi028 ceph-mon[31916]: pgmap v956: 129 pgs: 129 active+clean; 267 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:33:25.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 108015 15.65 MB/sec execute 245 sec latency 305.198 ms 2024-03-22T02:33:26.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 108395 15.68 MB/sec execute 246 sec latency 121.796 ms 2024-03-22T02:33:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:26 smithi120 ceph-mon[36803]: pgmap v957: 129 pgs: 129 active+clean; 271 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-03-22T02:33:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:26 smithi083 ceph-mon[36340]: pgmap v957: 129 pgs: 129 active+clean; 271 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-03-22T02:33:27.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:26 smithi028 ceph-mon[31916]: pgmap v957: 129 pgs: 129 active+clean; 271 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.2 MiB/s wr, 585 op/s 2024-03-22T02:33:27.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 108921 15.76 MB/sec execute 247 sec latency 127.696 ms 2024-03-22T02:33:27.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:27 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:28.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:27 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:27 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:33:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 -- 172.21.15.28:0/1921152779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560104f60 msgr2=0x7fc560107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 --2- 172.21.15.28:0/1921152779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560104f60 0x7fc560107350 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc554009900 tx=0x7fc55402bc30 comp rx=0 tx=0).stop 2024-03-22T02:33:28.662 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 -- 172.21.15.28:0/1921152779 shutdown_connections 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 --2- 172.21.15.28:0/1921152779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 0x7fc560109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 --2- 172.21.15.28:0/1921152779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560104f60 0x7fc560107350 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 --2- 172.21.15.28:0/1921152779 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560102630 0x7fc560104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 -- 172.21.15.28:0/1921152779 >> 172.21.15.28:0/1921152779 conn(0x7fc5600fc020 msgr2=0x7fc5600fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 -- 172.21.15.28:0/1921152779 shutdown_connections 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.660+0000 7fc565d25640 1 -- 172.21.15.28:0/1921152779 wait complete. 2024-03-22T02:33:28.663 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 Processor -- start 2024-03-22T02:33:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 -- start start 2024-03-22T02:33:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560104f60 0x7fc5601983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.665 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 0x7fc56019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d820 con 0x7fc560102630 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d6a0 con 0x7fc560104f60 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc565d25640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc56010d9a0 con 0x7fc560107890 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.661+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58158/0 (socket says 172.21.15.28:58158) 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 -- 172.21.15.28:0/3541215989 learned_addr learned my addr 172.21.15.28:0/3541215989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55effd640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560104f60 0x7fc5601983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55ffff640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 0x7fc56019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 -- 172.21.15.28:0/3541215989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 msgr2=0x7fc56019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 0x7fc56019f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.666 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 -- 172.21.15.28:0/3541215989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560104f60 msgr2=0x7fc5601983a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560104f60 0x7fc5601983a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 -- 172.21.15.28:0/3541215989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56019fa80 con 0x7fc560102630 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9f0 tx=0x7fc55000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55cff9640 1 -- 172.21.15.28:0/3541215989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55000cd80 con 0x7fc560102630 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56019fd70 con 0x7fc560102630 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55cff9640 1 -- 172.21.15.28:0/3541215989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc550010910 con 0x7fc560102630 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc55cff9640 1 -- 172.21.15.28:0/3541215989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500188f0 con 0x7fc560102630 2024-03-22T02:33:28.667 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.662+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5601a0220 con 0x7fc560102630 2024-03-22T02:33:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.665+0000 7fc55cff9640 1 -- 172.21.15.28:0/3541215989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fc550010ac0 con 0x7fc560102630 2024-03-22T02:33:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.665+0000 7fc55cff9640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 0x7fc5480a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.665+0000 7fc5689ec740 1 Processor -- start 2024-03-22T02:33:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55cff9640 1 -- 172.21.15.28:0/3541215989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc5500cfb90 con 0x7fc560102630 2024-03-22T02:33:28.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55effd640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 0x7fc5480a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 -- start start 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b805ed0 0x556e0b80a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b80d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e0b76a040 con 0x556e0b805ed0 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556e0b7129e0 con 0x556e0b80b040 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc5689ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556e0b765ff0 con 0x556e0b8054e0 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56398/0 (socket says 172.21.15.28:56398) 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 -- 172.21.15.28:0/2660297736 learned_addr learned my addr 172.21.15.28:0/2660297736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:28.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55ffff640 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b80d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55effd640 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b805ed0 0x556e0b80a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55effd640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 0x7fc5480a5480 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc554000c00 tx=0x7fc554005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 -- 172.21.15.28:0/2660297736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 msgr2=0x556e0b80d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b80d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 -- 172.21.15.28:0/2660297736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b805ed0 msgr2=0x556e0b80a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b805ed0 0x556e0b80a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 -- 172.21.15.28:0/2660297736 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e0b47e210 con 0x556e0b8054e0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.666+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc55009bef0 tx=0x7fc55009c0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77ea92f7b30526f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc545ffb640 1 -- 172.21.15.28:0/2660297736 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55009a260 con 0x556e0b8054e0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc545ffb640 1 -- 172.21.15.28:0/2660297736 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5500985d0 con 0x556e0b8054e0 2024-03-22T02:33:28.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc545ffb640 1 -- 172.21.15.28:0/2660297736 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550097f10 con 0x556e0b8054e0 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 -- 172.21.15.28:0/2660297736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 msgr2=0x556e0b805900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc55009bef0 tx=0x7fc55009c0a0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 -- 172.21.15.28:0/2660297736 shutdown_connections 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b80d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b805ed0 0x556e0b80a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 --2- 172.21.15.28:0/2660297736 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b805900 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 -- 172.21.15.28:0/2660297736 >> 172.21.15.28:0/2660297736 conn(0x556e0b800a50 msgr2=0x556e0b800e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 -- 172.21.15.28:0/2660297736 shutdown_connections 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.667+0000 7fc5689ec740 1 -- 172.21.15.28:0/2660297736 wait complete. 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 -- start start 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 Processor -- start 2024-03-22T02:33:28.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b80f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 0x556e0b80f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556e0b80df90 con 0x556e0b80b040 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x556e0b725db0 con 0x556e0b805ed0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x556e0b80e110 con 0x556e0b8054e0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc5689ec740 10 client.0 _ll_register_callbacks cb 0x556e0b47da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b80f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc55effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 0x556e0b80f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58172/0 (socket says 172.21.15.28:58172) 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.668+0000 7fc55ffff640 1 -- 172.21.15.28:0/3507568585 learned_addr learned my addr 172.21.15.28:0/3507568585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:28.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 -- 172.21.15.28:0/3507568585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 msgr2=0x556e0b80f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b80f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 -- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 msgr2=0x556e0b80f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 0x556e0b80f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 -- 172.21.15.28:0/3507568585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e0b54a620 con 0x556e0b80b040 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b80f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55effd640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 0x556e0b80f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc55ffff640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc558011d10 tx=0x7fc55800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481fa0c4086a153c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc53effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558009e30 con 0x556e0b80b040 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5500985d0 con 0x556e0b80b040 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc558004b20 con 0x556e0b80b040 2024-03-22T02:33:28.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc558024050 con 0x556e0b80b040 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc5689ec740 10 client.20082 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556e0b819e40 con 0x556e0b80b040 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.669+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5500985d0 con 0x556e0b80b040 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fc558019030 con 0x556e0b80b040 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 10 client.20082 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 10 client.20082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 20 client.20082 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b81d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 4 client.20082 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e0b777580 con 0x556e0b81b580 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b81d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc53effd640 10 client.20082 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:28.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.670+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b81d9b0 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=228ce0cf93c3d138 server_cookie=3ebc13bf60797781 in_seq=0 out_seq=0 2024-03-22T02:33:28.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.676+0000 7fc53effd640 1 -- 172.21.15.28:0/3507568585 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556e0b777580 con 0x556e0b81b580 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.676+0000 7fc53effd640 10 client.20082 handle_command_reply: tid=0 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.676+0000 7fc5689ec740 1 client.20082 shutdown 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 msgr2=0x556e0b8169b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc558011d10 tx=0x7fc55800f790 comp rx=0 tx=0).stop 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 shutdown_connections 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b81d9b0 crc :-1 s=READY pgs=410 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b8169b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc53effd640 0 client.20082 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b805ed0 0x556e0b80f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 --2- 172.21.15.28:0/3507568585 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b8054e0 0x556e0b80f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 >> 172.21.15.28:0/3507568585 conn(0x556e0b800a50 msgr2=0x556e0b7ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:28.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 shutdown_connections 2024-03-22T02:33:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 1 -- 172.21.15.28:0/3507568585 wait complete. 2024-03-22T02:33:28.682 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.677+0000 7fc5689ec740 20 client.20082 trim_cache size 0 max 16384 2024-03-22T02:33:28.687 INFO:tasks.workunit.client.0.smithi028.stdout: 10 109299 15.80 MB/sec execute 248 sec latency 94.992 ms 2024-03-22T02:33:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.694+0000 7fc5689ec740 1 Processor -- start 2024-03-22T02:33:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.694+0000 7fc5689ec740 1 -- start start 2024-03-22T02:33:28.697 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b816970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b7ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601ad1f0 con 0x556e0b8054e0 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601aaac0 con 0x556e0b80b040 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5689ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601a8910 con 0x556e0b805ed0 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b816970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58174/0 (socket says 172.21.15.28:58174) 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b816970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56410/0 (socket says 172.21.15.28:56410) 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 -- 172.21.15.28:0/550764835 learned_addr learned my addr 172.21.15.28:0/550764835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55ffff640 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b7ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 -- 172.21.15.28:0/550764835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 msgr2=0x556e0b816970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b816970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 -- 172.21.15.28:0/550764835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 msgr2=0x556e0b7ff3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b7ff3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 -- 172.21.15.28:0/550764835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e0b806760 con 0x556e0b8054e0 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cd570 tx=0x7fc5500a4a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a55d247e785e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.695+0000 7fc5457fa640 1 -- 172.21.15.28:0/550764835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5500985d0 con 0x556e0b8054e0 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5457fa640 1 -- 172.21.15.28:0/550764835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55009b3b0 con 0x556e0b8054e0 2024-03-22T02:33:28.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5457fa640 1 -- 172.21.15.28:0/550764835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc550096ad0 con 0x556e0b8054e0 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 -- 172.21.15.28:0/550764835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 msgr2=0x556e0b8043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc5500cd570 tx=0x7fc5500a4a00 comp rx=0 tx=0).stop 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 -- 172.21.15.28:0/550764835 shutdown_connections 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b80b040 0x556e0b7ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b816970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 --2- 172.21.15.28:0/550764835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b8054e0 0x556e0b8043c0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 -- 172.21.15.28:0/550764835 >> 172.21.15.28:0/550764835 conn(0x556e0b800a50 msgr2=0x556e0b803cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 -- 172.21.15.28:0/550764835 shutdown_connections 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.696+0000 7fc5689ec740 1 -- 172.21.15.28:0/550764835 wait complete. 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 -- start start 2024-03-22T02:33:28.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 Processor -- start 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 0x556e0b814320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b814860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601a8f80 con 0x556e0b80b040 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601aa3f0 con 0x556e0b8054e0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fc5601a9650 con 0x556e0b805ed0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc5689ec740 10 client.0 _ll_register_callbacks cb 0x7fc558009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55f7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 0x556e0b814320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58190/0 (socket says 172.21.15.28:58190) 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55ffff640 1 -- 172.21.15.28:0/102684284 learned_addr learned my addr 172.21.15.28:0/102684284 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55effd640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b814860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55ffff640 1 -- 172.21.15.28:0/102684284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 msgr2=0x556e0b814860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.697+0000 7fc55ffff640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b814860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc55ffff640 1 -- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 msgr2=0x556e0b814320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc55ffff640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 0x556e0b814320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc55ffff640 1 -- 172.21.15.28:0/102684284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556e0b815e60 con 0x556e0b80b040 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 0x556e0b814320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc55ffff640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc558011ce0 tx=0x7fc558011860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbbb47c79a1b9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558003970 con 0x556e0b80b040 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc55009b3b0 con 0x556e0b80b040 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fc558017040 con 0x556e0b80b040 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fc558037050 con 0x556e0b80b040 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc5689ec740 10 client.20088 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:28.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x556e0b885fb0 con 0x556e0b80b040 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc55009b3b0 con 0x556e0b80b040 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.698+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fc558019040 con 0x556e0b80b040 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 10 client.20088 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 10 client.20088 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 20 client.20088 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b888a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 4 client.20088 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556e0b7718d0 con 0x556e0b81b580 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b888a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:28.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc55f7fe640 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b888a10 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfd89ec80de1f58f server_cookie=18e56baf186c08c6 in_seq=0 out_seq=0 2024-03-22T02:33:28.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:28.699+0000 7fc53effd640 10 client.20088 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:28 smithi120 ceph-mon[36803]: pgmap v958: 129 pgs: 129 active+clean; 273 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-03-22T02:33:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:33:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:28 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:33:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:28 smithi083 ceph-mon[36340]: pgmap v958: 129 pgs: 129 active+clean; 273 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-03-22T02:33:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:33:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:28 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:33:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:28 smithi028 ceph-mon[31916]: pgmap v958: 129 pgs: 129 active+clean; 273 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.7 MiB/s wr, 567 op/s 2024-03-22T02:33:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:33:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:33:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:28 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:33:29.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 109348 15.75 MB/sec execute 249 sec latency 911.810 ms 2024-03-22T02:33:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.757+0000 7fc53effd640 1 -- 172.21.15.28:0/102684284 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556e0b7718d0 con 0x556e0b81b580 2024-03-22T02:33:29.760 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.757+0000 7fc53effd640 10 client.20088 handle_command_reply: tid=0 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.757+0000 7fc5689ec740 1 client.20088 shutdown 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 msgr2=0x556e0b815870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc558011ce0 tx=0x7fc558011860 comp rx=0 tx=0).stop 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 shutdown_connections 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x556e0b81b580 0x556e0b888a10 crc :-1 s=READY pgs=411 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556e0b80b040 0x556e0b815870 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc53effd640 0 client.20088 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x556e0b805ed0 0x556e0b814860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 --2- 172.21.15.28:0/102684284 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x556e0b8054e0 0x556e0b814320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.758+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 >> 172.21.15.28:0/102684284 conn(0x556e0b800a50 msgr2=0x556e0b806970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.759+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 shutdown_connections 2024-03-22T02:33:29.761 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.759+0000 7fc5689ec740 1 -- 172.21.15.28:0/102684284 wait complete. 2024-03-22T02:33:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.759+0000 7fc5689ec740 20 client.20088 trim_cache size 0 max 16384 2024-03-22T02:33:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:33:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:33:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:33:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 msgr2=0x7fc5480a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 0x7fc5480a5480 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc554000c00 tx=0x7fc554005d50 comp rx=0 tx=0).stop 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 msgr2=0x7fc560197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc55000e9f0 tx=0x7fc55000eeb0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 shutdown_connections 2024-03-22T02:33:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fc5480a2fc0 0x7fc5480a5480 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc560107890 0x7fc56019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fc560104f60 0x7fc5601983a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 --2- 172.21.15.28:0/3541215989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc560102630 0x7fc560197e60 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 >> 172.21.15.28:0/3541215989 conn(0x7fc5600fc020 msgr2=0x7fc560102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 shutdown_connections 2024-03-22T02:33:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:29.760+0000 7fc565d25640 1 -- 172.21.15.28:0/3541215989 wait complete. 2024-03-22T02:33:30.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 109545 15.74 MB/sec execute 250 sec latency 896.938 ms 2024-03-22T02:33:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:30 smithi120 ceph-mon[36803]: pgmap v959: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 444 op/s 2024-03-22T02:33:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:30 smithi083 ceph-mon[36340]: pgmap v959: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 444 op/s 2024-03-22T02:33:31.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:30 smithi028 ceph-mon[31916]: pgmap v959: 129 pgs: 129 active+clean; 273 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 444 op/s 2024-03-22T02:33:31.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 109795 15.75 MB/sec execute 251 sec latency 100.349 ms 2024-03-22T02:33:32.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 109994 15.71 MB/sec execute 252 sec latency 79.080 ms 2024-03-22T02:33:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:32 smithi120 ceph-mon[36803]: pgmap v960: 129 pgs: 129 active+clean; 269 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-03-22T02:33:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:32 smithi083 ceph-mon[36340]: pgmap v960: 129 pgs: 129 active+clean; 269 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-03-22T02:33:33.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:32 smithi028 ceph-mon[31916]: pgmap v960: 129 pgs: 129 active+clean; 269 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 5.5 MiB/s wr, 585 op/s 2024-03-22T02:33:33.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 110205 15.69 MB/sec execute 253 sec latency 166.084 ms 2024-03-22T02:33:33.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:33 smithi120 ceph-mon[36803]: pgmap v961: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 460 op/s 2024-03-22T02:33:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:33 smithi083 ceph-mon[36340]: pgmap v961: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 460 op/s 2024-03-22T02:33:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:33 smithi028 ceph-mon[31916]: pgmap v961: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 4.3 MiB/s wr, 460 op/s 2024-03-22T02:33:34.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 110445 15.66 MB/sec execute 254 sec latency 98.968 ms 2024-03-22T02:33:35.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 110607 15.63 MB/sec execute 255 sec latency 283.599 ms 2024-03-22T02:33:36.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:35 smithi120 ceph-mon[36803]: pgmap v962: 129 pgs: 129 active+clean; 264 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:33:36.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:35 smithi083 ceph-mon[36340]: pgmap v962: 129 pgs: 129 active+clean; 264 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:33:36.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:35 smithi028 ceph-mon[31916]: pgmap v962: 129 pgs: 129 active+clean; 264 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:33:36.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 110869 15.60 MB/sec execute 256 sec latency 123.832 ms 2024-03-22T02:33:37.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 111319 15.61 MB/sec execute 257 sec latency 74.045 ms 2024-03-22T02:33:38.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:37 smithi120 ceph-mon[36803]: pgmap v963: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.0 MiB/s wr, 553 op/s 2024-03-22T02:33:38.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:37 smithi083 ceph-mon[36340]: pgmap v963: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.0 MiB/s wr, 553 op/s 2024-03-22T02:33:38.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:37 smithi028 ceph-mon[31916]: pgmap v963: 129 pgs: 129 active+clean; 269 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 4.0 MiB/s wr, 553 op/s 2024-03-22T02:33:38.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 111851 15.67 MB/sec execute 258 sec latency 52.659 ms 2024-03-22T02:33:39.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 112395 15.73 MB/sec execute 259 sec latency 54.501 ms 2024-03-22T02:33:40.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:39 smithi120 ceph-mon[36803]: pgmap v964: 129 pgs: 129 active+clean; 271 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-03-22T02:33:40.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:39 smithi083 ceph-mon[36340]: pgmap v964: 129 pgs: 129 active+clean; 271 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-03-22T02:33:40.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:39 smithi028 ceph-mon[31916]: pgmap v964: 129 pgs: 129 active+clean; 271 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 3.8 MiB/s wr, 541 op/s 2024-03-22T02:33:40.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 112761 15.76 MB/sec execute 260 sec latency 153.810 ms 2024-03-22T02:33:41.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113055 15.80 MB/sec execute 261 sec latency 169.360 ms 2024-03-22T02:33:42.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:41 smithi120 ceph-mon[36803]: pgmap v965: 129 pgs: 129 active+clean; 275 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.5 MiB/s wr, 594 op/s 2024-03-22T02:33:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:41 smithi083 ceph-mon[36340]: pgmap v965: 129 pgs: 129 active+clean; 275 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.5 MiB/s wr, 594 op/s 2024-03-22T02:33:42.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:41 smithi028 ceph-mon[31916]: pgmap v965: 129 pgs: 129 active+clean; 275 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 4.5 MiB/s wr, 594 op/s 2024-03-22T02:33:42.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113333 15.80 MB/sec execute 262 sec latency 82.051 ms 2024-03-22T02:33:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 -- 172.21.15.28:0/1697474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744100ab0 msgr2=0x7fe744100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.856 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1697474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744100ab0 0x7fe744100ed0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fe734009900 tx=0x7fe73402bc30 comp rx=0 tx=0).stop 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 -- 172.21.15.28:0/1697474682 shutdown_connections 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1697474682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 0x7fe744108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1697474682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744101ca0 0x7fe744102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1697474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744100ab0 0x7fe744100ed0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 -- 172.21.15.28:0/1697474682 >> 172.21.15.28:0/1697474682 conn(0x7fe7440fc020 msgr2=0x7fe7440fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 -- 172.21.15.28:0/1697474682 shutdown_connections 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.853+0000 7fe74a32c640 1 -- 172.21.15.28:0/1697474682 wait complete. 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 Processor -- start 2024-03-22T02:33:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 -- start start 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 0x7fe744069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 0x7fe74419f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe74410c700 con 0x7fe744101ca0 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe74410c580 con 0x7fe744102660 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe74a32c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe74410c880 con 0x7fe744100ab0 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 0x7fe744069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 0x7fe74419f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.854+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 0x7fe744069d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39500/0 (socket says 172.21.15.28:39500) 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47242/0 (socket says 172.21.15.28:47242) 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe743fff640 1 -- 172.21.15.28:0/1024112697 learned_addr learned my addr 172.21.15.28:0/1024112697 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 -- 172.21.15.28:0/1024112697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 msgr2=0x7fe744069d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 0x7fe744069d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 -- 172.21.15.28:0/1024112697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 msgr2=0x7fe74419f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 0x7fe74419f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 -- 172.21.15.28:0/1024112697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74419faf0 con 0x7fe744101ca0 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7437fe640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fe738005fd0 tx=0x7fe73800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7417fa640 1 -- 172.21.15.28:0/1024112697 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738009c30 con 0x7fe744101ca0 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7417fa640 1 -- 172.21.15.28:0/1024112697 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe738009dd0 con 0x7fe744101ca0 2024-03-22T02:33:42.859 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe74419fd80 con 0x7fe744101ca0 2024-03-22T02:33:42.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe7417fa640 1 -- 172.21.15.28:0/1024112697 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738010460 con 0x7fe744101ca0 2024-03-22T02:33:42.860 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.855+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7441a0270 con 0x7fe744101ca0 2024-03-22T02:33:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.857+0000 7fe7417fa640 1 -- 172.21.15.28:0/1024112697 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fe738003f50 con 0x7fe744101ca0 2024-03-22T02:33:42.867 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 Processor -- start 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 -- start start 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b036900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7417fa640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 0x7fe72c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56261af9b040 con 0x56261b03c040 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56261af439e0 con 0x56261b0364e0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe74cff2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56261af96ff0 con 0x56261b036ed0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b036900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7437fe640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 0x7fe72c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.868 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47244/0 (socket says 172.21.15.28:47244) 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39506/0 (socket says 172.21.15.28:39506) 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7488c9640 1 -- 172.21.15.28:0/3457829439 learned_addr learned my addr 172.21.15.28:0/3457829439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe7417fa640 1 -- 172.21.15.28:0/1024112697 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe7380cf3f0 con 0x7fe744101ca0 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe743fff640 1 -- 172.21.15.28:0/3457829439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 msgr2=0x56261b036900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.858+0000 7fe743fff640 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b036900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe743fff640 1 -- 172.21.15.28:0/3457829439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 msgr2=0x56261b03e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe743fff640 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe743fff640 1 -- 172.21.15.28:0/3457829439 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56261acaf210 con 0x56261b036ed0 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe7488c9640 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe7437fe640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 0x7fe72c0a5480 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe73809b6f0 tx=0x7fe73801e690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe743fff640 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe73402c7b0 tx=0x7fe73402c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5813fd7b307ed925 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.869 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe72a7fc640 1 -- 172.21.15.28:0/3457829439 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734042070 con 0x56261b036ed0 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe72a7fc640 1 -- 172.21.15.28:0/3457829439 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73402cb80 con 0x56261b036ed0 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.859+0000 7fe72a7fc640 1 -- 172.21.15.28:0/3457829439 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734004ef0 con 0x56261b036ed0 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- 172.21.15.28:0/3457829439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 msgr2=0x56261b03b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe73402c7b0 tx=0x7fe73402c6d0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- 172.21.15.28:0/3457829439 shutdown_connections 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b03e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b03b9d0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- 172.21.15.28:0/3457829439 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b036900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- 172.21.15.28:0/3457829439 >> 172.21.15.28:0/3457829439 conn(0x56261b031a50 msgr2=0x56261b031e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:42.870 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- 172.21.15.28:0/3457829439 shutdown_connections 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- 172.21.15.28:0/3457829439 wait complete. 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 -- start start 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 Processor -- start 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.860+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b04be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56261b03ef90 con 0x56261b03c040 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56261af56db0 con 0x56261b036ed0 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56261b03f110 con 0x56261b0364e0 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 10 client.0 _ll_register_callbacks cb 0x56261acaea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b04be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.871 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe743fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe743fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:59338/0 (socket says 172.21.15.28:59338) 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39514/0 (socket says 172.21.15.28:39514) 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe743fff640 1 -- 172.21.15.28:0/2276760446 learned_addr learned my addr 172.21.15.28:0/2276760446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 -- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 msgr2=0x56261b044de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 -- 172.21.15.28:0/2276760446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 msgr2=0x56261b04be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b04be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 -- 172.21.15.28:0/2276760446 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56261ad7b620 con 0x56261b0364e0 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe743fff640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x56261b037730 tx=0x7fe73c0096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=884a6070f4dc0b02 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:33:42.872 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c024440 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe73c0103d0 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.861+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73402cb80 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe73c0245e0 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 10 client.29896 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56261b04f250 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73402cb80 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fe73c022020 con 0x56261b0364e0 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 10 client.29896 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:42.873 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 10 client.29896 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 20 client.29896 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b052d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 4 client.29896 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56261afa8580 con 0x56261b050920 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe743fff640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b052d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe743fff640 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b052d30 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf96603c54b998b3 server_cookie=dc167c1ea0073035 in_seq=0 out_seq=0 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.862+0000 7fe7237fe640 10 client.29896 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe7237fe640 1 -- 172.21.15.28:0/2276760446 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56261afa8580 con 0x56261b050920 2024-03-22T02:33:42.874 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe7237fe640 10 client.29896 handle_command_reply: tid=0 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 client.29896 shutdown 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 msgr2=0x56261b0448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x56261b037730 tx=0x7fe73c0096a0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 shutdown_connections 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b052d30 crc :-1 s=READY pgs=412 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b04be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b036ed0 0x56261b044de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe7237fe640 0 client.29896 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2276760446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b0364e0 0x56261b0448a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 >> 172.21.15.28:0/2276760446 conn(0x56261b031a50 msgr2=0x56261b030c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 shutdown_connections 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 1 -- 172.21.15.28:0/2276760446 wait complete. 2024-03-22T02:33:42.875 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.865+0000 7fe74cff2740 20 client.29896 trim_cache size 0 max 16384 2024-03-22T02:33:42.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 Processor -- start 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 -- start start 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b0303c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b030900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441aafa0 con 0x56261b03c040 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441a8870 con 0x56261b0364e0 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe74cff2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441a66c0 con 0x56261b036ed0 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b030900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:39528/0 (socket says 172.21.15.28:39528) 2024-03-22T02:33:42.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 -- 172.21.15.28:0/2252880022 learned_addr learned my addr 172.21.15.28:0/2252880022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b0303c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 -- 172.21.15.28:0/2252880022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 msgr2=0x56261b0303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b0303c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 -- 172.21.15.28:0/2252880022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 msgr2=0x56261b030900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b030900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe743fff640 1 -- 172.21.15.28:0/2252880022 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56261b0329c0 con 0x56261b036ed0 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.884+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b030900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.885+0000 7fe743fff640 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe73402c7b0 tx=0x7fe73402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11a141fe9d62a2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.885+0000 7fe729ffb640 1 -- 172.21.15.28:0/2252880022 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734042040 con 0x56261b036ed0 2024-03-22T02:33:42.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.885+0000 7fe729ffb640 1 -- 172.21.15.28:0/2252880022 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73402caa0 con 0x56261b036ed0 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.885+0000 7fe729ffb640 1 -- 172.21.15.28:0/2252880022 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe734004ef0 con 0x56261b036ed0 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- 172.21.15.28:0/2252880022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 msgr2=0x56261b04bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe73402c7b0 tx=0x7fe73402bc30 comp rx=0 tx=0).stop 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- 172.21.15.28:0/2252880022 shutdown_connections 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b03c040 0x56261b030900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b04bdb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2252880022 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b0364e0 0x56261b0303c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- 172.21.15.28:0/2252880022 >> 172.21.15.28:0/2252880022 conn(0x56261b031a50 msgr2=0x56261b035540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- 172.21.15.28:0/2252880022 shutdown_connections 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- 172.21.15.28:0/2252880022 wait complete. 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- start start 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 Processor -- start 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b049830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b03c040 0x56261b04a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441a6d30 con 0x56261b0364e0 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441a81a0 con 0x56261b03c040 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441a7400 con 0x56261b036ed0 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe74cff2740 10 client.0 _ll_register_callbacks cb 0x7fe73c012b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.893 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.886+0000 7fe7488c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47268/0 (socket says 172.21.15.28:47268) 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7488c9640 1 -- 172.21.15.28:0/2816616013 learned_addr learned my addr 172.21.15.28:0/2816616013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b049830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b03c040 0x56261b04a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 -- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b03c040 msgr2=0x56261b04a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b03c040 0x56261b04a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 -- 172.21.15.28:0/2816616013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 msgr2=0x56261b0492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 -- 172.21.15.28:0/2816616013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56261b04ae30 con 0x56261b036ed0 2024-03-22T02:33:42.894 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe743fff640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b049830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x56261b032920 tx=0x7fe73402c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6864563685208294 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7237fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73405b060 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe73402caa0 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.887+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fe7340050f0 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fe734042040 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 10 client.29902 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56261b0b6c70 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe73402caa0 con 0x56261b036ed0 2024-03-22T02:33:42.895 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fe73404ebb0 con 0x56261b036ed0 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 10 client.29902 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 10 client.29902 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 20 client.29902 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b0b90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 4 client.29902 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56261afa28d0 con 0x56261b050920 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.888+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b0b90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.889+0000 7fe7237fe640 10 client.29902 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:42.896 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:42.889+0000 7fe7488c9640 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b0b90d0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b5429dbf7c1d6e server_cookie=ce8018f2bdceb1e7 in_seq=0 out_seq=0 2024-03-22T02:33:43.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113415 15.76 MB/sec execute 263 sec latency 525.373 ms 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.074+0000 7fe7237fe640 1 -- 172.21.15.28:0/2816616013 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56261afa28d0 con 0x56261b050920 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.074+0000 7fe7237fe640 10 client.29902 handle_command_reply: tid=0 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.074+0000 7fe74cff2740 1 client.29902 shutdown 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.074+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 msgr2=0x56261b049830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.074+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b049830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x56261b032920 tx=0x7fe73402c3b0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 shutdown_connections 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56261b050920 0x56261b0b90d0 crc :-1 s=READY pgs=413 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56261b03c040 0x56261b04a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe7237fe640 0 client.29902 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56261b036ed0 0x56261b049830 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 --2- 172.21.15.28:0/2816616013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56261b0364e0 0x56261b0492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 >> 172.21.15.28:0/2816616013 conn(0x56261b031a50 msgr2=0x56261b0377a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 shutdown_connections 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 1 -- 172.21.15.28:0/2816616013 wait complete. 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.075+0000 7fe74cff2740 20 client.29902 trim_cache size 0 max 16384 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:33:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:33:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 msgr2=0x7fe72c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 0x7fe72c0a5480 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fe73809b6f0 tx=0x7fe73801e690 comp rx=0 tx=0).stop 2024-03-22T02:33:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 msgr2=0x7fe74419cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:44.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fe738005fd0 tx=0x7fe73800dde0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 shutdown_connections 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fe72c0a2fc0 0x7fe72c0a5480 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fe744102660 0x7fe74419f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe744101ca0 0x7fe74419cbc0 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 --2- 172.21.15.28:0/1024112697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe744100ab0 0x7fe744069d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 >> 172.21.15.28:0/1024112697 conn(0x7fe7440fc020 msgr2=0x7fe744104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.076+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 shutdown_connections 2024-03-22T02:33:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:44.077+0000 7fe74a32c640 1 -- 172.21.15.28:0/1024112697 wait complete. 2024-03-22T02:33:44.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:43 smithi120 ceph-mon[36803]: pgmap v966: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 411 op/s 2024-03-22T02:33:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:43 smithi083 ceph-mon[36340]: pgmap v966: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 411 op/s 2024-03-22T02:33:44.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:43 smithi028 ceph-mon[31916]: pgmap v966: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.4 MiB/s wr, 411 op/s 2024-03-22T02:33:44.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113612 15.74 MB/sec execute 264 sec latency 855.551 ms 2024-03-22T02:33:45.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113727 15.70 MB/sec execute 265 sec latency 300.204 ms 2024-03-22T02:33:46.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:45 smithi120 ceph-mon[36803]: pgmap v967: 129 pgs: 129 active+clean; 276 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 465 op/s 2024-03-22T02:33:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:45 smithi083 ceph-mon[36340]: pgmap v967: 129 pgs: 129 active+clean; 276 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 465 op/s 2024-03-22T02:33:46.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:45 smithi028 ceph-mon[31916]: pgmap v967: 129 pgs: 129 active+clean; 276 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.1 MiB/s wr, 465 op/s 2024-03-22T02:33:46.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 113894 15.66 MB/sec execute 266 sec latency 213.124 ms 2024-03-22T02:33:47.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 114155 15.63 MB/sec execute 267 sec latency 88.972 ms 2024-03-22T02:33:48.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:47 smithi120 ceph-mon[36803]: pgmap v968: 129 pgs: 129 active+clean; 274 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2024-03-22T02:33:48.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:47 smithi083 ceph-mon[36340]: pgmap v968: 129 pgs: 129 active+clean; 274 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2024-03-22T02:33:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:47 smithi028 ceph-mon[31916]: pgmap v968: 129 pgs: 129 active+clean; 274 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2024-03-22T02:33:48.688 INFO:tasks.workunit.client.0.smithi028.stdout: 10 114467 15.64 MB/sec execute 268 sec latency 107.864 ms 2024-03-22T02:33:49.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 114884 15.62 MB/sec execute 269 sec latency 88.233 ms 2024-03-22T02:33:50.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:49 smithi120 ceph-mon[36803]: pgmap v969: 129 pgs: 129 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 516 op/s 2024-03-22T02:33:50.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:49 smithi083 ceph-mon[36340]: pgmap v969: 129 pgs: 129 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 516 op/s 2024-03-22T02:33:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:49 smithi028 ceph-mon[31916]: pgmap v969: 129 pgs: 129 active+clean; 270 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.2 MiB/s wr, 516 op/s 2024-03-22T02:33:50.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 115194 15.62 MB/sec execute 270 sec latency 201.061 ms 2024-03-22T02:33:51.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 115552 15.68 MB/sec execute 271 sec latency 92.734 ms 2024-03-22T02:33:52.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:51 smithi120 ceph-mon[36803]: pgmap v970: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.0 MiB/s wr, 584 op/s 2024-03-22T02:33:52.266 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:51 smithi083 ceph-mon[36340]: pgmap v970: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.0 MiB/s wr, 584 op/s 2024-03-22T02:33:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:51 smithi028 ceph-mon[31916]: pgmap v970: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.0 MiB/s wr, 584 op/s 2024-03-22T02:33:52.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 116016 15.70 MB/sec execute 272 sec latency 92.181 ms 2024-03-22T02:33:53.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 116454 15.76 MB/sec execute 273 sec latency 78.111 ms 2024-03-22T02:33:54.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:53 smithi120 ceph-mon[36803]: pgmap v971: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 523 op/s 2024-03-22T02:33:54.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:53 smithi083 ceph-mon[36340]: pgmap v971: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 523 op/s 2024-03-22T02:33:54.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:53 smithi028 ceph-mon[31916]: pgmap v971: 129 pgs: 129 active+clean; 274 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.8 MiB/s wr, 523 op/s 2024-03-22T02:33:54.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 116745 15.78 MB/sec execute 274 sec latency 72.546 ms 2024-03-22T02:33:55.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117009 15.78 MB/sec execute 275 sec latency 211.838 ms 2024-03-22T02:33:56.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:55 smithi120 ceph-mon[36803]: pgmap v972: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 540 op/s 2024-03-22T02:33:56.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:55 smithi083 ceph-mon[36340]: pgmap v972: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 540 op/s 2024-03-22T02:33:56.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:55 smithi028 ceph-mon[31916]: pgmap v972: 129 pgs: 129 active+clean; 277 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.7 MiB/s wr, 540 op/s 2024-03-22T02:33:56.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117158 15.75 MB/sec execute 276 sec latency 207.040 ms 2024-03-22T02:33:57.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 -- 172.21.15.28:0/4056478737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 msgr2=0x7fb9f4107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 --2- 172.21.15.28:0/4056478737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f4107350 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009900 tx=0x7fb9dc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 -- 172.21.15.28:0/4056478737 shutdown_connections 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 --2- 172.21.15.28:0/4056478737 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4107890 0x7fb9f4109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 --2- 172.21.15.28:0/4056478737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f4107350 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 --2- 172.21.15.28:0/4056478737 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4102630 0x7fb9f4104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.187+0000 7fb9f88df640 1 -- 172.21.15.28:0/4056478737 >> 172.21.15.28:0/4056478737 conn(0x7fb9f40fc020 msgr2=0x7fb9f40fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- 172.21.15.28:0/4056478737 shutdown_connections 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- 172.21.15.28:0/4056478737 wait complete. 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 Processor -- start 2024-03-22T02:33:57.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- start start 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4102630 0x7fb9f419a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4107890 0x7fb9f41a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d820 con 0x7fb9f4104f60 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d6a0 con 0x7fb9f4107890 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.188+0000 7fb9f88df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f410d9a0 con 0x7fb9f4102630 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52464/0 (socket says 172.21.15.28:52464) 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/2121781434 learned_addr learned my addr 172.21.15.28:0/2121781434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4102630 0x7fb9f419a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4107890 0x7fb9f41a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/2121781434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4102630 msgr2=0x7fb9f419a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.192 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4102630 0x7fb9f419a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/2121781434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4107890 msgr2=0x7fb9f41a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4107890 0x7fb9f41a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/2121781434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f41a1c50 con 0x7fb9f4104f60 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9eb7fe640 1 -- 172.21.15.28:0/2121781434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc041070 con 0x7fb9f4104f60 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9eb7fe640 1 -- 172.21.15.28:0/2121781434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9dc004450 con 0x7fb9f4104f60 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f41a1ee0 con 0x7fb9f4104f60 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9eb7fe640 1 -- 172.21.15.28:0/2121781434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc034440 con 0x7fb9f4104f60 2024-03-22T02:33:57.193 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.189+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f41a2340 con 0x7fb9f4104f60 2024-03-22T02:33:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.191+0000 7fb9eb7fe640 1 -- 172.21.15.28:0/2121781434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb9dc044070 con 0x7fb9f4104f60 2024-03-22T02:33:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9eb7fe640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 0x7fb9e00a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 Processor -- start 2024-03-22T02:33:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9f259c640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 0x7fb9e00a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9eb7fe640 1 -- 172.21.15.28:0/2121781434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb9dc0f1330 con 0x7fb9f4104f60 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 -- start start 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e4fb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4fc040 0x562a6e4fe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a6e45b040 con 0x562a6e4f6ed0 2024-03-22T02:33:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562a6e4039e0 con 0x562a6e4f64e0 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.192+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a6e456ff0 con 0x562a6e4fc040 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 0x7fb9e00a5480 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009bd0 tx=0x7fb9ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e4fb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55020/0 (socket says 172.21.15.28:55020) 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4fc040 0x562a6e4fe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e4fb9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52474/0 (socket says 172.21.15.28:52474) 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 -- 172.21.15.28:0/503632337 learned_addr learned my addr 172.21.15.28:0/503632337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 -- 172.21.15.28:0/503632337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4fc040 msgr2=0x562a6e4fe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4fc040 0x562a6e4fe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 -- 172.21.15.28:0/503632337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 msgr2=0x562a6e4fb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e4fb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.205 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 -- 172.21.15.28:0/503632337 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6e16f210 con 0x562a6e4f64e0 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9f259c640 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec005bc0 tx=0x7fb9ec014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed4bea1470c2076 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9e8ff9640 1 -- 172.21.15.28:0/503632337 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec005020 con 0x562a6e4f64e0 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9e8ff9640 1 -- 172.21.15.28:0/503632337 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9ec0047a0 con 0x562a6e4f64e0 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.193+0000 7fb9e8ff9640 1 -- 172.21.15.28:0/503632337 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0267e0 con 0x562a6e4f64e0 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- 172.21.15.28:0/503632337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 msgr2=0x562a6e4f6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.206 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec005bc0 tx=0x7fb9ec014000 comp rx=0 tx=0).stop 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- 172.21.15.28:0/503632337 shutdown_connections 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4fc040 0x562a6e4fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e4fb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/503632337 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f64e0 0x562a6e4f6900 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- 172.21.15.28:0/503632337 >> 172.21.15.28:0/503632337 conn(0x562a6e4f1a50 msgr2=0x562a6e4f1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- 172.21.15.28:0/503632337 shutdown_connections 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- 172.21.15.28:0/503632337 wait complete. 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.194+0000 7fb9fb59e740 1 -- start start 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 Processor -- start 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e500480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4fc040 0x562a6e507a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a6e4feff0 con 0x562a6e4f6ed0 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x562a6e416db0 con 0x562a6e4fc040 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x562a6e4ff170 con 0x562a6e4f64e0 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9fb59e740 10 client.0 _ll_register_callbacks cb 0x562a6e16ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52488/0 (socket says 172.21.15.28:52488) 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e500480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/1622449476 learned_addr learned my addr 172.21.15.28:0/1622449476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4fc040 0x562a6e507a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/1622449476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 msgr2=0x562a6e500480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e500480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4fc040 msgr2=0x562a6e507a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4fc040 0x562a6e507a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 -- 172.21.15.28:0/1622449476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6e23b620 con 0x562a6e4f6ed0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc02cfd0 tx=0x7fb9dc0bd5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6650d0d4ee3b0dc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.195+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc04b040 con 0x562a6e4f6ed0 2024-03-22T02:33:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb9dc0035f0 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9ec0047a0 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb9dc004c80 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 10 client.20118 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562a6e50ad70 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9ec0047a0 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb9dc0c5b70 con 0x562a6e4f6ed0 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 10 client.20118 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 10 client.20118 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 20 client.20118 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e50c570 0x562a6e50e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 4 client.20118 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:33:57.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a6e468580 con 0x562a6e50c570 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.196+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e50c570 0x562a6e50e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.197+0000 7fb9d9ffb640 10 client.20118 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.197+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e50c570 0x562a6e50e9d0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19a0f1d544468428 server_cookie=a050587f74854357 in_seq=0 out_seq=0 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.198+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1622449476 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a6e468580 con 0x562a6e50c570 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.198+0000 7fb9d9ffb640 10 client.20118 handle_command_reply: tid=0 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.198+0000 7fb9fb59e740 1 client.20118 shutdown 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.198+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 msgr2=0x562a6e5009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.198+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc02cfd0 tx=0x7fb9dc0bd5c0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 shutdown_connections 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e50c570 0x562a6e50e9d0 crc :-1 s=READY pgs=414 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4fc040 0x562a6e507a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.211 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4f6ed0 0x562a6e5009c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1622449476 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e500480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9d9ffb640 0 client.20118 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 >> 172.21.15.28:0/1622449476 conn(0x562a6e4f1a50 msgr2=0x562a6e4f0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 shutdown_connections 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1622449476 wait complete. 2024-03-22T02:33:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.199+0000 7fb9fb59e740 20 client.20118 trim_cache size 0 max 16384 2024-03-22T02:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.213+0000 7fb9fb59e740 1 Processor -- start 2024-03-22T02:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.213+0000 7fb9fb59e740 1 -- start start 2024-03-22T02:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e4f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e4f1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41af490 con 0x562a6e4fc040 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41acd60 con 0x562a6e4f6ed0 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41aabb0 con 0x562a6e4f64e0 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52490/0 (socket says 172.21.15.28:52490) 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1085564502 learned_addr learned my addr 172.21.15.28:0/1085564502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e4f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e4f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1085564502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 msgr2=0x562a6e4f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e4f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1085564502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 msgr2=0x562a6e4f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e4f1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1085564502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6e4f27c0 con 0x562a6e4fc040 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4003ce0 tx=0x7fb9e4004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a0c4d93935a444 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9dbfff640 1 -- 172.21.15.28:0/1085564502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4010430 con 0x562a6e4fc040 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9dbfff640 1 -- 172.21.15.28:0/1085564502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9e4010ab0 con 0x562a6e4fc040 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.214+0000 7fb9dbfff640 1 -- 172.21.15.28:0/1085564502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e401bc00 con 0x562a6e4fc040 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1085564502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 msgr2=0x562a6e5079c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4003ce0 tx=0x7fb9e4004410 comp rx=0 tx=0).stop 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1085564502 shutdown_connections 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e5079c0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e4f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1085564502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e4f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1085564502 >> 172.21.15.28:0/1085564502 conn(0x562a6e4f1a50 msgr2=0x562a6e4f03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1085564502 shutdown_connections 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1085564502 wait complete. 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 -- start start 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.215+0000 7fb9fb59e740 1 Processor -- start 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e506c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41ab220 con 0x562a6e4fc040 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41ac690 con 0x562a6e4f6ed0 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f41ab8f0 con 0x562a6e4f64e0 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 10 client.0 _ll_register_callbacks cb 0x7fb9dc0b9310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e506c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f259c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:57582/0 (socket says 172.21.15.28:57582) 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f1d9b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:55050/0 (socket says 172.21.15.28:55050) 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f259c640 1 -- 172.21.15.28:0/1639751653 learned_addr learned my addr 172.21.15.28:0/1639751653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1639751653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 msgr2=0x562a6e5061c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 msgr2=0x562a6e506700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 -- 172.21.15.28:0/1639751653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6e572910 con 0x562a6e4fc040 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f1d9b640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9f2d9d640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e506c40 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4004910 tx=0x7fb9e4005ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9421fb3f35668259 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4010700 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb9e40108a0 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.216+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb9e4010ab0 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb9e4035050 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 10 client.20124 fetch_fsmap learned FSMap version 46 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x562a6e573b80 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb9e4010ab0 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb9e4026b30 con 0x562a6e4fc040 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 10 client.20124 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 10 client.20124 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 20 client.20124 populate_metadata read hostname 'smithi028' 2024-03-22T02:33:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e574580 0x562a6e5769a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:33:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 4 client.20124 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:33:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a6e4628d0 con 0x562a6e574580 2024-03-22T02:33:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e574580 0x562a6e5769a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:33:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9d9ffb640 10 client.20124 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:57.217+0000 7fb9f259c640 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e574580 0x562a6e5769a0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8510687b5c377896 server_cookie=be5a58c2c18c41d0 in_seq=0 out_seq=0 2024-03-22T02:33:57.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117242 15.71 MB/sec execute 277 sec latency 268.108 ms 2024-03-22T02:33:58.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:33:58 smithi083 ceph-mon[36340]: pgmap v973: 129 pgs: 129 active+clean; 277 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 480 op/s 2024-03-22T02:33:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9d9ffb640 1 -- 172.21.15.28:0/1639751653 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a6e4628d0 con 0x562a6e574580 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9d9ffb640 10 client.20124 handle_command_reply: tid=0 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 client.20124 shutdown 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 msgr2=0x562a6e506c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e506c40 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4004910 tx=0x7fb9e4005ac0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 shutdown_connections 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x562a6e574580 0x562a6e5769a0 crc :-1 s=READY pgs=415 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9d9ffb640 0 client.20124 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.304+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a6e4fc040 0x562a6e506c40 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x562a6e4f6ed0 0x562a6e506700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 1 --2- 172.21.15.28:0/1639751653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x562a6e4f64e0 0x562a6e5061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 >> 172.21.15.28:0/1639751653 conn(0x562a6e4f1a50 msgr2=0x562a6e4f0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:58.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 shutdown_connections 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 1 -- 172.21.15.28:0/1639751653 wait complete. 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.305+0000 7fb9fb59e740 20 client.20124 trim_cache size 0 max 16384 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:33:58.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 msgr2=0x7fb9e00a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 0x7fb9e00a5480 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009bd0 tx=0x7fb9ec005f90 comp rx=0 tx=0).stop 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 msgr2=0x7fb9f419a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc000c00 tx=0x7fb9dc02c640 comp rx=0 tx=0).stop 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 shutdown_connections 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb9e00a2fc0 0x7fb9e00a5480 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb9f4107890 0x7fb9f41a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f4104f60 0x7fb9f419a570 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.309 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 --2- 172.21.15.28:0/2121781434 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9f4102630 0x7fb9f419a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:33:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.306+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 >> 172.21.15.28:0/2121781434 conn(0x7fb9f40fc020 msgr2=0x7fb9f4101ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:33:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.307+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 shutdown_connections 2024-03-22T02:33:58.310 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:33:58.307+0000 7fb9f88df640 1 -- 172.21.15.28:0/2121781434 wait complete. 2024-03-22T02:33:58.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:33:58 smithi028 ceph-mon[31916]: pgmap v973: 129 pgs: 129 active+clean; 277 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 480 op/s 2024-03-22T02:33:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:33:58 smithi120 ceph-mon[36803]: pgmap v973: 129 pgs: 129 active+clean; 277 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.6 MiB/s wr, 480 op/s 2024-03-22T02:33:58.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117375 15.66 MB/sec execute 278 sec latency 657.859 ms 2024-03-22T02:33:59.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117633 15.64 MB/sec execute 279 sec latency 72.345 ms 2024-03-22T02:34:00.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:00 smithi083 ceph-mon[36340]: pgmap v974: 129 pgs: 129 active+clean; 280 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-03-22T02:34:00.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:00 smithi028 ceph-mon[31916]: pgmap v974: 129 pgs: 129 active+clean; 280 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-03-22T02:34:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:00 smithi120 ceph-mon[36803]: pgmap v974: 129 pgs: 129 active+clean; 280 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.2 MiB/s wr, 420 op/s 2024-03-22T02:34:00.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 117925 15.64 MB/sec execute 280 sec latency 97.800 ms 2024-03-22T02:34:01.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 118114 15.60 MB/sec execute 281 sec latency 227.298 ms 2024-03-22T02:34:02.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:02 smithi083 ceph-mon[36340]: pgmap v975: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 560 op/s 2024-03-22T02:34:02.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:02 smithi028 ceph-mon[31916]: pgmap v975: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 560 op/s 2024-03-22T02:34:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:02 smithi120 ceph-mon[36803]: pgmap v975: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.3 MiB/s wr, 560 op/s 2024-03-22T02:34:02.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 118656 15.63 MB/sec execute 282 sec latency 92.700 ms 2024-03-22T02:34:03.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 119110 15.69 MB/sec execute 283 sec latency 51.503 ms 2024-03-22T02:34:04.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:04 smithi083 ceph-mon[36340]: pgmap v976: 129 pgs: 129 active+clean; 276 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 504 op/s 2024-03-22T02:34:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:04 smithi028 ceph-mon[31916]: pgmap v976: 129 pgs: 129 active+clean; 276 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 504 op/s 2024-03-22T02:34:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:04 smithi120 ceph-mon[36803]: pgmap v976: 129 pgs: 129 active+clean; 276 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 504 op/s 2024-03-22T02:34:04.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 119645 15.72 MB/sec execute 284 sec latency 91.316 ms 2024-03-22T02:34:05.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 119984 15.76 MB/sec execute 285 sec latency 71.211 ms 2024-03-22T02:34:06.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:06 smithi083 ceph-mon[36340]: pgmap v977: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:34:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:06 smithi028 ceph-mon[31916]: pgmap v977: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:34:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:06 smithi120 ceph-mon[36803]: pgmap v977: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 5.3 MiB/s wr, 543 op/s 2024-03-22T02:34:06.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 120214 15.78 MB/sec execute 286 sec latency 170.475 ms 2024-03-22T02:34:07.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 120513 15.78 MB/sec execute 287 sec latency 53.363 ms 2024-03-22T02:34:08.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:08 smithi083 ceph-mon[36340]: pgmap v978: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 547 op/s 2024-03-22T02:34:08.300 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:08 smithi120 ceph-mon[36803]: pgmap v978: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 547 op/s 2024-03-22T02:34:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:08 smithi028 ceph-mon[31916]: pgmap v978: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.8 MiB/s wr, 547 op/s 2024-03-22T02:34:08.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 120714 15.76 MB/sec execute 288 sec latency 153.578 ms 2024-03-22T02:34:09.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 120883 15.72 MB/sec execute 289 sec latency 146.613 ms 2024-03-22T02:34:10.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:10 smithi028 ceph-mon[31916]: pgmap v979: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-03-22T02:34:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:10 smithi120 ceph-mon[36803]: pgmap v979: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-03-22T02:34:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:10 smithi083 ceph-mon[36340]: pgmap v979: 129 pgs: 129 active+clean; 275 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.6 MiB/s wr, 521 op/s 2024-03-22T02:34:10.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 121124 15.70 MB/sec execute 290 sec latency 56.900 ms 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 -- 172.21.15.28:0/1356498751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60100ab0 msgr2=0x7faf60100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 --2- 172.21.15.28:0/1356498751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60100ab0 0x7faf60100ed0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7faf50009900 tx=0x7faf5002bc30 comp rx=0 tx=0).stop 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 -- 172.21.15.28:0/1356498751 shutdown_connections 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 --2- 172.21.15.28:0/1356498751 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60102660 0x7faf60108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 --2- 172.21.15.28:0/1356498751 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60101ca0 0x7faf60102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 --2- 172.21.15.28:0/1356498751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60100ab0 0x7faf60100ed0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.429 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 -- 172.21.15.28:0/1356498751 >> 172.21.15.28:0/1356498751 conn(0x7faf600fc020 msgr2=0x7faf600fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 -- 172.21.15.28:0/1356498751 shutdown_connections 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 -- 172.21.15.28:0/1356498751 wait complete. 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.425+0000 7faf654c3640 1 Processor -- start 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 -- start start 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60100ab0 0x7faf60196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60102660 0x7faf6019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c720 con 0x7faf60101ca0 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c5a0 con 0x7faf60100ab0 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf654c3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c8a0 con 0x7faf60102660 2024-03-22T02:34:11.430 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45964/0 (socket says 172.21.15.28:45964) 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 -- 172.21.15.28:0/2256811682 learned_addr learned my addr 172.21.15.28:0/2256811682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf5effd640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60100ab0 0x7faf60196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60102660 0x7faf6019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 -- 172.21.15.28:0/2256811682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60102660 msgr2=0x7faf6019e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60102660 0x7faf6019e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 -- 172.21.15.28:0/2256811682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60100ab0 msgr2=0x7faf60196dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60100ab0 0x7faf60196dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 -- 172.21.15.28:0/2256811682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6019e9e0 con 0x7faf60101ca0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.426+0000 7faf57fff640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf4800e9f0 tx=0x7faf4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.427+0000 7faf5cff9640 1 -- 172.21.15.28:0/2256811682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4800ccc0 con 0x7faf60101ca0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.427+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6019ecd0 con 0x7faf60101ca0 2024-03-22T02:34:11.431 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.427+0000 7faf5cff9640 1 -- 172.21.15.28:0/2256811682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf4800ce60 con 0x7faf60101ca0 2024-03-22T02:34:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.427+0000 7faf5cff9640 1 -- 172.21.15.28:0/2256811682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf48010690 con 0x7faf60101ca0 2024-03-22T02:34:11.432 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.427+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf6019f1d0 con 0x7faf60101ca0 2024-03-22T02:34:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.429+0000 7faf5cff9640 1 -- 172.21.15.28:0/2256811682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faf480046c0 con 0x7faf60101ca0 2024-03-22T02:34:11.438 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.429+0000 7faf68184740 1 Processor -- start 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.429+0000 7faf68184740 1 -- start start 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5cff9640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 0x7faf4c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 0x55cb5eb924d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb5eaef040 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cb5ea979e0 con 0x55cb5eb8aed0 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf68184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cb5eaeaff0 con 0x55cb5eb90040 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 0x7faf4c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf57fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45980/0 (socket says 172.21.15.28:45980) 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5effd640 1 -- 172.21.15.28:0/757864521 learned_addr learned my addr 172.21.15.28:0/757864521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5cff9640 1 -- 172.21.15.28:0/2256811682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf480cfc50 con 0x7faf60101ca0 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 0x55cb5eb924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 -- 172.21.15.28:0/757864521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8aed0 msgr2=0x55cb5eb8f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 -- 172.21.15.28:0/757864521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 msgr2=0x55cb5eb8a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5f7fe640 1 -- 172.21.15.28:0/757864521 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb5e803210 con 0x55cb5eb90040 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.430+0000 7faf5effd640 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 0x7faf4c0a5480 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf58005fd0 tx=0x7faf58009f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 0x55cb5eb924d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faf5800fc20 tx=0x7faf5801d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf5867358c614913 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf557fa640 1 -- 172.21.15.28:0/757864521 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58024070 con 0x55cb5eb90040 2024-03-22T02:34:11.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf557fa640 1 -- 172.21.15.28:0/757864521 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58020c80 con 0x55cb5eb90040 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf557fa640 1 -- 172.21.15.28:0/757864521 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5801fd70 con 0x55cb5eb90040 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf68184740 1 -- 172.21.15.28:0/757864521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 msgr2=0x55cb5eb924d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.431+0000 7faf68184740 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 0x55cb5eb924d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faf5800fc20 tx=0x7faf5801d000 comp rx=0 tx=0).stop 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 -- 172.21.15.28:0/757864521 shutdown_connections 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb90040 0x55cb5eb924d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 --2- 172.21.15.28:0/757864521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb8a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 -- 172.21.15.28:0/757864521 >> 172.21.15.28:0/757864521 conn(0x55cb5eb85a50 msgr2=0x55cb5eb85e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 -- 172.21.15.28:0/757864521 shutdown_connections 2024-03-22T02:34:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 -- 172.21.15.28:0/757864521 wait complete. 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 -- start start 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.432+0000 7faf68184740 1 Processor -- start 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb96660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb96ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cb5eb951a0 con 0x55cb5eb90040 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55cb5eaaadb0 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55cb5eb95320 con 0x55cb5eb8aed0 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb96660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf57fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb96ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb96660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:58622/0 (socket says 172.21.15.28:58622) 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5effd640 1 -- 172.21.15.28:0/2654954858 learned_addr learned my addr 172.21.15.28:0/2654954858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45988/0 (socket says 172.21.15.28:45988) 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 10 client.0 _ll_register_callbacks cb 0x55cb5e802a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 -- 172.21.15.28:0/2654954858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 msgr2=0x55cb5eb96ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb96ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 -- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 msgr2=0x55cb5eb96660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb96660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 -- 172.21.15.28:0/2654954858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb5e8cf620 con 0x55cb5eb90040 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7faf580099a0 tx=0x7faf58009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2399f100110f539 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf467fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58024070 con 0x55cb5eb90040 2024-03-22T02:34:11.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf58020c80 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.433+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf5801fce0 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7faf58003970 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 10 client.20136 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55cb5eba0f30 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58020c80 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7faf58005570 con 0x55cb5eb90040 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 10 client.20136 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 10 client.20136 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 20 client.20136 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eba2520 0x55cb5eba4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 4 client.20136 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb5eafc580 con 0x55cb5eba2520 2024-03-22T02:34:11.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.434+0000 7faf5effd640 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eba2520 0x55cb5eba4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf467fc640 10 client.20136 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf5effd640 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eba2520 0x55cb5eba4910 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec2bc019a7f65842 server_cookie=f450dd7500b6af90 in_seq=0 out_seq=0 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf467fc640 1 -- 172.21.15.28:0/2654954858 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cb5eafc580 con 0x55cb5eba2520 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf467fc640 10 client.20136 handle_command_reply: tid=0 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf68184740 1 client.20136 shutdown 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 msgr2=0x55cb5eb9dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7faf580099a0 tx=0x7faf58009870 comp rx=0 tx=0).stop 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 shutdown_connections 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.435+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eba2520 0x55cb5eba4910 crc :-1 s=READY pgs=416 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb90040 0x55cb5eb9dbe0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb96ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.445 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 --2- 172.21.15.28:0/2654954858 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb96660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf467fc640 0 client.20136 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 >> 172.21.15.28:0/2654954858 conn(0x55cb5eb85a50 msgr2=0x55cb5eb84c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 shutdown_connections 2024-03-22T02:34:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 1 -- 172.21.15.28:0/2654954858 wait complete. 2024-03-22T02:34:11.446 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.436+0000 7faf68184740 20 client.20136 trim_cache size 0 max 16384 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 Processor -- start 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 -- start start 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb8f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf601ab050 con 0x55cb5eb8aed0 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf601a8920 con 0x55cb5eb85a50 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf68184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf601a6770 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45996/0 (socket says 172.21.15.28:45996) 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 -- 172.21.15.28:0/1247044081 learned_addr learned my addr 172.21.15.28:0/1247044081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf57fff640 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5effd640 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb8f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 -- 172.21.15.28:0/1247044081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8a4e0 msgr2=0x55cb5eb9dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 -- 172.21.15.28:0/1247044081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 msgr2=0x55cb5eb8f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb8f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 -- 172.21.15.28:0/1247044081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb5eba4c80 con 0x55cb5eb8aed0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7faf5800fc20 tx=0x7faf5800fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98072b4c9d3d1119 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf54ff9640 1 -- 172.21.15.28:0/1247044081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58045070 con 0x55cb5eb8aed0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf54ff9640 1 -- 172.21.15.28:0/1247044081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58009500 con 0x55cb5eb8aed0 2024-03-22T02:34:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.451+0000 7faf54ff9640 1 -- 172.21.15.28:0/1247044081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58024070 con 0x55cb5eb8aed0 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- 172.21.15.28:0/1247044081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 msgr2=0x55cb5eb8bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7faf5800fc20 tx=0x7faf5800fd70 comp rx=0 tx=0).stop 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- 172.21.15.28:0/1247044081 shutdown_connections 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb8bfe0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 --2- 172.21.15.28:0/1247044081 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb8f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- 172.21.15.28:0/1247044081 >> 172.21.15.28:0/1247044081 conn(0x55cb5eb96220 msgr2=0x55cb5eb8b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- 172.21.15.28:0/1247044081 shutdown_connections 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- 172.21.15.28:0/1247044081 wait complete. 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.452+0000 7faf68184740 1 -- start start 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 Processor -- start 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb9b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb9ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf601a6de0 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf601a8250 con 0x55cb5eb85a50 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf601a74b0 con 0x55cb5eb8aed0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf68184740 10 client.0 _ll_register_callbacks cb 0x7faf48005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf5effd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb9b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45998/0 (socket says 172.21.15.28:45998) 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 -- 172.21.15.28:0/79327194 learned_addr learned my addr 172.21.15.28:0/79327194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf5f7fe640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb9ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 -- 172.21.15.28:0/79327194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 msgr2=0x55cb5eb9ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb9ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 -- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 msgr2=0x55cb5eb9b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb9b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 -- 172.21.15.28:0/79327194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb5eb9d2c0 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf57fff640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7faf48098320 tx=0x7faf480672d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbf9aacdb8ba956 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf467fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.453+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf480a0060 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf4806b160 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf58009500 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7faf480a4920 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 10 client.20142 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55cb5ec08df0 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58009500 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7faf48067440 con 0x55cb5eb8a4e0 2024-03-22T02:34:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 10 client.20142 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 10 client.20142 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 20 client.20142 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eb90040 0x55cb5ec0b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 4 client.20142 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cb5eaf68d0 con 0x55cb5eb90040 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.454+0000 7faf5effd640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eb90040 0x55cb5ec0b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.455+0000 7faf467fc640 10 client.20142 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:11.455+0000 7faf5effd640 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eb90040 0x55cb5ec0b880 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4048eb03e558020c server_cookie=cfff6490d04caac2 in_seq=0 out_seq=0 2024-03-22T02:34:11.689 INFO:tasks.workunit.client.0.smithi028.stdout: 10 121207 15.66 MB/sec execute 291 sec latency 495.206 ms 2024-03-22T02:34:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:12 smithi028 ceph-mon[31916]: pgmap v980: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 460 op/s 2024-03-22T02:34:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:12 smithi120 ceph-mon[36803]: pgmap v980: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 460 op/s 2024-03-22T02:34:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:12 smithi083 ceph-mon[36340]: pgmap v980: 129 pgs: 129 active+clean; 273 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.4 MiB/s wr, 460 op/s 2024-03-22T02:34:12.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 121226 15.60 MB/sec execute 292 sec latency 927.053 ms 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf467fc640 1 -- 172.21.15.28:0/79327194 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cb5eaf68d0 con 0x55cb5eb90040 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf467fc640 10 client.20142 handle_command_reply: tid=0 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 client.20142 shutdown 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 msgr2=0x55cb5eb9bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7faf48098320 tx=0x7faf480672d0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 shutdown_connections 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55cb5eb90040 0x55cb5ec0b880 crc :-1 s=READY pgs=417 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf467fc640 0 client.20142 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55cb5eb8aed0 0x55cb5eb9ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.811 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb5eb8a4e0 0x55cb5eb9bcc0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 --2- 172.21.15.28:0/79327194 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55cb5eb85a50 0x55cb5eb9b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.809+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 >> 172.21.15.28:0/79327194 conn(0x55cb5eb96220 msgr2=0x55cb5eb8c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.810+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 shutdown_connections 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.810+0000 7faf68184740 1 -- 172.21.15.28:0/79327194 wait complete. 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.810+0000 7faf68184740 20 client.20142 trim_cache size 0 max 16384 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:34:12.812 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 msgr2=0x7faf4c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 0x7faf4c0a5480 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faf58005fd0 tx=0x7faf58009f30 comp rx=0 tx=0).stop 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 msgr2=0x7faf60197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7faf4800e9f0 tx=0x7faf4800edf0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 shutdown_connections 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf4c0a2fc0 0x7faf4c0a5480 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf60102660 0x7faf6019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf60101ca0 0x7faf60197300 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.813 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 --2- 172.21.15.28:0/2256811682 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf60100ab0 0x7faf60196dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 >> 172.21.15.28:0/2256811682 conn(0x7faf600fc020 msgr2=0x7faf60104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 shutdown_connections 2024-03-22T02:34:12.814 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:12.811+0000 7faf654c3640 1 -- 172.21.15.28:0/2256811682 wait complete. 2024-03-22T02:34:13.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 121443 15.58 MB/sec execute 293 sec latency 788.751 ms 2024-03-22T02:34:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:14 smithi028 ceph-mon[31916]: pgmap v981: 129 pgs: 129 active+clean; 280 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-22T02:34:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:14 smithi120 ceph-mon[36803]: pgmap v981: 129 pgs: 129 active+clean; 280 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-22T02:34:14.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:14 smithi083 ceph-mon[36340]: pgmap v981: 129 pgs: 129 active+clean; 280 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.8 MiB/s wr, 301 op/s 2024-03-22T02:34:14.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 121813 15.58 MB/sec execute 294 sec latency 85.791 ms 2024-03-22T02:34:15.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 122339 15.60 MB/sec execute 295 sec latency 54.620 ms 2024-03-22T02:34:16.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:16 smithi028 ceph-mon[31916]: pgmap v982: 129 pgs: 129 active+clean; 275 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 424 op/s 2024-03-22T02:34:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:16 smithi120 ceph-mon[36803]: pgmap v982: 129 pgs: 129 active+clean; 275 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 424 op/s 2024-03-22T02:34:16.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:16 smithi083 ceph-mon[36340]: pgmap v982: 129 pgs: 129 active+clean; 275 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 424 op/s 2024-03-22T02:34:16.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 122775 15.67 MB/sec execute 296 sec latency 181.769 ms 2024-03-22T02:34:17.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 123294 15.69 MB/sec execute 297 sec latency 80.269 ms 2024-03-22T02:34:18.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:18 smithi028 ceph-mon[31916]: pgmap v983: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:34:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:18 smithi120 ceph-mon[36803]: pgmap v983: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:34:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:18 smithi083 ceph-mon[36340]: pgmap v983: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.2 MiB/s wr, 530 op/s 2024-03-22T02:34:18.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 123608 15.73 MB/sec execute 298 sec latency 133.446 ms 2024-03-22T02:34:19.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 123906 15.76 MB/sec execute 299 sec latency 79.838 ms 2024-03-22T02:34:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:20 smithi028 ceph-mon[31916]: pgmap v984: 129 pgs: 129 active+clean; 270 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 497 op/s 2024-03-22T02:34:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:20 smithi120 ceph-mon[36803]: pgmap v984: 129 pgs: 129 active+clean; 270 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 497 op/s 2024-03-22T02:34:20.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:20 smithi083 ceph-mon[36340]: pgmap v984: 129 pgs: 129 active+clean; 270 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.9 MiB/s wr, 497 op/s 2024-03-22T02:34:20.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 124133 15.76 MB/sec execute 300 sec latency 95.215 ms 2024-03-22T02:34:21.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 124229 15.71 MB/sec execute 301 sec latency 381.585 ms 2024-03-22T02:34:22.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:22 smithi028 ceph-mon[31916]: pgmap v985: 129 pgs: 129 active+clean; 272 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 503 op/s 2024-03-22T02:34:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:22 smithi120 ceph-mon[36803]: pgmap v985: 129 pgs: 129 active+clean; 272 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 503 op/s 2024-03-22T02:34:22.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:22 smithi083 ceph-mon[36340]: pgmap v985: 129 pgs: 129 active+clean; 272 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 503 op/s 2024-03-22T02:34:22.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 124380 15.68 MB/sec execute 302 sec latency 167.673 ms 2024-03-22T02:34:23.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 124588 15.65 MB/sec execute 303 sec latency 85.773 ms 2024-03-22T02:34:24.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:24 smithi028 ceph-mon[31916]: pgmap v986: 129 pgs: 129 active+clean; 277 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 526 op/s 2024-03-22T02:34:24.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:24 smithi120 ceph-mon[36803]: pgmap v986: 129 pgs: 129 active+clean; 277 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 526 op/s 2024-03-22T02:34:24.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:24 smithi083 ceph-mon[36340]: pgmap v986: 129 pgs: 129 active+clean; 277 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.4 MiB/s wr, 526 op/s 2024-03-22T02:34:24.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 124937 15.64 MB/sec execute 304 sec latency 88.847 ms 2024-03-22T02:34:25.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 125286 15.64 MB/sec execute 305 sec latency 77.244 ms 2024-03-22T02:34:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 -- 172.21.15.28:0/3437905069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 msgr2=0x7f63a0108ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/3437905069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a0108ef0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6398009ca0 tx=0x7f639802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 -- 172.21.15.28:0/3437905069 shutdown_connections 2024-03-22T02:34:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/3437905069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a0108ef0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/3437905069 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0101ca0 0x7f63a0102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/3437905069 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0100ab0 0x7f63a0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 -- 172.21.15.28:0/3437905069 >> 172.21.15.28:0/3437905069 conn(0x7f63a00fc020 msgr2=0x7f63a00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 -- 172.21.15.28:0/3437905069 shutdown_connections 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.921+0000 7f63a7fe2640 1 -- 172.21.15.28:0/3437905069 wait complete. 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 Processor -- start 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 -- start start 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 0x7f63a0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a019e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63a010c730 con 0x7f63a0102660 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63a010c5b0 con 0x7f63a0101ca0 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a7fe2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a010c8b0 con 0x7f63a0100ab0 2024-03-22T02:34:25.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a019e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a557d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 0x7f63a0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a5d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a5d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:57658/0 (socket says 172.21.15.28:57658) 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a557d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 0x7f63a0197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:50878/0 (socket says 172.21.15.28:50878) 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.922+0000 7f63a5d7e640 1 -- 172.21.15.28:0/1069707338 learned_addr learned my addr 172.21.15.28:0/1069707338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 -- 172.21.15.28:0/1069707338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 msgr2=0x7f63a0197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 0x7f63a0197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 -- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 msgr2=0x7f63a019e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a019e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 -- 172.21.15.28:0/1069707338 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a019e910 con 0x7f63a0100ab0 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a657f640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a019e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f639000e9c0 tx=0x7f639000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f6396ffd640 1 -- 172.21.15.28:0/1069707338 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000cd80 con 0x7f63a0100ab0 2024-03-22T02:34:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a019ec00 con 0x7f63a0100ab0 2024-03-22T02:34:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f6396ffd640 1 -- 172.21.15.28:0/1069707338 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63900035f0 con 0x7f63a0100ab0 2024-03-22T02:34:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f6396ffd640 1 -- 172.21.15.28:0/1069707338 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390010690 con 0x7f63a0100ab0 2024-03-22T02:34:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.923+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a019f150 con 0x7f63a0100ab0 2024-03-22T02:34:25.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.925+0000 7f6396ffd640 1 -- 172.21.15.28:0/1069707338 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f63900037a0 con 0x7f63a0100ab0 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.925+0000 7f63aabc9740 1 Processor -- start 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 -- start start 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102ce9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102cfabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e102c53040 con 0x55e102ceaec0 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e102bfb9e0 con 0x55e102cf87e0 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63aabc9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e102c4eff0 con 0x55e102ce9b50 2024-03-22T02:34:25.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42000/0 (socket says 172.21.15.28:42000) 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 -- 172.21.15.28:0/2816794185 learned_addr learned my addr 172.21.15.28:0/2816794185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f6396ffd640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 0x7f638c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102ce9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a557d640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102cfabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 0x7f638c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 -- 172.21.15.28:0/2816794185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 msgr2=0x55e102ce9fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102ce9fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f6396ffd640 1 -- 172.21.15.28:0/1069707338 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f63900cfc80 con 0x7f63a0100ab0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 -- 172.21.15.28:0/2816794185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 msgr2=0x55e102cfabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102cfabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a657f640 1 -- 172.21.15.28:0/2816794185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e102967210 con 0x55e102ceaec0 2024-03-22T02:34:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.926+0000 7f63a557d640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102cfabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f63a657f640 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f63980324c0 tx=0x7f639802cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f419908ba6bba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f638bfff640 1 -- 172.21.15.28:0/2816794185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398050070 con 0x55e102ceaec0 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f638bfff640 1 -- 172.21.15.28:0/2816794185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6398034cd0 con 0x55e102ceaec0 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f638bfff640 1 -- 172.21.15.28:0/2816794185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398039040 con 0x55e102ceaec0 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f63a657f640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 0x7f638c0a5480 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f639804d000 tx=0x7f6398048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f63aabc9740 1 -- 172.21.15.28:0/2816794185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 msgr2=0x55e102ce9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.927+0000 7f63aabc9740 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f63980324c0 tx=0x7f639802cef0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 -- 172.21.15.28:0/2816794185 shutdown_connections 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102cfabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102ce9fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 --2- 172.21.15.28:0/2816794185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102ce9610 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 -- 172.21.15.28:0/2816794185 >> 172.21.15.28:0/2816794185 conn(0x55e102cebc60 msgr2=0x55e102cec060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 -- 172.21.15.28:0/2816794185 shutdown_connections 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 -- 172.21.15.28:0/2816794185 wait complete. 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 -- start start 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 Processor -- start 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.928+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ceaec0 0x55e102cfce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102d03e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e102cfb410 con 0x55e102ce9b50 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e102c0edb0 con 0x55e102cf87e0 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42014/0 (socket says 172.21.15.28:42014) 2024-03-22T02:34:25.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 -- 172.21.15.28:0/4237930855 learned_addr learned my addr 172.21.15.28:0/4237930855 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e102cfb590 con 0x55e102ceaec0 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a557d640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ceaec0 0x55e102cfce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a657f640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102d03e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 10 client.0 _ll_register_callbacks cb 0x55e102966a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 -- 172.21.15.28:0/4237930855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ceaec0 msgr2=0x55e102cfce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ceaec0 0x55e102cfce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 -- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 msgr2=0x55e102d03e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102d03e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 -- 172.21.15.28:0/4237930855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e102a33620 con 0x55e102ce9b50 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a657f640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102d03e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6398034cd0 con 0x55e102ce9b50 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f63900058c0 tx=0x7f639009a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3911a6d487ac399 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63897fa640 10 client.20154 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63897fa640 1 -- 172.21.15.28:0/4237930855 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900a0090 con 0x55e102ce9b50 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63897fa640 1 -- 172.21.15.28:0/4237930855 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f639009b390 con 0x55e102ce9b50 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63897fa640 1 -- 172.21.15.28:0/4237930855 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f639009b590 con 0x55e102ce9b50 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 10 client.20154 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.929+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=46}) -- 0x55e102d071b0 con 0x55e102ce9b50 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63897fa640 1 -- 172.21.15.28:0/4237930855 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f6390097c00 con 0x55e102ce9b50 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 10 client.20154 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 10 client.20154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 20 client.20154 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d08770 0x55e102d0ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 4 client.20154 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e102c60580 con 0x55e102d08770 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63a557d640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d08770 0x55e102d0ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63a557d640 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d08770 0x55e102d0ab80 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9169eccbbccd37ac server_cookie=781ec58ff7ff8f15 in_seq=0 out_seq=0 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.930+0000 7f63897fa640 10 client.20154 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63897fa640 1 -- 172.21.15.28:0/4237930855 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e102c60580 con 0x55e102d08770 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63897fa640 10 client.20154 handle_command_reply: tid=0 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 client.20154 shutdown 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 msgr2=0x55e102cfc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f63900058c0 tx=0x7f639009a140 comp rx=0 tx=0).stop 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 shutdown_connections 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d08770 0x55e102d0ab80 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102d03e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ceaec0 0x55e102cfce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63897fa640 0 client.20154 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 --2- 172.21.15.28:0/4237930855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ce9b50 0x55e102cfc910 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 >> 172.21.15.28:0/4237930855 conn(0x55e102cebc60 msgr2=0x55e102ce8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 shutdown_connections 2024-03-22T02:34:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 1 -- 172.21.15.28:0/4237930855 wait complete. 2024-03-22T02:34:25.941 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.932+0000 7f63aabc9740 20 client.20154 trim_cache size 0 max 16384 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 Processor -- start 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 -- start start 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102cfcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102ce93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01aafb0 con 0x55e102ceaec0 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a8880 con 0x55e102cf87e0 2024-03-22T02:34:25.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63aabc9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a66d0 con 0x55e102ce9b50 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63a557d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63a557d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42018/0 (socket says 172.21.15.28:42018) 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63a657f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102ce93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63a557d640 1 -- 172.21.15.28:0/188345192 learned_addr learned my addr 172.21.15.28:0/188345192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.948+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102cfcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 -- 172.21.15.28:0/188345192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 msgr2=0x55e102cfcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102cfcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 -- 172.21.15.28:0/188345192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 msgr2=0x55e102ce93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102ce93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 -- 172.21.15.28:0/188345192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e102d0b200 con 0x55e102ceaec0 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a657f640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102ce93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102cfcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f63a557d640 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x55e102d0b430 tx=0x7f639c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1daa6938c8b585 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f638b7fe640 1 -- 172.21.15.28:0/188345192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c01f050 con 0x55e102ceaec0 2024-03-22T02:34:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f638b7fe640 1 -- 172.21.15.28:0/188345192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639c01f6d0 con 0x55e102ceaec0 2024-03-22T02:34:25.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.949+0000 7f638b7fe640 1 -- 172.21.15.28:0/188345192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c010d60 con 0x55e102ceaec0 2024-03-22T02:34:25.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- 172.21.15.28:0/188345192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 msgr2=0x55e102d03e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:25.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x55e102d0b430 tx=0x7f639c005f90 comp rx=0 tx=0).stop 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- 172.21.15.28:0/188345192 shutdown_connections 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102cf87e0 0x55e102ce93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102ceaec0 0x55e102d03e20 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- 172.21.15.28:0/188345192 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102cfcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- 172.21.15.28:0/188345192 >> 172.21.15.28:0/188345192 conn(0x55e102cebc60 msgr2=0x55e102ceeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- 172.21.15.28:0/188345192 shutdown_connections 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- 172.21.15.28:0/188345192 wait complete. 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- start start 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 Processor -- start 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102d01590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 0x55e102d01ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a6d40 con 0x55e102cf87e0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a81b0 con 0x55e102ceaec0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.950+0000 7f63aabc9740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f63a01a7410 con 0x55e102ce9b50 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63aabc9740 10 client.0 _ll_register_callbacks cb 0x7f639009bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a557d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 0x55e102d01ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42020/0 (socket says 172.21.15.28:42020) 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a5d7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102d01590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 -- 172.21.15.28:0/3341849893 learned_addr learned my addr 172.21.15.28:0/3341849893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 -- 172.21.15.28:0/3341849893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 msgr2=0x55e102d01590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102d01590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 -- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 msgr2=0x55e102d01ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 0x55e102d01ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 -- 172.21.15.28:0/3341849893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e102d030d0 con 0x55e102cf87e0 2024-03-22T02:34:25.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102d01590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a557d640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 0x55e102d01ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63a657f640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6398009ca0 tx=0x7f639802cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b89baa0c05061605 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398039070 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f63980349f0 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.951+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f639c01f6d0 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6398050040 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63aabc9740 10 client.20160 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e102d6eb90 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639c01f6d0 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f639803f070 con 0x55e102cf87e0 2024-03-22T02:34:25.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63aabc9740 10 client.20160 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63aabc9740 10 client.20160 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.952+0000 7f63aabc9740 20 client.20160 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d6f110 0x55e102d71500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63aabc9740 4 client.20160 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e102c5a8d0 con 0x55e102d6f110 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d6f110 0x55e102d71500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63a5d7e640 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d6f110 0x55e102d71500 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bde3e5440234e1a0 server_cookie=c550d98a68b3dac6 in_seq=0 out_seq=0 2024-03-22T02:34:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:25.953+0000 7f63897fa640 10 client.20160 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:26.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:26 smithi028 ceph-mon[31916]: pgmap v987: 129 pgs: 129 active+clean; 276 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:34:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:26 smithi120 ceph-mon[36803]: pgmap v987: 129 pgs: 129 active+clean; 276 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:34:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:26 smithi083 ceph-mon[36340]: pgmap v987: 129 pgs: 129 active+clean; 276 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:34:26.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 125461 15.60 MB/sec execute 306 sec latency 506.237 ms 2024-03-22T02:34:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63897fa640 1 -- 172.21.15.28:0/3341849893 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e102c5a8d0 con 0x55e102d6f110 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63897fa640 10 client.20160 handle_command_reply: tid=0 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 client.20160 shutdown 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 msgr2=0x55e102d02ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f6398009ca0 tx=0x7f639802cca0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 shutdown_connections 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e102d6f110 0x55e102d71500 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e102cf87e0 0x55e102d02ae0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e102ceaec0 0x55e102d01ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 --2- 172.21.15.28:0/3341849893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e102ce9b50 0x55e102d01590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 >> 172.21.15.28:0/3341849893 conn(0x55e102cebc60 msgr2=0x55e102cef750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.227+0000 7f63897fa640 0 client.20160 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 shutdown_connections 2024-03-22T02:34:27.229 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63aabc9740 1 -- 172.21.15.28:0/3341849893 wait complete. 2024-03-22T02:34:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63aabc9740 20 client.20160 trim_cache size 0 max 16384 2024-03-22T02:34:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:34:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:34:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:34:27.230 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 msgr2=0x7f638c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 0x7f638c0a5480 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f639804d000 tx=0x7f6398048000 comp rx=0 tx=0).stop 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 msgr2=0x7f63a0196d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f639000e9c0 tx=0x7f639000ee80 comp rx=0 tx=0).stop 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 shutdown_connections 2024-03-22T02:34:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f638c0a2fc0 0x7f638c0a5480 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63a0102660 0x7f63a019e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f63a0101ca0 0x7f63a0197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 --2- 172.21.15.28:0/1069707338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f63a0100ab0 0x7f63a0196d00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 >> 172.21.15.28:0/1069707338 conn(0x7f63a00fc020 msgr2=0x7f63a0104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 shutdown_connections 2024-03-22T02:34:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:27.228+0000 7f63a7fe2640 1 -- 172.21.15.28:0/1069707338 wait complete. 2024-03-22T02:34:27.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 125732 15.59 MB/sec execute 307 sec latency 677.390 ms 2024-03-22T02:34:28.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:28 smithi028 ceph-mon[31916]: pgmap v988: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-03-22T02:34:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:28 smithi120 ceph-mon[36803]: pgmap v988: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-03-22T02:34:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:28 smithi083 ceph-mon[36340]: pgmap v988: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.7 MiB/s wr, 411 op/s 2024-03-22T02:34:28.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 126151 15.63 MB/sec execute 308 sec latency 57.617 ms 2024-03-22T02:34:29.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:29 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:29.381 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:29 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:29.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:29 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:34:29.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 126598 15.67 MB/sec execute 309 sec latency 132.145 ms 2024-03-22T02:34:30.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:30 smithi028 ceph-mon[31916]: pgmap v989: 129 pgs: 129 active+clean; 280 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-03-22T02:34:30.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:30 smithi120 ceph-mon[36803]: pgmap v989: 129 pgs: 129 active+clean; 280 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-03-22T02:34:30.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:30 smithi083 ceph-mon[36340]: pgmap v989: 129 pgs: 129 active+clean; 280 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 353 op/s 2024-03-22T02:34:30.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 126919 15.68 MB/sec execute 310 sec latency 373.826 ms 2024-03-22T02:34:31.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:31 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:31 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:31 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:31.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 127086 15.67 MB/sec execute 311 sec latency 387.508 ms 2024-03-22T02:34:32.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 127343 15.70 MB/sec execute 312 sec latency 488.922 ms 2024-03-22T02:34:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:32 smithi028 ceph-mon[31916]: pgmap v990: 129 pgs: 129 active+clean; 285 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 517 op/s 2024-03-22T02:34:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:32 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:32.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:32 smithi120 ceph-mon[36803]: pgmap v990: 129 pgs: 129 active+clean; 285 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 517 op/s 2024-03-22T02:34:32.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:32.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:32 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:32 smithi083 ceph-mon[36340]: pgmap v990: 129 pgs: 129 active+clean; 285 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.9 MiB/s wr, 517 op/s 2024-03-22T02:34:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:32 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:33.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 127620 15.69 MB/sec execute 313 sec latency 59.682 ms 2024-03-22T02:34:33.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:33.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:33.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:33.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:33.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:33 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:34.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:33 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:34:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:34:34.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:33 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:34:34.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 127799 15.67 MB/sec execute 314 sec latency 177.428 ms 2024-03-22T02:34:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:34 smithi120 ceph-mon[36803]: pgmap v991: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 530 op/s 2024-03-22T02:34:35.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:34 smithi083 ceph-mon[36340]: pgmap v991: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 530 op/s 2024-03-22T02:34:35.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:34 smithi028 ceph-mon[31916]: pgmap v991: 129 pgs: 129 active+clean; 273 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.4 MiB/s wr, 530 op/s 2024-03-22T02:34:35.690 INFO:tasks.workunit.client.0.smithi028.stdout: 10 127989 15.64 MB/sec execute 315 sec latency 103.119 ms 2024-03-22T02:34:36.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 128197 15.62 MB/sec execute 316 sec latency 294.509 ms 2024-03-22T02:34:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:36 smithi083 ceph-mon[36340]: pgmap v992: 129 pgs: 129 active+clean; 274 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 537 op/s 2024-03-22T02:34:37.032 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:36 smithi120 ceph-mon[36803]: pgmap v992: 129 pgs: 129 active+clean; 274 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 537 op/s 2024-03-22T02:34:37.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:36 smithi028 ceph-mon[31916]: pgmap v992: 129 pgs: 129 active+clean; 274 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.7 MiB/s wr, 537 op/s 2024-03-22T02:34:37.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 128399 15.60 MB/sec execute 317 sec latency 469.497 ms 2024-03-22T02:34:38.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 128723 15.59 MB/sec execute 318 sec latency 98.063 ms 2024-03-22T02:34:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:38 smithi083 ceph-mon[36340]: pgmap v993: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-03-22T02:34:39.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:38 smithi028 ceph-mon[31916]: pgmap v993: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-03-22T02:34:39.139 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:38 smithi120 ceph-mon[36803]: pgmap v993: 129 pgs: 129 active+clean; 276 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.3 MiB/s wr, 493 op/s 2024-03-22T02:34:39.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 129125 15.58 MB/sec execute 319 sec latency 70.132 ms 2024-03-22T02:34:40.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 -- 172.21.15.28:0/1241431021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 msgr2=0x7faf901024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 --2- 172.21.15.28:0/1241431021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf901024b0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7faf80009930 tx=0x7faf8002bc30 comp rx=0 tx=0).stop 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 -- 172.21.15.28:0/1241431021 shutdown_connections 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 --2- 172.21.15.28:0/1241431021 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf901016f0 0x7faf90108f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 --2- 172.21.15.28:0/1241431021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf90100d30 0x7faf901011b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 --2- 172.21.15.28:0/1241431021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf901024b0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 -- 172.21.15.28:0/1241431021 >> 172.21.15.28:0/1241431021 conn(0x7faf900fb7e0 msgr2=0x7faf900fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 -- 172.21.15.28:0/1241431021 shutdown_connections 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.319+0000 7faf954a4640 1 -- 172.21.15.28:0/1241431021 wait complete. 2024-03-22T02:34:40.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 Processor -- start 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 -- start start 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf90100d30 0x7faf90100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c740 con 0x7faf90102090 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c5c0 con 0x7faf90100d30 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf954a4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf9010c8c0 con 0x7faf901016f0 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34116/0 (socket says 172.21.15.28:34116) 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:51292/0 (socket says 172.21.15.28:51292) 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8f7fe640 1 -- 172.21.15.28:0/2302064102 learned_addr learned my addr 172.21.15.28:0/2302064102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:40.325 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.320+0000 7faf8effd640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf90100d30 0x7faf90100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 -- 172.21.15.28:0/2302064102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf90100d30 msgr2=0x7faf90100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf90100d30 0x7faf90100760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 -- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 msgr2=0x7faf9019f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 -- 172.21.15.28:0/2302064102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9019faa0 con 0x7faf901016f0 2024-03-22T02:34:40.326 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf8400e9f0 tx=0x7faf8400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf77fff640 1 -- 172.21.15.28:0/2302064102 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8400ccc0 con 0x7faf901016f0 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9019fd90 con 0x7faf901016f0 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf77fff640 1 -- 172.21.15.28:0/2302064102 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf84005020 con 0x7faf901016f0 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf77fff640 1 -- 172.21.15.28:0/2302064102 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840049e0 con 0x7faf901016f0 2024-03-22T02:34:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.321+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf901a0230 con 0x7faf901016f0 2024-03-22T02:34:40.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 Processor -- start 2024-03-22T02:34:40.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 -- start start 2024-03-22T02:34:40.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf77fff640 1 -- 172.21.15.28:0/2302064102 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faf84005560 con 0x7faf901016f0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff6269d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565ff586040 con 0x5565ff6214e0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565ff52e9e0 con 0x5565ff627040 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf98169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565ff581ff0 con 0x5565ff621ed0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8effd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff6269d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34132/0 (socket says 172.21.15.28:34132) 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 -- 172.21.15.28:0/4287053455 learned_addr learned my addr 172.21.15.28:0/4287053455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 -- 172.21.15.28:0/4287053455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 msgr2=0x5565ff6269d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff6269d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 -- 172.21.15.28:0/4287053455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 msgr2=0x5565ff6294d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.324+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6294d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf8f7fe640 1 -- 172.21.15.28:0/4287053455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ff29a210 con 0x5565ff6214e0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf77fff640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 0x7faf780a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7faf880118b0 tx=0x7faf88011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a403407692effb2e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf8effd640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 0x7faf780a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf77fff640 1 -- 172.21.15.28:0/2302064102 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf840cf7c0 con 0x7faf901016f0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf757fa640 1 -- 172.21.15.28:0/4287053455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8801e070 con 0x5565ff6214e0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf757fa640 1 -- 172.21.15.28:0/4287053455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf88011310 con 0x5565ff6214e0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.325+0000 7faf757fa640 1 -- 172.21.15.28:0/4287053455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf88019e60 con 0x5565ff6214e0 2024-03-22T02:34:40.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- 172.21.15.28:0/4287053455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 msgr2=0x5565ff621900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf8effd640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 0x7faf780a5480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faf8002c7b0 tx=0x7faf80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7faf880118b0 tx=0x7faf88011d70 comp rx=0 tx=0).stop 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- 172.21.15.28:0/4287053455 shutdown_connections 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6294d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff6269d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- 172.21.15.28:0/4287053455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff621900 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- 172.21.15.28:0/4287053455 >> 172.21.15.28:0/4287053455 conn(0x5565ff61ca50 msgr2=0x5565ff61ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- 172.21.15.28:0/4287053455 shutdown_connections 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- 172.21.15.28:0/4287053455 wait complete. 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- start start 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 Processor -- start 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff62b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6329d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5565ff629ff0 con 0x5565ff621ed0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5565ff541db0 con 0x5565ff627040 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5565ff62a170 con 0x5565ff6214e0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf98169740 10 client.0 _ll_register_callbacks cb 0x5565ff299a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.326+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34144/0 (socket says 172.21.15.28:34144) 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 -- 172.21.15.28:0/3958602912 learned_addr learned my addr 172.21.15.28:0/3958602912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8effd640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff62b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 -- 172.21.15.28:0/3958602912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 msgr2=0x5565ff62b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff62b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.336 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 -- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 msgr2=0x5565ff6329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 -- 172.21.15.28:0/3958602912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ff366620 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7faf840a3f60 tx=0x7faf8409b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c3d047b42f63c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8401c070 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf8409bce0 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf88011310 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7faf8409c180 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf98169740 10 client.20172 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5565ff635e10 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.327+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf840b3070 con 0x5565ff621ed0 2024-03-22T02:34:40.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7faf840a4000 con 0x5565ff621ed0 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 10 client.20172 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 10 client.20172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 20 client.20172 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff637500 0x5565ff639940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 4 client.20172 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565ff593580 con 0x5565ff637500 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf8effd640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff637500 0x5565ff639940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf8effd640 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff637500 0x5565ff639940 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5d8756854a9316d server_cookie=f7f83e2e3597956f in_seq=0 out_seq=0 2024-03-22T02:34:40.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.328+0000 7faf72ff5640 10 client.20172 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf72ff5640 1 -- 172.21.15.28:0/3958602912 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5565ff593580 con 0x5565ff637500 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf72ff5640 10 client.20172 handle_command_reply: tid=0 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf98169740 1 client.20172 shutdown 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 msgr2=0x5565ff62b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7faf840a3f60 tx=0x7faf8409b910 comp rx=0 tx=0).stop 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.334+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 shutdown_connections 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff637500 0x5565ff639940 crc :-1 s=READY pgs=420 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff627040 0x5565ff6329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff621ed0 0x5565ff62b990 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 --2- 172.21.15.28:0/3958602912 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff62b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf72ff5640 0 client.20172 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:40.340 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 >> 172.21.15.28:0/3958602912 conn(0x5565ff61ca50 msgr2=0x5565ff61bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 shutdown_connections 2024-03-22T02:34:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 1 -- 172.21.15.28:0/3958602912 wait complete. 2024-03-22T02:34:40.341 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.335+0000 7faf98169740 20 client.20172 trim_cache size 0 max 16384 2024-03-22T02:34:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.350+0000 7faf98169740 1 Processor -- start 2024-03-22T02:34:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.350+0000 7faf98169740 1 -- start start 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.350+0000 7faf98169740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff61ca50 0x5565ff62b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.350+0000 7faf98169740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf98169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff61b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf98169740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf901ad210 con 0x5565ff6214e0 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf98169740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf901aaae0 con 0x5565ff61ca50 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf98169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a8930 con 0x5565ff621ed0 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34150/0 (socket says 172.21.15.28:34150) 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 -- 172.21.15.28:0/1544285960 learned_addr learned my addr 172.21.15.28:0/1544285960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8effd640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff61ca50 0x5565ff62b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff61b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 -- 172.21.15.28:0/1544285960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 msgr2=0x5565ff61b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff61b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 -- 172.21.15.28:0/1544285960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff61ca50 msgr2=0x5565ff62b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff61ca50 0x5565ff62b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 -- 172.21.15.28:0/1544285960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565ff62a510 con 0x5565ff6214e0 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff61b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7faf84097ee0 tx=0x7faf840054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9767d91deb7213 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf74ff9640 1 -- 172.21.15.28:0/1544285960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8401c070 con 0x5565ff6214e0 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf74ff9640 1 -- 172.21.15.28:0/1544285960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf840b3070 con 0x5565ff6214e0 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.351+0000 7faf74ff9640 1 -- 172.21.15.28:0/1544285960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8409bce0 con 0x5565ff6214e0 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- 172.21.15.28:0/1544285960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 msgr2=0x5565ff632990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7faf84097ee0 tx=0x7faf840054d0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- 172.21.15.28:0/1544285960 shutdown_connections 2024-03-22T02:34:40.360 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff621ed0 0x5565ff61b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff6214e0 0x5565ff632990 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 --2- 172.21.15.28:0/1544285960 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff61ca50 0x5565ff62b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- 172.21.15.28:0/1544285960 >> 172.21.15.28:0/1544285960 conn(0x5565ff62b010 msgr2=0x5565ff620540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- 172.21.15.28:0/1544285960 shutdown_connections 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- 172.21.15.28:0/1544285960 wait complete. 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.352+0000 7faf98169740 1 -- start start 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 Processor -- start 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff6327c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff621ed0 0x5565ff630890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a8fa0 con 0x5565ff61ca50 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faf901aa410 con 0x5565ff621ed0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faf901a9670 con 0x5565ff6214e0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf98169740 10 client.0 _ll_register_callbacks cb 0x7faf840053d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34158/0 (socket says 172.21.15.28:34158) 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 -- 172.21.15.28:0/3474768537 learned_addr learned my addr 172.21.15.28:0/3474768537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff6327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8f7fe640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff621ed0 0x5565ff630890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 -- 172.21.15.28:0/3474768537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 msgr2=0x5565ff6327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff6327c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 -- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff621ed0 msgr2=0x5565ff630890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff621ed0 0x5565ff630890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 -- 172.21.15.28:0/3474768537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565ff61d4c0 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf8effd640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7faf90068fd0 tx=0x7faf8003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39ba11dd66ee410a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.353+0000 7faf72ff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80051540 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faf800516e0 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf840b3070 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7faf8005f050 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 10 client.20178 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5565ff69e040 con 0x5565ff61ca50 2024-03-22T02:34:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf840b3070 con 0x5565ff61ca50 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7faf800035f0 con 0x5565ff61ca50 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 10 client.20178 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 10 client.20178 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 20 client.20178 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff627040 0x5565ff6a0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 4 client.20178 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5565ff58d8d0 con 0x5565ff627040 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.354+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff627040 0x5565ff6a0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.355+0000 7faf8e7fc640 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff627040 0x5565ff6a0a60 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9959ace1b766082c server_cookie=6dc1ac7bd54108ac in_seq=0 out_seq=0 2024-03-22T02:34:40.364 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:40.355+0000 7faf72ff5640 10 client.20178 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:40.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 129481 15.58 MB/sec execute 320 sec latency 210.968 ms 2024-03-22T02:34:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:40 smithi120 ceph-mon[36803]: pgmap v994: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-03-22T02:34:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:40 smithi083 ceph-mon[36340]: pgmap v994: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-03-22T02:34:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:40 smithi028 ceph-mon[31916]: pgmap v994: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.9 MiB/s wr, 505 op/s 2024-03-22T02:34:41.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 129557 15.56 MB/sec execute 321 sec latency 792.439 ms 2024-03-22T02:34:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf72ff5640 1 -- 172.21.15.28:0/3474768537 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5565ff58d8d0 con 0x5565ff627040 2024-03-22T02:34:41.832 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf72ff5640 10 client.20178 handle_command_reply: tid=0 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 client.20178 shutdown 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 msgr2=0x5565ff632280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7faf90068fd0 tx=0x7faf8003f040 comp rx=0 tx=0).stop 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 shutdown_connections 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5565ff627040 0x5565ff6a0a60 crc :-1 s=READY pgs=421 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5565ff621ed0 0x5565ff630890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf72ff5640 0 client.20178 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5565ff6214e0 0x5565ff6327c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 --2- 172.21.15.28:0/3474768537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565ff61ca50 0x5565ff632280 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 >> 172.21.15.28:0/3474768537 conn(0x5565ff62b010 msgr2=0x5565ff620540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 shutdown_connections 2024-03-22T02:34:41.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 1 -- 172.21.15.28:0/3474768537 wait complete. 2024-03-22T02:34:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.830+0000 7faf98169740 20 client.20178 trim_cache size 0 max 16384 2024-03-22T02:34:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:34:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:34:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:34:41.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:34:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 msgr2=0x7faf780a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 0x7faf780a5480 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7faf8002c7b0 tx=0x7faf80000bc0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 msgr2=0x7faf9019cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faf8400e9f0 tx=0x7faf8400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 shutdown_connections 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faf780a2fc0 0x7faf780a5480 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf90102090 0x7faf9019f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf901016f0 0x7faf9019cbf0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 --2- 172.21.15.28:0/2302064102 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faf90100d30 0x7faf90100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 >> 172.21.15.28:0/2302064102 conn(0x7faf900fb7e0 msgr2=0x7faf90105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 shutdown_connections 2024-03-22T02:34:41.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:41.831+0000 7faf954a4640 1 -- 172.21.15.28:0/2302064102 wait complete. 2024-03-22T02:34:42.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 129909 15.61 MB/sec execute 322 sec latency 247.515 ms 2024-03-22T02:34:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:42 smithi120 ceph-mon[36803]: pgmap v995: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-03-22T02:34:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:42 smithi083 ceph-mon[36340]: pgmap v995: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-03-22T02:34:43.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:42 smithi028 ceph-mon[31916]: pgmap v995: 129 pgs: 129 active+clean; 281 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 5.0 MiB/s wr, 454 op/s 2024-03-22T02:34:43.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 130366 15.63 MB/sec execute 323 sec latency 138.030 ms 2024-03-22T02:34:44.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 130685 15.65 MB/sec execute 324 sec latency 281.143 ms 2024-03-22T02:34:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:44 smithi120 ceph-mon[36803]: pgmap v996: 129 pgs: 129 active+clean; 281 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 372 op/s 2024-03-22T02:34:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:44 smithi083 ceph-mon[36340]: pgmap v996: 129 pgs: 129 active+clean; 281 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 372 op/s 2024-03-22T02:34:45.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:44 smithi028 ceph-mon[31916]: pgmap v996: 129 pgs: 129 active+clean; 281 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 4.2 MiB/s wr, 372 op/s 2024-03-22T02:34:45.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 130922 15.66 MB/sec execute 325 sec latency 332.708 ms 2024-03-22T02:34:46.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 131168 15.64 MB/sec execute 326 sec latency 61.270 ms 2024-03-22T02:34:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:46 smithi120 ceph-mon[36803]: pgmap v997: 129 pgs: 129 active+clean; 268 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:34:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:46 smithi083 ceph-mon[36340]: pgmap v997: 129 pgs: 129 active+clean; 268 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:34:47.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:46 smithi028 ceph-mon[31916]: pgmap v997: 129 pgs: 129 active+clean; 268 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.8 MiB/s wr, 516 op/s 2024-03-22T02:34:47.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 131282 15.62 MB/sec execute 327 sec latency 232.959 ms 2024-03-22T02:34:48.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 131439 15.59 MB/sec execute 328 sec latency 150.415 ms 2024-03-22T02:34:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:48 smithi120 ceph-mon[36803]: pgmap v998: 129 pgs: 129 active+clean; 269 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 519 op/s 2024-03-22T02:34:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:48 smithi083 ceph-mon[36340]: pgmap v998: 129 pgs: 129 active+clean; 269 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 519 op/s 2024-03-22T02:34:49.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:48 smithi028 ceph-mon[31916]: pgmap v998: 129 pgs: 129 active+clean; 269 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 519 op/s 2024-03-22T02:34:49.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 131665 15.57 MB/sec execute 329 sec latency 89.359 ms 2024-03-22T02:34:50.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 131932 15.56 MB/sec execute 330 sec latency 70.262 ms 2024-03-22T02:34:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:50 smithi120 ceph-mon[36803]: pgmap v999: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 509 op/s 2024-03-22T02:34:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:50 smithi083 ceph-mon[36340]: pgmap v999: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 509 op/s 2024-03-22T02:34:51.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:50 smithi028 ceph-mon[31916]: pgmap v999: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.1 MiB/s wr, 509 op/s 2024-03-22T02:34:51.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 132346 15.56 MB/sec execute 331 sec latency 74.600 ms 2024-03-22T02:34:52.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 132546 15.54 MB/sec execute 332 sec latency 182.017 ms 2024-03-22T02:34:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:52 smithi120 ceph-mon[36803]: pgmap v1000: 129 pgs: 129 active+clean; 273 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-03-22T02:34:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:52 smithi083 ceph-mon[36340]: pgmap v1000: 129 pgs: 129 active+clean; 273 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-03-22T02:34:53.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:52 smithi028 ceph-mon[31916]: pgmap v1000: 129 pgs: 129 active+clean; 273 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.0 MiB/s wr, 514 op/s 2024-03-22T02:34:53.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 132999 15.55 MB/sec execute 333 sec latency 82.106 ms 2024-03-22T02:34:54.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 133435 15.60 MB/sec execute 334 sec latency 73.081 ms 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 -- 172.21.15.28:0/3152970033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 msgr2=0x7fa190104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 --2- 172.21.15.28:0/3152970033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190104a20 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fa188009900 tx=0x7fa18802bc30 comp rx=0 tx=0).stop 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 -- 172.21.15.28:0/3152970033 shutdown_connections 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 --2- 172.21.15.28:0/3152970033 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190107890 0x7fa190109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 --2- 172.21.15.28:0/3152970033 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190104f60 0x7fa190107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 --2- 172.21.15.28:0/3152970033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190104a20 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 -- 172.21.15.28:0/3152970033 >> 172.21.15.28:0/3152970033 conn(0x7fa1900fc020 msgr2=0x7fa1900fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:54.946 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 -- 172.21.15.28:0/3152970033 shutdown_connections 2024-03-22T02:34:54.947 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.945+0000 7fa1984e1640 1 -- 172.21.15.28:0/3152970033 wait complete. 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 Processor -- start 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 -- start start 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 0x7fa1901983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190107890 0x7fa19019f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010d830 con 0x7fa190102630 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010d6b0 con 0x7fa190107890 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa1984e1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa19010d9b0 con 0x7fa190104f60 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 0x7fa1901983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59330/0 (socket says 172.21.15.28:59330) 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa19627d640 1 -- 172.21.15.28:0/2715120302 learned_addr learned my addr 172.21.15.28:0/2715120302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa196a7e640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190107890 0x7fa19019f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 -- 172.21.15.28:0/2715120302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190107890 msgr2=0x7fa19019f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190107890 0x7fa19019f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 -- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 msgr2=0x7fa190197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.946+0000 7fa195a7c640 1 -- 172.21.15.28:0/2715120302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19019fa90 con 0x7fa190104f60 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa195a7c640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 0x7fa1901983b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa18000d6a0 tx=0x7fa18000daa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa1877fe640 1 -- 172.21.15.28:0/2715120302 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180017070 con 0x7fa190104f60 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa19019fd80 con 0x7fa190104f60 2024-03-22T02:34:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1901a0250 con 0x7fa190104f60 2024-03-22T02:34:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa1877fe640 1 -- 172.21.15.28:0/2715120302 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1800047a0 con 0x7fa190104f60 2024-03-22T02:34:54.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.947+0000 7fa1877fe640 1 -- 172.21.15.28:0/2715120302 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180009c10 con 0x7fa190104f60 2024-03-22T02:34:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.950+0000 7fa1877fe640 1 -- 172.21.15.28:0/2715120302 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa180005220 con 0x7fa190104f60 2024-03-22T02:34:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 Processor -- start 2024-03-22T02:34:54.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 -- start start 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f403900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f4089d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f40b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55948f368040 con 0x55948f4034e0 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55948f3109e0 con 0x55948f403ed0 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa19b19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55948f363ff0 con 0x55948f409040 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa1877fe640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 0x7fa17c0a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa196a7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f403900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa195a7c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f40b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa196a7e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f403900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59336/0 (socket says 172.21.15.28:59336) 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.951+0000 7fa196a7e640 1 -- 172.21.15.28:0/1431531955 learned_addr learned my addr 172.21.15.28:0/1431531955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa196a7e640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 0x7fa17c0a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa1877fe640 1 -- 172.21.15.28:0/2715120302 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa1800cf2e0 con 0x7fa190104f60 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa19627d640 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f4089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 -- 172.21.15.28:0/1431531955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 msgr2=0x55948f4089d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f4089d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 -- 172.21.15.28:0/1431531955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 msgr2=0x55948f403900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f403900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 -- 172.21.15.28:0/1431531955 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948f07c210 con 0x55948f409040 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa195a7c640 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f40b4d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa1800cca20 tx=0x7fa18009b490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=871ca5b8b717a6f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa196a7e640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 0x7fa17c0a9890 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa18c005c40 tx=0x7fa18c009c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa184bfe640 1 -- 172.21.15.28:0/1431531955 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa180020070 con 0x55948f409040 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa184bfe640 1 -- 172.21.15.28:0/1431531955 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa18009bd00 con 0x55948f409040 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.952+0000 7fa184bfe640 1 -- 172.21.15.28:0/1431531955 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1800a4010 con 0x55948f409040 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 -- 172.21.15.28:0/1431531955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 msgr2=0x55948f40b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f40b4d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa1800cca20 tx=0x7fa18009b490 comp rx=0 tx=0).stop 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 -- 172.21.15.28:0/1431531955 shutdown_connections 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f40b4d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f4089d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1431531955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f403900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 -- 172.21.15.28:0/1431531955 >> 172.21.15.28:0/1431531955 conn(0x55948f3fea50 msgr2=0x55948f3fee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 -- 172.21.15.28:0/1431531955 shutdown_connections 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.953+0000 7fa19b19b740 1 -- 172.21.15.28:0/1431531955 wait complete. 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 -- start start 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 Processor -- start 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f40d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f409040 0x55948f414950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55948f40bff0 con 0x55948f409040 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55948f323db0 con 0x55948f403ed0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55948f40c170 con 0x55948f4034e0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19b19b740 10 client.0 _ll_register_callbacks cb 0x55948f07ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19627d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa195a7c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f40d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19627d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:49168/0 (socket says 172.21.15.28:49168) 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa19627d640 1 -- 172.21.15.28:0/3342835054 learned_addr learned my addr 172.21.15.28:0/3342835054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.954+0000 7fa196a7e640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f409040 0x55948f414950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 -- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 msgr2=0x55948f40d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f40d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 -- 172.21.15.28:0/3342835054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f409040 msgr2=0x55948f414950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f409040 0x55948f414950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 -- 172.21.15.28:0/3342835054 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948f148620 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19627d640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa188009230 tx=0x7fa188004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f01a2caa4c6ff135 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188005d50 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa1880046c0 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa18009bd00 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa1880048c0 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19b19b740 10 client.29983 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.955+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55948f417d30 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1880053b0 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fa188003820 con 0x55948f4034e0 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 10 client.29983 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 10 client.29983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:54.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 20 client.29983 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f419910 0x55948f41bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 4 client.29983 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa18009bd00 con 0x55948f419910 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa195a7c640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f419910 0x55948f41bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa179ffb640 10 client.29983 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:54.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.956+0000 7fa195a7c640 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f419910 0x55948f41bd70 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3811368903e5af4 server_cookie=ba6c75af63959fa5 in_seq=0 out_seq=0 2024-03-22T02:34:54.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.964+0000 7fa179ffb640 1 -- 172.21.15.28:0/3342835054 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa18009bd00 con 0x55948f419910 2024-03-22T02:34:54.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.964+0000 7fa179ffb640 10 client.29983 handle_command_reply: tid=0 2024-03-22T02:34:54.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.964+0000 7fa19b19b740 1 client.29983 shutdown 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 msgr2=0x55948f40d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa188009230 tx=0x7fa188004270 comp rx=0 tx=0).stop 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 shutdown_connections 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f419910 0x55948f41bd70 crc :-1 s=READY pgs=422 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f409040 0x55948f414950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f40d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3342835054 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f4034e0 0x55948f40d3d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa179ffb640 0 client.29983 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 >> 172.21.15.28:0/3342835054 conn(0x55948f3fea50 msgr2=0x55948f3fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 shutdown_connections 2024-03-22T02:34:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 1 -- 172.21.15.28:0/3342835054 wait complete. 2024-03-22T02:34:54.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.965+0000 7fa19b19b740 20 client.29983 trim_cache size 0 max 16384 2024-03-22T02:34:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.982+0000 7fa19b19b740 1 Processor -- start 2024-03-22T02:34:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 -- start start 2024-03-22T02:34:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.984 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f414870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f420670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa174012690 con 0x55948f4034e0 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa17400ff60 con 0x55948f403ed0 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19b19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa17400ddb0 con 0x55948f409040 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59346/0 (socket says 172.21.15.28:59346) 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 -- 172.21.15.28:0/1622864851 learned_addr learned my addr 172.21.15.28:0/1622864851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa196a7e640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f420670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa195a7c640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f414870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 -- 172.21.15.28:0/1622864851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 msgr2=0x55948f420670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f420670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 -- 172.21.15.28:0/1622864851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 msgr2=0x55948f414870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.985 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f414870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 -- 172.21.15.28:0/1622864851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948f408090 con 0x55948f4034e0 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa196a7e640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f420670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.983+0000 7fa19627d640 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fa188003e80 tx=0x7fa18802cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df335eddce687b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.984+0000 7fa17bfff640 1 -- 172.21.15.28:0/1622864851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188059070 con 0x55948f4034e0 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.984+0000 7fa17bfff640 1 -- 172.21.15.28:0/1622864851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188003ad0 con 0x55948f4034e0 2024-03-22T02:34:54.986 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.984+0000 7fa17bfff640 1 -- 172.21.15.28:0/1622864851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188004bc0 con 0x55948f4034e0 2024-03-22T02:34:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- 172.21.15.28:0/1622864851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 msgr2=0x55948f405030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fa188003e80 tx=0x7fa18802cd20 comp rx=0 tx=0).stop 2024-03-22T02:34:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- 172.21.15.28:0/1622864851 shutdown_connections 2024-03-22T02:34:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f409040 0x55948f420670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.989 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f403ed0 0x55948f414870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 --2- 172.21.15.28:0/1622864851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f405030 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- 172.21.15.28:0/1622864851 >> 172.21.15.28:0/1622864851 conn(0x55948f3fea50 msgr2=0x55948f404920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- 172.21.15.28:0/1622864851 shutdown_connections 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- 172.21.15.28:0/1622864851 wait complete. 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 -- start start 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 Processor -- start 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.985+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f403ed0 0x55948f4120a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f409040 0x55948f4130b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa17400e420 con 0x55948f4034e0 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa17400f890 con 0x55948f409040 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa17400eaf0 con 0x55948f403ed0 2024-03-22T02:34:54.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 10 client.0 _ll_register_callbacks cb 0x7fa18802c670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59354/0 (socket says 172.21.15.28:59354) 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 -- 172.21.15.28:0/3080796250 learned_addr learned my addr 172.21.15.28:0/3080796250 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa196a7e640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f409040 0x55948f4130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa195a7c640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f403ed0 0x55948f4120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 -- 172.21.15.28:0/3080796250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f403ed0 msgr2=0x55948f4120a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f403ed0 0x55948f4120a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 -- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f409040 msgr2=0x55948f4130b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f409040 0x55948f4130b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 -- 172.21.15.28:0/3080796250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948f4136a0 con 0x55948f4034e0 2024-03-22T02:34:54.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19627d640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa188009850 tx=0x7fa18802cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de3074e105e0af9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa188059030 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa1880035f0 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.986+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa188003ad0 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa18806d050 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 10 client.20196 fetch_fsmap learned FSMap version 46 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55948f4149b0 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa188003ad0 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fa18805e450 con 0x55948f4034e0 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 10 client.20196 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 10 client.20196 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 20 client.20196 populate_metadata read hostname 'smithi028' 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f47fee0 0x55948f4934a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:34:54.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 4 client.20196 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:34:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55948f375580 con 0x55948f47fee0 2024-03-22T02:34:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.987+0000 7fa195a7c640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f47fee0 0x55948f4934a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:34:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.988+0000 7fa195a7c640 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f47fee0 0x55948f4934a0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d264bcd4ddc2fcdf server_cookie=2531ba3c32ad32cb in_seq=0 out_seq=0 2024-03-22T02:34:54.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:54.988+0000 7fa179ffb640 10 client.20196 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:54 smithi120 ceph-mon[36803]: pgmap v1001: 129 pgs: 129 active+clean; 276 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:34:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:54 smithi083 ceph-mon[36340]: pgmap v1001: 129 pgs: 129 active+clean; 276 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:34:55.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:54 smithi028 ceph-mon[31916]: pgmap v1001: 129 pgs: 129 active+clean; 276 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-03-22T02:34:55.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 133591 15.60 MB/sec execute 335 sec latency 558.966 ms 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.081+0000 7fa179ffb640 1 -- 172.21.15.28:0/3080796250 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55948f375580 con 0x55948f47fee0 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.081+0000 7fa179ffb640 10 client.20196 handle_command_reply: tid=0 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.081+0000 7fa19b19b740 1 client.20196 shutdown 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 msgr2=0x55948f411b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fa188009850 tx=0x7fa18802cdd0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 shutdown_connections 2024-03-22T02:34:56.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55948f47fee0 0x55948f4934a0 crc :-1 s=READY pgs=423 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa179ffb640 0 client.20196 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55948f409040 0x55948f4130b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55948f403ed0 0x55948f4120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 --2- 172.21.15.28:0/3080796250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55948f4034e0 0x55948f411b60 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 >> 172.21.15.28:0/3080796250 conn(0x55948f3fea50 msgr2=0x55948f402540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 shutdown_connections 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 1 -- 172.21.15.28:0/3080796250 wait complete. 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.082+0000 7fa19b19b740 20 client.20196 trim_cache size 0 max 16384 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:34:56.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:34:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:34:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:34:56.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 msgr2=0x7fa17c0a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 0x7fa17c0a9890 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa18c005c40 tx=0x7fa18c009c90 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 msgr2=0x7fa1901983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 0x7fa1901983b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa18000d6a0 tx=0x7fa18000daa0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 shutdown_connections 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa17c0a73d0 0x7fa17c0a9890 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa190107890 0x7fa19019f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa190104f60 0x7fa1901983b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 --2- 172.21.15.28:0/2715120302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa190102630 0x7fa190197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 >> 172.21.15.28:0/2715120302 conn(0x7fa1900fc020 msgr2=0x7fa190102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 shutdown_connections 2024-03-22T02:34:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:34:56.083+0000 7fa1984e1640 1 -- 172.21.15.28:0/2715120302 wait complete. 2024-03-22T02:34:56.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 133948 15.60 MB/sec execute 336 sec latency 685.926 ms 2024-03-22T02:34:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:56 smithi120 ceph-mon[36803]: pgmap v1002: 129 pgs: 129 active+clean; 278 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-03-22T02:34:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:56 smithi083 ceph-mon[36340]: pgmap v1002: 129 pgs: 129 active+clean; 278 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-03-22T02:34:57.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:56 smithi028 ceph-mon[31916]: pgmap v1002: 129 pgs: 129 active+clean; 278 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.2 MiB/s wr, 466 op/s 2024-03-22T02:34:57.691 INFO:tasks.workunit.client.0.smithi028.stdout: 10 134191 15.61 MB/sec execute 337 sec latency 233.723 ms 2024-03-22T02:34:58.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 134423 15.61 MB/sec execute 338 sec latency 95.219 ms 2024-03-22T02:34:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:34:58 smithi120 ceph-mon[36803]: pgmap v1003: 129 pgs: 129 active+clean; 272 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2024-03-22T02:34:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:34:58 smithi083 ceph-mon[36340]: pgmap v1003: 129 pgs: 129 active+clean; 272 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2024-03-22T02:34:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:34:58 smithi028 ceph-mon[31916]: pgmap v1003: 129 pgs: 129 active+clean; 272 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.9 MiB/s wr, 369 op/s 2024-03-22T02:34:59.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 134678 15.60 MB/sec execute 339 sec latency 111.449 ms 2024-03-22T02:35:00.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 134917 15.60 MB/sec execute 340 sec latency 106.378 ms 2024-03-22T02:35:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:00 smithi120 ceph-mon[36803]: pgmap v1004: 129 pgs: 129 active+clean; 275 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-22T02:35:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:00 smithi083 ceph-mon[36340]: pgmap v1004: 129 pgs: 129 active+clean; 275 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-22T02:35:01.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:00 smithi028 ceph-mon[31916]: pgmap v1004: 129 pgs: 129 active+clean; 275 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 457 op/s 2024-03-22T02:35:01.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 135132 15.58 MB/sec execute 341 sec latency 139.436 ms 2024-03-22T02:35:02.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 135271 15.55 MB/sec execute 342 sec latency 359.481 ms 2024-03-22T02:35:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:02 smithi120 ceph-mon[36803]: pgmap v1005: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 563 op/s 2024-03-22T02:35:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:02 smithi083 ceph-mon[36340]: pgmap v1005: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 563 op/s 2024-03-22T02:35:03.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:02 smithi028 ceph-mon[31916]: pgmap v1005: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 563 op/s 2024-03-22T02:35:03.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 135525 15.54 MB/sec execute 343 sec latency 94.276 ms 2024-03-22T02:35:04.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 135876 15.53 MB/sec execute 344 sec latency 176.528 ms 2024-03-22T02:35:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:04 smithi120 ceph-mon[36803]: pgmap v1006: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-03-22T02:35:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:04 smithi083 ceph-mon[36340]: pgmap v1006: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-03-22T02:35:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:04 smithi028 ceph-mon[31916]: pgmap v1006: 129 pgs: 129 active+clean; 270 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 530 op/s 2024-03-22T02:35:05.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 136155 15.52 MB/sec execute 345 sec latency 85.830 ms 2024-03-22T02:35:06.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 136651 15.54 MB/sec execute 346 sec latency 95.189 ms 2024-03-22T02:35:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:06 smithi120 ceph-mon[36803]: pgmap v1007: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-22T02:35:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:06 smithi083 ceph-mon[36340]: pgmap v1007: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-22T02:35:07.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:06 smithi028 ceph-mon[31916]: pgmap v1007: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-22T02:35:07.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 136971 15.56 MB/sec execute 347 sec latency 115.340 ms 2024-03-22T02:35:08.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 137428 15.60 MB/sec execute 348 sec latency 83.929 ms 2024-03-22T02:35:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:08 smithi120 ceph-mon[36803]: pgmap v1008: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:35:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:08 smithi083 ceph-mon[36340]: pgmap v1008: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:35:09.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:08 smithi028 ceph-mon[31916]: pgmap v1008: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.3 MiB/s wr, 553 op/s 2024-03-22T02:35:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 -- 172.21.15.28:0/1431261453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c101ca0 msgr2=0x7f588c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 --2- 172.21.15.28:0/1431261453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c101ca0 0x7f588c102120 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f588002bc30 comp rx=0 tx=0).stop 2024-03-22T02:35:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 -- 172.21.15.28:0/1431261453 shutdown_connections 2024-03-22T02:35:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 --2- 172.21.15.28:0/1431261453 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c102660 0x7f588c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 --2- 172.21.15.28:0/1431261453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c101ca0 0x7f588c102120 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 --2- 172.21.15.28:0/1431261453 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c100ab0 0x7f588c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 -- 172.21.15.28:0/1431261453 >> 172.21.15.28:0/1431261453 conn(0x7f588c0fc020 msgr2=0x7f588c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 -- 172.21.15.28:0/1431261453 shutdown_connections 2024-03-22T02:35:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.188+0000 7f589199a640 1 -- 172.21.15.28:0/1431261453 wait complete. 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 Processor -- start 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 -- start start 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c100ab0 0x7f588c197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c101ca0 0x7f588c198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10d850 con 0x7f588c102660 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10d6d0 con 0x7f588c101ca0 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f589199a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f588c10d9d0 con 0x7f588c100ab0 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f588b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f588b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38076/0 (socket says 172.21.15.28:38076) 2024-03-22T02:35:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f588b7fe640 1 -- 172.21.15.28:0/1682216688 learned_addr learned my addr 172.21.15.28:0/1682216688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f588a7fc640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c101ca0 0x7f588c198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.189+0000 7f588affd640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c100ab0 0x7f588c197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 -- 172.21.15.28:0/1682216688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c100ab0 msgr2=0x7f588c197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c100ab0 0x7f588c197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 -- 172.21.15.28:0/1682216688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c101ca0 msgr2=0x7f588c198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c101ca0 0x7f588c198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 -- 172.21.15.28:0/1682216688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f588c19fa40 con 0x7f588c102660 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f588b7fe640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f587c00eaa0 tx=0x7f587c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f5890998640 1 -- 172.21.15.28:0/1682216688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c00cdb0 con 0x7f588c102660 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f5890998640 1 -- 172.21.15.28:0/1682216688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f587c005020 con 0x7f588c102660 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f5890998640 1 -- 172.21.15.28:0/1682216688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c0116b0 con 0x7f588c102660 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588c19fd30 con 0x7f588c102660 2024-03-22T02:35:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.190+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f588c1a01a0 con 0x7f588c102660 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.192+0000 7f5890998640 1 -- 172.21.15.28:0/1682216688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f587c005560 con 0x7f588c102660 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.192+0000 7f5894583740 1 Processor -- start 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 -- start start 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9acb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ace4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2c9a2b040 con 0x55e2c9ac64e0 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2c99d39e0 con 0x55e2c9acc040 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5894583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e2c9a26ff0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38080/0 (socket says 172.21.15.28:38080) 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 -- 172.21.15.28:0/1995439241 learned_addr learned my addr 172.21.15.28:0/1995439241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588affd640 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9acb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ace4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5890998640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 0x7f58740a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588b7fe640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 0x7f58740a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 -- 172.21.15.28:0/1995439241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 msgr2=0x55e2c9acb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9acb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 -- 172.21.15.28:0/1995439241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 msgr2=0x55e2c9ace4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ace4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f5890998640 1 -- 172.21.15.28:0/1682216688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f587c0cf960 con 0x7f588c102660 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 -- 172.21.15.28:0/1995439241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c973f210 con 0x55e2c9ac64e0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.193+0000 7f588a7fc640 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55985635b227ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5871ffb640 1 -- 172.21.15.28:0/1995439241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880004230 con 0x55e2c9ac64e0 2024-03-22T02:35:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f588b7fe640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 0x7f58740a5480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f587c0cd3c0 tx=0x7f587c066d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5871ffb640 1 -- 172.21.15.28:0/1995439241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58800043d0 con 0x55e2c9ac64e0 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5871ffb640 1 -- 172.21.15.28:0/1995439241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800346a0 con 0x55e2c9ac64e0 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 -- 172.21.15.28:0/1995439241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 msgr2=0x55e2c9ac6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880005fd0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 -- 172.21.15.28:0/1995439241 shutdown_connections 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ace4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9acb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 --2- 172.21.15.28:0/1995439241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ac6900 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 -- 172.21.15.28:0/1995439241 >> 172.21.15.28:0/1995439241 conn(0x55e2c9ac1a50 msgr2=0x55e2c9ac1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:09.203 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 -- 172.21.15.28:0/1995439241 shutdown_connections 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.194+0000 7f5894583740 1 -- 172.21.15.28:0/1995439241 wait complete. 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 -- start start 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 Processor -- start 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2c9acef90 con 0x55e2c9acc040 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e2c99e6db0 con 0x55e2c9ac64e0 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e2c9acf110 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f5894583740 10 client.0 _ll_register_callbacks cb 0x55e2c973ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f588affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f588affd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:45436/0 (socket says 172.21.15.28:45436) 2024-03-22T02:35:09.207 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f588affd640 1 -- 172.21.15.28:0/2228965728 learned_addr learned my addr 172.21.15.28:0/2228965728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f588a7fc640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.195+0000 7f588a7fc640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.208 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 -- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 msgr2=0x55e2c9ad0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 -- 172.21.15.28:0/2228965728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 msgr2=0x55e2c9ad7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.209 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 -- 172.21.15.28:0/2228965728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c980b620 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588a7fc640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:35:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f588affd640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5884012700 tx=0x7f5884012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97657b34b1951e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.210 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f586affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884004050 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58800043d0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f58840041f0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5884025050 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f5894583740 10 client.30001 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e2c9adaea0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.196+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58800043d0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f5884003a60 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 10 client.30001 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 10 client.30001 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 20 client.30001 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9adc630 0x55e2c9adea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 4 client.30001 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2c9a38580 con 0x55e2c9adc630 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f588a7fc640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9adc630 0x55e2c9adea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f588a7fc640 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9adc630 0x55e2c9adea20 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed07cf3983501864 server_cookie=dfd84b4c63109ad in_seq=0 out_seq=0 2024-03-22T02:35:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.197+0000 7f586affd640 10 client.30001 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f586affd640 1 -- 172.21.15.28:0/2228965728 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e2c9a38580 con 0x55e2c9adc630 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f586affd640 10 client.30001 handle_command_reply: tid=0 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 client.30001 shutdown 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 msgr2=0x55e2c9ad09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5884012700 tx=0x7f5884012bc0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 shutdown_connections 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9adc630 0x55e2c9adea20 crc :-1 s=READY pgs=424 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad09d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f586affd640 0 client.30001 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 --2- 172.21.15.28:0/2228965728 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 >> 172.21.15.28:0/2228965728 conn(0x55e2c9ac1a50 msgr2=0x55e2c9ac0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 shutdown_connections 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 1 -- 172.21.15.28:0/2228965728 wait complete. 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.199+0000 7f5894583740 20 client.30001 trim_cache size 0 max 16384 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 Processor -- start 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 -- start start 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac64e0 0x55e2c9acb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ac03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1ad180 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1aaa50 con 0x55e2c9acc040 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f5894583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1a88a0 con 0x55e2c9ac64e0 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f588affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f588affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38094/0 (socket says 172.21.15.28:38094) 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f588affd640 1 -- 172.21.15.28:0/3245005796 learned_addr learned my addr 172.21.15.28:0/3245005796 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.215+0000 7f588a7fc640 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac64e0 0x55e2c9acb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588a7fc640 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ac03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 -- 172.21.15.28:0/3245005796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac64e0 msgr2=0x55e2c9acb4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac64e0 0x55e2c9acb4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 -- 172.21.15.28:0/3245005796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 msgr2=0x55e2c9ac03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ac03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 -- 172.21.15.28:0/3245005796 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c9aded90 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f588affd640 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f5884003dd0 tx=0x7f5884010710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdc80ecadb9fe8c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f58717fa640 1 -- 172.21.15.28:0/3245005796 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58840108d0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f58717fa640 1 -- 172.21.15.28:0/3245005796 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5884005070 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.216+0000 7f58717fa640 1 -- 172.21.15.28:0/3245005796 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884003810 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- 172.21.15.28:0/3245005796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 msgr2=0x55e2c9ad79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f5884003dd0 tx=0x7f5884010710 comp rx=0 tx=0).stop 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- 172.21.15.28:0/3245005796 shutdown_connections 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9acc040 0x55e2c9ac03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad79d0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 --2- 172.21.15.28:0/3245005796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac64e0 0x55e2c9acb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- 172.21.15.28:0/3245005796 >> 172.21.15.28:0/3245005796 conn(0x55e2c9ac1a50 msgr2=0x55e2c9ac5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- 172.21.15.28:0/3245005796 shutdown_connections 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- 172.21.15.28:0/3245005796 wait complete. 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.217+0000 7f5894583740 1 -- start start 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 Processor -- start 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1a8f10 con 0x55e2c9acc040 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1aa380 con 0x55e2c9ac64e0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f588c1a95e0 con 0x55e2c9ac6ed0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f5894583740 10 client.0 _ll_register_callbacks cb 0x7f5884009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38102/0 (socket says 172.21.15.28:38102) 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 -- 172.21.15.28:0/4223000165 learned_addr learned my addr 172.21.15.28:0/4223000165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588affd640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 -- 172.21.15.28:0/4223000165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 msgr2=0x55e2c9ad7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 -- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 msgr2=0x55e2c9ad72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 -- 172.21.15.28:0/4223000165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c9ad6030 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f588a7fc640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfae2070231ba0f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.218+0000 7f586affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800043d0 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f5880005050 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5884005070 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5880067050 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 10 client.20214 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e2c9b43370 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5884005070 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f58800035f0 con 0x55e2c9acc040 2024-03-22T02:35:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 10 client.20214 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 10 client.20214 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 20 client.20214 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9b43870 0x55e2c9b45c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 4 client.20214 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e2c9a328d0 con 0x55e2c9b43870 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.219+0000 7f588affd640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9b43870 0x55e2c9b45c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.220+0000 7f586affd640 10 client.20214 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:09.220+0000 7f588affd640 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9b43870 0x55e2c9b45c60 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db9cc772b58ae60 server_cookie=b904504730569624 in_seq=0 out_seq=0 2024-03-22T02:35:09.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 137695 15.61 MB/sec execute 349 sec latency 390.491 ms 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.376+0000 7f586affd640 1 -- 172.21.15.28:0/4223000165 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e2c9a328d0 con 0x55e2c9b43870 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.376+0000 7f586affd640 10 client.20214 handle_command_reply: tid=0 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.376+0000 7f5894583740 1 client.20214 shutdown 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 msgr2=0x55e2c9ad58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f5880036040 comp rx=0 tx=0).stop 2024-03-22T02:35:10.378 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 shutdown_connections 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e2c9b43870 0x55e2c9b45c60 crc :-1 s=READY pgs=425 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e2c9acc040 0x55e2c9ad58f0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e2c9ac6ed0 0x55e2c9ad7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f586affd640 0 client.20214 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 --2- 172.21.15.28:0/4223000165 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e2c9ac64e0 0x55e2c9ad72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 >> 172.21.15.28:0/4223000165 conn(0x55e2c9ac1a50 msgr2=0x55e2c9ac5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 shutdown_connections 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 1 -- 172.21.15.28:0/4223000165 wait complete. 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.377+0000 7f5894583740 20 client.20214 trim_cache size 0 max 16384 2024-03-22T02:35:10.379 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:35:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:35:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:35:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:35:10.380 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 msgr2=0x7f58740a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 0x7f58740a5480 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f587c0cd3c0 tx=0x7f587c066d20 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 msgr2=0x7f588c19f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f587c00eaa0 tx=0x7f587c00ef60 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 shutdown_connections 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f58740a2fc0 0x7f58740a5480 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f588c102660 0x7f588c19f3d0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f588c101ca0 0x7f588c198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 --2- 172.21.15.28:0/1682216688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f588c100ab0 0x7f588c197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 >> 172.21.15.28:0/1682216688 conn(0x7f588c0fc020 msgr2=0x7f588c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 shutdown_connections 2024-03-22T02:35:10.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:10.378+0000 7f589199a640 1 -- 172.21.15.28:0/1682216688 wait complete. 2024-03-22T02:35:10.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 137777 15.58 MB/sec execute 350 sec latency 1008.504 ms 2024-03-22T02:35:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:10 smithi120 ceph-mon[36803]: pgmap v1009: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-03-22T02:35:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:10 smithi083 ceph-mon[36340]: pgmap v1009: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-03-22T02:35:11.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:10 smithi028 ceph-mon[31916]: pgmap v1009: 129 pgs: 129 active+clean; 279 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 502 op/s 2024-03-22T02:35:11.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 138046 15.59 MB/sec execute 351 sec latency 93.031 ms 2024-03-22T02:35:12.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 138212 15.57 MB/sec execute 352 sec latency 383.342 ms 2024-03-22T02:35:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:12 smithi120 ceph-mon[36803]: pgmap v1010: 129 pgs: 129 active+clean; 280 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 431 op/s 2024-03-22T02:35:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:12 smithi083 ceph-mon[36340]: pgmap v1010: 129 pgs: 129 active+clean; 280 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 431 op/s 2024-03-22T02:35:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:12 smithi028 ceph-mon[31916]: pgmap v1010: 129 pgs: 129 active+clean; 280 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 431 op/s 2024-03-22T02:35:13.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 138448 15.56 MB/sec execute 353 sec latency 205.964 ms 2024-03-22T02:35:14.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 138661 15.55 MB/sec execute 354 sec latency 100.665 ms 2024-03-22T02:35:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:14 smithi120 ceph-mon[36803]: pgmap v1011: 129 pgs: 129 active+clean; 282 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 437 op/s 2024-03-22T02:35:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:14 smithi083 ceph-mon[36340]: pgmap v1011: 129 pgs: 129 active+clean; 282 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 437 op/s 2024-03-22T02:35:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:14 smithi028 ceph-mon[31916]: pgmap v1011: 129 pgs: 129 active+clean; 282 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.7 MiB/s wr, 437 op/s 2024-03-22T02:35:15.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 138885 15.53 MB/sec execute 355 sec latency 104.929 ms 2024-03-22T02:35:16.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 139234 15.51 MB/sec execute 356 sec latency 69.461 ms 2024-03-22T02:35:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:16 smithi120 ceph-mon[36803]: pgmap v1012: 129 pgs: 129 active+clean; 274 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 561 op/s 2024-03-22T02:35:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:16 smithi083 ceph-mon[36340]: pgmap v1012: 129 pgs: 129 active+clean; 274 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 561 op/s 2024-03-22T02:35:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:16 smithi028 ceph-mon[31916]: pgmap v1012: 129 pgs: 129 active+clean; 274 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.3 MiB/s wr, 561 op/s 2024-03-22T02:35:17.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 139493 15.50 MB/sec execute 357 sec latency 187.528 ms 2024-03-22T02:35:18.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 139833 15.51 MB/sec execute 358 sec latency 155.732 ms 2024-03-22T02:35:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:18 smithi120 ceph-mon[36803]: pgmap v1013: 129 pgs: 129 active+clean; 274 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 496 op/s 2024-03-22T02:35:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:18 smithi083 ceph-mon[36340]: pgmap v1013: 129 pgs: 129 active+clean; 274 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 496 op/s 2024-03-22T02:35:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:18 smithi028 ceph-mon[31916]: pgmap v1013: 129 pgs: 129 active+clean; 274 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.8 MiB/s wr, 496 op/s 2024-03-22T02:35:19.692 INFO:tasks.workunit.client.0.smithi028.stdout: 10 140196 15.52 MB/sec execute 359 sec latency 193.519 ms 2024-03-22T02:35:20.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 140650 15.57 MB/sec execute 360 sec latency 60.978 ms 2024-03-22T02:35:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:20 smithi120 ceph-mon[36803]: pgmap v1014: 129 pgs: 129 active+clean; 281 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 532 op/s 2024-03-22T02:35:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:20 smithi083 ceph-mon[36340]: pgmap v1014: 129 pgs: 129 active+clean; 281 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 532 op/s 2024-03-22T02:35:21.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:20 smithi028 ceph-mon[31916]: pgmap v1014: 129 pgs: 129 active+clean; 281 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 532 op/s 2024-03-22T02:35:21.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 141119 15.59 MB/sec execute 361 sec latency 93.212 ms 2024-03-22T02:35:22.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 141419 15.61 MB/sec execute 362 sec latency 150.143 ms 2024-03-22T02:35:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:22 smithi120 ceph-mon[36803]: pgmap v1015: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 519 op/s 2024-03-22T02:35:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:22 smithi083 ceph-mon[36340]: pgmap v1015: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 519 op/s 2024-03-22T02:35:23.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:22 smithi028 ceph-mon[31916]: pgmap v1015: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 519 op/s 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3515330865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18100ab0 msgr2=0x7fdc18100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3515330865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18100ab0 0x7fdc18100ed0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009900 tx=0x7fdc0802bc30 comp rx=0 tx=0).stop 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3515330865 shutdown_connections 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3515330865 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18102660 0x7fdc18108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3515330865 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18101ca0 0x7fdc18102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3515330865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18100ab0 0x7fdc18100ed0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3515330865 >> 172.21.15.28:0/3515330865 conn(0x7fdc180fc020 msgr2=0x7fdc180fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3515330865 shutdown_connections 2024-03-22T02:35:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.470+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3515330865 wait complete. 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 Processor -- start 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 -- start start 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 0x7fdc1819a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 0x7fdc1819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810d850 con 0x7fdc18102660 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810d6d0 con 0x7fdc18100ab0 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1e53e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810d9d0 con 0x7fdc18101ca0 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.471+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 0x7fdc1819a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.475 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 0x7fdc1819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50036/0 (socket says 172.21.15.28:50036) 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 0x7fdc1819a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42130/0 (socket says 172.21.15.28:42130) 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc1cadb640 1 -- 172.21.15.28:0/3968654078 learned_addr learned my addr 172.21.15.28:0/3968654078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 -- 172.21.15.28:0/3968654078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 msgr2=0x7fdc1819a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 0x7fdc1819a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 -- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 msgr2=0x7fdc181a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 -- 172.21.15.28:0/3968654078 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc181a1c90 con 0x7fdc18101ca0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc177fe640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 0x7fdc1819a5b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00e9f0 tx=0x7fdc0c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc157fa640 1 -- 172.21.15.28:0/3968654078 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c00ccc0 con 0x7fdc18101ca0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc181a1f80 con 0x7fdc18101ca0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc157fa640 1 -- 172.21.15.28:0/3968654078 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c010910 con 0x7fdc18101ca0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc157fa640 1 -- 172.21.15.28:0/3968654078 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c018960 con 0x7fdc18101ca0 2024-03-22T02:35:23.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.472+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc181a2490 con 0x7fdc18101ca0 2024-03-22T02:35:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.474+0000 7fdc157fa640 1 -- 172.21.15.28:0/3968654078 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fdc0c010ac0 con 0x7fdc18101ca0 2024-03-22T02:35:23.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 Processor -- start 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 -- start start 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 0x55b96c4a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b96c407040 con 0x55b96c4a8040 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b96c3af9e0 con 0x55b96c4a2ed0 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc211ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b96c402ff0 con 0x55b96c4a24e0 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 0x55b96c4a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc157fa640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 0x7fdc000a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:47252/0 (socket says 172.21.15.28:47252) 2024-03-22T02:35:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 0x55b96c4a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:42142/0 (socket says 172.21.15.28:42142) 2024-03-22T02:35:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50038/0 (socket says 172.21.15.28:50038) 2024-03-22T02:35:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2171084452 learned_addr learned my addr 172.21.15.28:0/2171084452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:23.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.475+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 0x7fdc000a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2171084452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 msgr2=0x55b96c4a79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 0x55b96c4a79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2171084452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 msgr2=0x55b96c4aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc157fa640 1 -- 172.21.15.28:0/3968654078 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fdc0c0cfbc0 con 0x7fdc18101ca0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2171084452 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b96c11b210 con 0x55b96c4a24e0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000ff20 tx=0x7fdc100138f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f3f6a8e1bdd3167 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 0x7fdc000a5480 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc10013df0 tx=0x7fdc10020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdbfe7fc640 1 -- 172.21.15.28:0/2171084452 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1000f470 con 0x55b96c4a24e0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdbfe7fc640 1 -- 172.21.15.28:0/2171084452 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc10023030 con 0x55b96c4a24e0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.476+0000 7fdbfe7fc640 1 -- 172.21.15.28:0/2171084452 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10010690 con 0x55b96c4a24e0 2024-03-22T02:35:23.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- 172.21.15.28:0/2171084452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 msgr2=0x55b96c4a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fdc1000ff20 tx=0x7fdc100138f0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- 172.21.15.28:0/2171084452 shutdown_connections 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a2ed0 0x55b96c4a79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2171084452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a24e0 0x55b96c4a2900 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- 172.21.15.28:0/2171084452 >> 172.21.15.28:0/2171084452 conn(0x55b96c49da50 msgr2=0x55b96c49de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- 172.21.15.28:0/2171084452 shutdown_connections 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- 172.21.15.28:0/2171084452 wait complete. 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 -- start start 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.477+0000 7fdc211ff740 1 Processor -- start 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.491 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4ac970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a8040 0x55b96c4b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b96c4aaf90 con 0x55b96c4a24e0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b96c3c2db0 con 0x55b96c4a8040 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b96c4ab110 con 0x55b96c4a2ed0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc211ff740 10 client.0 _ll_register_callbacks cb 0x55b96c11aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50044/0 (socket says 172.21.15.28:50044) 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 -- 172.21.15.28:0/2813276443 learned_addr learned my addr 172.21.15.28:0/2813276443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc177fe640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4ac970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a8040 0x55b96c4b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 -- 172.21.15.28:0/2813276443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 msgr2=0x55b96c4ac970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4ac970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.492 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 -- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a8040 msgr2=0x55b96c4b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a8040 0x55b96c4b39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 -- 172.21.15.28:0/2813276443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b96c1e7620 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc177fe640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdc17fff640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009900 tx=0x7fdc08004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266b947428127a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.478+0000 7fdbf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc08005d50 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fdc08005230 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc10023030 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fdc0804f050 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 10 client.20226 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b96c4b6de0 con 0x55b96c4a24e0 2024-03-22T02:35:23.493 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc10023030 con 0x55b96c4a24e0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fdc08003710 con 0x55b96c4a24e0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 10 client.20226 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 10 client.20226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 20 client.20226 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c4b84b0 0x55b96c4ba8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 4 client.20226 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.479+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b96c414580 con 0x55b96c4b84b0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.480+0000 7fdc177fe640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c4b84b0 0x55b96c4ba8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.480+0000 7fdbf77fe640 10 client.20226 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.480+0000 7fdc177fe640 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c4b84b0 0x55b96c4ba8e0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d674b2d1b540932c server_cookie=7c60043795eba803 in_seq=0 out_seq=0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2813276443 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b96c414580 con 0x55b96c4b84b0 2024-03-22T02:35:23.494 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdbf77fe640 10 client.20226 handle_command_reply: tid=0 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 client.20226 shutdown 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 msgr2=0x55b96c4ac430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009900 tx=0x7fdc08004250 comp rx=0 tx=0).stop 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 shutdown_connections 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c4b84b0 0x55b96c4ba8e0 crc :-1 s=READY pgs=426 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a8040 0x55b96c4b39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4ac970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2813276443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a24e0 0x55b96c4ac430 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdbf77fe640 0 client.20226 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.481+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 >> 172.21.15.28:0/2813276443 conn(0x55b96c49da50 msgr2=0x55b96c49cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.482+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 shutdown_connections 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.482+0000 7fdc211ff740 1 -- 172.21.15.28:0/2813276443 wait complete. 2024-03-22T02:35:23.495 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.482+0000 7fdc211ff740 20 client.20226 trim_cache size 0 max 16384 2024-03-22T02:35:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 Processor -- start 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 -- start start 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181af3f0 con 0x55b96c4a8040 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181accc0 con 0x55b96c4a24e0 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc211ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181aab10 con 0x55b96c4a2ed0 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50050/0 (socket says 172.21.15.28:50050) 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc1cadb640 1 -- 172.21.15.28:0/4015571228 learned_addr learned my addr 172.21.15.28:0/4015571228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc177fe640 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.504 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.498+0000 7fdc17fff640 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 -- 172.21.15.28:0/4015571228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 msgr2=0x55b96c4b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 -- 172.21.15.28:0/4015571228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 msgr2=0x55b96c4a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 -- 172.21.15.28:0/4015571228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b96c4a2120 con 0x55b96c4a8040 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009aa0 tx=0x7fdc10009a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fe4e26118021986 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdbfdffb640 1 -- 172.21.15.28:0/4015571228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc10037070 con 0x55b96c4a8040 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdbfdffb640 1 -- 172.21.15.28:0/4015571228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc1000f840 con 0x55b96c4a8040 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.499+0000 7fdbfdffb640 1 -- 172.21.15.28:0/4015571228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc100056f0 con 0x55b96c4a8040 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- 172.21.15.28:0/4015571228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 msgr2=0x55b96c4ab720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fdc10009aa0 tx=0x7fdc10009a30 comp rx=0 tx=0).stop 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- 172.21.15.28:0/4015571228 shutdown_connections 2024-03-22T02:35:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4ab720 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 --2- 172.21.15.28:0/4015571228 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- 172.21.15.28:0/4015571228 >> 172.21.15.28:0/4015571228 conn(0x55b96c49da50 msgr2=0x55b96c4a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- 172.21.15.28:0/4015571228 shutdown_connections 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- 172.21.15.28:0/4015571228 wait complete. 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 -- start start 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.500+0000 7fdc211ff740 1 Processor -- start 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181ab180 con 0x55b96c4a8040 2024-03-22T02:35:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181ac5f0 con 0x55b96c4a24e0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fdc181ab850 con 0x55b96c4a2ed0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 10 client.0 _ll_register_callbacks cb 0x7fdc08009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc17fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50052/0 (socket says 172.21.15.28:50052) 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2616465501 learned_addr learned my addr 172.21.15.28:0/2616465501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc177fe640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2616465501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 msgr2=0x55b96c4b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 msgr2=0x55b96c4b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 -- 172.21.15.28:0/2616465501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b96c4b27d0 con 0x55b96c4a8040 2024-03-22T02:35:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc1cadb640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fdc10035040 tx=0x7fdc10005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e53b6261aa6056 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdbf77fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc1003f070 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fdc10005050 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc1000f840 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.501+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fdc10048050 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 10 client.20232 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b96c4b3ae0 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc1000f840 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fdc10042020 con 0x55b96c4a8040 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 10 client.20232 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:23.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 10 client.20232 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 20 client.20232 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c51ec50 0x55b96c521040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 4 client.20232 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b96c40e8d0 con 0x55b96c51ec50 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc17fff640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c51ec50 0x55b96c521040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdbf77fe640 10 client.20232 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:23.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:23.502+0000 7fdc17fff640 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c51ec50 0x55b96c521040 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2266b297ff140e86 server_cookie=4406dffaae9b3a49 in_seq=0 out_seq=0 2024-03-22T02:35:23.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 141632 15.61 MB/sec execute 363 sec latency 295.107 ms 2024-03-22T02:35:24.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 141713 15.57 MB/sec execute 364 sec latency 375.243 ms 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.783+0000 7fdbf77fe640 1 -- 172.21.15.28:0/2616465501 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b96c40e8d0 con 0x55b96c51ec50 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdbf77fe640 10 client.20232 handle_command_reply: tid=0 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 client.20232 shutdown 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 msgr2=0x55b96c4b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fdc10035040 tx=0x7fdc10005ea0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 shutdown_connections 2024-03-22T02:35:24.786 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b96c51ec50 0x55b96c521040 crc :-1 s=READY pgs=427 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.784+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b96c4a8040 0x55b96c4b21e0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdbf77fe640 0 client.20232 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b96c4a2ed0 0x55b96c4b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 1 --2- 172.21.15.28:0/2616465501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b96c4a24e0 0x55b96c4b0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 >> 172.21.15.28:0/2616465501 conn(0x55b96c49da50 msgr2=0x55b96c4a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 shutdown_connections 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 1 -- 172.21.15.28:0/2616465501 wait complete. 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.785+0000 7fdc211ff740 20 client.20232 trim_cache size 0 max 16384 2024-03-22T02:35:24.787 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:35:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:35:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:35:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:35:24.788 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 msgr2=0x7fdc000a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 0x7fdc000a5480 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fdc10013df0 tx=0x7fdc10020000 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 msgr2=0x7fdc1819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 0x7fdc1819a5b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c00e9f0 tx=0x7fdc0c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 shutdown_connections 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fdc000a2fc0 0x7fdc000a5480 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102660 0x7fdc181a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdc18101ca0 0x7fdc1819a5b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 --2- 172.21.15.28:0/3968654078 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fdc18100ab0 0x7fdc1819a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 >> 172.21.15.28:0/3968654078 conn(0x7fdc180fc020 msgr2=0x7fdc181098b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 shutdown_connections 2024-03-22T02:35:24.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:24.786+0000 7fdc1e53e640 1 -- 172.21.15.28:0/3968654078 wait complete. 2024-03-22T02:35:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:24 smithi120 ceph-mon[36803]: pgmap v1016: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-03-22T02:35:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:24 smithi083 ceph-mon[36340]: pgmap v1016: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-03-22T02:35:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:24 smithi028 ceph-mon[31916]: pgmap v1016: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.3 MiB/s wr, 494 op/s 2024-03-22T02:35:25.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 141930 15.57 MB/sec execute 365 sec latency 154.416 ms 2024-03-22T02:35:26.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 142172 15.55 MB/sec execute 366 sec latency 69.943 ms 2024-03-22T02:35:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:26 smithi120 ceph-mon[36803]: pgmap v1017: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 503 op/s 2024-03-22T02:35:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:26 smithi083 ceph-mon[36340]: pgmap v1017: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 503 op/s 2024-03-22T02:35:27.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:26 smithi028 ceph-mon[31916]: pgmap v1017: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 503 op/s 2024-03-22T02:35:27.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 142404 15.54 MB/sec execute 367 sec latency 100.766 ms 2024-03-22T02:35:28.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:27 smithi028 ceph-mon[31916]: pgmap v1018: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 433 op/s 2024-03-22T02:35:28.248 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:27 smithi120 ceph-mon[36803]: pgmap v1018: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 433 op/s 2024-03-22T02:35:28.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:27 smithi083 ceph-mon[36340]: pgmap v1018: 129 pgs: 129 active+clean; 282 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 433 op/s 2024-03-22T02:35:28.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 142607 15.51 MB/sec execute 368 sec latency 270.965 ms 2024-03-22T02:35:29.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 142907 15.50 MB/sec execute 369 sec latency 83.661 ms 2024-03-22T02:35:30.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 143228 15.50 MB/sec execute 370 sec latency 63.209 ms 2024-03-22T02:35:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:30 smithi120 ceph-mon[36803]: pgmap v1019: 129 pgs: 129 active+clean; 279 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-03-22T02:35:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:30 smithi083 ceph-mon[36340]: pgmap v1019: 129 pgs: 129 active+clean; 279 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-03-22T02:35:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:30 smithi028 ceph-mon[31916]: pgmap v1019: 129 pgs: 129 active+clean; 279 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 476 op/s 2024-03-22T02:35:31.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 143633 15.52 MB/sec execute 371 sec latency 86.818 ms 2024-03-22T02:35:32.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 144115 15.57 MB/sec execute 372 sec latency 61.617 ms 2024-03-22T02:35:32.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:32 smithi120 ceph-mon[36803]: pgmap v1020: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-03-22T02:35:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:32 smithi083 ceph-mon[36340]: pgmap v1020: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-03-22T02:35:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:32 smithi028 ceph-mon[31916]: pgmap v1020: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 522 op/s 2024-03-22T02:35:33.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 144401 15.57 MB/sec execute 373 sec latency 177.267 ms 2024-03-22T02:35:34.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 144782 15.58 MB/sec execute 374 sec latency 112.578 ms 2024-03-22T02:35:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:34 smithi120 ceph-mon[36803]: pgmap v1021: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 512 op/s 2024-03-22T02:35:34.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:34.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:34.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:34 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:34 smithi083 ceph-mon[36340]: pgmap v1021: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 512 op/s 2024-03-22T02:35:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:34 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:34 smithi028 ceph-mon[31916]: pgmap v1021: 129 pgs: 129 active+clean; 283 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 512 op/s 2024-03-22T02:35:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:35:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:34 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:35.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 144990 15.58 MB/sec execute 375 sec latency 131.098 ms 2024-03-22T02:35:35.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:35.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:35.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:35.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:35 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:35 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:35 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 145206 15.58 MB/sec execute 376 sec latency 192.067 ms 2024-03-22T02:35:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:36 smithi120 ceph-mon[36803]: pgmap v1022: 129 pgs: 129 active+clean; 283 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 537 op/s 2024-03-22T02:35:36.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:36.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:36 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:36 smithi083 ceph-mon[36340]: pgmap v1022: 129 pgs: 129 active+clean; 283 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 537 op/s 2024-03-22T02:35:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:36 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:36 smithi028 ceph-mon[31916]: pgmap v1022: 129 pgs: 129 active+clean; 283 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 537 op/s 2024-03-22T02:35:37.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:36 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 145436 15.56 MB/sec execute 377 sec latency 124.164 ms 2024-03-22T02:35:37.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.950+0000 7f49e839a640 1 -- 172.21.15.28:0/2833743354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e0108b90 msgr2=0x7f49e010afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:37.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.950+0000 7f49e839a640 1 --2- 172.21.15.28:0/2833743354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e0108b90 0x7f49e010afc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009900 tx=0x7f49dc02bc50 comp rx=0 tx=0).stop 2024-03-22T02:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.956+0000 7f49e839a640 1 -- 172.21.15.28:0/2833743354 shutdown_connections 2024-03-22T02:35:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.956+0000 7f49e839a640 1 --2- 172.21.15.28:0/2833743354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e010deb0 0x7f49e010e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.956+0000 7f49e839a640 1 --2- 172.21.15.28:0/2833743354 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e010d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.956+0000 7f49e839a640 1 --2- 172.21.15.28:0/2833743354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e0108b90 0x7f49e010afc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:37.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.956+0000 7f49e839a640 1 -- 172.21.15.28:0/2833743354 >> 172.21.15.28:0/2833743354 conn(0x7f49e006ebd0 msgr2=0x7f49e0071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:37.963 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.961+0000 7f49e839a640 1 -- 172.21.15.28:0/2833743354 shutdown_connections 2024-03-22T02:35:37.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.963+0000 7f49e839a640 1 -- 172.21.15.28:0/2833743354 wait complete. 2024-03-22T02:35:37.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.964+0000 7f49e839a640 1 Processor -- start 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 -- start start 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 0x7f49e01a6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e01a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 0x7f49e01ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49e011c640 con 0x7f49e0108b90 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49e011c4c0 con 0x7f49e010b500 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.965+0000 7f49e839a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49e011c7c0 con 0x7f49e010deb0 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.967+0000 7f49e6136640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 0x7f49e01a6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.967+0000 7f49e6136640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 0x7f49e01a6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59442/0 (socket says 172.21.15.28:59442) 2024-03-22T02:35:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.967+0000 7f49e6136640 1 -- 172.21.15.28:0/4235614126 learned_addr learned my addr 172.21.15.28:0/4235614126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.967+0000 7f49e6937640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 0x7f49e01ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.972 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.968+0000 7f49e5935640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e01a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e5935640 1 -- 172.21.15.28:0/4235614126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 msgr2=0x7f49e01ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e5935640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 0x7f49e01ae180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e5935640 1 -- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 msgr2=0x7f49e01a6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e5935640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 0x7f49e01a6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e5935640 1 -- 172.21.15.28:0/4235614126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e01103f0 con 0x7f49e010b500 2024-03-22T02:35:37.976 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.971+0000 7f49e6937640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 0x7f49e01ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:35:37.983 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.979+0000 7f49e5935640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e01a7110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c970 tx=0x7f49d000ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.983+0000 7f49d77fe640 1 -- 172.21.15.28:0/4235614126 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0007bb0 con 0x7f49e010b500 2024-03-22T02:35:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.983+0000 7f49d77fe640 1 -- 172.21.15.28:0/4235614126 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d0007d50 con 0x7f49e010b500 2024-03-22T02:35:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.983+0000 7f49d77fe640 1 -- 172.21.15.28:0/4235614126 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d000f450 con 0x7f49e010b500 2024-03-22T02:35:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.984+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49e01106e0 con 0x7f49e010b500 2024-03-22T02:35:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.984+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49e0110be0 con 0x7f49e010b500 2024-03-22T02:35:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.990+0000 7f49eaf84740 1 Processor -- start 2024-03-22T02:35:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.990+0000 7f49eaf84740 1 -- start start 2024-03-22T02:35:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c27477900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c2747c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c273dc040 con 0x555c27477ed0 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c273849e0 con 0x555c274774e0 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49eaf84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555c273d7ff0 con 0x555c2747d040 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49e5935640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49e5935640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:43728/0 (socket says 172.21.15.28:43728) 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49e5935640 1 -- 172.21.15.28:0/3357248974 learned_addr learned my addr 172.21.15.28:0/3357248974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.991+0000 7f49e6937640 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c27477900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e6136640 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c2747c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:37.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:35:37.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e5935640 1 -- 172.21.15.28:0/3357248974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 msgr2=0x555c27477900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e5935640 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c27477900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e5935640 1 -- 172.21.15.28:0/3357248974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 msgr2=0x555c2747c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e5935640 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c2747c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.994+0000 7f49e5935640 1 -- 172.21.15.28:0/3357248974 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c270f0210 con 0x555c2747d040 2024-03-22T02:35:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.996+0000 7f49e5935640 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c940 tx=0x7f49d00048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43c9283387ba8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.996+0000 7f49d77fe640 1 -- 172.21.15.28:0/4235614126 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f49d000f5f0 con 0x7f49e010b500 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.997+0000 7f49d77fe640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 0x7f49cc0a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.997+0000 7f49d77fe640 1 -- 172.21.15.28:0/4235614126 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f49d002a070 con 0x7f49e010b500 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.998+0000 7f49e6937640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 0x7f49cc0a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.998+0000 7f49d4bfe640 1 -- 172.21.15.28:0/3357248974 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d00990b0 con 0x555c2747d040 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.998+0000 7f49d4bfe640 1 -- 172.21.15.28:0/3357248974 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d009d510 con 0x555c2747d040 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:37.998+0000 7f49d4bfe640 1 -- 172.21.15.28:0/3357248974 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d009c770 con 0x555c2747d040 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.000+0000 7f49eaf84740 1 -- 172.21.15.28:0/3357248974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 msgr2=0x555c2747f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.000+0000 7f49eaf84740 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c940 tx=0x7f49d00048f0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49e6937640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 0x7f49cc0a56c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x555c27478730 tx=0x7f49d8009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49eaf84740 1 -- 172.21.15.28:0/3357248974 shutdown_connections 2024-03-22T02:35:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49eaf84740 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c2747f4d0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49eaf84740 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c2747c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49eaf84740 1 --2- 172.21.15.28:0/3357248974 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c27477900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.005+0000 7f49eaf84740 1 -- 172.21.15.28:0/3357248974 >> 172.21.15.28:0/3357248974 conn(0x555c27472a50 msgr2=0x555c27472e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.011+0000 7f49eaf84740 1 -- 172.21.15.28:0/3357248974 shutdown_connections 2024-03-22T02:35:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.012+0000 7f49eaf84740 1 -- 172.21.15.28:0/3357248974 wait complete. 2024-03-22T02:35:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.012+0000 7f49eaf84740 1 -- start start 2024-03-22T02:35:38.013 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.012+0000 7f49eaf84740 1 Processor -- start 2024-03-22T02:35:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:35:38.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c274814e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c274889f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555c2747fff0 con 0x555c27477ed0 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x555c27397db0 con 0x555c274774e0 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x555c27480170 con 0x555c2747d040 2024-03-22T02:35:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.014+0000 7f49eaf84740 10 client.0 _ll_register_callbacks cb 0x555c270efa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.015+0000 7f49e5935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.015+0000 7f49e5935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59466/0 (socket says 172.21.15.28:59466) 2024-03-22T02:35:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.015+0000 7f49e5935640 1 -- 172.21.15.28:0/808648257 learned_addr learned my addr 172.21.15.28:0/808648257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.015+0000 7f49e6937640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c274889f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.016+0000 7f49e5935640 1 -- 172.21.15.28:0/808648257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 msgr2=0x555c274889f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.016+0000 7f49e5935640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c274889f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.016+0000 7f49e5935640 1 -- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 msgr2=0x555c274814e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.016+0000 7f49e5935640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c274814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.017+0000 7f49e5935640 1 -- 172.21.15.28:0/808648257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c271bc620 con 0x555c27477ed0 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.017+0000 7f49e5935640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f49d00d0b90 tx=0x7f49d00a5790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f618ebd5e3e67c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.017+0000 7f49c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.017+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d0026070 con 0x555c27477ed0 2024-03-22T02:35:38.020 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.017+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f49d0099480 con 0x555c27477ed0 2024-03-22T02:35:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.018+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d009d510 con 0x555c27477ed0 2024-03-22T02:35:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.018+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f49d009c950 con 0x555c27477ed0 2024-03-22T02:35:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.018+0000 7f49eaf84740 10 client.20244 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.018+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x555c2748bd60 con 0x555c27477ed0 2024-03-22T02:35:38.021 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.018+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d009d510 con 0x555c27477ed0 2024-03-22T02:35:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f49d00a5300 con 0x555c27477ed0 2024-03-22T02:35:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 10 client.20244 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 10 client.20244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 20 client.20244 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2748d690 0x555c2748fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 4 client.20244 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:35:38.024 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.019+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c273e9580 con 0x555c2748d690 2024-03-22T02:35:38.026 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.024+0000 7f49e6136640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2748d690 0x555c2748fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.048+0000 7f49e6136640 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2748d690 0x555c2748fb30 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a740f08e9404d2a server_cookie=c937c8b0b5d5b368 in_seq=0 out_seq=0 2024-03-22T02:35:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.048+0000 7f49c9ffb640 10 client.20244 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.057+0000 7f49c9ffb640 1 -- 172.21.15.28:0/808648257 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555c273e9580 con 0x555c2748d690 2024-03-22T02:35:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.057+0000 7f49c9ffb640 10 client.20244 handle_command_reply: tid=0 2024-03-22T02:35:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.057+0000 7f49eaf84740 1 client.20244 shutdown 2024-03-22T02:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.060+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 msgr2=0x555c27481a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.060+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f49d00d0b90 tx=0x7f49d00a5790 comp rx=0 tx=0).stop 2024-03-22T02:35:38.062 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 shutdown_connections 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2748d690 0x555c2748fb30 crc :-1 s=READY pgs=428 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c2747d040 0x555c274889f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27481a20 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 --2- 172.21.15.28:0/808648257 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c274774e0 0x555c274814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 >> 172.21.15.28:0/808648257 conn(0x555c27472a50 msgr2=0x555c27471c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:38.063 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.062+0000 7f49c9ffb640 0 client.20244 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:38.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.068+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 shutdown_connections 2024-03-22T02:35:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.070+0000 7f49eaf84740 1 -- 172.21.15.28:0/808648257 wait complete. 2024-03-22T02:35:38.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:35:38.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:35:38.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:35:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.072+0000 7f49eaf84740 20 client.20244 trim_cache size 0 max 16384 2024-03-22T02:35:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 Processor -- start 2024-03-22T02:35:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 -- start start 2024-03-22T02:35:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c27472a50 0x555c2748f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 0x555c274889b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 0x555c27480570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4012580 con 0x555c274774e0 2024-03-22T02:35:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.107+0000 7f49eaf84740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49c400fe50 con 0x555c27477ed0 2024-03-22T02:35:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.108+0000 7f49eaf84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49c400dd50 con 0x555c27472a50 2024-03-22T02:35:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.108+0000 7f49e5935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 0x555c274889b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.108+0000 7f49e5935640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 0x555c274889b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59480/0 (socket says 172.21.15.28:59480) 2024-03-22T02:35:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.108+0000 7f49e5935640 1 -- 172.21.15.28:0/513903799 learned_addr learned my addr 172.21.15.28:0/513903799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.110+0000 7f49e6937640 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 0x555c27480570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.111+0000 7f49e6136640 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c27472a50 0x555c2748f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 -- 172.21.15.28:0/513903799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c27472a50 msgr2=0x555c2748f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c27472a50 0x555c2748f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 -- 172.21.15.28:0/513903799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 msgr2=0x555c274889b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 0x555c274889b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 -- 172.21.15.28:0/513903799 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c274782b0 con 0x555c27477ed0 2024-03-22T02:35:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.115+0000 7f49e6937640 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 0x555c27480570 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009de0 tx=0x7f49d8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bcffea6397096e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.119+0000 7f49cbfff640 1 -- 172.21.15.28:0/513903799 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d8004d20 con 0x555c27477ed0 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.119+0000 7f49cbfff640 1 -- 172.21.15.28:0/513903799 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8023070 con 0x555c27477ed0 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.119+0000 7f49cbfff640 1 -- 172.21.15.28:0/513903799 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d801f870 con 0x555c27477ed0 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 -- 172.21.15.28:0/513903799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 msgr2=0x555c27480570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 0x555c27480570 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009de0 tx=0x7f49d8015000 comp rx=0 tx=0).stop 2024-03-22T02:35:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 -- 172.21.15.28:0/513903799 shutdown_connections 2024-03-22T02:35:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27477ed0 0x555c27480570 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c274774e0 0x555c274889b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 --2- 172.21.15.28:0/513903799 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c27472a50 0x555c2748f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.120+0000 7f49eaf84740 1 -- 172.21.15.28:0/513903799 >> 172.21.15.28:0/513903799 conn(0x555c2747cb80 msgr2=0x555c27475cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.124+0000 7f49eaf84740 1 -- 172.21.15.28:0/513903799 shutdown_connections 2024-03-22T02:35:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 -- 172.21.15.28:0/513903799 wait complete. 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 -- start start 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 Processor -- start 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27472a50 0x555c27488970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c274774e0 0x555c274854f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f49c400e310 con 0x555c27477ed0 2024-03-22T02:35:38.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f49c400f780 con 0x555c27472a50 2024-03-22T02:35:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f49c400e9e0 con 0x555c274774e0 2024-03-22T02:35:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.128+0000 7f49eaf84740 10 client.0 _ll_register_callbacks cb 0x7f49d00a0f60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.129+0000 7f49e6937640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.129+0000 7f49e6937640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59496/0 (socket says 172.21.15.28:59496) 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.129+0000 7f49e6937640 1 -- 172.21.15.28:0/2286196869 learned_addr learned my addr 172.21.15.28:0/2286196869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 -- 172.21.15.28:0/2286196869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c274774e0 msgr2=0x555c274854f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c274774e0 0x555c274854f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 -- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27472a50 msgr2=0x555c27488970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27472a50 0x555c27488970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 -- 172.21.15.28:0/2286196869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c274f1aa0 con 0x555c27477ed0 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49e6937640 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f49d801f7e0 tx=0x7f49d8009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86cff15760d7d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49d801fc90 con 0x555c27477ed0 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.130+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f49d8033430 con 0x555c27477ed0 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.131+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49d8023070 con 0x555c27477ed0 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.131+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f49d801fe30 con 0x555c27477ed0 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.131+0000 7f49eaf84740 10 client.20250 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.131+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x555c274f2db0 con 0x555c27477ed0 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49d8023070 con 0x555c27477ed0 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f49d8027870 con 0x555c27477ed0 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 10 client.20250 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 10 client.20250 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 20 client.20250 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2747d040 0x555c274f5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 4 client.20250 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:35:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.132+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c273e38d0 con 0x555c2747d040 2024-03-22T02:35:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.141+0000 7f49e6136640 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2747d040 0x555c274f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:38.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.148+0000 7f49e6136640 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2747d040 0x555c274f5630 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16183424c5ae5a03 server_cookie=2cec27da54a9060a in_seq=0 out_seq=0 2024-03-22T02:35:38.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:38.149+0000 7f49c9ffb640 10 client.20250 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:38.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 145515 15.54 MB/sec execute 378 sec latency 310.067 ms 2024-03-22T02:35:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:38 smithi120 ceph-mon[36803]: pgmap v1023: 129 pgs: 129 active+clean; 285 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:35:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:38 smithi083 ceph-mon[36340]: pgmap v1023: 129 pgs: 129 active+clean; 285 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:35:39.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:38 smithi028 ceph-mon[31916]: pgmap v1023: 129 pgs: 129 active+clean; 285 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.1 MiB/s wr, 436 op/s 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49c9ffb640 1 -- 172.21.15.28:0/2286196869 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c273e38d0 con 0x555c2747d040 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49c9ffb640 10 client.20250 handle_command_reply: tid=0 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 client.20250 shutdown 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 msgr2=0x555c27485b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f49d801f7e0 tx=0x7f49d8009e80 comp rx=0 tx=0).stop 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 shutdown_connections 2024-03-22T02:35:39.487 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x555c2747d040 0x555c274f5630 crc :-1 s=READY pgs=429 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555c27477ed0 0x555c27485b20 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555c274774e0 0x555c274854f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 --2- 172.21.15.28:0/2286196869 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x555c27472a50 0x555c27488970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 >> 172.21.15.28:0/2286196869 conn(0x555c2747cb80 msgr2=0x555c27479200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49c9ffb640 0 client.20250 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 shutdown_connections 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 1 -- 172.21.15.28:0/2286196869 wait complete. 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.486+0000 7f49eaf84740 20 client.20250 trim_cache size 0 max 16384 2024-03-22T02:35:39.488 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:35:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:35:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:35:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:35:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 msgr2=0x7f49cc0a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:39.489 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 0x7f49cc0a56c0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x555c27478730 tx=0x7f49d8009700 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 msgr2=0x7f49e01a7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e01a7110 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f49d000c970 tx=0x7f49d000ce30 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 shutdown_connections 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f49cc0a3200 0x7f49cc0a56c0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f49e010deb0 0x7f49e01ae180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f49e010b500 0x7f49e01a7110 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 --2- 172.21.15.28:0/4235614126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0108b90 0x7f49e01a6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 >> 172.21.15.28:0/4235614126 conn(0x7f49e006ebd0 msgr2=0x7f49e010d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 shutdown_connections 2024-03-22T02:35:39.490 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:39.487+0000 7f49e839a640 1 -- 172.21.15.28:0/4235614126 wait complete. 2024-03-22T02:35:39.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 145598 15.51 MB/sec execute 379 sec latency 771.307 ms 2024-03-22T02:35:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:39 smithi120 ceph-mon[36803]: pgmap v1024: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 372 op/s 2024-03-22T02:35:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:39 smithi083 ceph-mon[36340]: pgmap v1024: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 372 op/s 2024-03-22T02:35:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:39 smithi028 ceph-mon[31916]: pgmap v1024: 129 pgs: 129 active+clean; 286 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.3 MiB/s wr, 372 op/s 2024-03-22T02:35:40.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 145814 15.49 MB/sec execute 380 sec latency 83.114 ms 2024-03-22T02:35:41.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 146107 15.49 MB/sec execute 381 sec latency 79.365 ms 2024-03-22T02:35:42.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:42 smithi083 ceph-mon[36340]: pgmap v1025: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 453 op/s 2024-03-22T02:35:42.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:42 smithi028 ceph-mon[31916]: pgmap v1025: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 453 op/s 2024-03-22T02:35:42.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:42 smithi120 ceph-mon[36803]: pgmap v1025: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.7 MiB/s wr, 453 op/s 2024-03-22T02:35:42.693 INFO:tasks.workunit.client.0.smithi028.stdout: 10 146483 15.48 MB/sec execute 382 sec latency 73.089 ms 2024-03-22T02:35:43.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 146666 15.45 MB/sec execute 383 sec latency 147.915 ms 2024-03-22T02:35:44.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:44 smithi083 ceph-mon[36340]: pgmap v1026: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 415 op/s 2024-03-22T02:35:44.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:44 smithi028 ceph-mon[31916]: pgmap v1026: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 415 op/s 2024-03-22T02:35:44.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:44 smithi120 ceph-mon[36803]: pgmap v1026: 129 pgs: 129 active+clean; 279 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.6 MiB/s wr, 415 op/s 2024-03-22T02:35:44.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 147008 15.47 MB/sec execute 384 sec latency 97.300 ms 2024-03-22T02:35:45.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 147423 15.48 MB/sec execute 385 sec latency 98.892 ms 2024-03-22T02:35:46.265 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:46 smithi083 ceph-mon[36340]: pgmap v1027: 129 pgs: 129 active+clean; 277 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.4 MiB/s wr, 546 op/s 2024-03-22T02:35:46.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:46 smithi028 ceph-mon[31916]: pgmap v1027: 129 pgs: 129 active+clean; 277 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.4 MiB/s wr, 546 op/s 2024-03-22T02:35:46.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:46 smithi120 ceph-mon[36803]: pgmap v1027: 129 pgs: 129 active+clean; 277 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.4 MiB/s wr, 546 op/s 2024-03-22T02:35:46.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 147827 15.52 MB/sec execute 386 sec latency 84.879 ms 2024-03-22T02:35:47.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 148319 15.55 MB/sec execute 387 sec latency 100.529 ms 2024-03-22T02:35:48.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:48 smithi028 ceph-mon[31916]: pgmap v1028: 129 pgs: 129 active+clean; 280 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 509 op/s 2024-03-22T02:35:48.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:48 smithi120 ceph-mon[36803]: pgmap v1028: 129 pgs: 129 active+clean; 280 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 509 op/s 2024-03-22T02:35:48.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:48 smithi083 ceph-mon[36340]: pgmap v1028: 129 pgs: 129 active+clean; 280 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.6 MiB/s wr, 509 op/s 2024-03-22T02:35:48.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 148532 15.55 MB/sec execute 388 sec latency 208.380 ms 2024-03-22T02:35:49.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 148740 15.55 MB/sec execute 389 sec latency 97.962 ms 2024-03-22T02:35:50.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:50 smithi028 ceph-mon[31916]: pgmap v1029: 129 pgs: 129 active+clean; 279 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-22T02:35:50.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:50 smithi120 ceph-mon[36803]: pgmap v1029: 129 pgs: 129 active+clean; 279 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-22T02:35:50.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:50 smithi083 ceph-mon[36340]: pgmap v1029: 129 pgs: 129 active+clean; 279 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 4.5 MiB/s wr, 509 op/s 2024-03-22T02:35:50.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 148994 15.55 MB/sec execute 390 sec latency 111.312 ms 2024-03-22T02:35:51.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 149240 15.53 MB/sec execute 391 sec latency 71.246 ms 2024-03-22T02:35:52.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:52 smithi028 ceph-mon[31916]: pgmap v1030: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 524 op/s 2024-03-22T02:35:52.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:52 smithi120 ceph-mon[36803]: pgmap v1030: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 524 op/s 2024-03-22T02:35:52.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:52 smithi083 ceph-mon[36340]: pgmap v1030: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 524 op/s 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 -- 172.21.15.28:0/2415731196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58102630 msgr2=0x7f7f58104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/2415731196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58102630 0x7f7f58104a20 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009ca0 tx=0x7f7f5002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 -- 172.21.15.28:0/2415731196 shutdown_connections 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/2415731196 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58107890 0x7f7f58109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/2415731196 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58104f60 0x7f7f58107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/2415731196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58102630 0x7f7f58104a20 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 -- 172.21.15.28:0/2415731196 >> 172.21.15.28:0/2415731196 conn(0x7f7f580fc020 msgr2=0x7f7f580fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 -- 172.21.15.28:0/2415731196 shutdown_connections 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 -- 172.21.15.28:0/2415731196 wait complete. 2024-03-22T02:35:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.585+0000 7f7f5d8af640 1 Processor -- start 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 -- start start 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 0x7f7f58197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58107890 0x7f7f5819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5810d850 con 0x7f7f58104f60 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5810d6d0 con 0x7f7f58107890 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f5d8af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7f5810d9d0 con 0x7f7f58102630 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55068/0 (socket says 172.21.15.28:55068) 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 -- 172.21.15.28:0/4067215564 learned_addr learned my addr 172.21.15.28:0/4067215564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 0x7f7f58197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f577fe640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58107890 0x7f7f5819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 -- 172.21.15.28:0/4067215564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 msgr2=0x7f7f58197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 0x7f7f58197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 -- 172.21.15.28:0/4067215564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58107890 msgr2=0x7f7f5819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58107890 0x7f7f5819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 -- 172.21.15.28:0/4067215564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f5819fa30 con 0x7f7f58104f60 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 0x7f7f58197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.586+0000 7f7f567fc640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005fd0 tx=0x7f7f4c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.587+0000 7f7f5c8ad640 1 -- 172.21.15.28:0/4067215564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c017070 con 0x7f7f58104f60 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.587+0000 7f7f5c8ad640 1 -- 172.21.15.28:0/4067215564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c009c30 con 0x7f7f58104f60 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.587+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f5819fd20 con 0x7f7f58104f60 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.587+0000 7f7f5c8ad640 1 -- 172.21.15.28:0/4067215564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c010460 con 0x7f7f58104f60 2024-03-22T02:35:52.591 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.587+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f581a0210 con 0x7f7f58104f60 2024-03-22T02:35:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.589+0000 7f7f5c8ad640 1 -- 172.21.15.28:0/4067215564 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f7f4c003e70 con 0x7f7f58104f60 2024-03-22T02:35:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 Processor -- start 2024-03-22T02:35:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 -- start start 2024-03-22T02:35:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f5c8ad640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 0x7f7f400a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6affc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6afff4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c6af5c040 con 0x564c6aff7ed0 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564c6af049e0 con 0x564c6affd040 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f60574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c6af57ff0 con 0x564c6aff74e0 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6affc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 0x7f7f400a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46322/0 (socket says 172.21.15.28:46322) 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f577fe640 1 -- 172.21.15.28:0/696169870 learned_addr learned my addr 172.21.15.28:0/696169870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f577fe640 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6afff4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.590+0000 7f7f5c8ad640 1 -- 172.21.15.28:0/4067215564 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f7f4c0cf460 con 0x7f7f58104f60 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 -- 172.21.15.28:0/696169870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 msgr2=0x564c6aff7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 -- 172.21.15.28:0/696169870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 msgr2=0x564c6afff4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6afff4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 -- 172.21.15.28:0/696169870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c6ac70210 con 0x564c6aff7ed0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f577fe640 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 0x7f7f400a5480 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006010 tx=0x7f7f50032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f567fc640 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6affc9d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0ccec0 tx=0x7f7f4c06e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b02feea7751654c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f3dffb640 1 -- 172.21.15.28:0/696169870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0a8d90 con 0x564c6aff7ed0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f3dffb640 1 -- 172.21.15.28:0/696169870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c009de0 con 0x564c6aff7ed0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.591+0000 7f7f3dffb640 1 -- 172.21.15.28:0/696169870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0860f0 con 0x564c6aff7ed0 2024-03-22T02:35:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- 172.21.15.28:0/696169870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 msgr2=0x564c6affc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6affc9d0 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0ccec0 tx=0x7f7f4c06e4b0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- 172.21.15.28:0/696169870 shutdown_connections 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6afff4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6affc9d0 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- 172.21.15.28:0/696169870 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6aff7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- 172.21.15.28:0/696169870 >> 172.21.15.28:0/696169870 conn(0x564c6aff2a50 msgr2=0x564c6aff2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- 172.21.15.28:0/696169870 shutdown_connections 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- 172.21.15.28:0/696169870 wait complete. 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- start start 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 Processor -- start 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6b001510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff7ed0 0x564c6b001a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564c6affff90 con 0x564c6affd040 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x564c6af17db0 con 0x564c6aff74e0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564c6b000110 con 0x564c6aff7ed0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.592+0000 7f7f60574740 10 client.0 _ll_register_callbacks cb 0x564c6ac6fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6b001510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55086/0 (socket says 172.21.15.28:55086) 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff7ed0 0x564c6b001a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 -- 172.21.15.28:0/789782586 learned_addr learned my addr 172.21.15.28:0/789782586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 -- 172.21.15.28:0/789782586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff7ed0 msgr2=0x564c6b001a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff7ed0 0x564c6b001a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 -- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 msgr2=0x564c6b001510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6b001510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 -- 172.21.15.28:0/789782586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c6ad3c620 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f577fe640 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004530 tx=0x7f7f480039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd1752723808a23 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48011ae0 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f7f48011c80 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f4c009de0 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7f48029050 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f60574740 10 client.20262 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.593+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x564c6b00bec0 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c009de0 con 0x564c6affd040 2024-03-22T02:35:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7f4801d070 con 0x564c6affd040 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 10 client.20262 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 10 client.20262 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 20 client.20262 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b00d600 0x564c6b00fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 4 client.20262 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c6af69580 con 0x564c6b00d600 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b00d600 0x564c6b00fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b00d600 0x564c6b00fa70 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82eeeda51cae728b server_cookie=aecf5739a194639f in_seq=0 out_seq=0 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.594+0000 7f7f36ffd640 10 client.20262 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f36ffd640 1 -- 172.21.15.28:0/789782586 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564c6af69580 con 0x564c6b00d600 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f36ffd640 10 client.20262 handle_command_reply: tid=0 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 client.20262 shutdown 2024-03-22T02:35:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 msgr2=0x564c6b008a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f7f48004530 tx=0x7f7f480039c0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 shutdown_connections 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b00d600 0x564c6b00fa70 crc :-1 s=READY pgs=430 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6affd040 0x564c6b008a90 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff7ed0 0x564c6b001a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 --2- 172.21.15.28:0/789782586 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6b001510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f36ffd640 0 client.20262 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 >> 172.21.15.28:0/789782586 conn(0x564c6aff2a50 msgr2=0x564c6aff1c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 shutdown_connections 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 1 -- 172.21.15.28:0/789782586 wait complete. 2024-03-22T02:35:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.596+0000 7f7f60574740 20 client.20262 trim_cache size 0 max 16384 2024-03-22T02:35:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.611+0000 7f7f60574740 1 Processor -- start 2024-03-22T02:35:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.611+0000 7f7f60574740 1 -- start start 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.611+0000 7f7f60574740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6affc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.611+0000 7f7f60574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.611+0000 7f7f60574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 0x564c6aff13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f60574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581ad230 con 0x564c6aff7ed0 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f60574740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581aab00 con 0x564c6aff74e0 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f60574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581a8950 con 0x564c6affd040 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6affc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55100/0 (socket says 172.21.15.28:55100) 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6affc890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:49650/0 (socket says 172.21.15.28:49650) 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f567fc640 1 -- 172.21.15.28:0/2344495039 learned_addr learned my addr 172.21.15.28:0/2344495039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 0x564c6aff13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 -- 172.21.15.28:0/2344495039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 msgr2=0x564c6affc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6affc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 -- 172.21.15.28:0/2344495039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 msgr2=0x564c6b008a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 -- 172.21.15.28:0/2344495039 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c6aff3880 con 0x564c6affd040 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f567fc640 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f577fe640 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 0x564c6aff13c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7f48000c00 tx=0x7f7f48005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1defa69ec989e752 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f3d7fa640 1 -- 172.21.15.28:0/2344495039 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f48032070 con 0x564c6affd040 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f3d7fa640 1 -- 172.21.15.28:0/2344495039 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f48004350 con 0x564c6affd040 2024-03-22T02:35:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.612+0000 7f7f3d7fa640 1 -- 172.21.15.28:0/2344495039 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4800fb00 con 0x564c6affd040 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 -- 172.21.15.28:0/2344495039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 msgr2=0x564c6aff13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 0x564c6aff13c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7f48000c00 tx=0x7f7f48005740 comp rx=0 tx=0).stop 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 -- 172.21.15.28:0/2344495039 shutdown_connections 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6affd040 0x564c6aff13c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b008a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 --2- 172.21.15.28:0/2344495039 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6aff74e0 0x564c6affc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 -- 172.21.15.28:0/2344495039 >> 172.21.15.28:0/2344495039 conn(0x564c6aff2a50 msgr2=0x564c6aff6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 -- 172.21.15.28:0/2344495039 shutdown_connections 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.613+0000 7f7f60574740 1 -- 172.21.15.28:0/2344495039 wait complete. 2024-03-22T02:35:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 -- start start 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 Processor -- start 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b0066a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6b0076b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581a8fc0 con 0x564c6aff7ed0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581aa430 con 0x564c6affd040 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f7f581a9690 con 0x564c6aff74e0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f60574740 10 client.0 _ll_register_callbacks cb 0x7f7f48009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.614+0000 7f7f567fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b0066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f577fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6b0076b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46354/0 (socket says 172.21.15.28:46354) 2024-03-22T02:35:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1610134893 learned_addr learned my addr 172.21.15.28:0/1610134893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 msgr2=0x564c6b0076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6b0076b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1610134893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 msgr2=0x564c6b0066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b0066a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 -- 172.21.15.28:0/1610134893 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c6b007ca0 con 0x564c6aff74e0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f567fc640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b0066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f577fe640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6b0076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f56ffd640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009ca0 tx=0x7f7f5002c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0cec31a29c5981 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f50039040 con 0x564c6aff74e0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f48004350 con 0x564c6aff74e0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.615+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f7f500349f0 con 0x564c6aff74e0 2024-03-22T02:35:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7f50061050 con 0x564c6aff74e0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 10 client.30052 fetch_fsmap learned FSMap version 46 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x564c6b0739b0 con 0x564c6aff74e0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f48004350 con 0x564c6aff74e0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7f5003f540 con 0x564c6aff74e0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 10 client.30052 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 10 client.30052 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 20 client.30052 populate_metadata read hostname 'smithi028' 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b0743e0 0x564c6b076800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 4 client.30052 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564c6af638d0 con 0x564c6b0743e0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f567fc640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b0743e0 0x564c6b076800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.616+0000 7f7f36ffd640 10 client.30052 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:52.617+0000 7f7f567fc640 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b0743e0 0x564c6b076800 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c68e68a444c96dd server_cookie=ed6b1f06f1db52b1 in_seq=0 out_seq=0 2024-03-22T02:35:52.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 149505 15.52 MB/sec execute 392 sec latency 107.275 ms 2024-03-22T02:35:53.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 149548 15.49 MB/sec execute 393 sec latency 665.667 ms 2024-03-22T02:35:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f36ffd640 1 -- 172.21.15.28:0/1610134893 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564c6af638d0 con 0x564c6b0743e0 2024-03-22T02:35:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f36ffd640 10 client.30052 handle_command_reply: tid=0 2024-03-22T02:35:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 client.30052 shutdown 2024-03-22T02:35:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 msgr2=0x564c6b006160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:54.223 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7f50009ca0 tx=0x7f7f5002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 shutdown_connections 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564c6b0743e0 0x564c6b076800 crc :-1 s=READY pgs=431 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564c6affd040 0x564c6b0076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564c6aff7ed0 0x564c6b0066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f36ffd640 0 client.30052 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 --2- 172.21.15.28:0/1610134893 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564c6aff74e0 0x564c6b006160 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 >> 172.21.15.28:0/1610134893 conn(0x564c6aff2a50 msgr2=0x564c6aff8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 shutdown_connections 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 1 -- 172.21.15.28:0/1610134893 wait complete. 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.221+0000 7f7f60574740 20 client.30052 trim_cache size 0 max 16384 2024-03-22T02:35:54.224 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:35:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:35:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:35:54.225 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 msgr2=0x7f7f400a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 0x7f7f400a5480 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7f50006010 tx=0x7f7f50032590 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 msgr2=0x7f7f58198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c005fd0 tx=0x7f7f4c00dde0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 shutdown_connections 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f7f400a2fc0 0x7f7f400a5480 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f7f58107890 0x7f7f5819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f58104f60 0x7f7f58198350 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 --2- 172.21.15.28:0/4067215564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7f58102630 0x7f7f58197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:35:54.226 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 >> 172.21.15.28:0/4067215564 conn(0x7f7f580fc020 msgr2=0x7f7f58102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:35:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 shutdown_connections 2024-03-22T02:35:54.227 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:35:54.222+0000 7f7f5d8af640 1 -- 172.21.15.28:0/4067215564 wait complete. 2024-03-22T02:35:54.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:54 smithi028 ceph-mon[31916]: pgmap v1031: 129 pgs: 129 active+clean; 281 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 381 op/s 2024-03-22T02:35:54.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:54 smithi120 ceph-mon[36803]: pgmap v1031: 129 pgs: 129 active+clean; 281 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 381 op/s 2024-03-22T02:35:54.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:54 smithi083 ceph-mon[36340]: pgmap v1031: 129 pgs: 129 active+clean; 281 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 381 op/s 2024-03-22T02:35:54.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 149800 15.47 MB/sec execute 394 sec latency 399.123 ms 2024-03-22T02:35:55.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 150089 15.45 MB/sec execute 395 sec latency 92.491 ms 2024-03-22T02:35:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:56 smithi028 ceph-mon[31916]: pgmap v1032: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 519 op/s 2024-03-22T02:35:56.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:56 smithi120 ceph-mon[36803]: pgmap v1032: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 519 op/s 2024-03-22T02:35:56.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:56 smithi083 ceph-mon[36340]: pgmap v1032: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 5.1 MiB/s wr, 519 op/s 2024-03-22T02:35:56.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 150330 15.45 MB/sec execute 396 sec latency 156.867 ms 2024-03-22T02:35:57.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 150765 15.47 MB/sec execute 397 sec latency 109.133 ms 2024-03-22T02:35:58.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:35:58 smithi028 ceph-mon[31916]: pgmap v1033: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 500 op/s 2024-03-22T02:35:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:35:58 smithi120 ceph-mon[36803]: pgmap v1033: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 500 op/s 2024-03-22T02:35:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:35:58 smithi083 ceph-mon[36340]: pgmap v1033: 129 pgs: 129 active+clean; 288 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 3.8 MiB/s wr, 500 op/s 2024-03-22T02:35:58.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 151167 15.50 MB/sec execute 398 sec latency 139.753 ms 2024-03-22T02:35:59.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 151460 15.51 MB/sec execute 399 sec latency 115.581 ms 2024-03-22T02:36:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:00 smithi028 ceph-mon[31916]: pgmap v1034: 129 pgs: 129 active+clean; 283 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 508 op/s 2024-03-22T02:36:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:00 smithi120 ceph-mon[36803]: pgmap v1034: 129 pgs: 129 active+clean; 283 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 508 op/s 2024-03-22T02:36:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:00 smithi083 ceph-mon[36340]: pgmap v1034: 129 pgs: 129 active+clean; 283 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.9 MiB/s wr, 508 op/s 2024-03-22T02:36:00.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 151901 15.53 MB/sec execute 400 sec latency 114.224 ms 2024-03-22T02:36:01.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 152194 15.55 MB/sec execute 401 sec latency 87.036 ms 2024-03-22T02:36:02.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:02 smithi028 ceph-mon[31916]: pgmap v1035: 129 pgs: 129 active+clean; 278 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 563 op/s 2024-03-22T02:36:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:02 smithi120 ceph-mon[36803]: pgmap v1035: 129 pgs: 129 active+clean; 278 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 563 op/s 2024-03-22T02:36:02.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:02 smithi083 ceph-mon[36340]: pgmap v1035: 129 pgs: 129 active+clean; 278 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.4 MiB/s wr, 563 op/s 2024-03-22T02:36:02.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 152434 15.53 MB/sec execute 402 sec latency 93.920 ms 2024-03-22T02:36:03.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 152659 15.53 MB/sec execute 403 sec latency 238.192 ms 2024-03-22T02:36:04.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:04 smithi028 ceph-mon[31916]: pgmap v1036: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 549 op/s 2024-03-22T02:36:04.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:04 smithi120 ceph-mon[36803]: pgmap v1036: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 549 op/s 2024-03-22T02:36:04.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:04 smithi083 ceph-mon[36340]: pgmap v1036: 129 pgs: 129 active+clean; 280 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 549 op/s 2024-03-22T02:36:04.694 INFO:tasks.workunit.client.0.smithi028.stdout: 10 152838 15.51 MB/sec execute 404 sec latency 324.024 ms 2024-03-22T02:36:05.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 153043 15.50 MB/sec execute 405 sec latency 91.344 ms 2024-03-22T02:36:06.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:06 smithi028 ceph-mon[31916]: pgmap v1037: 129 pgs: 129 active+clean; 289 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 587 op/s 2024-03-22T02:36:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:06 smithi120 ceph-mon[36803]: pgmap v1037: 129 pgs: 129 active+clean; 289 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 587 op/s 2024-03-22T02:36:06.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:06 smithi083 ceph-mon[36340]: pgmap v1037: 129 pgs: 129 active+clean; 289 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 5.0 MiB/s wr, 587 op/s 2024-03-22T02:36:06.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 153368 15.49 MB/sec execute 406 sec latency 110.780 ms 2024-03-22T02:36:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 -- 172.21.15.28:0/1582091007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24100ab0 msgr2=0x7f5f24100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 --2- 172.21.15.28:0/1582091007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24100ab0 0x7f5f24100ed0 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f5f14009900 tx=0x7f5f1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:36:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 -- 172.21.15.28:0/1582091007 shutdown_connections 2024-03-22T02:36:07.321 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 --2- 172.21.15.28:0/1582091007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24102660 0x7f5f24108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 --2- 172.21.15.28:0/1582091007 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 0x7f5f24102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 --2- 172.21.15.28:0/1582091007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24100ab0 0x7f5f24100ed0 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 -- 172.21.15.28:0/1582091007 >> 172.21.15.28:0/1582091007 conn(0x7f5f240fc020 msgr2=0x7f5f240fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 -- 172.21.15.28:0/1582091007 shutdown_connections 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.318+0000 7f5f2a393640 1 -- 172.21.15.28:0/1582091007 wait complete. 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 Processor -- start 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 -- start start 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24100ab0 0x7f5f24197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 0x7f5f241983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.322 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c700 con 0x7f5f24102660 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c580 con 0x7f5f24101ca0 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f2a393640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2410c880 con 0x7f5f24100ab0 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37716/0 (socket says 172.21.15.28:37716) 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f28930640 1 -- 172.21.15.28:0/3333898862 learned_addr learned my addr 172.21.15.28:0/3333898862 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24100ab0 0x7f5f24197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.319+0000 7f5f237fe640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 0x7f5f241983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 -- 172.21.15.28:0/3333898862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24100ab0 msgr2=0x7f5f24197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24100ab0 0x7f5f24197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.323 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 -- 172.21.15.28:0/3333898862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 msgr2=0x7f5f241983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 0x7f5f241983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 -- 172.21.15.28:0/3333898862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f2419fab0 con 0x7f5f24102660 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f28930640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00efd0 tx=0x7f5f1c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f217fa640 1 -- 172.21.15.28:0/3333898862 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c003b70 con 0x7f5f24102660 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f217fa640 1 -- 172.21.15.28:0/3333898862 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f1c003d10 con 0x7f5f24102660 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f2419fda0 con 0x7f5f24102660 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f217fa640 1 -- 172.21.15.28:0/3333898862 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0109f0 con 0x7f5f24102660 2024-03-22T02:36:07.324 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.320+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f241a0210 con 0x7f5f24102660 2024-03-22T02:36:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.322+0000 7f5f217fa640 1 -- 172.21.15.28:0/3333898862 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5f1c004020 con 0x7f5f24102660 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.322+0000 7f5f2cf78740 1 Processor -- start 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 -- start start 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f217fa640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 0x7f5f0c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69ccfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 0x55bb69cd28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb69c2b040 con 0x55bb69ccb2e0 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb69bd39e0 con 0x55bb69cd0450 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb69c26ff0 con 0x55bb69cca8f0 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f237fe640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 0x7f5f0c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69ccfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:38788/0 (socket says 172.21.15.28:38788) 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f23fff640 1 -- 172.21.15.28:0/3153707691 learned_addr learned my addr 172.21.15.28:0/3153707691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 0x55bb69cd28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f217fa640 1 -- 172.21.15.28:0/3333898862 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f5f1c0cfc60 con 0x7f5f24102660 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 -- 172.21.15.28:0/3153707691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 msgr2=0x55bb69ccad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 -- 172.21.15.28:0/3153707691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 msgr2=0x55bb69ccfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69ccfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 -- 172.21.15.28:0/3153707691 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb6993f210 con 0x55bb69cd0450 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f23fff640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f237fe640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69ccfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f237fe640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 0x7f5f0c0a5480 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5f241993a0 tx=0x7f5f18009b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.323+0000 7f5f28930640 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 0x55bb69cd28e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c09beb0 tx=0x7f5f1c0104c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a53b1727002d66 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f0a7fc640 1 -- 172.21.15.28:0/3153707691 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c004710 con 0x55bb69cd0450 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f0a7fc640 1 -- 172.21.15.28:0/3153707691 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f1c09c620 con 0x55bb69cd0450 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f0a7fc640 1 -- 172.21.15.28:0/3153707691 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0cd400 con 0x55bb69cd0450 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 -- 172.21.15.28:0/3153707691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 msgr2=0x55bb69cd28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 0x55bb69cd28e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c09beb0 tx=0x7f5f1c0104c0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 -- 172.21.15.28:0/3153707691 shutdown_connections 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cd0450 0x55bb69cd28e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69ccfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/3153707691 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cca8f0 0x55bb69ccad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 -- 172.21.15.28:0/3153707691 >> 172.21.15.28:0/3153707691 conn(0x55bb69cc1a50 msgr2=0x55bb69cc1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 -- 172.21.15.28:0/3153707691 shutdown_connections 2024-03-22T02:36:07.332 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.324+0000 7f5f2cf78740 1 -- 172.21.15.28:0/3153707691 wait complete. 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 -- start start 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 Processor -- start 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bb69cd3340 con 0x55bb69cd0450 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55bb69be6db0 con 0x55bb69cca8f0 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55bb69cd34c0 con 0x55bb69ccb2e0 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f2cf78740 10 client.0 _ll_register_callbacks cb 0x55bb6993ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37724/0 (socket says 172.21.15.28:37724) 2024-03-22T02:36:07.333 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 -- 172.21.15.28:0/1162388437 learned_addr learned my addr 172.21.15.28:0/1162388437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f237fe640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 -- 172.21.15.28:0/1162388437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 msgr2=0x55bb69cd4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 -- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 msgr2=0x55bb69cd4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.325+0000 7f5f28930640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f28930640 1 -- 172.21.15.28:0/1162388437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb69a0b620 con 0x55bb69cd0450 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f23fff640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f28930640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0a4400 tx=0x7f5f1c003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d24429a7acbec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c09bf20 con 0x55bb69cd0450 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f5f1c010b90 con 0x55bb69cd0450 2024-03-22T02:36:07.334 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f1c09c620 con 0x55bb69cd0450 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c0a0070 con 0x55bb69cd0450 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 10 client.20280 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bb69cdf120 con 0x55bb69cd0450 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f1c09c620 con 0x55bb69cd0450 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f5f1c0b7020 con 0x55bb69cd0450 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 10 client.20280 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 10 client.20280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 20 client.20280 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69ce0470 0x55bb69ce2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 4 client.20280 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.326+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb69c38580 con 0x55bb69ce0470 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.327+0000 7f5f23fff640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69ce0470 0x55bb69ce2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.327+0000 7f5f23fff640 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69ce0470 0x55bb69ce2860 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c24a13341150671 server_cookie=8dfe3e9fa58c15cc in_seq=0 out_seq=0 2024-03-22T02:36:07.335 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.327+0000 7f5f037fe640 10 client.20280 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f037fe640 1 -- 172.21.15.28:0/1162388437 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb69c38580 con 0x55bb69ce0470 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f037fe640 10 client.20280 handle_command_reply: tid=0 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 client.20280 shutdown 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 msgr2=0x55bb69cdbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0a4400 tx=0x7f5f1c003ec0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 shutdown_connections 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69ce0470 0x55bb69ce2860 crc :-1 s=READY pgs=432 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69cdbdb0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1162388437 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.337 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.332+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 >> 172.21.15.28:0/1162388437 conn(0x55bb69cc1a50 msgr2=0x55bb69cc0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.333+0000 7f5f037fe640 0 client.20280 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.333+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 shutdown_connections 2024-03-22T02:36:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.333+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1162388437 wait complete. 2024-03-22T02:36:07.338 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.333+0000 7f5f2cf78740 20 client.20280 trim_cache size 0 max 16384 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 Processor -- start 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 -- start start 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cc53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cd0450 0x55bb69cc03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241aaf90 con 0x55bb69ccb2e0 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241a8860 con 0x55bb69cca8f0 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241a66b0 con 0x55bb69cd0450 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37726/0 (socket says 172.21.15.28:37726) 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f237fe640 1 -- 172.21.15.28:0/1502295298 learned_addr learned my addr 172.21.15.28:0/1502295298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:07.351 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f28930640 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cd0450 0x55bb69cc03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.348+0000 7f5f23fff640 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cc53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 -- 172.21.15.28:0/1502295298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cd0450 msgr2=0x55bb69cc03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cd0450 0x55bb69cc03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 -- 172.21.15.28:0/1502295298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 msgr2=0x55bb69cc53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cc53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 -- 172.21.15.28:0/1502295298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb69cbd4b0 con 0x55bb69ccb2e0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f237fe640 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f5f18011330 tx=0x7f5f18021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f89162a0431da4 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f09ffb640 1 -- 172.21.15.28:0/1502295298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1802d070 con 0x55bb69ccb2e0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f09ffb640 1 -- 172.21.15.28:0/1502295298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f18003ad0 con 0x55bb69ccb2e0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.349+0000 7f5f09ffb640 1 -- 172.21.15.28:0/1502295298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18028760 con 0x55bb69ccb2e0 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1502295298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 msgr2=0x55bb69cdbd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f5f18011330 tx=0x7f5f18021000 comp rx=0 tx=0).stop 2024-03-22T02:36:07.352 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1502295298 shutdown_connections 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69cd0450 0x55bb69cc03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69ccb2e0 0x55bb69cdbd70 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/1502295298 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cc53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1502295298 >> 172.21.15.28:0/1502295298 conn(0x55bb69cc1a50 msgr2=0x55bb69cc4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1502295298 shutdown_connections 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- 172.21.15.28:0/1502295298 wait complete. 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- start start 2024-03-22T02:36:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 Processor -- start 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241a6d20 con 0x55bb69cd0450 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241a8190 con 0x55bb69cca8f0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.350+0000 7f5f2cf78740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5f241a73f0 con 0x55bb69ccb2e0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f2cf78740 10 client.0 _ll_register_callbacks cb 0x7f5f180114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37728/0 (socket says 172.21.15.28:37728) 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f237fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f23fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd8db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:46434/0 (socket says 172.21.15.28:46434) 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 -- 172.21.15.28:0/308682580 learned_addr learned my addr 172.21.15.28:0/308682580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 -- 172.21.15.28:0/308682580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 msgr2=0x55bb69cd92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 -- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 msgr2=0x55bb69cd8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 -- 172.21.15.28:0/308682580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb69d44d30 con 0x55bb69cd0450 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f28930640 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0a0230 tx=0x7f5f1c004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a4472f7bec37e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f037fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c09c620 con 0x55bb69cd0450 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f5f1c0850d0 con 0x55bb69cd0450 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f18003ad0 con 0x55bb69cd0450 2024-03-22T02:36:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c0b5020 con 0x55bb69cd0450 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f2cf78740 10 client.20286 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.351+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55bb69d46040 con 0x55bb69cd0450 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f18003ad0 con 0x55bb69cd0450 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f5f1c0b7020 con 0x55bb69cd0450 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 10 client.20286 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 10 client.20286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 20 client.20286 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69d46880 0x55bb69d48ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 4 client.20286 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bb69cbd2b0 con 0x55bb69d46880 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f23fff640 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69d46880 0x55bb69d48ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f037fe640 10 client.20286 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:07.352+0000 7f5f23fff640 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69d46880 0x55bb69d48ca0 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e3731d51a751946 server_cookie=80680c5ddc69bac8 in_seq=0 out_seq=0 2024-03-22T02:36:07.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 153639 15.47 MB/sec execute 407 sec latency 245.695 ms 2024-03-22T02:36:08.316 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:08 smithi083 ceph-mon[36340]: pgmap v1038: 129 pgs: 129 active+clean; 292 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 431 op/s 2024-03-22T02:36:08.321 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:08 smithi028 ceph-mon[31916]: pgmap v1038: 129 pgs: 129 active+clean; 292 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 431 op/s 2024-03-22T02:36:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:08 smithi120 ceph-mon[36803]: pgmap v1038: 129 pgs: 129 active+clean; 292 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.1 MiB/s wr, 431 op/s 2024-03-22T02:36:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f037fe640 1 -- 172.21.15.28:0/308682580 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bb69cbd2b0 con 0x55bb69d46880 2024-03-22T02:36:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f037fe640 10 client.20286 handle_command_reply: tid=0 2024-03-22T02:36:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 client.20286 shutdown 2024-03-22T02:36:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 msgr2=0x55bb69d44740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:08.551 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0a0230 tx=0x7f5f1c004920 comp rx=0 tx=0).stop 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 shutdown_connections 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55bb69d46880 0x55bb69d48ca0 crc :-1 s=READY pgs=433 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bb69cd0450 0x55bb69d44740 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55bb69ccb2e0 0x55bb69cd92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 --2- 172.21.15.28:0/308682580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55bb69cca8f0 0x55bb69cd8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f037fe640 0 client.20286 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 >> 172.21.15.28:0/308682580 conn(0x55bb69cc1a50 msgr2=0x55bb69ccbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 shutdown_connections 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 1 -- 172.21.15.28:0/308682580 wait complete. 2024-03-22T02:36:08.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.550+0000 7f5f2cf78740 20 client.20286 trim_cache size 0 max 16384 2024-03-22T02:36:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:36:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:36:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:36:08.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 msgr2=0x7f5f0c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 0x7f5f0c0a5480 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5f241993a0 tx=0x7f5f18009b60 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 msgr2=0x7f5f2419f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00efd0 tx=0x7f5f1c00c6a0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 shutdown_connections 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5f0c0a2fc0 0x7f5f0c0a5480 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f24102660 0x7f5f2419f440 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5f24101ca0 0x7f5f241983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 --2- 172.21.15.28:0/3333898862 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f24100ab0 0x7f5f24197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:08.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 >> 172.21.15.28:0/3333898862 conn(0x7f5f240fc020 msgr2=0x7f5f240fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 shutdown_connections 2024-03-22T02:36:08.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:08.551+0000 7f5f2a393640 1 -- 172.21.15.28:0/3333898862 wait complete. 2024-03-22T02:36:08.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 153696 15.44 MB/sec execute 408 sec latency 833.299 ms 2024-03-22T02:36:09.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 153949 15.43 MB/sec execute 409 sec latency 188.756 ms 2024-03-22T02:36:10.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:10 smithi028 ceph-mon[31916]: pgmap v1039: 129 pgs: 129 active+clean; 293 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 391 op/s 2024-03-22T02:36:10.500 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:10 smithi120 ceph-mon[36803]: pgmap v1039: 129 pgs: 129 active+clean; 293 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 391 op/s 2024-03-22T02:36:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:10 smithi083 ceph-mon[36340]: pgmap v1039: 129 pgs: 129 active+clean; 293 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.8 MiB/s wr, 391 op/s 2024-03-22T02:36:10.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 154368 15.46 MB/sec execute 410 sec latency 60.536 ms 2024-03-22T02:36:11.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 154824 15.50 MB/sec execute 411 sec latency 64.994 ms 2024-03-22T02:36:12.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:12 smithi028 ceph-mon[31916]: pgmap v1040: 129 pgs: 129 active+clean; 286 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 528 op/s 2024-03-22T02:36:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:12 smithi120 ceph-mon[36803]: pgmap v1040: 129 pgs: 129 active+clean; 286 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 528 op/s 2024-03-22T02:36:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:12 smithi083 ceph-mon[36340]: pgmap v1040: 129 pgs: 129 active+clean; 286 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.3 MiB/s wr, 528 op/s 2024-03-22T02:36:12.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 155312 15.52 MB/sec execute 412 sec latency 70.068 ms 2024-03-22T02:36:13.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 155586 15.53 MB/sec execute 413 sec latency 86.915 ms 2024-03-22T02:36:14.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:14 smithi028 ceph-mon[31916]: pgmap v1041: 129 pgs: 129 active+clean; 278 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 505 op/s 2024-03-22T02:36:14.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:14 smithi120 ceph-mon[36803]: pgmap v1041: 129 pgs: 129 active+clean; 278 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 505 op/s 2024-03-22T02:36:14.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:14 smithi083 ceph-mon[36340]: pgmap v1041: 129 pgs: 129 active+clean; 278 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.8 MiB/s wr, 505 op/s 2024-03-22T02:36:14.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 155789 15.52 MB/sec execute 414 sec latency 419.358 ms 2024-03-22T02:36:15.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 155983 15.51 MB/sec execute 415 sec latency 107.652 ms 2024-03-22T02:36:16.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:16 smithi028 ceph-mon[31916]: pgmap v1042: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.6 MiB/s wr, 570 op/s 2024-03-22T02:36:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:16 smithi120 ceph-mon[36803]: pgmap v1042: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.6 MiB/s wr, 570 op/s 2024-03-22T02:36:16.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:16 smithi083 ceph-mon[36340]: pgmap v1042: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.6 MiB/s wr, 570 op/s 2024-03-22T02:36:16.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 156253 15.51 MB/sec execute 416 sec latency 87.910 ms 2024-03-22T02:36:17.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 156527 15.51 MB/sec execute 417 sec latency 65.116 ms 2024-03-22T02:36:18.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:18 smithi028 ceph-mon[31916]: pgmap v1043: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 533 op/s 2024-03-22T02:36:18.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:18 smithi120 ceph-mon[36803]: pgmap v1043: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 533 op/s 2024-03-22T02:36:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:18 smithi083 ceph-mon[36340]: pgmap v1043: 129 pgs: 129 active+clean; 282 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.7 MiB/s wr, 533 op/s 2024-03-22T02:36:18.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 156843 15.50 MB/sec execute 418 sec latency 104.136 ms 2024-03-22T02:36:19.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 157001 15.47 MB/sec execute 419 sec latency 428.381 ms 2024-03-22T02:36:20.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:20 smithi028 ceph-mon[31916]: pgmap v1044: 129 pgs: 129 active+clean; 283 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.3 MiB/s wr, 495 op/s 2024-03-22T02:36:20.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:20 smithi120 ceph-mon[36803]: pgmap v1044: 129 pgs: 129 active+clean; 283 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.3 MiB/s wr, 495 op/s 2024-03-22T02:36:20.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:20 smithi083 ceph-mon[36340]: pgmap v1044: 129 pgs: 129 active+clean; 283 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.3 MiB/s wr, 495 op/s 2024-03-22T02:36:20.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 157288 15.46 MB/sec execute 420 sec latency 103.786 ms 2024-03-22T02:36:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.671+0000 7f0374dd1640 1 -- 172.21.15.28:0/3832404432 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368032a00 con 0x7f0370107890 2024-03-22T02:36:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.671+0000 7f0378037640 1 -- 172.21.15.28:0/3832404432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370107890 msgr2=0x7f0370109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.672 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.671+0000 7f0378037640 1 --2- 172.21.15.28:0/3832404432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370107890 0x7f0370109ce0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0368009ca0 tx=0x7f036802c120 comp rx=0 tx=0).stop 2024-03-22T02:36:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.673+0000 7f0378037640 1 -- 172.21.15.28:0/3832404432 shutdown_connections 2024-03-22T02:36:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.673+0000 7f0378037640 1 --2- 172.21.15.28:0/3832404432 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370107890 0x7f0370109ce0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.673+0000 7f0378037640 1 --2- 172.21.15.28:0/3832404432 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0370104f60 0x7f0370107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.673+0000 7f0378037640 1 --2- 172.21.15.28:0/3832404432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370102630 0x7f0370104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.673+0000 7f0378037640 1 -- 172.21.15.28:0/3832404432 >> 172.21.15.28:0/3832404432 conn(0x7f03700fc020 msgr2=0x7f03700fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.675+0000 7f0378037640 1 -- 172.21.15.28:0/3832404432 shutdown_connections 2024-03-22T02:36:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.677+0000 7f0378037640 1 -- 172.21.15.28:0/3832404432 wait complete. 2024-03-22T02:36:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.677+0000 7f0378037640 1 Processor -- start 2024-03-22T02:36:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 -- start start 2024-03-22T02:36:21.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0370102630 0x7f0370110ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370104f60 0x7f0370111420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0370102020 con 0x7f0370107890 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0370101ea0 con 0x7f0370102630 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.678+0000 7f0378037640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03701021a0 con 0x7f0370104f60 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.679+0000 7f03765d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.679+0000 7f03765d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57096/0 (socket says 172.21.15.28:57096) 2024-03-22T02:36:21.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.679+0000 7f03765d4640 1 -- 172.21.15.28:0/2912454573 learned_addr learned my addr 172.21.15.28:0/2912454573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.680+0000 7f03765d4640 1 -- 172.21.15.28:0/2912454573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370104f60 msgr2=0x7f0370111420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.680+0000 7f03765d4640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370104f60 0x7f0370111420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.680+0000 7f03765d4640 1 -- 172.21.15.28:0/2912454573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0370102630 msgr2=0x7f0370110ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.680 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.680+0000 7f03765d4640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0370102630 0x7f0370110ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.681 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.680+0000 7f03765d4640 1 -- 172.21.15.28:0/2912454573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03701185c0 con 0x7f0370107890 2024-03-22T02:36:21.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.682+0000 7f03765d4640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f036802c660 tx=0x7f0368035cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.683+0000 7f0366ffd640 1 -- 172.21.15.28:0/2912454573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368039040 con 0x7f0370107890 2024-03-22T02:36:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.683+0000 7f0366ffd640 1 -- 172.21.15.28:0/2912454573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0368044070 con 0x7f0370107890 2024-03-22T02:36:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.683+0000 7f0366ffd640 1 -- 172.21.15.28:0/2912454573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0368004460 con 0x7f0370107890 2024-03-22T02:36:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.683+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f037006b5b0 con 0x7f0370107890 2024-03-22T02:36:21.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.683+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f037006bad0 con 0x7f0370107890 2024-03-22T02:36:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.687+0000 7f0366ffd640 1 -- 172.21.15.28:0/2912454573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0368004600 con 0x7f0370107890 2024-03-22T02:36:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 Processor -- start 2024-03-22T02:36:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 -- start start 2024-03-22T02:36:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1157900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e115f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564e10bc040 con 0x5564e1157ed0 2024-03-22T02:36:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564e10649e0 con 0x5564e11574e0 2024-03-22T02:36:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.688+0000 7f037ac1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5564e10b7ff0 con 0x5564e115d040 2024-03-22T02:36:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.689+0000 7f03755d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.689+0000 7f0366ffd640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 0x7f035c0a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.689+0000 7f0375dd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.689+0000 7f0375dd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57110/0 (socket says 172.21.15.28:57110) 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.689+0000 7f0375dd3640 1 -- 172.21.15.28:0/2710470238 learned_addr learned my addr 172.21.15.28:0/2710470238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0366ffd640 1 -- 172.21.15.28:0/2912454573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f03680f1ed0 con 0x7f0370107890 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f03765d4640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 0x7f035c0a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 -- 172.21.15.28:0/2710470238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 msgr2=0x5564e115f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e115f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 -- 172.21.15.28:0/2710470238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 msgr2=0x5564e1157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 -- 172.21.15.28:0/2710470238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564e0dd0210 con 0x5564e1157ed0 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.690+0000 7f0375dd3640 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f036c009c40 tx=0x7f036c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96df7b4ef8b57823 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.691+0000 7f035bfff640 1 -- 172.21.15.28:0/2710470238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c005020 con 0x5564e1157ed0 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.691+0000 7f035bfff640 1 -- 172.21.15.28:0/2710470238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c0047a0 con 0x5564e1157ed0 2024-03-22T02:36:21.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.691+0000 7f035bfff640 1 -- 172.21.15.28:0/2710470238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c010760 con 0x5564e1157ed0 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.692+0000 7f037ac1a740 1 -- 172.21.15.28:0/2710470238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 msgr2=0x5564e115c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.692+0000 7f037ac1a740 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f036c009c40 tx=0x7f036c00e720 comp rx=0 tx=0).stop 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 -- 172.21.15.28:0/2710470238 shutdown_connections 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e115f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e115c9d0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 --2- 172.21.15.28:0/2710470238 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 -- 172.21.15.28:0/2710470238 >> 172.21.15.28:0/2710470238 conn(0x5564e1152a50 msgr2=0x5564e1152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f03765d4640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 0x7f035c0a5670 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f036804bff0 tx=0x7f036802c120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 -- 172.21.15.28:0/2710470238 shutdown_connections 2024-03-22T02:36:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 -- 172.21.15.28:0/2710470238 wait complete. 2024-03-22T02:36:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 -- start start 2024-03-22T02:36:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.693+0000 7f037ac1a740 1 Processor -- start 2024-03-22T02:36:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e1161a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e115d040 0x5564e1168ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5564e115fff0 con 0x5564e11574e0 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5564e1077db0 con 0x5564e115d040 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5564e1160170 con 0x5564e1157ed0 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f037ac1a740 10 client.0 _ll_register_callbacks cb 0x5564e0dcfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f03755d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f03755d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57118/0 (socket says 172.21.15.28:57118) 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f03755d2640 1 -- 172.21.15.28:0/418332269 learned_addr learned my addr 172.21.15.28:0/418332269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f03755d2640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e115d040 0x5564e1168ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.694+0000 7f0375dd3640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e1161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 157640 15.47 MB/sec execute 421 sec latency 102.484 ms 2024-03-22T02:36:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 -- 172.21.15.28:0/418332269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 msgr2=0x5564e1161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e1161a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 -- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e115d040 msgr2=0x5564e1168ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e115d040 0x5564e1168ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 -- 172.21.15.28:0/418332269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564e0e9c620 con 0x5564e11574e0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03755d2640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f0360009bd0 tx=0x7f0360001060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33afd721ebeddeb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03597fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360012040 con 0x5564e11574e0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.695+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f0360008d30 con 0x5564e11574e0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f036c0047a0 con 0x5564e11574e0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0360005020 con 0x5564e11574e0 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 10 client.20298 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5564e116bc90 con 0x5564e11574e0 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0360004770 con 0x5564e11574e0 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f0360018040 con 0x5564e11574e0 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 10 client.20298 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 10 client.20298 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 20 client.20298 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e116d310 0x5564e116f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 4 client.20298 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.696+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f036c0047a0 con 0x5564e116d310 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.697+0000 7f0375dd3640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e116d310 0x5564e116f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.697+0000 7f0375dd3640 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e116d310 0x5564e116f740 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4ae0cb74afeaa1 server_cookie=dae0f5a2e2bcdb1f in_seq=0 out_seq=0 2024-03-22T02:36:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.697+0000 7f03597fa640 10 client.20298 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.702+0000 7f03597fa640 1 -- 172.21.15.28:0/418332269 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f036c0047a0 con 0x5564e116d310 2024-03-22T02:36:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.702+0000 7f03597fa640 10 client.20298 handle_command_reply: tid=0 2024-03-22T02:36:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.702+0000 7f037ac1a740 1 client.20298 shutdown 2024-03-22T02:36:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 msgr2=0x5564e1161540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f0360009bd0 tx=0x7f0360001060 comp rx=0 tx=0).stop 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 shutdown_connections 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e116d310 0x5564e116f740 crc :-1 s=READY pgs=434 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e115d040 0x5564e1168ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e1161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 --2- 172.21.15.28:0/418332269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e11574e0 0x5564e1161540 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 >> 172.21.15.28:0/418332269 conn(0x5564e1152a50 msgr2=0x5564e1151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f03597fa640 0 client.20298 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 shutdown_connections 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 1 -- 172.21.15.28:0/418332269 wait complete. 2024-03-22T02:36:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.703+0000 7f037ac1a740 20 client.20298 trim_cache size 0 max 16384 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 Processor -- start 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 -- start start 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f0375dd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f03755d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f03755d2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:40230/0 (socket says 172.21.15.28:40230) 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f03755d2640 1 -- 172.21.15.28:0/743611514 learned_addr learned my addr 172.21.15.28:0/743611514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f0375dd3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57122/0 (socket says 172.21.15.28:57122) 2024-03-22T02:36:21.726 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e11619e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c4970 con 0x5564e1157ed0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c2240 con 0x5564e11574e0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c0090 con 0x5564e115d040 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.724+0000 7f03755d2640 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e11619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 -- 172.21.15.28:0/743611514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 msgr2=0x5564e11619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e11619e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 -- 172.21.15.28:0/743611514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 msgr2=0x5564e1160bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 -- 172.21.15.28:0/743611514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564e11537c0 con 0x5564e1157ed0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f03755d2640 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f0375dd3640 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x5564e11539c0 tx=0x7f036c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5cfbde4dc3ce179 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f035b7fe640 1 -- 172.21.15.28:0/743611514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c01d040 con 0x5564e1157ed0 2024-03-22T02:36:21.727 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f035b7fe640 1 -- 172.21.15.28:0/743611514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c028070 con 0x5564e1157ed0 2024-03-22T02:36:21.728 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.725+0000 7f035b7fe640 1 -- 172.21.15.28:0/743611514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036c029520 con 0x5564e1157ed0 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 msgr2=0x5564e1168a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x5564e11539c0 tx=0x7f036c005d30 comp rx=0 tx=0).stop 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 shutdown_connections 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e115d040 0x5564e11619e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e1157ed0 0x5564e1168a80 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.731 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 --2- 172.21.15.28:0/743611514 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1160bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 >> 172.21.15.28:0/743611514 conn(0x5564e1152a50 msgr2=0x5564e1155cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 shutdown_connections 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- 172.21.15.28:0/743611514 wait complete. 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 -- start start 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.726+0000 7f037ac1a740 1 Processor -- start 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1165f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e11664d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c0700 con 0x5564e115d040 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c1b70 con 0x5564e11574e0 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f03701c0dd0 con 0x5564e1157ed0 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 10 client.0 _ll_register_callbacks cb 0x7f0360007190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.732 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57128/0 (socket says 172.21.15.28:57128) 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 -- 172.21.15.28:0/3356343590 learned_addr learned my addr 172.21.15.28:0/3356343590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f0375dd3640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e11664d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1165f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 -- 172.21.15.28:0/3356343590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 msgr2=0x5564e11664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e11664d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 -- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 msgr2=0x5564e1165f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1165f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 -- 172.21.15.28:0/3356343590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564e1167ad0 con 0x5564e115d040 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03755d2640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0360005fd0 tx=0x7f0360009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb672c2c77acdcd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03597fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03600055d0 con 0x5564e115d040 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f0360004770 con 0x5564e115d040 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.727+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f036c028070 con 0x5564e115d040 2024-03-22T02:36:21.733 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0360004970 con 0x5564e115d040 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 10 client.20304 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5564e1168de0 con 0x5564e115d040 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f036c028070 con 0x5564e115d040 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f0360018030 con 0x5564e115d040 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 10 client.20304 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 10 client.20304 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 20 client.20304 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e11d3e40 0x5564e11d6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 4 client.20304 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5564e10c9580 con 0x5564e11d3e40 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.728+0000 7f0375dd3640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e11d3e40 0x5564e11d6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.729+0000 7f0375dd3640 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e11d3e40 0x5564e11d6260 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=826f999bcfdf4ac9 server_cookie=4f9c5db9872a5c61 in_seq=0 out_seq=0 2024-03-22T02:36:21.734 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:21.729+0000 7f03597fa640 10 client.20304 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:22.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:22 smithi028 ceph-mon[31916]: pgmap v1045: 129 pgs: 129 active+clean; 293 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:36:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:22 smithi120 ceph-mon[36803]: pgmap v1045: 129 pgs: 129 active+clean; 293 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:36:22.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:22 smithi083 ceph-mon[36340]: pgmap v1045: 129 pgs: 129 active+clean; 293 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:36:22.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 157676 15.44 MB/sec execute 422 sec latency 722.189 ms 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f03597fa640 1 -- 172.21.15.28:0/3356343590 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5564e10c9580 con 0x5564e11d3e40 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f03597fa640 10 client.20304 handle_command_reply: tid=0 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 client.20304 shutdown 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 msgr2=0x5564e11674e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f0360005fd0 tx=0x7f0360009f90 comp rx=0 tx=0).stop 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 shutdown_connections 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5564e11d3e40 0x5564e11d6260 crc :-1 s=READY pgs=435 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5564e115d040 0x5564e11674e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f03597fa640 0 client.20304 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5564e1157ed0 0x5564e11664d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.833 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 --2- 172.21.15.28:0/3356343590 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5564e11574e0 0x5564e1165f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 >> 172.21.15.28:0/3356343590 conn(0x5564e1152a50 msgr2=0x5564e1156540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 shutdown_connections 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 1 -- 172.21.15.28:0/3356343590 wait complete. 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.831+0000 7f037ac1a740 20 client.20304 trim_cache size 0 max 16384 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:36:22.834 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:36:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 msgr2=0x7f035c0a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 0x7f035c0a5670 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f036804bff0 tx=0x7f036802c120 comp rx=0 tx=0).stop 2024-03-22T02:36:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 msgr2=0x7f03701ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f036802c660 tx=0x7f0368035cf0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.835 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 shutdown_connections 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f035c0a31b0 0x7f035c0a5670 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0370107890 0x7f03701ae630 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0370104f60 0x7f0370111420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 --2- 172.21.15.28:0/2912454573 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0370102630 0x7f0370110ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 >> 172.21.15.28:0/2912454573 conn(0x7f03700fc020 msgr2=0x7f0370102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 shutdown_connections 2024-03-22T02:36:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:22.832+0000 7f0378037640 1 -- 172.21.15.28:0/2912454573 wait complete. 2024-03-22T02:36:23.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 158126 15.47 MB/sec execute 423 sec latency 761.264 ms 2024-03-22T02:36:24.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:24 smithi120 ceph-mon[36803]: pgmap v1046: 129 pgs: 129 active+clean; 298 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 338 op/s 2024-03-22T02:36:24.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:24 smithi083 ceph-mon[36340]: pgmap v1046: 129 pgs: 129 active+clean; 298 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 338 op/s 2024-03-22T02:36:24.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:24 smithi028 ceph-mon[31916]: pgmap v1046: 129 pgs: 129 active+clean; 298 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.4 MiB/s wr, 338 op/s 2024-03-22T02:36:24.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 158510 15.49 MB/sec execute 424 sec latency 199.882 ms 2024-03-22T02:36:25.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 158822 15.49 MB/sec execute 425 sec latency 99.321 ms 2024-03-22T02:36:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:26 smithi120 ceph-mon[36803]: pgmap v1047: 129 pgs: 129 active+clean; 283 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-03-22T02:36:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:26 smithi083 ceph-mon[36340]: pgmap v1047: 129 pgs: 129 active+clean; 283 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-03-22T02:36:26.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:26 smithi028 ceph-mon[31916]: pgmap v1047: 129 pgs: 129 active+clean; 283 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.2 MiB/s wr, 508 op/s 2024-03-22T02:36:26.695 INFO:tasks.workunit.client.0.smithi028.stdout: 10 159138 15.51 MB/sec execute 426 sec latency 56.284 ms 2024-03-22T02:36:27.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 159403 15.50 MB/sec execute 427 sec latency 134.206 ms 2024-03-22T02:36:28.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:28 smithi120 ceph-mon[36803]: pgmap v1048: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 510 op/s 2024-03-22T02:36:28.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:28 smithi083 ceph-mon[36340]: pgmap v1048: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 510 op/s 2024-03-22T02:36:28.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:28 smithi028 ceph-mon[31916]: pgmap v1048: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.3 MiB/s wr, 510 op/s 2024-03-22T02:36:28.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 159668 15.50 MB/sec execute 428 sec latency 80.649 ms 2024-03-22T02:36:29.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 159819 15.49 MB/sec execute 429 sec latency 292.814 ms 2024-03-22T02:36:30.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 160027 15.47 MB/sec execute 430 sec latency 115.615 ms 2024-03-22T02:36:30.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:30 smithi120 ceph-mon[36803]: pgmap v1049: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 498 op/s 2024-03-22T02:36:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:30 smithi083 ceph-mon[36340]: pgmap v1049: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 498 op/s 2024-03-22T02:36:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:30 smithi028 ceph-mon[31916]: pgmap v1049: 129 pgs: 129 active+clean; 278 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 4.3 MiB/s wr, 498 op/s 2024-03-22T02:36:31.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 160311 15.47 MB/sec execute 431 sec latency 92.468 ms 2024-03-22T02:36:32.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 160664 15.46 MB/sec execute 432 sec latency 67.957 ms 2024-03-22T02:36:32.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:32 smithi120 ceph-mon[36803]: pgmap v1050: 129 pgs: 129 active+clean; 285 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-03-22T02:36:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:32 smithi083 ceph-mon[36340]: pgmap v1050: 129 pgs: 129 active+clean; 285 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-03-22T02:36:32.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:32 smithi028 ceph-mon[31916]: pgmap v1050: 129 pgs: 129 active+clean; 285 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 5.3 MiB/s wr, 569 op/s 2024-03-22T02:36:33.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 161035 15.47 MB/sec execute 433 sec latency 86.187 ms 2024-03-22T02:36:34.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 161300 15.47 MB/sec execute 434 sec latency 202.464 ms 2024-03-22T02:36:34.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:34 smithi120 ceph-mon[36803]: pgmap v1051: 129 pgs: 129 active+clean; 286 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 515 op/s 2024-03-22T02:36:34.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:34 smithi083 ceph-mon[36340]: pgmap v1051: 129 pgs: 129 active+clean; 286 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 515 op/s 2024-03-22T02:36:34.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:34 smithi028 ceph-mon[31916]: pgmap v1051: 129 pgs: 129 active+clean; 286 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 515 op/s 2024-03-22T02:36:35.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 161703 15.49 MB/sec execute 435 sec latency 105.633 ms 2024-03-22T02:36:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 -- 172.21.15.28:0/914438048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 msgr2=0x7ff3fc103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 --2- 172.21.15.28:0/914438048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc103360 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009900 tx=0x7ff3ec02bc80 comp rx=0 tx=0).stop 2024-03-22T02:36:35.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 -- 172.21.15.28:0/914438048 shutdown_connections 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 --2- 172.21.15.28:0/914438048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc1038a0 0x7ff3fc10a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 --2- 172.21.15.28:0/914438048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc103360 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 --2- 172.21.15.28:0/914438048 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 -- 172.21.15.28:0/914438048 >> 172.21.15.28:0/914438048 conn(0x7ff3fc0fb7e0 msgr2=0x7ff3fc0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 -- 172.21.15.28:0/914438048 shutdown_connections 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.960+0000 7ff40101c640 1 -- 172.21.15.28:0/914438048 wait complete. 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 Processor -- start 2024-03-22T02:36:35.962 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 -- start start 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc1038a0 0x7ff3fc19f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10d850 con 0x7ff3fc102ee0 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10d6d0 con 0x7ff3fc1038a0 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff40101c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc10d9d0 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.964 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44266/0 (socket says 172.21.15.28:44266) 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff3fad9d640 1 -- 172.21.15.28:0/2905127322 learned_addr learned my addr 172.21.15.28:0/2905127322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.961+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc1038a0 0x7ff3fc19f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 -- 172.21.15.28:0/2905127322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc1038a0 msgr2=0x7ff3fc19f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc1038a0 0x7ff3fc19f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 -- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 msgr2=0x7ff3fc198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 -- 172.21.15.28:0/2905127322 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc19fa20 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e9c0 tx=0x7ff3f000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3e3fff640 1 -- 172.21.15.28:0/2905127322 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000cd80 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3e3fff640 1 -- 172.21.15.28:0/2905127322 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f00035f0 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff3e3fff640 1 -- 172.21.15.28:0/2905127322 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0010690 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.965 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.962+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc19fd10 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.963+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc1a0180 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.966 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff3e3fff640 1 -- 172.21.15.28:0/2905127322 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff3f00037a0 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 Processor -- start 2024-03-22T02:36:35.967 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 -- start start 2024-03-22T02:36:35.968 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff3e3fff640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 0x7ff3e40a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e9450 0x56446c5eb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56446c544040 con 0x56446c5e9450 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56446c4ec9e0 con 0x56446c5e38f0 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff403c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56446c53fff0 con 0x56446c5e42e0 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.966+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44874/0 (socket says 172.21.15.28:44874) 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44280/0 (socket says 172.21.15.28:44280) 2024-03-22T02:36:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fb59e640 1 -- 172.21.15.28:0/4171966811 learned_addr learned my addr 172.21.15.28:0/4171966811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e9450 0x56446c5eb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 0x7ff3e40a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3e3fff640 1 -- 172.21.15.28:0/2905127322 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7ff3f00cfb60 con 0x7ff3fc0ffe70 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 -- 172.21.15.28:0/4171966811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 msgr2=0x56446c5e3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 -- 172.21.15.28:0/4171966811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e9450 msgr2=0x56446c5eb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e9450 0x56446c5eb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 -- 172.21.15.28:0/4171966811 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446c258210 con 0x56446c5e42e0 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00cd5c0 tx=0x7ff3f0014d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15b24111fcd35d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.968+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 0x7ff3e40a5500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec005ab0 tx=0x7ff3ec02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.967+0000 7ff3e17fa640 1 -- 172.21.15.28:0/4171966811 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f009c580 con 0x56446c5e42e0 2024-03-22T02:36:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.968+0000 7ff3e17fa640 1 -- 172.21.15.28:0/4171966811 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f009be80 con 0x56446c5e42e0 2024-03-22T02:36:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.968+0000 7ff3e17fa640 1 -- 172.21.15.28:0/4171966811 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0014da0 con 0x56446c5e42e0 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 -- 172.21.15.28:0/4171966811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 msgr2=0x56446c5e8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00cd5c0 tx=0x7ff3f0014d30 comp rx=0 tx=0).stop 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 -- 172.21.15.28:0/4171966811 shutdown_connections 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e9450 0x56446c5eb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e8de0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 --2- 172.21.15.28:0/4171966811 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5e3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 -- 172.21.15.28:0/4171966811 >> 172.21.15.28:0/4171966811 conn(0x56446c5daa50 msgr2=0x56446c5dae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.969+0000 7ff403c07740 1 -- 172.21.15.28:0/4171966811 shutdown_connections 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.970+0000 7ff403c07740 1 -- 172.21.15.28:0/4171966811 wait complete. 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.970+0000 7ff403c07740 1 -- start start 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.971+0000 7ff403c07740 1 Processor -- start 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.971+0000 7ff403c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 0x56446c5e67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.971+0000 7ff403c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.971+0000 7ff403c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5e7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44284/0 (socket says 172.21.15.28:44284) 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 0x56446c5e67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 0x56446c5e67f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58196/0 (socket says 172.21.15.28:58196) 2024-03-22T02:36:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fa59c640 1 -- 172.21.15.28:0/2364340308 learned_addr learned my addr 172.21.15.28:0/2364340308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5e7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56446c5df500 con 0x56446c5e38f0 2024-03-22T02:36:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56446c4ffdb0 con 0x56446c5e9450 2024-03-22T02:36:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56446c5df680 con 0x56446c5e42e0 2024-03-22T02:36:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.972+0000 7ff403c07740 10 client.0 _ll_register_callbacks cb 0x56446c257a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 -- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 msgr2=0x56446c5e7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5e7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 -- 172.21.15.28:0/2364340308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 msgr2=0x56446c5e67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 0x56446c5e67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446c324620 con 0x56446c5e42e0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec036040 tx=0x7ff3ec009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39c30dd70277c3b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3deff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec04e070 con 0x56446c5e42e0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7ff3ec0046c0 con 0x56446c5e42e0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f009be80 con 0x56446c5e42e0 2024-03-22T02:36:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7ff3ec0048c0 con 0x56446c5e42e0 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff403c07740 10 client.30094 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.973+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56446c5fc650 con 0x56446c5e42e0 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f009be80 con 0x56446c5e42e0 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7ff3ec049070 con 0x56446c5e42e0 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 10 client.30094 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 10 client.30094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 20 client.30094 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c6001e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 4 client.30094 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56446c551580 con 0x56446c5fdd90 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c6001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff3deff5640 10 client.30094 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.974+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c6001e0 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fdbbf1c0852f133 server_cookie=1db61f8048c43cc6 in_seq=0 out_seq=0 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.978+0000 7ff3deff5640 1 -- 172.21.15.28:0/2364340308 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56446c551580 con 0x56446c5fdd90 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.978+0000 7ff3deff5640 10 client.30094 handle_command_reply: tid=0 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 client.30094 shutdown 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 msgr2=0x56446c5e6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec036040 tx=0x7ff3ec009260 comp rx=0 tx=0).stop 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 shutdown_connections 2024-03-22T02:36:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c6001e0 crc :-1 s=READY pgs=436 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5e7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e42e0 0x56446c5e6d30 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff3deff5640 0 client.30094 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 --2- 172.21.15.28:0/2364340308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e38f0 0x56446c5e67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 >> 172.21.15.28:0/2364340308 conn(0x56446c5daa50 msgr2=0x56446c5d9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 shutdown_connections 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 1 -- 172.21.15.28:0/2364340308 wait complete. 2024-03-22T02:36:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.979+0000 7ff403c07740 20 client.30094 trim_cache size 0 max 16384 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.995+0000 7ff403c07740 1 Processor -- start 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 -- start start 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 0x56446c5d93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5d9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.001 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1ab060 con 0x56446c5e42e0 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1a8930 con 0x56446c5e9450 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff403c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1a6780 con 0x56446c5e38f0 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 0x56446c5d93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58208/0 (socket says 172.21.15.28:58208) 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fa59c640 1 -- 172.21.15.28:0/1966680226 learned_addr learned my addr 172.21.15.28:0/1966680226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5d9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 -- 172.21.15.28:0/1966680226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 msgr2=0x56446c5d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5d9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 -- 172.21.15.28:0/1966680226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 msgr2=0x56446c5d9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 -- 172.21.15.28:0/1966680226 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446c5da380 con 0x56446c5e38f0 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 0x56446c5d93c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00cd5c0 tx=0x7ff3f0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90aae99646f1e8f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.996+0000 7ff3e0ff9640 1 -- 172.21.15.28:0/1966680226 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f009c580 con 0x56446c5e38f0 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff3e0ff9640 1 -- 172.21.15.28:0/1966680226 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f009be80 con 0x56446c5e38f0 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff3e0ff9640 1 -- 172.21.15.28:0/1966680226 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f00a4400 con 0x56446c5e38f0 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 -- 172.21.15.28:0/1966680226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 msgr2=0x56446c5d93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 0x56446c5d93c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00cd5c0 tx=0x7ff3f0000bc0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 -- 172.21.15.28:0/1966680226 shutdown_connections 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e9450 0x56446c5d9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5d9900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.003 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 --2- 172.21.15.28:0/1966680226 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e38f0 0x56446c5d93c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 -- 172.21.15.28:0/1966680226 >> 172.21.15.28:0/1966680226 conn(0x56446c5daa50 msgr2=0x56446c5e67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 -- 172.21.15.28:0/1966680226 shutdown_connections 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.997+0000 7ff403c07740 1 -- 172.21.15.28:0/1966680226 wait complete. 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 -- start start 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 Processor -- start 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5f7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1a6df0 con 0x56446c5e42e0 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1a8260 con 0x56446c5e38f0 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1a74c0 con 0x56446c5e9450 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff403c07740 10 client.0 _ll_register_callbacks cb 0x7ff3f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5f7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fad9d640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44904/0 (socket says 172.21.15.28:44904) 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44306/0 (socket says 172.21.15.28:44306) 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fad9d640 1 -- 172.21.15.28:0/40203061 learned_addr learned my addr 172.21.15.28:0/40203061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 -- 172.21.15.28:0/40203061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 msgr2=0x56446c5f8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 -- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 msgr2=0x56446c5f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 -- 172.21.15.28:0/40203061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446c5f94a0 con 0x56446c5e42e0 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fb59e640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:36.005 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3fa59c640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5f7ef0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec003870 tx=0x7ff3ec02cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc12752d0f7f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.998+0000 7ff3deff5640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3ec04e040 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7ff3ec004ef0 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3f009be80 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7ff3ec0050f0 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 10 client.20322 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x56446c664120 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3f009be80 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7ff3ec049040 con 0x56446c5e42e0 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 10 client.20322 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 10 client.20322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:36.006 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 20 client.20322 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c666a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 4 client.20322 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56446c5e65e0 con 0x56446c5fdd90 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:35.999+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c666a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:36.000+0000 7ff3fad9d640 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c666a40 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41bcc6af9a3e8742 server_cookie=88359a75cd8b1db9 in_seq=0 out_seq=0 2024-03-22T02:36:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:36.000+0000 7ff3deff5640 10 client.20322 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:36.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 161839 15.47 MB/sec execute 436 sec latency 555.968 ms 2024-03-22T02:36:36.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:36 smithi120 ceph-mon[36803]: pgmap v1052: 129 pgs: 129 active+clean; 289 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 559 op/s 2024-03-22T02:36:36.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:36 smithi083 ceph-mon[36340]: pgmap v1052: 129 pgs: 129 active+clean; 289 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 559 op/s 2024-03-22T02:36:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:36 smithi028 ceph-mon[31916]: pgmap v1052: 129 pgs: 129 active+clean; 289 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.4 MiB/s wr, 559 op/s 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff3deff5640 1 -- 172.21.15.28:0/40203061 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56446c5e65e0 con 0x56446c5fdd90 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff3deff5640 10 client.20322 handle_command_reply: tid=0 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 client.20322 shutdown 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 msgr2=0x56446c5f7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5f7ef0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec003870 tx=0x7ff3ec02cd00 comp rx=0 tx=0).stop 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 shutdown_connections 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x56446c5fdd90 0x56446c666a40 crc :-1 s=READY pgs=437 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56446c5e9450 0x56446c5f8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446c5e42e0 0x56446c5f7ef0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 --2- 172.21.15.28:0/40203061 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x56446c5e38f0 0x56446c5f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.127 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff3deff5640 0 client.20322 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 >> 172.21.15.28:0/40203061 conn(0x56446c5daa50 msgr2=0x56446c5d93c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 shutdown_connections 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 1 -- 172.21.15.28:0/40203061 wait complete. 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.125+0000 7ff403c07740 20 client.20322 trim_cache size 0 max 16384 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:36:37.128 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 msgr2=0x7ff3e40a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 0x7ff3e40a5500 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec005ab0 tx=0x7ff3ec02df90 comp rx=0 tx=0).stop 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 msgr2=0x7ff3fc197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e9c0 tx=0x7ff3f000ee80 comp rx=0 tx=0).stop 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 shutdown_connections 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff3e40a3040 0x7ff3e40a5500 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.129 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff3fc1038a0 0x7ff3fc19f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3fc102ee0 0x7ff3fc198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 --2- 172.21.15.28:0/2905127322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff3fc0ffe70 0x7ff3fc197e00 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 >> 172.21.15.28:0/2905127322 conn(0x7ff3fc0fb7e0 msgr2=0x7ff3fc1061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 shutdown_connections 2024-03-22T02:36:37.130 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:37.126+0000 7ff40101c640 1 -- 172.21.15.28:0/2905127322 wait complete. 2024-03-22T02:36:37.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 162228 15.50 MB/sec execute 437 sec latency 660.428 ms 2024-03-22T02:36:37.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:37 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:37.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:37 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:37.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:37 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:36:38.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 162517 15.49 MB/sec execute 438 sec latency 109.778 ms 2024-03-22T02:36:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:38 smithi120 ceph-mon[36803]: pgmap v1053: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2024-03-22T02:36:38.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:38.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:38.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:36:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:38 smithi083 ceph-mon[36340]: pgmap v1053: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2024-03-22T02:36:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:38.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:36:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:38 smithi028 ceph-mon[31916]: pgmap v1053: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.8 MiB/s wr, 356 op/s 2024-03-22T02:36:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:36:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:36:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:36:39.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 162706 15.48 MB/sec execute 439 sec latency 300.016 ms 2024-03-22T02:36:40.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 162965 15.48 MB/sec execute 440 sec latency 116.054 ms 2024-03-22T02:36:40.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:40 smithi120 ceph-mon[36803]: pgmap v1054: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-03-22T02:36:40.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:40 smithi083 ceph-mon[36340]: pgmap v1054: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-03-22T02:36:40.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:40 smithi028 ceph-mon[31916]: pgmap v1054: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.4 MiB/s wr, 362 op/s 2024-03-22T02:36:41.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 163195 15.47 MB/sec execute 441 sec latency 99.659 ms 2024-03-22T02:36:42.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 163499 15.48 MB/sec execute 442 sec latency 89.479 ms 2024-03-22T02:36:42.748 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:42 smithi120 ceph-mon[36803]: pgmap v1055: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-03-22T02:36:42.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:42 smithi083 ceph-mon[36340]: pgmap v1055: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-03-22T02:36:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:42 smithi028 ceph-mon[31916]: pgmap v1055: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.5 MiB/s wr, 520 op/s 2024-03-22T02:36:43.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 163780 15.47 MB/sec execute 443 sec latency 132.161 ms 2024-03-22T02:36:44.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 164007 15.45 MB/sec execute 444 sec latency 208.440 ms 2024-03-22T02:36:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:44 smithi083 ceph-mon[36340]: pgmap v1056: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2024-03-22T02:36:44.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:44 smithi028 ceph-mon[31916]: pgmap v1056: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2024-03-22T02:36:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:44 smithi120 ceph-mon[36803]: pgmap v1056: 129 pgs: 129 active+clean; 278 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.5 MiB/s wr, 496 op/s 2024-03-22T02:36:45.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 164209 15.44 MB/sec execute 445 sec latency 216.882 ms 2024-03-22T02:36:46.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 164488 15.43 MB/sec execute 446 sec latency 143.548 ms 2024-03-22T02:36:46.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:46 smithi083 ceph-mon[36340]: pgmap v1057: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-22T02:36:46.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:46 smithi028 ceph-mon[31916]: pgmap v1057: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-22T02:36:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:46 smithi120 ceph-mon[36803]: pgmap v1057: 129 pgs: 129 active+clean; 290 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 578 op/s 2024-03-22T02:36:47.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 164843 15.44 MB/sec execute 447 sec latency 94.029 ms 2024-03-22T02:36:48.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 165293 15.47 MB/sec execute 448 sec latency 108.294 ms 2024-03-22T02:36:48.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:48 smithi083 ceph-mon[36340]: pgmap v1058: 129 pgs: 129 active+clean; 291 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 511 op/s 2024-03-22T02:36:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:48 smithi028 ceph-mon[31916]: pgmap v1058: 129 pgs: 129 active+clean; 291 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 511 op/s 2024-03-22T02:36:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:48 smithi120 ceph-mon[36803]: pgmap v1058: 129 pgs: 129 active+clean; 291 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 511 op/s 2024-03-22T02:36:49.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 165727 15.50 MB/sec execute 449 sec latency 126.657 ms 2024-03-22T02:36:50.268 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/988493086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc100ac0 msgr2=0x7fb1dc100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/988493086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc100ee0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009900 tx=0x7fb1cc02bc30 comp rx=0 tx=0).stop 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/988493086 shutdown_connections 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/988493086 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc102670 0x7fb1dc108f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/988493086 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/988493086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc100ee0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/988493086 >> 172.21.15.28:0/988493086 conn(0x7fb1dc0fc030 msgr2=0x7fb1dc0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/988493086 shutdown_connections 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.265+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/988493086 wait complete. 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 Processor -- start 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 -- start start 2024-03-22T02:36:50.269 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc102670 0x7fb1dc19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10c740 con 0x7fb1dc101cb0 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10c5c0 con 0x7fb1dc102670 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1e1d4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc10c8c0 con 0x7fb1dc100ac0 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51912/0 (socket says 172.21.15.28:51912) 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/335261236 learned_addr learned my addr 172.21.15.28:0/335261236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc102670 0x7fb1dc19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.266+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/335261236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc100ac0 msgr2=0x7fb1dc197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/335261236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc102670 msgr2=0x7fb1dc19f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc102670 0x7fb1dc19f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/335261236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc19fa60 con 0x7fb1dc101cb0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400e9f0 tx=0x7fb1c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d97fa640 1 -- 172.21.15.28:0/335261236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c400cd80 con 0x7fb1dc101cb0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d97fa640 1 -- 172.21.15.28:0/335261236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1c4010910 con 0x7fb1dc101cb0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1d97fa640 1 -- 172.21.15.28:0/335261236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4018990 con 0x7fb1dc101cb0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc19fd50 con 0x7fb1dc101cb0 2024-03-22T02:36:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.267+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc1a0240 con 0x7fb1dc101cb0 2024-03-22T02:36:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.269+0000 7fb1d97fa640 1 -- 172.21.15.28:0/335261236 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb1c4010ac0 con 0x7fb1dc101cb0 2024-03-22T02:36:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 Processor -- start 2024-03-22T02:36:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 -- start start 2024-03-22T02:36:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe470900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 0x560bfe4784d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bfe3d5040 con 0x560bfe470ed0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560bfe37d9e0 con 0x560bfe4704e0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560bfe3d0ff0 con 0x560bfe476040 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe470900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51914/0 (socket says 172.21.15.28:51914) 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1d97fa640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 0x7fb1c80a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 -- 172.21.15.28:0/2794913729 learned_addr learned my addr 172.21.15.28:0/2794913729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 0x560bfe4784d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 0x7fb1c80a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 -- 172.21.15.28:0/2794913729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 msgr2=0x560bfe4784d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 0x560bfe4784d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 -- 172.21.15.28:0/2794913729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 msgr2=0x560bfe470900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe470900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 -- 172.21.15.28:0/2794913729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfe0e9210 con 0x560bfe470ed0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 0x560bfe4784d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1d97fa640 1 -- 172.21.15.28:0/335261236 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb1c40cfb10 con 0x7fb1dc101cb0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.270+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009900 tx=0x7fb1cc009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e109c67910214c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1d1ffb640 1 -- 172.21.15.28:0/2794913729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc02cb80 con 0x560bfe470ed0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 0x7fb1c80a5480 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x560bfe471730 tx=0x7fb1d4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1d1ffb640 1 -- 172.21.15.28:0/2794913729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1cc005040 con 0x560bfe470ed0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1d1ffb640 1 -- 172.21.15.28:0/2794913729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1cc004920 con 0x560bfe470ed0 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2794913729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 msgr2=0x560bfe4759d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fb1cc009900 tx=0x7fb1cc009560 comp rx=0 tx=0).stop 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2794913729 shutdown_connections 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe476040 0x560bfe4784d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe470ed0 0x560bfe4759d0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2794913729 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe470900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.271+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2794913729 >> 172.21.15.28:0/2794913729 conn(0x560bfe46ba50 msgr2=0x560bfe46be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2794913729 shutdown_connections 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2794913729 wait complete. 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- start start 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 Processor -- start 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe47a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe4819e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560bfe478f90 con 0x560bfe476040 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x560bfe390db0 con 0x560bfe4704e0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.272+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x560bfe479110 con 0x560bfe470ed0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1e4a0b740 10 client.0 _ll_register_callbacks cb 0x560bfe0e8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe4819e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:41580/0 (socket says 172.21.15.28:41580) 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/2474574529 learned_addr learned my addr 172.21.15.28:0/2474574529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe47a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 msgr2=0x560bfe47a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe47a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/2474574529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 msgr2=0x560bfe4819e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe4819e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 -- 172.21.15.28:0/2474574529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfe1b5620 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe4819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c6a0 tx=0x7fb1c409b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68788f9ee459945e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1baffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c40bad80 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1cc005040 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.273+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb1c4018b30 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb1c4018d30 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 10 client.30112 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560bfe484e70 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1cc005040 con 0x560bfe470ed0 2024-03-22T02:36:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb1c4097c10 con 0x560bfe470ed0 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 10 client.30112 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 10 client.30112 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 20 client.30112 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe486590 0x560bfe4889e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 4 client.30112 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560bfe3e2580 con 0x560bfe486590 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.274+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe486590 0x560bfe4889e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.275+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe486590 0x560bfe4889e0 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97df5c8732ec7cff server_cookie=aca3e4f9458e5940 in_seq=0 out_seq=0 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.275+0000 7fb1baffd640 10 client.30112 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1baffd640 1 -- 172.21.15.28:0/2474574529 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb1cc0509c0 con 0x560bfe486590 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1baffd640 10 client.30112 handle_command_reply: tid=0 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 client.30112 shutdown 2024-03-22T02:36:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 msgr2=0x560bfe47a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c6a0 tx=0x7fb1c409b920 comp rx=0 tx=0).stop 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 shutdown_connections 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe486590 0x560bfe4889e0 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe4819e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1baffd640 0 client.30112 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe47a9a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/2474574529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe47a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 >> 172.21.15.28:0/2474574529 conn(0x560bfe46ba50 msgr2=0x560bfe46ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 shutdown_connections 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/2474574529 wait complete. 2024-03-22T02:36:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.277+0000 7fb1e4a0b740 20 client.30112 trim_cache size 0 max 16384 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 Processor -- start 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 -- start start 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe4754c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4819a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1ad150 con 0x560bfe476040 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1aaa20 con 0x560bfe4704e0 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1a8870 con 0x560bfe470ed0 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4819a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51934/0 (socket says 172.21.15.28:51934) 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3799879871 learned_addr learned my addr 172.21.15.28:0/3799879871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe4754c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3799879871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 msgr2=0x560bfe4819a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4819a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3799879871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 msgr2=0x560bfe4754c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe4754c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3799879871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bfe471760 con 0x560bfe476040 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4819a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.294+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe4754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.295+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4003a20 tx=0x7fb1d4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25519a428b87484d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.295+0000 7fb1d17fa640 1 -- 172.21.15.28:0/3799879871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4026070 con 0x560bfe476040 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.295+0000 7fb1d17fa640 1 -- 172.21.15.28:0/3799879871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4004730 con 0x560bfe476040 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.295+0000 7fb1d17fa640 1 -- 172.21.15.28:0/3799879871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4022db0 con 0x560bfe476040 2024-03-22T02:36:50.303 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3799879871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 msgr2=0x560bfe46a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4003a20 tx=0x7fb1d4005bd0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3799879871 shutdown_connections 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe46a3c0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4819a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3799879871 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe4754c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3799879871 >> 172.21.15.28:0/3799879871 conn(0x560bfe46ba50 msgr2=0x560bfe46f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3799879871 shutdown_connections 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3799879871 wait complete. 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- start start 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 Processor -- start 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe481370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.304 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1a8ee0 con 0x560bfe476040 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1aa350 con 0x560bfe4704e0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1a95b0 con 0x560bfe470ed0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1e4a0b740 10 client.0 _ll_register_callbacks cb 0x7fb1c400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1d2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.296+0000 7fb1dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51950/0 (socket says 172.21.15.28:51950) 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3524899615 learned_addr learned my addr 172.21.15.28:0/3524899615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1db7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe481370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3524899615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 msgr2=0x560bfe4818b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4818b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.305 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 msgr2=0x560bfe481370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe481370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 -- 172.21.15.28:0/3524899615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bfe4801b0 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1d2ffd640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1dbfff640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4004a80 tx=0x7fb1d4004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cded9140181b3b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1baffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4026030 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1d4004730 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb1d4022d40 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fb1d403e050 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1e4a0b740 10 client.20340 fetch_fsmap learned FSMap version 46 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x560bfe4ed450 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.297+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1d4004730 con 0x560bfe476040 2024-03-22T02:36:50.306 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fb1d401f040 con 0x560bfe476040 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 10 client.20340 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 10 client.20340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 20 client.20340 populate_metadata read hostname 'smithi028' 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe4eddd0 0x560bfe4f01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 4 client.20340 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560bfe3dc8d0 con 0x560bfe4eddd0 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe4eddd0 0x560bfe4f01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1baffd640 10 client.20340 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:50.307 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:50.298+0000 7fb1db7fe640 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe4eddd0 0x560bfe4f01f0 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94c1cb62ab1a961a server_cookie=3852435ee041d388 in_seq=0 out_seq=0 2024-03-22T02:36:50.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 165903 15.48 MB/sec execute 450 sec latency 283.250 ms 2024-03-22T02:36:50.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:50 smithi083 ceph-mon[36340]: pgmap v1059: 129 pgs: 129 active+clean; 292 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 492 op/s 2024-03-22T02:36:50.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:50 smithi028 ceph-mon[31916]: pgmap v1059: 129 pgs: 129 active+clean; 292 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 492 op/s 2024-03-22T02:36:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:50 smithi120 ceph-mon[36803]: pgmap v1059: 129 pgs: 129 active+clean; 292 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 492 op/s 2024-03-22T02:36:51.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 166033 15.46 MB/sec execute 451 sec latency 524.705 ms 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.704+0000 7fb1baffd640 1 -- 172.21.15.28:0/3524899615 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560bfe3dc8d0 con 0x560bfe4eddd0 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.704+0000 7fb1baffd640 10 client.20340 handle_command_reply: tid=0 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.704+0000 7fb1e4a0b740 1 client.20340 shutdown 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.704+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 msgr2=0x560bfe47f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.704+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4004a80 tx=0x7fb1d4004970 comp rx=0 tx=0).stop 2024-03-22T02:36:51.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 shutdown_connections 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x560bfe4eddd0 0x560bfe4f01f0 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1baffd640 0 client.20340 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560bfe476040 0x560bfe47f980 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x560bfe470ed0 0x560bfe4818b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 --2- 172.21.15.28:0/3524899615 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x560bfe4704e0 0x560bfe481370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 >> 172.21.15.28:0/3524899615 conn(0x560bfe46ba50 msgr2=0x560bfe46f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 shutdown_connections 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 1 -- 172.21.15.28:0/3524899615 wait complete. 2024-03-22T02:36:51.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.705+0000 7fb1e4a0b740 20 client.20340 trim_cache size 0 max 16384 2024-03-22T02:36:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:36:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:36:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:36:51.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 msgr2=0x7fb1c80a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 0x7fb1c80a5480 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x560bfe471730 tx=0x7fb1d4004ab0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 msgr2=0x7fb1dc198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400e9f0 tx=0x7fb1c400eeb0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 shutdown_connections 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb1c80a2fc0 0x7fb1c80a5480 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb1dc102670 0x7fb1dc19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1dc101cb0 0x7fb1dc198380 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 --2- 172.21.15.28:0/335261236 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb1dc100ac0 0x7fb1dc197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 >> 172.21.15.28:0/335261236 conn(0x7fb1dc0fc030 msgr2=0x7fb1dc0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 shutdown_connections 2024-03-22T02:36:51.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:36:51.706+0000 7fb1e1d4b640 1 -- 172.21.15.28:0/335261236 wait complete. 2024-03-22T02:36:52.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 166300 15.46 MB/sec execute 452 sec latency 101.124 ms 2024-03-22T02:36:52.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:52 smithi083 ceph-mon[36340]: pgmap v1060: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 499 op/s 2024-03-22T02:36:52.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:52 smithi028 ceph-mon[31916]: pgmap v1060: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 499 op/s 2024-03-22T02:36:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:52 smithi120 ceph-mon[36803]: pgmap v1060: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.2 MiB/s wr, 499 op/s 2024-03-22T02:36:53.696 INFO:tasks.workunit.client.0.smithi028.stdout: 10 166583 15.48 MB/sec execute 453 sec latency 121.138 ms 2024-03-22T02:36:54.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 166816 15.46 MB/sec execute 454 sec latency 70.569 ms 2024-03-22T02:36:54.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:54 smithi083 ceph-mon[36340]: pgmap v1061: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 383 op/s 2024-03-22T02:36:54.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:54 smithi028 ceph-mon[31916]: pgmap v1061: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 383 op/s 2024-03-22T02:36:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:54 smithi120 ceph-mon[36803]: pgmap v1061: 129 pgs: 129 active+clean; 290 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 383 op/s 2024-03-22T02:36:55.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 166961 15.45 MB/sec execute 455 sec latency 447.421 ms 2024-03-22T02:36:56.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 167225 15.45 MB/sec execute 456 sec latency 99.837 ms 2024-03-22T02:36:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:56 smithi028 ceph-mon[31916]: pgmap v1062: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-03-22T02:36:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:56 smithi120 ceph-mon[36803]: pgmap v1062: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-03-22T02:36:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:56 smithi083 ceph-mon[36340]: pgmap v1062: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.1 MiB/s wr, 525 op/s 2024-03-22T02:36:57.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 167542 15.43 MB/sec execute 457 sec latency 86.655 ms 2024-03-22T02:36:58.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 167880 15.43 MB/sec execute 458 sec latency 84.450 ms 2024-03-22T02:36:58.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:36:58 smithi028 ceph-mon[31916]: pgmap v1063: 129 pgs: 129 active+clean; 285 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-03-22T02:36:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:36:58 smithi120 ceph-mon[36803]: pgmap v1063: 129 pgs: 129 active+clean; 285 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-03-22T02:36:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:36:58 smithi083 ceph-mon[36340]: pgmap v1063: 129 pgs: 129 active+clean; 285 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.2 MiB/s wr, 505 op/s 2024-03-22T02:36:59.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 168298 15.45 MB/sec execute 459 sec latency 92.047 ms 2024-03-22T02:37:00.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 168514 15.44 MB/sec execute 460 sec latency 341.561 ms 2024-03-22T02:37:00.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:00 smithi028 ceph-mon[31916]: pgmap v1064: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 525 op/s 2024-03-22T02:37:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:00 smithi120 ceph-mon[36803]: pgmap v1064: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 525 op/s 2024-03-22T02:37:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:00 smithi083 ceph-mon[36340]: pgmap v1064: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 525 op/s 2024-03-22T02:37:01.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 168957 15.47 MB/sec execute 461 sec latency 62.787 ms 2024-03-22T02:37:02.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 169375 15.49 MB/sec execute 462 sec latency 68.359 ms 2024-03-22T02:37:02.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:02 smithi028 ceph-mon[31916]: pgmap v1065: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 563 op/s 2024-03-22T02:37:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:02 smithi120 ceph-mon[36803]: pgmap v1065: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 563 op/s 2024-03-22T02:37:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:02 smithi083 ceph-mon[36340]: pgmap v1065: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.8 MiB/s wr, 563 op/s 2024-03-22T02:37:03.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 169718 15.49 MB/sec execute 463 sec latency 84.227 ms 2024-03-22T02:37:04.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170003 15.50 MB/sec execute 464 sec latency 100.383 ms 2024-03-22T02:37:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:04 smithi028 ceph-mon[31916]: pgmap v1066: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 473 op/s 2024-03-22T02:37:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:04 smithi120 ceph-mon[36803]: pgmap v1066: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 473 op/s 2024-03-22T02:37:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:04 smithi083 ceph-mon[36340]: pgmap v1066: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.0 MiB/s wr, 473 op/s 2024-03-22T02:37:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f736a7fc640 1 -- 172.21.15.28:0/3730359620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364032a00 con 0x7f736c1059a0 2024-03-22T02:37:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 -- 172.21.15.28:0/3730359620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c1059a0 msgr2=0x7f736c10c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 --2- 172.21.15.28:0/3730359620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c1059a0 0x7f736c10c230 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f7364009ca0 tx=0x7f736402c120 comp rx=0 tx=0).stop 2024-03-22T02:37:05.403 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 -- 172.21.15.28:0/3730359620 shutdown_connections 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 --2- 172.21.15.28:0/3730359620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c1059a0 0x7f736c10c230 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 --2- 172.21.15.28:0/3730359620 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c104fe0 0x7f736c105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 --2- 172.21.15.28:0/3730359620 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c103df0 0x7f736c104210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.401+0000 7f7371c8f640 1 -- 172.21.15.28:0/3730359620 >> 172.21.15.28:0/3730359620 conn(0x7f736c0ff380 msgr2=0x7f736c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.402+0000 7f7371c8f640 1 -- 172.21.15.28:0/3730359620 shutdown_connections 2024-03-22T02:37:05.404 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.402+0000 7f7371c8f640 1 -- 172.21.15.28:0/3730359620 wait complete. 2024-03-22T02:37:05.405 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.403+0000 7f7371c8f640 1 Processor -- start 2024-03-22T02:37:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 -- start start 2024-03-22T02:37:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 0x7f736c199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 0x7f736c1a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.406 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f736c10fb30 con 0x7f736c104fe0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f736c10f9b0 con 0x7f736c103df0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f7371c8f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f736c10fcb0 con 0x7f736c1059a0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 0x7f736c1a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 0x7f736c1a1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:54860/0 (socket says 172.21.15.28:54860) 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736bfff640 1 -- 172.21.15.28:0/610324444 learned_addr learned my addr 172.21.15.28:0/610324444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736b7fe640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 0x7f736c199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.407 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.404+0000 7f736affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48348/0 (socket says 172.21.15.28:48348) 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 -- 172.21.15.28:0/610324444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 msgr2=0x7f736c1a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 0x7f736c1a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 -- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 msgr2=0x7f736c19a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 -- 172.21.15.28:0/610324444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f736c1a1bf0 con 0x7f736c103df0 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736affd640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:05.408 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f736b7fe640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 0x7f736c199fd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f735c009fd0 tx=0x7f735c00b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.405+0000 7f7368ff9640 1 -- 172.21.15.28:0/610324444 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c015020 con 0x7f736c103df0 2024-03-22T02:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.406+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f736c1a1ee0 con 0x7f736c103df0 2024-03-22T02:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.406+0000 7f7368ff9640 1 -- 172.21.15.28:0/610324444 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c011d10 con 0x7f736c103df0 2024-03-22T02:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.406+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f736c1a23d0 con 0x7f736c103df0 2024-03-22T02:37:05.409 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.406+0000 7f7368ff9640 1 -- 172.21.15.28:0/610324444 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c00f2e0 con 0x7f736c103df0 2024-03-22T02:37:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.409+0000 7f7368ff9640 1 -- 172.21.15.28:0/610324444 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f735c018020 con 0x7f736c103df0 2024-03-22T02:37:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.410+0000 7f7368ff9640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 0x7f73540a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.410+0000 7f736affd640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 0x7f73540a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.410+0000 7f7368ff9640 1 -- 172.21.15.28:0/610324444 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f735c0cefc0 con 0x7f736c103df0 2024-03-22T02:37:05.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.411+0000 7f736affd640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 0x7f73540a5480 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f736c19b4e0 tx=0x7f736000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.437+0000 7f7374872740 1 Processor -- start 2024-03-22T02:37:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.437+0000 7f7374872740 1 -- start start 2024-03-22T02:37:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a249d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.439 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a274d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564062984040 con 0x564062a1f4e0 2024-03-22T02:37:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56406292c9e0 con 0x564062a1fed0 2024-03-22T02:37:05.440 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f7374872740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x56406297fff0 con 0x564062a25040 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f736b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f736b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48360/0 (socket says 172.21.15.28:48360) 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.438+0000 7f736b7fe640 1 -- 172.21.15.28:0/3077275186 learned_addr learned my addr 172.21.15.28:0/3077275186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 -- 172.21.15.28:0/3077275186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 msgr2=0x564062a274d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a274d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 -- 172.21.15.28:0/3077275186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 msgr2=0x564062a249d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.441 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a249d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 -- 172.21.15.28:0/3077275186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564062698210 con 0x564062a1f4e0 2024-03-22T02:37:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.439+0000 7f736b7fe640 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f735c09b270 tx=0x7f735c09aee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a305084cb2b64005 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.442 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.440+0000 7f7351ffb640 1 -- 172.21.15.28:0/3077275186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c09b2e0 con 0x564062a1f4e0 2024-03-22T02:37:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.440+0000 7f7351ffb640 1 -- 172.21.15.28:0/3077275186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f735c095e50 con 0x564062a1f4e0 2024-03-22T02:37:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.440+0000 7f7351ffb640 1 -- 172.21.15.28:0/3077275186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f735c0b4210 con 0x564062a1f4e0 2024-03-22T02:37:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.441+0000 7f7374872740 1 -- 172.21.15.28:0/3077275186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 msgr2=0x564062a1f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.441+0000 7f7374872740 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f735c09b270 tx=0x7f735c09aee0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 -- 172.21.15.28:0/3077275186 shutdown_connections 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a274d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a249d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 --2- 172.21.15.28:0/3077275186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a1f900 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 -- 172.21.15.28:0/3077275186 >> 172.21.15.28:0/3077275186 conn(0x564062a1aa50 msgr2=0x564062a1ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 -- 172.21.15.28:0/3077275186 shutdown_connections 2024-03-22T02:37:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.442+0000 7f7374872740 1 -- 172.21.15.28:0/3077275186 wait complete. 2024-03-22T02:37:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.451+0000 7f7374872740 1 -- start start 2024-03-22T02:37:05.453 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.451+0000 7f7374872740 1 Processor -- start 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1f4e0 0x564062a31df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a39370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564062a27ff0 con 0x564062a1fed0 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x56406293fdb0 con 0x564062a1f4e0 2024-03-22T02:37:05.454 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x564062a28170 con 0x564062a25040 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f7374872740 10 client.0 _ll_register_callbacks cb 0x564062697a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f736affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f736b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1f4e0 0x564062a31df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f736affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48370/0 (socket says 172.21.15.28:48370) 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.452+0000 7f736affd640 1 -- 172.21.15.28:0/190379276 learned_addr learned my addr 172.21.15.28:0/190379276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.453+0000 7f736bfff640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a39370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.455 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 -- 172.21.15.28:0/190379276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 msgr2=0x564062a39370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a39370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 -- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1f4e0 msgr2=0x564062a31df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1f4e0 0x564062a31df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 -- 172.21.15.28:0/190379276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564062764620 con 0x564062a1fed0 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f736affd640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7360010e20 tx=0x7f73600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3d960463930efd server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f734affd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.454+0000 7f734affd640 1 -- 172.21.15.28:0/190379276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360016070 con 0x564062a1fed0 2024-03-22T02:37:05.456 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.455+0000 7f734affd640 1 -- 172.21.15.28:0/190379276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f7360027680 con 0x564062a1fed0 2024-03-22T02:37:05.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.463+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f735c095e50 con 0x564062a1fed0 2024-03-22T02:37:05.465 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.464+0000 7f734affd640 1 -- 172.21.15.28:0/190379276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7360027880 con 0x564062a1fed0 2024-03-22T02:37:05.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.464+0000 7f7374872740 10 client.20352 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:05.466 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.464+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=46}) -- 0x564062a2c0f0 con 0x564062a1fed0 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f734affd640 1 -- 172.21.15.28:0/190379276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7360027a20 con 0x564062a1fed0 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 10 client.20352 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 10 client.20352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 20 client.20352 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062a2d690 0x564062a2faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 4 client.20352 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:37:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.466+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564062991580 con 0x564062a2d690 2024-03-22T02:37:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.468+0000 7f736b7fe640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062a2d690 0x564062a2faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.470+0000 7f736b7fe640 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062a2d690 0x564062a2faf0 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed9355c7917fa7ff server_cookie=172e2091ce0079a2 in_seq=0 out_seq=0 2024-03-22T02:37:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.470+0000 7f734affd640 10 client.20352 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f734affd640 1 -- 172.21.15.28:0/190379276 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564062991580 con 0x564062a2d690 2024-03-22T02:37:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f734affd640 10 client.20352 handle_command_reply: tid=0 2024-03-22T02:37:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 client.20352 shutdown 2024-03-22T02:37:05.476 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 msgr2=0x564062a32330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f7360010e20 tx=0x7f73600046c0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 shutdown_connections 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062a2d690 0x564062a2faf0 crc :-1 s=READY pgs=440 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a39370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1fed0 0x564062a32330 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.474+0000 7f7374872740 1 --2- 172.21.15.28:0/190379276 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1f4e0 0x564062a31df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.475+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 >> 172.21.15.28:0/190379276 conn(0x564062a1aa50 msgr2=0x564062a19c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.475+0000 7f734affd640 0 client.20352 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:05.477 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.475+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 shutdown_connections 2024-03-22T02:37:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.476+0000 7f7374872740 1 -- 172.21.15.28:0/190379276 wait complete. 2024-03-22T02:37:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.476+0000 7f7374872740 20 client.20352 trim_cache size 0 max 16384 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 Processor -- start 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 -- start start 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a1f4e0 0x564062a2f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1ab140 con 0x564062a25040 2024-03-22T02:37:05.507 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a8a10 con 0x564062a1fed0 2024-03-22T02:37:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.505+0000 7f7374872740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a6860 con 0x564062a1f4e0 2024-03-22T02:37:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48382/0 (socket says 172.21.15.28:48382) 2024-03-22T02:37:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 -- 172.21.15.28:0/914037417 learned_addr learned my addr 172.21.15.28:0/914037417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 -- 172.21.15.28:0/914037417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a1f4e0 msgr2=0x564062a2f6a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a1f4e0 0x564062a2f6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 -- 172.21.15.28:0/914037417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 msgr2=0x564062a39330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.506+0000 7f736bfff640 1 -- 172.21.15.28:0/914037417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564062a1b7c0 con 0x564062a25040 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.507+0000 7f736bfff640 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7364037040 tx=0x7f73640323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94399af8532fca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.507+0000 7f73517fa640 1 -- 172.21.15.28:0/914037417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364044070 con 0x564062a25040 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.507+0000 7f73517fa640 1 -- 172.21.15.28:0/914037417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7364003970 con 0x564062a25040 2024-03-22T02:37:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.507+0000 7f73517fa640 1 -- 172.21.15.28:0/914037417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7364039040 con 0x564062a25040 2024-03-22T02:37:05.510 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.508+0000 7f7374872740 1 -- 172.21.15.28:0/914037417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 msgr2=0x564062a20760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.508+0000 7f7374872740 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f7364037040 tx=0x7f73640323f0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.508+0000 7f7374872740 1 -- 172.21.15.28:0/914037417 shutdown_connections 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.509+0000 7f7374872740 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a25040 0x564062a20760 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.509+0000 7f7374872740 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.509+0000 7f7374872740 1 --2- 172.21.15.28:0/914037417 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a1f4e0 0x564062a2f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.509+0000 7f7374872740 1 -- 172.21.15.28:0/914037417 >> 172.21.15.28:0/914037417 conn(0x564062a1aa50 msgr2=0x564062a1e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:05.514 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.513+0000 7f7374872740 1 -- 172.21.15.28:0/914037417 shutdown_connections 2024-03-22T02:37:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.515+0000 7f7374872740 1 -- 172.21.15.28:0/914037417 wait complete. 2024-03-22T02:37:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.516+0000 7f7374872740 1 -- start start 2024-03-22T02:37:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.516+0000 7f7374872740 1 Processor -- start 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.517+0000 7f7374872740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a38df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a359f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a6ed0 con 0x564062a1f4e0 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a8340 con 0x564062a1fed0 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f736c1a75a0 con 0x564062a25040 2024-03-22T02:37:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.518+0000 7f7374872740 10 client.0 _ll_register_callbacks cb 0x7f736000ac00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.519+0000 7f736bfff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a359f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.519+0000 7f736affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.519+0000 7f736affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44386/0 (socket says 172.21.15.28:44386) 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.519+0000 7f736b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a38df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.519+0000 7f736affd640 1 -- 172.21.15.28:0/2464735178 learned_addr learned my addr 172.21.15.28:0/2464735178 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 -- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 msgr2=0x564062a39330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 -- 172.21.15.28:0/2464735178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 msgr2=0x564062a38df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a38df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 -- 172.21.15.28:0/2464735178 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564062a36220 con 0x564062a25040 2024-03-22T02:37:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f736bfff640 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a359f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f736402c730 tx=0x7f7364032930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27057ee59786ca2a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.520+0000 7f734affd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:37:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.521+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736404f070 con 0x564062a25040 2024-03-22T02:37:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.521+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f73640042f0 con 0x564062a25040 2024-03-22T02:37:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.521+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7364003970 con 0x564062a25040 2024-03-22T02:37:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.522+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f7364059050 con 0x564062a25040 2024-03-22T02:37:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.522+0000 7f7374872740 10 client.30127 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.522+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x564062aacdf0 con 0x564062a25040 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7364003970 con 0x564062a25040 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f7364040c90 con 0x564062a25040 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 10 client.30127 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 10 client.30127 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 20 client.30127 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062aad740 0x564062aafb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:05.527 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 4 client.30127 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:37:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.525+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56406298b8d0 con 0x564062aad740 2024-03-22T02:37:05.528 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.526+0000 7f736b7fe640 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062aad740 0x564062aafb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.531+0000 7f736b7fe640 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062aad740 0x564062aafb60 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74fe8de414327028 server_cookie=ee3965d53bb73510 in_seq=0 out_seq=0 2024-03-22T02:37:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:05.531+0000 7f734affd640 10 client.30127 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:05.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170113 15.48 MB/sec execute 465 sec latency 418.663 ms 2024-03-22T02:37:06.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170134 15.45 MB/sec execute 466 sec latency 1094.997 ms 2024-03-22T02:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.761+0000 7f734affd640 1 -- 172.21.15.28:0/2464735178 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56406298b8d0 con 0x564062aad740 2024-03-22T02:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.761+0000 7f734affd640 10 client.30127 handle_command_reply: tid=0 2024-03-22T02:37:06.764 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.761+0000 7f7374872740 1 client.30127 shutdown 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.761+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 msgr2=0x564062a359f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.761+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a359f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f736402c730 tx=0x7f7364032930 comp rx=0 tx=0).stop 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 shutdown_connections 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x564062aad740 0x564062aafb60 crc :-1 s=READY pgs=441 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x564062a25040 0x564062a359f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f734affd640 0 client.30127 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x564062a1fed0 0x564062a39330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 --2- 172.21.15.28:0/2464735178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564062a1f4e0 0x564062a38df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 >> 172.21.15.28:0/2464735178 conn(0x564062a1aa50 msgr2=0x564062a193c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 shutdown_connections 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 1 -- 172.21.15.28:0/2464735178 wait complete. 2024-03-22T02:37:06.765 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.762+0000 7f7374872740 20 client.30127 trim_cache size 0 max 16384 2024-03-22T02:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:37:06.766 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.764+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 msgr2=0x7f73540a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.764+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 0x7f73540a5480 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f736c19b4e0 tx=0x7f736000a2d0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.764+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 msgr2=0x7f736c199fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.764+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 0x7f736c199fd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f735c009fd0 tx=0x7f735c00b7b0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.764+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 shutdown_connections 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f73540a2fc0 0x7f73540a5480 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f736c1059a0 0x7f736c1a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f736c104fe0 0x7f736c19a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 --2- 172.21.15.28:0/610324444 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f736c103df0 0x7f736c199fd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 >> 172.21.15.28:0/610324444 conn(0x7f736c0ff380 msgr2=0x7f736c108210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 shutdown_connections 2024-03-22T02:37:06.768 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:06.765+0000 7f7371c8f640 1 -- 172.21.15.28:0/610324444 wait complete. 2024-03-22T02:37:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:06 smithi028 ceph-mon[31916]: pgmap v1067: 129 pgs: 129 active+clean; 294 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-03-22T02:37:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:06 smithi120 ceph-mon[36803]: pgmap v1067: 129 pgs: 129 active+clean; 294 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-03-22T02:37:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:06 smithi083 ceph-mon[36340]: pgmap v1067: 129 pgs: 129 active+clean; 294 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.9 MiB/s wr, 510 op/s 2024-03-22T02:37:07.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170335 15.43 MB/sec execute 467 sec latency 875.552 ms 2024-03-22T02:37:08.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170620 15.44 MB/sec execute 468 sec latency 66.544 ms 2024-03-22T02:37:08.799 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:08 smithi120 ceph-mon[36803]: pgmap v1068: 129 pgs: 129 active+clean; 298 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.6 MiB/s wr, 399 op/s 2024-03-22T02:37:08.816 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:08 smithi083 ceph-mon[36340]: pgmap v1068: 129 pgs: 129 active+clean; 298 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.6 MiB/s wr, 399 op/s 2024-03-22T02:37:08.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:08 smithi028 ceph-mon[31916]: pgmap v1068: 129 pgs: 129 active+clean; 298 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.6 MiB/s wr, 399 op/s 2024-03-22T02:37:09.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 170919 15.43 MB/sec execute 469 sec latency 81.937 ms 2024-03-22T02:37:10.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 171132 15.41 MB/sec execute 470 sec latency 225.732 ms 2024-03-22T02:37:10.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:10 smithi028 ceph-mon[31916]: pgmap v1069: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 411 op/s 2024-03-22T02:37:10.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:10 smithi120 ceph-mon[36803]: pgmap v1069: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 411 op/s 2024-03-22T02:37:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:10 smithi083 ceph-mon[36340]: pgmap v1069: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.4 MiB/s wr, 411 op/s 2024-03-22T02:37:11.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 171406 15.41 MB/sec execute 471 sec latency 106.123 ms 2024-03-22T02:37:12.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 171768 15.41 MB/sec execute 472 sec latency 106.518 ms 2024-03-22T02:37:12.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:12 smithi028 ceph-mon[31916]: pgmap v1070: 129 pgs: 129 active+clean; 282 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 506 op/s 2024-03-22T02:37:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:12 smithi120 ceph-mon[36803]: pgmap v1070: 129 pgs: 129 active+clean; 282 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 506 op/s 2024-03-22T02:37:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:12 smithi083 ceph-mon[36340]: pgmap v1070: 129 pgs: 129 active+clean; 282 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.3 MiB/s wr, 506 op/s 2024-03-22T02:37:13.697 INFO:tasks.workunit.client.0.smithi028.stdout: 10 172129 15.43 MB/sec execute 473 sec latency 90.086 ms 2024-03-22T02:37:14.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 172610 15.46 MB/sec execute 474 sec latency 106.956 ms 2024-03-22T02:37:14.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:14 smithi028 ceph-mon[31916]: pgmap v1071: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 515 op/s 2024-03-22T02:37:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:14 smithi120 ceph-mon[36803]: pgmap v1071: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 515 op/s 2024-03-22T02:37:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:14 smithi083 ceph-mon[36340]: pgmap v1071: 129 pgs: 129 active+clean; 287 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 515 op/s 2024-03-22T02:37:15.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 172861 15.46 MB/sec execute 475 sec latency 257.647 ms 2024-03-22T02:37:16.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 173170 15.46 MB/sec execute 476 sec latency 91.256 ms 2024-03-22T02:37:16.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:16 smithi028 ceph-mon[31916]: pgmap v1072: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:37:16.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:16 smithi120 ceph-mon[36803]: pgmap v1072: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:37:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:16 smithi083 ceph-mon[36340]: pgmap v1072: 129 pgs: 129 active+clean; 288 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 538 op/s 2024-03-22T02:37:17.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 173408 15.46 MB/sec execute 477 sec latency 117.010 ms 2024-03-22T02:37:18.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 173703 15.46 MB/sec execute 478 sec latency 73.039 ms 2024-03-22T02:37:18.821 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:18 smithi028 ceph-mon[31916]: pgmap v1073: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 532 op/s 2024-03-22T02:37:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:18 smithi120 ceph-mon[36803]: pgmap v1073: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 532 op/s 2024-03-22T02:37:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:18 smithi083 ceph-mon[36340]: pgmap v1073: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 4.0 MiB/s wr, 532 op/s 2024-03-22T02:37:19.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 173965 15.46 MB/sec execute 479 sec latency 83.963 ms 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3920490637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0101ce0 msgr2=0x7fd0a0102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3920490637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0101ce0 0x7fd0a0102160 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd094009900 tx=0x7fd09402bc30 comp rx=0 tx=0).stop 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3920490637 shutdown_connections 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3920490637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 0x7fd0a0108f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3920490637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0101ce0 0x7fd0a0102160 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.889 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3920490637 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0100af0 0x7fd0a0100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3920490637 >> 172.21.15.28:0/3920490637 conn(0x7fd0a00fc060 msgr2=0x7fd0a00fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3920490637 shutdown_connections 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3920490637 wait complete. 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 Processor -- start 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.886+0000 7fd0a7ca3640 1 -- start start 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0101ce0 0x7fd0a019caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 0x7fd0a019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a010c770 con 0x7fd0a0100af0 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a010c5f0 con 0x7fd0a0101ce0 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a010c8f0 con 0x7fd0a01026a0 2024-03-22T02:37:19.890 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52654/0 (socket says 172.21.15.28:52654) 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3578094501 learned_addr learned my addr 172.21.15.28:0/3578094501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 0x7fd0a019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a523e640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0101ce0 0x7fd0a019caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3578094501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 msgr2=0x7fd0a019f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 0x7fd0a019f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3578094501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0101ce0 msgr2=0x7fd0a019caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0101ce0 0x7fd0a019caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3578094501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0a019f9f0 con 0x7fd0a0100af0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fd09800e9c0 tx=0x7fd09800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd09effd640 1 -- 172.21.15.28:0/3578094501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09800cd80 con 0x7fd0a0100af0 2024-03-22T02:37:19.891 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd09effd640 1 -- 172.21.15.28:0/3578094501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0980035f0 con 0x7fd0a0100af0 2024-03-22T02:37:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0a019fce0 con 0x7fd0a0100af0 2024-03-22T02:37:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd09effd640 1 -- 172.21.15.28:0/3578094501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd098010690 con 0x7fd0a0100af0 2024-03-22T02:37:19.892 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.887+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0a01a01f0 con 0x7fd0a0100af0 2024-03-22T02:37:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.889+0000 7fd09effd640 1 -- 172.21.15.28:0/3578094501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd0980037a0 con 0x7fd0a0100af0 2024-03-22T02:37:19.898 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 Processor -- start 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 -- start start 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd09effd640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 0x7fd0880a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559848996ed0 0x55984899b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x55984899e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5598488fb040 con 0x55984899c040 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5598488a39e0 con 0x559848996ed0 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0aa885740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x5598488f6ff0 con 0x5598489964e0 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd09effd640 1 -- 172.21.15.28:0/3578094501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd0980cfb60 con 0x7fd0a0100af0 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.890+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 0x7fd0880a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.899 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a6240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x55984899e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a523e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559848996ed0 0x55984899b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58710/0 (socket says 172.21.15.28:58710) 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/1948822295 learned_addr learned my addr 172.21.15.28:0/1948822295 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/1948822295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559848996ed0 msgr2=0x55984899b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559848996ed0 0x55984899b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/1948822295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 msgr2=0x55984899e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x55984899e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/1948822295 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55984860f210 con 0x5598489964e0 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 0x7fd0880a5500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd090005fd0 tx=0x7fd090004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.900 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd0980a4360 tx=0x7fd09809ba70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f1a9d275c21826 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd087fff640 1 -- 172.21.15.28:0/1948822295 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0980bc070 con 0x5598489964e0 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd087fff640 1 -- 172.21.15.28:0/1948822295 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd09809be80 con 0x5598489964e0 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.891+0000 7fd087fff640 1 -- 172.21.15.28:0/1948822295 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0980984f0 con 0x5598489964e0 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 -- 172.21.15.28:0/1948822295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 msgr2=0x559848996900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd0980a4360 tx=0x7fd09809ba70 comp rx=0 tx=0).stop 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 -- 172.21.15.28:0/1948822295 shutdown_connections 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x55984899e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x559848996ed0 0x55984899b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 --2- 172.21.15.28:0/1948822295 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x559848996900 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 -- 172.21.15.28:0/1948822295 >> 172.21.15.28:0/1948822295 conn(0x559848991a50 msgr2=0x559848991e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:19.901 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 -- 172.21.15.28:0/1948822295 shutdown_connections 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.892+0000 7fd0aa885740 1 -- 172.21.15.28:0/1948822295 wait complete. 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 -- start start 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 Processor -- start 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489a79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55984899ef90 con 0x559848996ed0 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x5598488b6db0 con 0x55984899c040 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55984899f110 con 0x5598489964e0 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0aa885740 10 client.0 _ll_register_callbacks cb 0x55984860ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:19.902 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a523e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:58720/0 (socket says 172.21.15.28:58720) 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3967525710 learned_addr learned my addr 172.21.15.28:0/3967525710 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489a79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 msgr2=0x5598489a79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489a79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3967525710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 msgr2=0x5598489a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a5a3f640 1 -- 172.21.15.28:0/3967525710 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598486db620 con 0x5598489964e0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.893+0000 7fd0a523e640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd09808e270 tx=0x7fd0980a49d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9019a4e146cb6d0a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:37:19.903 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09808e4f0 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd09809b2d0 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd09809be80 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd0980bc070 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 10 client.30139 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55984899ef90 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd09809be80 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fd0980a1160 con 0x5598489964e0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 10 client.30139 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 10 client.30139 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 20 client.30139 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5598489abf50 0x5598489ae340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 4 client.30139 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.894+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559848908580 con 0x5598489abf50 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.895+0000 7fd0a523e640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5598489abf50 0x5598489ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.904 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.895+0000 7fd0a523e640 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5598489abf50 0x5598489ae340 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd73358637b5227e server_cookie=f248885526beb23a in_seq=0 out_seq=0 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.895+0000 7fd0857fa640 10 client.30139 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0857fa640 1 -- 172.21.15.28:0/3967525710 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559848908580 con 0x5598489abf50 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0857fa640 10 client.30139 handle_command_reply: tid=0 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 client.30139 shutdown 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 msgr2=0x5598489a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd09808e270 tx=0x7fd0980a49d0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 shutdown_connections 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x5598489abf50 0x5598489ae340 crc :-1 s=READY pgs=442 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489a79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0857fa640 0 client.30139 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3967525710 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a0420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.896+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 >> 172.21.15.28:0/3967525710 conn(0x559848991a50 msgr2=0x559848990c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:19.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.897+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 shutdown_connections 2024-03-22T02:37:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.897+0000 7fd0aa885740 1 -- 172.21.15.28:0/3967525710 wait complete. 2024-03-22T02:37:19.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.897+0000 7fd0aa885740 20 client.30139 trim_cache size 0 max 16384 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 Processor -- start 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 -- start start 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489953c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01ad220 con 0x559848996ed0 2024-03-22T02:37:19.920 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01aaaf0 con 0x55984899c040 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0aa885740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01a8940 con 0x5598489964e0 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0a523e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0a523e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52682/0 (socket says 172.21.15.28:52682) 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0a5a3f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0a523e640 1 -- 172.21.15.28:0/2908474764 learned_addr learned my addr 172.21.15.28:0/2908474764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.918+0000 7fd0a6240640 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 -- 172.21.15.28:0/2908474764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 msgr2=0x5598489a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 -- 172.21.15.28:0/2908474764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 msgr2=0x5598489953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.921 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 -- 172.21.15.28:0/2908474764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55984899b120 con 0x559848996ed0 2024-03-22T02:37:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0a523e640 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd09402c7b0 tx=0x7fd094005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2661d3b0a79b2d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0877fe640 1 -- 172.21.15.28:0/2908474764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094005d50 con 0x559848996ed0 2024-03-22T02:37:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0877fe640 1 -- 172.21.15.28:0/2908474764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd094046d10 con 0x559848996ed0 2024-03-22T02:37:19.922 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.919+0000 7fd0877fe640 1 -- 172.21.15.28:0/2908474764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094005460 con 0x559848996ed0 2024-03-22T02:37:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- 172.21.15.28:0/2908474764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 msgr2=0x5598489a7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:19.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fd09402c7b0 tx=0x7fd094005fd0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- 172.21.15.28:0/2908474764 shutdown_connections 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55984899c040 0x5598489953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559848996ed0 0x5598489a7960 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 --2- 172.21.15.28:0/2908474764 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5598489964e0 0x5598489a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- 172.21.15.28:0/2908474764 >> 172.21.15.28:0/2908474764 conn(0x559848991a50 msgr2=0x559848994cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- 172.21.15.28:0/2908474764 shutdown_connections 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- 172.21.15.28:0/2908474764 wait complete. 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 -- start start 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.920+0000 7fd0aa885740 1 Processor -- start 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5598489964e0 0x5598489a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559848996ed0 0x5598489a5350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01a8fb0 con 0x55984899c040 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01aa420 con 0x5598489964e0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd0a01a9680 con 0x559848996ed0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0aa885740 10 client.0 _ll_register_callbacks cb 0x7fd0900093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52698/0 (socket says 172.21.15.28:52698) 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a523e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559848996ed0 0x5598489a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 -- 172.21.15.28:0/3371267403 learned_addr learned my addr 172.21.15.28:0/3371267403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5598489964e0 0x5598489a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 -- 172.21.15.28:0/3371267403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559848996ed0 msgr2=0x5598489a5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559848996ed0 0x5598489a5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 -- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5598489964e0 msgr2=0x5598489a4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5598489964e0 0x5598489a4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 -- 172.21.15.28:0/3371267403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598489a6950 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0a6240640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd0900206b0 tx=0x7fd090004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a79ea660a325d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0857fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.921+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd090019040 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd090004b20 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd094046d10 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd090037050 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 10 client.20376 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x5598489a7cd0 con 0x55984899c040 2024-03-22T02:37:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd094046d10 con 0x55984899c040 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fd090004610 con 0x55984899c040 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 10 client.20376 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 10 client.20376 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 20 client.20376 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559848a12a90 0x559848a14eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 4 client.20376 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.922+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5598489028d0 con 0x559848a12a90 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.923+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559848a12a90 0x559848a14eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.924+0000 7fd0857fa640 10 client.20376 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:19.924+0000 7fd0a5a3f640 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559848a12a90 0x559848a14eb0 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c13e33973064855b server_cookie=3adbf3d9915d890b in_seq=0 out_seq=0 2024-03-22T02:37:20.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 174085 15.44 MB/sec execute 480 sec latency 578.891 ms 2024-03-22T02:37:20.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:20 smithi028 ceph-mon[31916]: pgmap v1074: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-03-22T02:37:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:20 smithi120 ceph-mon[36803]: pgmap v1074: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-03-22T02:37:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:20 smithi083 ceph-mon[36340]: pgmap v1074: 129 pgs: 129 active+clean; 287 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.5 MiB/s wr, 472 op/s 2024-03-22T02:37:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0857fa640 1 -- 172.21.15.28:0/3371267403 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5598489028d0 con 0x559848a12a90 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0857fa640 10 client.20376 handle_command_reply: tid=0 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 client.20376 shutdown 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 msgr2=0x5598489a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fd0900206b0 tx=0x7fd090004a40 comp rx=0 tx=0).stop 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 shutdown_connections 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x559848a12a90 0x559848a14eb0 crc :-1 s=READY pgs=443 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0857fa640 0 client.20376 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55984899c040 0x5598489a6360 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559848996ed0 0x5598489a5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 --2- 172.21.15.28:0/3371267403 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x5598489964e0 0x5598489a4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 >> 172.21.15.28:0/3371267403 conn(0x559848991a50 msgr2=0x5598489a06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 shutdown_connections 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.551+0000 7fd0aa885740 1 -- 172.21.15.28:0/3371267403 wait complete. 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.552+0000 7fd0aa885740 20 client.20376 trim_cache size 0 max 16384 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:37:21.554 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 msgr2=0x7fd0880a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 0x7fd0880a5500 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fd090005fd0 tx=0x7fd090004ab0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 msgr2=0x7fd0a006a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7fd09800e9c0 tx=0x7fd09800ee80 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 shutdown_connections 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd0880a3040 0x7fd0880a5500 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0a01026a0 0x7fd0a019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd0a0101ce0 0x7fd0a019caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 --2- 172.21.15.28:0/3578094501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0a0100af0 0x7fd0a006a950 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 >> 172.21.15.28:0/3578094501 conn(0x7fd0a00fc060 msgr2=0x7fd0a0104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 shutdown_connections 2024-03-22T02:37:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:21.553+0000 7fd0a7ca3640 1 -- 172.21.15.28:0/3578094501 wait complete. 2024-03-22T02:37:21.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 174163 15.42 MB/sec execute 481 sec latency 1251.745 ms 2024-03-22T02:37:22.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 174399 15.41 MB/sec execute 482 sec latency 141.566 ms 2024-03-22T02:37:22.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:22 smithi028 ceph-mon[31916]: pgmap v1075: 129 pgs: 129 active+clean; 290 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 459 op/s 2024-03-22T02:37:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:22 smithi120 ceph-mon[36803]: pgmap v1075: 129 pgs: 129 active+clean; 290 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 459 op/s 2024-03-22T02:37:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:22 smithi083 ceph-mon[36340]: pgmap v1075: 129 pgs: 129 active+clean; 290 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.7 MiB/s wr, 459 op/s 2024-03-22T02:37:23.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 174666 15.39 MB/sec execute 483 sec latency 93.747 ms 2024-03-22T02:37:24.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 175037 15.39 MB/sec execute 484 sec latency 101.508 ms 2024-03-22T02:37:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:24 smithi120 ceph-mon[36803]: pgmap v1076: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2024-03-22T02:37:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:24 smithi083 ceph-mon[36340]: pgmap v1076: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2024-03-22T02:37:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:24 smithi028 ceph-mon[31916]: pgmap v1076: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.8 MiB/s wr, 411 op/s 2024-03-22T02:37:25.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 175355 15.40 MB/sec execute 485 sec latency 256.720 ms 2024-03-22T02:37:26.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 175715 15.42 MB/sec execute 486 sec latency 124.499 ms 2024-03-22T02:37:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:26 smithi120 ceph-mon[36803]: pgmap v1077: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 491 op/s 2024-03-22T02:37:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:26 smithi083 ceph-mon[36340]: pgmap v1077: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 491 op/s 2024-03-22T02:37:27.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:26 smithi028 ceph-mon[31916]: pgmap v1077: 129 pgs: 129 active+clean; 295 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 491 op/s 2024-03-22T02:37:27.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 176169 15.44 MB/sec execute 487 sec latency 106.655 ms 2024-03-22T02:37:28.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 176522 15.45 MB/sec execute 488 sec latency 103.334 ms 2024-03-22T02:37:28.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:28 smithi120 ceph-mon[36803]: pgmap v1078: 129 pgs: 129 active+clean; 285 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:37:29.018 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:28 smithi083 ceph-mon[36340]: pgmap v1078: 129 pgs: 129 active+clean; 285 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:37:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:28 smithi028 ceph-mon[31916]: pgmap v1078: 129 pgs: 129 active+clean; 285 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 553 op/s 2024-03-22T02:37:29.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 176863 15.46 MB/sec execute 489 sec latency 133.525 ms 2024-03-22T02:37:30.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 177078 15.45 MB/sec execute 490 sec latency 217.778 ms 2024-03-22T02:37:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:30 smithi120 ceph-mon[36803]: pgmap v1079: 129 pgs: 129 active+clean; 289 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-22T02:37:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:30 smithi083 ceph-mon[36340]: pgmap v1079: 129 pgs: 129 active+clean; 289 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-22T02:37:31.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:30 smithi028 ceph-mon[31916]: pgmap v1079: 129 pgs: 129 active+clean; 289 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 539 op/s 2024-03-22T02:37:31.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 177266 15.45 MB/sec execute 491 sec latency 430.034 ms 2024-03-22T02:37:32.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 177482 15.44 MB/sec execute 492 sec latency 107.121 ms 2024-03-22T02:37:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:32 smithi120 ceph-mon[36803]: pgmap v1080: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-03-22T02:37:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:32 smithi083 ceph-mon[36340]: pgmap v1080: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-03-22T02:37:33.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:32 smithi028 ceph-mon[31916]: pgmap v1080: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.0 MiB/s wr, 555 op/s 2024-03-22T02:37:33.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 177773 15.44 MB/sec execute 493 sec latency 85.476 ms 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 -- 172.21.15.28:0/3269975890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208101ca0 msgr2=0x7f4208102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 --2- 172.21.15.28:0/3269975890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208101ca0 0x7f4208102120 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009900 tx=0x7f41f002bc30 comp rx=0 tx=0).stop 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 -- 172.21.15.28:0/3269975890 shutdown_connections 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 --2- 172.21.15.28:0/3269975890 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208102660 0x7f4208108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 --2- 172.21.15.28:0/3269975890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208101ca0 0x7f4208102120 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 --2- 172.21.15.28:0/3269975890 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208100ab0 0x7f4208100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.674 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 -- 172.21.15.28:0/3269975890 >> 172.21.15.28:0/3269975890 conn(0x7f42080fc020 msgr2=0x7f42080fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 -- 172.21.15.28:0/3269975890 shutdown_connections 2024-03-22T02:37:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 -- 172.21.15.28:0/3269975890 wait complete. 2024-03-22T02:37:34.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.673+0000 7f420d393640 1 Processor -- start 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 -- start start 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208101ca0 0x7f4208198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208102660 0x7f420819f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f420810d850 con 0x7f4208100ab0 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f420810d6d0 con 0x7f4208101ca0 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f420d393640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f420810d9d0 con 0x7f4208102660 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32786/0 (socket says 172.21.15.28:32786) 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 -- 172.21.15.28:0/4055733223 learned_addr learned my addr 172.21.15.28:0/4055733223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:34.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f42077fe640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208102660 0x7f420819f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f42067fc640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208101ca0 0x7f4208198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 -- 172.21.15.28:0/4055733223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208102660 msgr2=0x7f420819f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208102660 0x7f420819f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 -- 172.21.15.28:0/4055733223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208101ca0 msgr2=0x7f4208198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208101ca0 0x7f4208198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.674+0000 7f4206ffd640 1 -- 172.21.15.28:0/4055733223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420819fa40 con 0x7f4208100ab0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f4206ffd640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f420000e9c0 tx=0x7f420000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f41fffff640 1 -- 172.21.15.28:0/4055733223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f420000cd80 con 0x7f4208100ab0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f41fffff640 1 -- 172.21.15.28:0/4055733223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4200010910 con 0x7f4208100ab0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420819fd30 con 0x7f4208100ab0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f41fffff640 1 -- 172.21.15.28:0/4055733223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4200018950 con 0x7f4208100ab0 2024-03-22T02:37:34.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.675+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42081a01a0 con 0x7f4208100ab0 2024-03-22T02:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.677+0000 7f41fffff640 1 -- 172.21.15.28:0/4055733223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f420001f020 con 0x7f4208100ab0 2024-03-22T02:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.677+0000 7f420ff7f740 1 Processor -- start 2024-03-22T02:37:34.683 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.677+0000 7f420ff7f740 1 -- start start 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bc6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 0x561857bc98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f41fffff640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 0x7f41f40a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561857b22040 con 0x561857bc7450 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561857aca9e0 con 0x561857bc18f0 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f420ff7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561857b1dff0 con 0x561857bc22e0 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 0x561857bc98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bc6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42077fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:45670/0 (socket says 172.21.15.28:45670) 2024-03-22T02:37:34.684 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42077fe640 1 -- 172.21.15.28:0/272948301 learned_addr learned my addr 172.21.15.28:0/272948301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f41fffff640 1 -- 172.21.15.28:0/4055733223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f42000cf8a0 con 0x7f4208100ab0 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42077fe640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 0x7f41f40a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 -- 172.21.15.28:0/272948301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 msgr2=0x561857bc6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bc6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 -- 172.21.15.28:0/272948301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 msgr2=0x561857bc1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42067fc640 1 -- 172.21.15.28:0/272948301 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561857836210 con 0x561857bc7450 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f4206ffd640 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bc6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.678+0000 7f42077fe640 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.679+0000 7f42067fc640 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 0x561857bc98e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009230 tx=0x7f41f0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1700421d47e88e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.679+0000 7f42077fe640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 0x7f41f40a5480 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x561857bc2b40 tx=0x7f41f8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.685 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.679+0000 7f41fd7fa640 1 -- 172.21.15.28:0/272948301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f002cb80 con 0x561857bc7450 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.679+0000 7f41fd7fa640 1 -- 172.21.15.28:0/272948301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f0005040 con 0x561857bc7450 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.679+0000 7f41fd7fa640 1 -- 172.21.15.28:0/272948301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f0004900 con 0x561857bc7450 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- 172.21.15.28:0/272948301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 msgr2=0x561857bc98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 0x561857bc98e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f41f0009230 tx=0x7f41f0009560 comp rx=0 tx=0).stop 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- 172.21.15.28:0/272948301 shutdown_connections 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc7450 0x561857bc98e0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bc6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- 172.21.15.28:0/272948301 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bc1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- 172.21.15.28:0/272948301 >> 172.21.15.28:0/272948301 conn(0x561857bb8a50 msgr2=0x561857bb8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- 172.21.15.28:0/272948301 shutdown_connections 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- 172.21.15.28:0/272948301 wait complete. 2024-03-22T02:37:34.686 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 -- start start 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 Processor -- start 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.680+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bcbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bd2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561857bca3a0 con 0x561857bc18f0 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x561857adddb0 con 0x561857bc7450 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x561857bca520 con 0x561857bc22e0 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32814/0 (socket says 172.21.15.28:32814) 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 -- 172.21.15.28:0/2907995967 learned_addr learned my addr 172.21.15.28:0/2907995967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 10 client.0 _ll_register_callbacks cb 0x561857835a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f42067fc640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bcbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.687 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 -- 172.21.15.28:0/2907995967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 msgr2=0x561857bcbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bcbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 -- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 msgr2=0x561857bd2e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bd2e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 -- 172.21.15.28:0/2907995967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561857902620 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f4206ffd640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f42000058c0 tx=0x7f420006a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3543550cda24f7e server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f41ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4200066b70 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f4200010ac0 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41f0005040 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.681+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f420009c2c0 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 10 client.20388 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561857bd6240 con 0x561857bc18f0 2024-03-22T02:37:34.688 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41f0005040 con 0x561857bc18f0 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f42000a4140 con 0x561857bc18f0 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 10 client.20388 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 10 client.20388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 20 client.20388 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857bd9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 4 client.20388 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561857b2f580 con 0x561857bd7980 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f42067fc640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857bd9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f41ee7fc640 10 client.20388 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:34.689 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.682+0000 7f42067fc640 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857bd9d70 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3e7d397024eb48 server_cookie=19f36f0c123199bd in_seq=0 out_seq=0 2024-03-22T02:37:34.690 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f41ee7fc640 1 -- 172.21.15.28:0/2907995967 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561857b2f580 con 0x561857bd7980 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f41ee7fc640 10 client.20388 handle_command_reply: tid=0 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 client.20388 shutdown 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 msgr2=0x561857bcb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f42000058c0 tx=0x7f420006a7b0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 shutdown_connections 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857bd9d70 crc :-1 s=READY pgs=444 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f41ee7fc640 0 client.20388 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bd2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bcbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 --2- 172.21.15.28:0/2907995967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcb890 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 >> 172.21.15.28:0/2907995967 conn(0x561857bb8a50 msgr2=0x561857bb7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:34.691 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 shutdown_connections 2024-03-22T02:37:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 1 -- 172.21.15.28:0/2907995967 wait complete. 2024-03-22T02:37:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.685+0000 7f420ff7f740 20 client.20388 trim_cache size 0 max 16384 2024-03-22T02:37:34.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 178053 15.43 MB/sec execute 494 sec latency 114.677 ms 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.700+0000 7f420ff7f740 1 Processor -- start 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.700+0000 7f420ff7f740 1 -- start start 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bd2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bb73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42081aaee0 con 0x561857bc18f0 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42081a87b0 con 0x561857bc7450 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f420ff7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42081a6600 con 0x561857bc22e0 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32822/0 (socket says 172.21.15.28:32822) 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 -- 172.21.15.28:0/542910463 learned_addr learned my addr 172.21.15.28:0/542910463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f42067fc640 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bd2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f42077fe640 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bb73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 -- 172.21.15.28:0/542910463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 msgr2=0x561857bd2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bd2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 -- 172.21.15.28:0/542910463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 msgr2=0x561857bb73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bb73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 -- 172.21.15.28:0/542910463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561857bb97c0 con 0x561857bc18f0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.701+0000 7f4206ffd640 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f4200099e70 tx=0x7f420009a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f1c9c66027a992 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f41fcff9640 1 -- 172.21.15.28:0/542910463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4200097e60 con 0x561857bc18f0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f41fcff9640 1 -- 172.21.15.28:0/542910463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f420009bbc0 con 0x561857bc18f0 2024-03-22T02:37:34.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f41fcff9640 1 -- 172.21.15.28:0/542910463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4200010ac0 con 0x561857bc18f0 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 -- 172.21.15.28:0/542910463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 msgr2=0x561857bcbbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f4200099e70 tx=0x7f420009a0d0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 -- 172.21.15.28:0/542910463 shutdown_connections 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc7450 0x561857bb73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc22e0 0x561857bd2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 --2- 172.21.15.28:0/542910463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc18f0 0x561857bcbbb0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 -- 172.21.15.28:0/542910463 >> 172.21.15.28:0/542910463 conn(0x561857bb8a50 msgr2=0x561857bcb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 -- 172.21.15.28:0/542910463 shutdown_connections 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.702+0000 7f420ff7f740 1 -- 172.21.15.28:0/542910463 wait complete. 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 -- start start 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 Processor -- start 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bd0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.709 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 0x561857bd1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42081a6c70 con 0x561857bc22e0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f42081a80e0 con 0x561857bc18f0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f42081a7340 con 0x561857bc7450 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f420ff7f740 10 client.0 _ll_register_callbacks cb 0x7f4200010e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32834/0 (socket says 172.21.15.28:32834) 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 -- 172.21.15.28:0/1063649688 learned_addr learned my addr 172.21.15.28:0/1063649688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f4206ffd640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bd0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42077fe640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 0x561857bd1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 -- 172.21.15.28:0/1063649688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 msgr2=0x561857bd1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.710 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 0x561857bd1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 -- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 msgr2=0x561857bd0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bd0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42067fc640 1 -- 172.21.15.28:0/1063649688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561857bd23b0 con 0x561857bc22e0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f4206ffd640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bd0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.703+0000 7f42077fe640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 0x561857bd1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f42067fc640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x561857bb99c0 tx=0x7f41f0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885bdeb66d881bab server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f0051ba0 con 0x561857bc22e0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f41f0051400 con 0x561857bc22e0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f420009bbc0 con 0x561857bc22e0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f41f0051600 con 0x561857bc22e0 2024-03-22T02:37:34.711 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 10 client.20394 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x561857c3e100 con 0x561857bc22e0 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f420009bbc0 con 0x561857bc22e0 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f41f0005410 con 0x561857bc22e0 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 10 client.20394 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 10 client.20394 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 20 client.20394 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857c40680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 4 client.20394 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561857b298d0 con 0x561857bd7980 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.704+0000 7f4206ffd640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857c40680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.705+0000 7f41ee7fc640 10 client.20394 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:34.712 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:34.705+0000 7f4206ffd640 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857c40680 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9141e75eb6749d5b server_cookie=53383b39d49128ba in_seq=0 out_seq=0 2024-03-22T02:37:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:34 smithi120 ceph-mon[36803]: pgmap v1081: 129 pgs: 129 active+clean; 294 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 494 op/s 2024-03-22T02:37:35.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:34 smithi083 ceph-mon[36340]: pgmap v1081: 129 pgs: 129 active+clean; 294 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 494 op/s 2024-03-22T02:37:35.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:34 smithi028 ceph-mon[31916]: pgmap v1081: 129 pgs: 129 active+clean; 294 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.7 MiB/s wr, 494 op/s 2024-03-22T02:37:35.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 178105 15.40 MB/sec execute 495 sec latency 755.556 ms 2024-03-22T02:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f41ee7fc640 1 -- 172.21.15.28:0/1063649688 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561857b298d0 con 0x561857bd7980 2024-03-22T02:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f41ee7fc640 10 client.20394 handle_command_reply: tid=0 2024-03-22T02:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 client.20394 shutdown 2024-03-22T02:37:36.278 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 msgr2=0x561857bd0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x561857bb99c0 tx=0x7f41f0009260 comp rx=0 tx=0).stop 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 shutdown_connections 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x561857bd7980 0x561857c40680 crc :-1 s=READY pgs=445 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x561857bc7450 0x561857bd1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561857bc22e0 0x561857bd0db0 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f41ee7fc640 0 client.20394 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.276+0000 7f420ff7f740 1 --2- 172.21.15.28:0/1063649688 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x561857bc18f0 0x561857bd0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 >> 172.21.15.28:0/1063649688 conn(0x561857bb8a50 msgr2=0x561857bc2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 shutdown_connections 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420ff7f740 1 -- 172.21.15.28:0/1063649688 wait complete. 2024-03-22T02:37:36.279 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420ff7f740 20 client.20394 trim_cache size 0 max 16384 2024-03-22T02:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:37:36.280 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 msgr2=0x7f41f40a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 0x7f41f40a5480 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x561857bc2b40 tx=0x7f41f8009500 comp rx=0 tx=0).stop 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 msgr2=0x7f4208197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.277+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f420000e9c0 tx=0x7f420000edc0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 shutdown_connections 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f41f40a2fc0 0x7f41f40a5480 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4208102660 0x7f420819f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.281 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4208101ca0 0x7f4208198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 --2- 172.21.15.28:0/4055733223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4208100ab0 0x7f4208197e20 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 >> 172.21.15.28:0/4055733223 conn(0x7f42080fc020 msgr2=0x7f4208104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 shutdown_connections 2024-03-22T02:37:36.282 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:36.278+0000 7f420d393640 1 -- 172.21.15.28:0/4055733223 wait complete. 2024-03-22T02:37:36.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 178291 15.38 MB/sec execute 496 sec latency 859.452 ms 2024-03-22T02:37:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:36 smithi120 ceph-mon[36803]: pgmap v1082: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-03-22T02:37:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:36 smithi083 ceph-mon[36340]: pgmap v1082: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-03-22T02:37:37.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:36 smithi028 ceph-mon[31916]: pgmap v1082: 129 pgs: 129 active+clean; 293 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 444 op/s 2024-03-22T02:37:37.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 178526 15.37 MB/sec execute 497 sec latency 127.317 ms 2024-03-22T02:37:38.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 178900 15.38 MB/sec execute 498 sec latency 86.516 ms 2024-03-22T02:37:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:38 smithi120 ceph-mon[36803]: pgmap v1083: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 490 op/s 2024-03-22T02:37:38.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:38 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:38 smithi083 ceph-mon[36340]: pgmap v1083: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 490 op/s 2024-03-22T02:37:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:38 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:39.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:38 smithi028 ceph-mon[31916]: pgmap v1083: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 5.4 MiB/s wr, 490 op/s 2024-03-22T02:37:39.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:38 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:37:39.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 179232 15.40 MB/sec execute 499 sec latency 74.225 ms 2024-03-22T02:37:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:39.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:39.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:37:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:37:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:37:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:37:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:37:40.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 179645 15.41 MB/sec execute 500 sec latency 116.612 ms 2024-03-22T02:37:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:40 smithi120 ceph-mon[36803]: pgmap v1084: 129 pgs: 129 active+clean; 291 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 498 op/s 2024-03-22T02:37:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:40 smithi083 ceph-mon[36340]: pgmap v1084: 129 pgs: 129 active+clean; 291 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 498 op/s 2024-03-22T02:37:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:40 smithi028 ceph-mon[31916]: pgmap v1084: 129 pgs: 129 active+clean; 291 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.8 MiB/s wr, 498 op/s 2024-03-22T02:37:41.698 INFO:tasks.workunit.client.0.smithi028.stdout: 10 179931 15.42 MB/sec execute 501 sec latency 277.332 ms 2024-03-22T02:37:42.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 180229 15.42 MB/sec execute 502 sec latency 133.472 ms 2024-03-22T02:37:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:42 smithi120 ceph-mon[36803]: pgmap v1085: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:37:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:42 smithi083 ceph-mon[36340]: pgmap v1085: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:37:43.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:42 smithi028 ceph-mon[31916]: pgmap v1085: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.4 MiB/s wr, 481 op/s 2024-03-22T02:37:43.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 180515 15.41 MB/sec execute 503 sec latency 109.802 ms 2024-03-22T02:37:44.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 180809 15.42 MB/sec execute 504 sec latency 71.018 ms 2024-03-22T02:37:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:44 smithi120 ceph-mon[36803]: pgmap v1086: 129 pgs: 129 active+clean; 287 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 500 op/s 2024-03-22T02:37:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:44 smithi083 ceph-mon[36340]: pgmap v1086: 129 pgs: 129 active+clean; 287 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 500 op/s 2024-03-22T02:37:45.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:44 smithi028 ceph-mon[31916]: pgmap v1086: 129 pgs: 129 active+clean; 287 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 4.0 MiB/s wr, 500 op/s 2024-03-22T02:37:45.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 181089 15.42 MB/sec execute 505 sec latency 78.819 ms 2024-03-22T02:37:46.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 181286 15.42 MB/sec execute 506 sec latency 346.195 ms 2024-03-22T02:37:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:46 smithi120 ceph-mon[36803]: pgmap v1087: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 515 op/s 2024-03-22T02:37:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:46 smithi083 ceph-mon[36340]: pgmap v1087: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 515 op/s 2024-03-22T02:37:47.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:46 smithi028 ceph-mon[31916]: pgmap v1087: 129 pgs: 129 active+clean; 285 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.9 MiB/s wr, 515 op/s 2024-03-22T02:37:47.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 181533 15.40 MB/sec execute 507 sec latency 133.528 ms 2024-03-22T02:37:48.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 181835 15.40 MB/sec execute 508 sec latency 76.049 ms 2024-03-22T02:37:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:48 smithi120 ceph-mon[36803]: pgmap v1088: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 542 op/s 2024-03-22T02:37:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:48 smithi083 ceph-mon[36340]: pgmap v1088: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 542 op/s 2024-03-22T02:37:49.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:48 smithi028 ceph-mon[31916]: pgmap v1088: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.3 MiB/s wr, 542 op/s 2024-03-22T02:37:49.381 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2958879263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 msgr2=0x7f3d98104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2958879263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98104a20 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009900 tx=0x7f3d8802bc30 comp rx=0 tx=0).stop 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2958879263 shutdown_connections 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2958879263 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98107890 0x7f3d98109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2958879263 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98104f60 0x7f3d98107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2958879263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98104a20 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2958879263 >> 172.21.15.28:0/2958879263 conn(0x7f3d980fc020 msgr2=0x7f3d980fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2958879263 shutdown_connections 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2958879263 wait complete. 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 Processor -- start 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- start start 2024-03-22T02:37:49.382 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98104f60 0x7f3d98198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98107890 0x7f3d9819f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9810d850 con 0x7f3d98102630 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9810d6d0 con 0x7f3d98107890 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.379+0000 7f3d9ef1e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d9810d9d0 con 0x7f3d98104f60 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59578/0 (socket says 172.21.15.28:59578) 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 -- 172.21.15.28:0/2266227965 learned_addr learned my addr 172.21.15.28:0/2266227965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98107890 0x7f3d9819f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d97fff640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98104f60 0x7f3d98198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 -- 172.21.15.28:0/2266227965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98104f60 msgr2=0x7f3d98198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.383 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98104f60 0x7f3d98198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 -- 172.21.15.28:0/2266227965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98107890 msgr2=0x7f3d9819f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98107890 0x7f3d9819f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 -- 172.21.15.28:0/2266227965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d9819fa30 con 0x7f3d98102630 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009230 tx=0x7f3d88005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d95ffb640 1 -- 172.21.15.28:0/2266227965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88041070 con 0x7f3d98102630 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d9819fcc0 con 0x7f3d98102630 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d95ffb640 1 -- 172.21.15.28:0/2266227965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d880044d0 con 0x7f3d98102630 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d95ffb640 1 -- 172.21.15.28:0/2266227965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d88034440 con 0x7f3d98102630 2024-03-22T02:37:49.384 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.380+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d981a01b0 con 0x7f3d98102630 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3d95ffb640 1 -- 172.21.15.28:0/2266227965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f3d88044070 con 0x7f3d98102630 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 Processor -- start 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 -- start start 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85e2e0 0x55ab2a862de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.390 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.383+0000 7f3da1be2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a7be040 con 0x55ab2a863450 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3da1be2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a7669e0 con 0x55ab2a85d8f0 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3da1be2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a7b9ff0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d97fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59586/0 (socket says 172.21.15.28:59586) 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:44128/0 (socket says 172.21.15.28:44128) 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85e2e0 0x55ab2a862de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d97fff640 1 -- 172.21.15.28:0/1115737382 learned_addr learned my addr 172.21.15.28:0/1115737382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d95ffb640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 0x7f3d800a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1115737382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85e2e0 msgr2=0x55ab2a862de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.391 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85e2e0 0x55ab2a862de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1115737382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 msgr2=0x55ab2a8658e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1115737382 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab2a4d2210 con 0x55ab2a85d8f0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d97fff640 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d95ffb640 1 -- 172.21.15.28:0/2266227965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f3d880f13d0 con 0x7f3d98102630 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 0x7f3d800a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.384+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009f30 tx=0x7f3d90010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a26b2008445e0e1a server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3d7effd640 1 -- 172.21.15.28:0/1115737382 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90016070 con 0x55ab2a85d8f0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 0x7f3d800a5480 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3d90021000 tx=0x7f3d90024000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3d7effd640 1 -- 172.21.15.28:0/1115737382 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d90012e00 con 0x55ab2a85d8f0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3d7effd640 1 -- 172.21.15.28:0/1115737382 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9001cc50 con 0x55ab2a85d8f0 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 -- 172.21.15.28:0/1115737382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 msgr2=0x55ab2a85dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.392 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3d90009f30 tx=0x7f3d90010800 comp rx=0 tx=0).stop 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 -- 172.21.15.28:0/1115737382 shutdown_connections 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a8658e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85e2e0 0x55ab2a862de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1115737382 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a85dd10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 -- 172.21.15.28:0/1115737382 >> 172.21.15.28:0/1115737382 conn(0x55ab2a854a50 msgr2=0x55ab2a854e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 -- 172.21.15.28:0/1115737382 shutdown_connections 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.385+0000 7f3da1be2740 1 -- 172.21.15.28:0/1115737382 wait complete. 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 -- start start 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 Processor -- start 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a867950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a867e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.393 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a866400 con 0x55ab2a863450 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a779db0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55ab2a866580 con 0x55ab2a85d8f0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3da1be2740 10 client.0 _ll_register_callbacks cb 0x55ab2a4d1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59602/0 (socket says 172.21.15.28:59602) 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a867950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3d97fff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a867e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.386+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1598451322 learned_addr learned my addr 172.21.15.28:0/1598451322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1598451322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 msgr2=0x55ab2a867950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a867950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 msgr2=0x55ab2a867e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a867e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/1598451322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab2a59e620 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3d9001c510 tx=0x7f3d9001c600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b68d1c16fa500fa server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90016050 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f3d9001cc60 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d90012e00 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3d9003d050 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3da1be2740 10 client.20406 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ab2a872240 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d90012e00 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.387+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f3d9000f040 con 0x55ab2a863450 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 10 client.20406 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 10 client.20406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 20 client.20406 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a8737e0 0x55ab2a875bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 4 client.20406 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab2a7cb580 con 0x55ab2a8737e0 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a8737e0 0x55ab2a875bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3d77fff640 10 client.20406 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.388+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a8737e0 0x55ab2a875bd0 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4da3667b0eade6ca server_cookie=a93c1d09ae3efc07 in_seq=0 out_seq=0 2024-03-22T02:37:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.392+0000 7f3d77fff640 1 -- 172.21.15.28:0/1598451322 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ab2a7cb580 con 0x55ab2a8737e0 2024-03-22T02:37:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.392+0000 7f3d77fff640 10 client.20406 handle_command_reply: tid=0 2024-03-22T02:37:49.397 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.392+0000 7f3da1be2740 1 client.20406 shutdown 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.392+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 msgr2=0x55ab2a86eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.392+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f3d9001c510 tx=0x7f3d9001c600 comp rx=0 tx=0).stop 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 shutdown_connections 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a8737e0 0x55ab2a875bd0 crc :-1 s=READY pgs=446 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a863450 0x55ab2a86eed0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a867e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1598451322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a867950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 >> 172.21.15.28:0/1598451322 conn(0x55ab2a854a50 msgr2=0x55ab2a853c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3d77fff640 0 client.20406 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 shutdown_connections 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 1 -- 172.21.15.28:0/1598451322 wait complete. 2024-03-22T02:37:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.393+0000 7f3da1be2740 20 client.20406 trim_cache size 0 max 16384 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 Processor -- start 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 -- start start 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a8533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981aaf60 con 0x55ab2a854a50 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981a8830 con 0x55ab2a85e2e0 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3da1be2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981a6680 con 0x55ab2a85d8f0 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a8533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59610/0 (socket says 172.21.15.28:59610) 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 -- 172.21.15.28:0/1420938733 learned_addr learned my addr 172.21.15.28:0/1420938733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d97fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:46666/0 (socket says 172.21.15.28:46666) 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 -- 172.21.15.28:0/1420938733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 msgr2=0x55ab2a86ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 -- 172.21.15.28:0/1420938733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 msgr2=0x55ab2a8533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a8533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.409+0000 7f3d9ccba640 1 -- 172.21.15.28:0/1420938733 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab2a85dcd0 con 0x55ab2a854a50 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009430 tx=0x7f3d880f3040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673fca43fcf2f5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3d7e7fc640 1 -- 172.21.15.28:0/1420938733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d8804b040 con 0x55ab2a854a50 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3d7e7fc640 1 -- 172.21.15.28:0/1420938733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d880bd7a0 con 0x55ab2a854a50 2024-03-22T02:37:49.412 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3d7e7fc640 1 -- 172.21.15.28:0/1420938733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d880c1840 con 0x55ab2a854a50 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 -- 172.21.15.28:0/1420938733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 msgr2=0x55ab2a862930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009430 tx=0x7f3d880f3040 comp rx=0 tx=0).stop 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 -- 172.21.15.28:0/1420938733 shutdown_connections 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85e2e0 0x55ab2a8533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 --2- 172.21.15.28:0/1420938733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a854a50 0x55ab2a862930 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.410+0000 7f3da1be2740 1 -- 172.21.15.28:0/1420938733 >> 172.21.15.28:0/1420938733 conn(0x55ab2a867510 msgr2=0x55ab2a858540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- 172.21.15.28:0/1420938733 shutdown_connections 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- 172.21.15.28:0/1420938733 wait complete. 2024-03-22T02:37:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- start start 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 Processor -- start 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a854a50 0x55ab2a86c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981a6cf0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981a8160 con 0x55ab2a85d8f0 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f3d981a73c0 con 0x55ab2a854a50 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3da1be2740 10 client.0 _ll_register_callbacks cb 0x7f3d90020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3d9d4bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59622/0 (socket says 172.21.15.28:59622) 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.411+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/3032809502 learned_addr learned my addr 172.21.15.28:0/3032809502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a854a50 0x55ab2a86c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d97fff640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/3032809502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a854a50 msgr2=0x55ab2a86c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a854a50 0x55ab2a86c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 msgr2=0x55ab2a86cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 -- 172.21.15.28:0/3032809502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab2a86e2d0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d9d4bb640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3d90005d70 tx=0x7f3d900102e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167bbdced1e6238d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d77fff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90016030 con 0x55ab2a85e2e0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f3d90012e00 con 0x55ab2a85e2e0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d880bd7a0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f3d9001ce70 con 0x55ab2a85e2e0 2024-03-22T02:37:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3da1be2740 10 client.20412 fetch_fsmap learned FSMap version 46 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.412+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55ab2a8d9ce0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d880bd7a0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f3d90028ce0 con 0x55ab2a85e2e0 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 10 client.20412 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 10 client.20412 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 20 client.20412 populate_metadata read hostname 'smithi028' 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a863450 0x55ab2a8dc690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 4 client.20412 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ab2a7c58d0 con 0x55ab2a863450 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a863450 0x55ab2a8dc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3d77fff640 10 client.20412 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:49.419 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:49.413+0000 7f3d9ccba640 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a863450 0x55ab2a8dc690 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=958f48858d89ac0b server_cookie=ec0020aec8c68227 in_seq=0 out_seq=0 2024-03-22T02:37:49.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 182097 15.40 MB/sec execute 509 sec latency 269.059 ms 2024-03-22T02:37:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3d77fff640 1 -- 172.21.15.28:0/3032809502 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ab2a7c58d0 con 0x55ab2a863450 2024-03-22T02:37:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3d77fff640 10 client.20412 handle_command_reply: tid=0 2024-03-22T02:37:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 client.20412 shutdown 2024-03-22T02:37:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 msgr2=0x55ab2a86dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f3d90005d70 tx=0x7f3d900102e0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 shutdown_connections 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55ab2a863450 0x55ab2a8dc690 crc :-1 s=READY pgs=447 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3d77fff640 0 client.20412 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ab2a85e2e0 0x55ab2a86dce0 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55ab2a85d8f0 0x55ab2a86cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 --2- 172.21.15.28:0/3032809502 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ab2a854a50 0x55ab2a86c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 >> 172.21.15.28:0/3032809502 conn(0x55ab2a867510 msgr2=0x55ab2a858540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 shutdown_connections 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 1 -- 172.21.15.28:0/3032809502 wait complete. 2024-03-22T02:37:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.674+0000 7f3da1be2740 20 client.20412 trim_cache size 0 max 16384 2024-03-22T02:37:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:37:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:37:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:37:50.677 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 msgr2=0x7f3d800a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 0x7f3d800a5480 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3d90021000 tx=0x7f3d90024000 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 msgr2=0x7f3d98197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009230 tx=0x7f3d88005d50 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 shutdown_connections 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f3d800a2fc0 0x7f3d800a5480 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f3d98107890 0x7f3d9819f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3d98104f60 0x7f3d98198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 --2- 172.21.15.28:0/2266227965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d98102630 0x7f3d98197e10 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 >> 172.21.15.28:0/2266227965 conn(0x7f3d980fc020 msgr2=0x7f3d98102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:37:50.678 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 shutdown_connections 2024-03-22T02:37:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:37:50.675+0000 7f3d9ef1e640 1 -- 172.21.15.28:0/2266227965 wait complete. 2024-03-22T02:37:50.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 182117 15.37 MB/sec execute 510 sec latency 1225.396 ms 2024-03-22T02:37:50.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:50 smithi120 ceph-mon[36803]: pgmap v1089: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-03-22T02:37:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:50 smithi083 ceph-mon[36340]: pgmap v1089: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-03-22T02:37:51.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:50 smithi028 ceph-mon[31916]: pgmap v1089: 129 pgs: 129 active+clean; 296 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 403 op/s 2024-03-22T02:37:51.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 182450 15.37 MB/sec execute 511 sec latency 192.524 ms 2024-03-22T02:37:52.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 182687 15.37 MB/sec execute 512 sec latency 117.864 ms 2024-03-22T02:37:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:52 smithi120 ceph-mon[36803]: pgmap v1090: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 325 op/s 2024-03-22T02:37:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:52 smithi083 ceph-mon[36340]: pgmap v1090: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 325 op/s 2024-03-22T02:37:53.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:52 smithi028 ceph-mon[31916]: pgmap v1090: 129 pgs: 129 active+clean; 295 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.3 MiB/s wr, 325 op/s 2024-03-22T02:37:53.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 183124 15.39 MB/sec execute 513 sec latency 64.273 ms 2024-03-22T02:37:54.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 183539 15.40 MB/sec execute 514 sec latency 62.109 ms 2024-03-22T02:37:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:54 smithi120 ceph-mon[36803]: pgmap v1091: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2024-03-22T02:37:55.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:54 smithi083 ceph-mon[36340]: pgmap v1091: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2024-03-22T02:37:55.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:54 smithi028 ceph-mon[31916]: pgmap v1091: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 411 op/s 2024-03-22T02:37:55.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 183875 15.40 MB/sec execute 515 sec latency 76.845 ms 2024-03-22T02:37:56.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 184059 15.39 MB/sec execute 516 sec latency 186.052 ms 2024-03-22T02:37:56.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:56 smithi120 ceph-mon[36803]: pgmap v1092: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:37:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:56 smithi083 ceph-mon[36340]: pgmap v1092: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:37:57.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:56 smithi028 ceph-mon[31916]: pgmap v1092: 129 pgs: 129 active+clean; 281 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.0 MiB/s wr, 528 op/s 2024-03-22T02:37:57.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 184305 15.40 MB/sec execute 517 sec latency 107.654 ms 2024-03-22T02:37:58.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 184627 15.41 MB/sec execute 518 sec latency 80.349 ms 2024-03-22T02:37:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:37:58 smithi120 ceph-mon[36803]: pgmap v1093: 129 pgs: 129 active+clean; 283 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 558 op/s 2024-03-22T02:37:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:37:58 smithi083 ceph-mon[36340]: pgmap v1093: 129 pgs: 129 active+clean; 283 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 558 op/s 2024-03-22T02:37:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:37:58 smithi028 ceph-mon[31916]: pgmap v1093: 129 pgs: 129 active+clean; 283 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.3 MiB/s wr, 558 op/s 2024-03-22T02:37:59.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 184900 15.41 MB/sec execute 519 sec latency 88.462 ms 2024-03-22T02:38:00.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 185154 15.39 MB/sec execute 520 sec latency 82.016 ms 2024-03-22T02:38:00.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:00 smithi120 ceph-mon[36803]: pgmap v1094: 129 pgs: 129 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 534 op/s 2024-03-22T02:38:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:00 smithi083 ceph-mon[36340]: pgmap v1094: 129 pgs: 129 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 534 op/s 2024-03-22T02:38:01.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:00 smithi028 ceph-mon[31916]: pgmap v1094: 129 pgs: 129 active+clean; 283 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 4.0 MiB/s wr, 534 op/s 2024-03-22T02:38:01.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 185340 15.39 MB/sec execute 521 sec latency 284.190 ms 2024-03-22T02:38:02.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 185541 15.37 MB/sec execute 522 sec latency 145.738 ms 2024-03-22T02:38:02.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:02 smithi120 ceph-mon[36803]: pgmap v1095: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 529 op/s 2024-03-22T02:38:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:02 smithi083 ceph-mon[36340]: pgmap v1095: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 529 op/s 2024-03-22T02:38:03.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:02 smithi028 ceph-mon[31916]: pgmap v1095: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.8 MiB/s wr, 529 op/s 2024-03-22T02:38:03.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 185862 15.37 MB/sec execute 523 sec latency 115.678 ms 2024-03-22T02:38:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 -- 172.21.15.28:0/3073123251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 msgr2=0x7f4f2010a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.789 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 --2- 172.21.15.28:0/3073123251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2010a000 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f4f18009ca0 tx=0x7f4f1802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 -- 172.21.15.28:0/3073123251 shutdown_connections 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 --2- 172.21.15.28:0/3073123251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2010a000 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 --2- 172.21.15.28:0/3073123251 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20102db0 0x7f4f20103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 --2- 172.21.15.28:0/3073123251 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20101bc0 0x7f4f20101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 -- 172.21.15.28:0/3073123251 >> 172.21.15.28:0/3073123251 conn(0x7f4f200fd150 msgr2=0x7f4f200ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 -- 172.21.15.28:0/3073123251 shutdown_connections 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.786+0000 7f4f270df640 1 -- 172.21.15.28:0/3073123251 wait complete. 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 Processor -- start 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 -- start start 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.790 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20102db0 0x7f4f20198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2010d820 con 0x7f4f20103770 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2010d6a0 con 0x7f4f20101bc0 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f270df640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f2010d9a0 con 0x7f4f20102db0 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f24e7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20102db0 0x7f4f20198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56680/0 (socket says 172.21.15.28:56680) 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 -- 172.21.15.28:0/1895848254 learned_addr learned my addr 172.21.15.28:0/1895848254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f24e7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:39284/0 (socket says 172.21.15.28:39284) 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 -- 172.21.15.28:0/1895848254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20102db0 msgr2=0x7f4f20198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.791 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20102db0 0x7f4f20198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 -- 172.21.15.28:0/1895848254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 msgr2=0x7f4f20197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f2567c640 1 -- 172.21.15.28:0/1895848254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f2019f9d0 con 0x7f4f20103770 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.787+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f2567c640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f4f18037040 tx=0x7f4f18032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f15ffb640 1 -- 172.21.15.28:0/1895848254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18041020 con 0x7f4f20103770 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f2019fc60 con 0x7f4f20103770 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f15ffb640 1 -- 172.21.15.28:0/1895848254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f18034cd0 con 0x7f4f20103770 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f15ffb640 1 -- 172.21.15.28:0/1895848254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f18039040 con 0x7f4f20103770 2024-03-22T02:38:03.792 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.788+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f201a01a0 con 0x7f4f20103770 2024-03-22T02:38:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.790+0000 7f4f15ffb640 1 -- 172.21.15.28:0/1895848254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4f18044020 con 0x7f4f20103770 2024-03-22T02:38:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.790+0000 7f4f29da7740 1 Processor -- start 2024-03-22T02:38:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.790+0000 7f4f29da7740 1 -- start start 2024-03-22T02:38:03.799 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 0x55a03a357de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a358450 0x55a03a35a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a03a2b3040 con 0x55a03a3528f0 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a03a25b9e0 con 0x55a03a3532e0 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f29da7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a03a2aeff0 con 0x55a03a358450 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56696/0 (socket says 172.21.15.28:56696) 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 -- 172.21.15.28:0/1789396139 learned_addr learned my addr 172.21.15.28:0/1789396139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 0x55a03a357de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a358450 0x55a03a35a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f15ffb640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 0x7f4f080a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.800 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 -- 172.21.15.28:0/1789396139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a358450 msgr2=0x55a03a35a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f2567c640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 0x7f4f080a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a358450 0x55a03a35a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 -- 172.21.15.28:0/1789396139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 msgr2=0x55a03a357de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 0x55a03a357de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 -- 172.21.15.28:0/1789396139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a039fc7210 con 0x55a03a3528f0 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 0x55a03a357de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f15ffb640 1 -- 172.21.15.28:0/1895848254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f4f180f1750 con 0x7f4f20103770 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009ec0 tx=0x7f4f100097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fce76bbf8724146 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f06ffd640 1 -- 172.21.15.28:0/1789396139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10013d10 con 0x55a03a3528f0 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.791+0000 7f4f06ffd640 1 -- 172.21.15.28:0/1789396139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f1000fda0 con 0x55a03a3528f0 2024-03-22T02:38:03.801 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f06ffd640 1 -- 172.21.15.28:0/1789396139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10010c40 con 0x55a03a3528f0 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f2567c640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 0x7f4f080a5480 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4f180b9ed0 tx=0x7f4f180a8040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 -- 172.21.15.28:0/1789396139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 msgr2=0x55a03a352d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009ec0 tx=0x7f4f100097a0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 -- 172.21.15.28:0/1789396139 shutdown_connections 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a358450 0x55a03a35a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a3532e0 0x55a03a357de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1789396139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a352d10 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 -- 172.21.15.28:0/1789396139 >> 172.21.15.28:0/1789396139 conn(0x55a03a349a50 msgr2=0x55a03a349e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 -- 172.21.15.28:0/1789396139 shutdown_connections 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.792+0000 7f4f29da7740 1 -- 172.21.15.28:0/1789396139 wait complete. 2024-03-22T02:38:03.802 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 -- start start 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 Processor -- start 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a35ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a363e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a03a35b400 con 0x55a03a3532e0 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a03a26edb0 con 0x55a03a358450 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a03a35b580 con 0x55a03a3528f0 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f29da7740 10 client.0 _ll_register_callbacks cb 0x55a039fc6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a35ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:42322/0 (socket says 172.21.15.28:42322) 2024-03-22T02:38:03.803 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f17fff640 1 -- 172.21.15.28:0/1309989784 learned_addr learned my addr 172.21.15.28:0/1309989784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a363e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 -- 172.21.15.28:0/1309989784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 msgr2=0x55a03a35c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 -- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 msgr2=0x55a03a363e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a363e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.793+0000 7f4f24e7b640 1 -- 172.21.15.28:0/1309989784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03a093620 con 0x55a03a3532e0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a35ce00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x55a03a3536e0 tx=0x7f4f1c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e866b7e1be41b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c0120d0 con 0x55a03a3532e0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f1000fda0 con 0x55a03a3532e0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f4f1c0047a0 con 0x55a03a3532e0 2024-03-22T02:38:03.804 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4f1c025050 con 0x55a03a3532e0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 10 client.20424 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a03a367280 con 0x55a03a3532e0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f1000fda0 con 0x55a03a3532e0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f4f1c0104c0 con 0x55a03a3532e0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 10 client.20424 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 10 client.20424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.794+0000 7f4f29da7740 20 client.20424 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3689c0 0x55a03a36ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4f29da7740 4 client.20424 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a03a2c0580 con 0x55a03a3689c0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3689c0 0x55a03a36ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4f17fff640 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3689c0 0x55a03a36ae00 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ed7635302188d0e server_cookie=f51367f3c5d1b60e in_seq=0 out_seq=0 2024-03-22T02:38:03.805 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.795+0000 7f4efffff640 10 client.20424 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4efffff640 1 -- 172.21.15.28:0/1309989784 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a03a2c0580 con 0x55a03a3689c0 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4efffff640 10 client.20424 handle_command_reply: tid=0 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 client.20424 shutdown 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 msgr2=0x55a03a35ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a35ce00 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x55a03a3536e0 tx=0x7f4f1c00f790 comp rx=0 tx=0).stop 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 shutdown_connections 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3689c0 0x55a03a36ae00 crc :-1 s=READY pgs=448 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a363e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a35ce00 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4efffff640 0 client.20424 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:03.806 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 --2- 172.21.15.28:0/1309989784 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.796+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 >> 172.21.15.28:0/1309989784 conn(0x55a03a349a50 msgr2=0x55a03a348c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.797+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 shutdown_connections 2024-03-22T02:38:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.797+0000 7f4f29da7740 1 -- 172.21.15.28:0/1309989784 wait complete. 2024-03-22T02:38:03.807 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.797+0000 7f4f29da7740 20 client.20424 trim_cache size 0 max 16384 2024-03-22T02:38:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.811+0000 7f4f29da7740 1 Processor -- start 2024-03-22T02:38:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 -- start start 2024-03-22T02:38:03.817 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a3483c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201aaf80 con 0x55a03a3532e0 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201a8850 con 0x55a03a358450 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f29da7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201a66a0 con 0x55a03a3528f0 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56710/0 (socket says 172.21.15.28:56710) 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 -- 172.21.15.28:0/644913704 learned_addr learned my addr 172.21.15.28:0/644913704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f17fff640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f17fff640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a3483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.818 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 -- 172.21.15.28:0/644913704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 msgr2=0x55a03a35c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 -- 172.21.15.28:0/644913704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 msgr2=0x55a03a3483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a3483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 -- 172.21.15.28:0/644913704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a03a36b480 con 0x55a03a3532e0 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f17fff640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c005880 tx=0x7f4f1c0059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a93b4768432c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f067fc640 1 -- 172.21.15.28:0/644913704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c02e070 con 0x55a03a3532e0 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.812+0000 7f4f067fc640 1 -- 172.21.15.28:0/644913704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f1c0120d0 con 0x55a03a3532e0 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f067fc640 1 -- 172.21.15.28:0/644913704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c01c950 con 0x55a03a3532e0 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- 172.21.15.28:0/644913704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 msgr2=0x55a03a363e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c005880 tx=0x7f4f1c0059b0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- 172.21.15.28:0/644913704 shutdown_connections 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a3483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.819 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3532e0 0x55a03a363e00 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 --2- 172.21.15.28:0/644913704 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3528f0 0x55a03a35c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- 172.21.15.28:0/644913704 >> 172.21.15.28:0/644913704 conn(0x55a03a349a50 msgr2=0x55a03a34d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- 172.21.15.28:0/644913704 shutdown_connections 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- 172.21.15.28:0/644913704 wait complete. 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.813+0000 7f4f29da7740 1 -- start start 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 Processor -- start 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.820 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 0x55a03a363b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a361bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201a6d10 con 0x55a03a3528f0 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201a8180 con 0x55a03a358450 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f4f201a73e0 con 0x55a03a3532e0 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f29da7740 10 client.0 _ll_register_callbacks cb 0x7f4f1c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-03-22T02:38:03.821 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 -- 172.21.15.28:0/3789948849 learned_addr learned my addr 172.21.15.28:0/3789948849 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 0x55a03a363b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a361bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 -- 172.21.15.28:0/3789948849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 msgr2=0x55a03a363b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 0x55a03a363b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 -- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 msgr2=0x55a03a361bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a361bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 -- 172.21.15.28:0/3789948849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a03a35b980 con 0x55a03a3528f0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 0x55a03a363b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4f17fff640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4f10013790 tx=0x7f4f10005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2b088cc9d10eb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4efffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.814+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f10037060 con 0x55a03a3528f0 2024-03-22T02:38:03.822 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f1c0120d0 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f4f10013270 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4f10030450 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 10 client.20430 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a03a3cf4b0 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f1c0120d0 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f4f1000f9e0 con 0x55a03a3528f0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 10 client.20430 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 10 client.20430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 20 client.20430 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3cfcd0 0x55a03a3d20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 4 client.20430 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a03a2ba8d0 con 0x55a03a3cfcd0 2024-03-22T02:38:03.823 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.815+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3cfcd0 0x55a03a3d20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.816+0000 7f4efffff640 10 client.20430 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:03.824 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:03.816+0000 7f4f24e7b640 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3cfcd0 0x55a03a3d20f0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=108aed2bdf866557 server_cookie=42812736d0678411 in_seq=0 out_seq=0 2024-03-22T02:38:04.699 INFO:tasks.workunit.client.0.smithi028.stdout: 10 185921 15.35 MB/sec execute 524 sec latency 770.883 ms 2024-03-22T02:38:04.905 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4efffff640 1 -- 172.21.15.28:0/3789948849 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a03a2ba8d0 con 0x55a03a3cfcd0 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4efffff640 10 client.20430 handle_command_reply: tid=0 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 client.20430 shutdown 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 msgr2=0x55a03a3635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f4f10013790 tx=0x7f4f10005f90 comp rx=0 tx=0).stop 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 shutdown_connections 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a03a3cfcd0 0x55a03a3d20f0 crc :-1 s=READY pgs=449 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a03a358450 0x55a03a361bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4efffff640 0 client.20430 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a03a3532e0 0x55a03a363b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 --2- 172.21.15.28:0/3789948849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a03a3528f0 0x55a03a3635e0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 >> 172.21.15.28:0/3789948849 conn(0x55a03a349a50 msgr2=0x55a03a3483c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:04.906 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 shutdown_connections 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 1 -- 172.21.15.28:0/3789948849 wait complete. 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.904+0000 7f4f29da7740 20 client.20430 trim_cache size 0 max 16384 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:38:04.907 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 msgr2=0x7f4f080a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 0x7f4f080a5480 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f4f180b9ed0 tx=0x7f4f180a8040 comp rx=0 tx=0).stop 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 msgr2=0x7f4f2019f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f4f18037040 tx=0x7f4f18032f40 comp rx=0 tx=0).stop 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 shutdown_connections 2024-03-22T02:38:04.908 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f4f080a2fc0 0x7f4f080a5480 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f20103770 0x7f4f2019f330 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4f20102db0 0x7f4f20198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 --2- 172.21.15.28:0/1895848254 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4f20101bc0 0x7f4f20197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 >> 172.21.15.28:0/1895848254 conn(0x7f4f200fd150 msgr2=0x7f4f20105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 shutdown_connections 2024-03-22T02:38:04.909 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:04.905+0000 7f4f270df640 1 -- 172.21.15.28:0/1895848254 wait complete. 2024-03-22T02:38:04.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:04 smithi120 ceph-mon[36803]: pgmap v1096: 129 pgs: 129 active+clean; 289 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:38:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:04 smithi083 ceph-mon[36340]: pgmap v1096: 129 pgs: 129 active+clean; 289 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:38:05.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:04 smithi028 ceph-mon[31916]: pgmap v1096: 129 pgs: 129 active+clean; 289 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.0 MiB/s wr, 525 op/s 2024-03-22T02:38:05.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 186228 15.35 MB/sec execute 525 sec latency 850.088 ms 2024-03-22T02:38:06.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 186423 15.34 MB/sec execute 526 sec latency 303.266 ms 2024-03-22T02:38:06.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:06 smithi120 ceph-mon[36803]: pgmap v1097: 129 pgs: 129 active+clean; 292 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-22T02:38:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:06 smithi083 ceph-mon[36340]: pgmap v1097: 129 pgs: 129 active+clean; 292 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-22T02:38:07.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:06 smithi028 ceph-mon[31916]: pgmap v1097: 129 pgs: 129 active+clean; 292 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-03-22T02:38:07.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 186787 15.35 MB/sec execute 527 sec latency 125.020 ms 2024-03-22T02:38:08.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 187194 15.36 MB/sec execute 528 sec latency 96.589 ms 2024-03-22T02:38:08.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:08 smithi120 ceph-mon[36803]: pgmap v1098: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-03-22T02:38:09.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:08 smithi083 ceph-mon[36340]: pgmap v1098: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-03-22T02:38:09.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:08 smithi028 ceph-mon[31916]: pgmap v1098: 129 pgs: 129 active+clean; 288 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.4 MiB/s wr, 444 op/s 2024-03-22T02:38:09.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 187492 15.36 MB/sec execute 529 sec latency 81.967 ms 2024-03-22T02:38:10.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 187820 15.38 MB/sec execute 530 sec latency 143.961 ms 2024-03-22T02:38:10.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:10 smithi120 ceph-mon[36803]: pgmap v1099: 129 pgs: 129 active+clean; 290 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 544 op/s 2024-03-22T02:38:11.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:10 smithi083 ceph-mon[36340]: pgmap v1099: 129 pgs: 129 active+clean; 290 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 544 op/s 2024-03-22T02:38:11.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:10 smithi028 ceph-mon[31916]: pgmap v1099: 129 pgs: 129 active+clean; 290 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 4.8 MiB/s wr, 544 op/s 2024-03-22T02:38:11.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 188081 15.38 MB/sec execute 531 sec latency 273.266 ms 2024-03-22T02:38:12.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 188293 15.37 MB/sec execute 532 sec latency 485.880 ms 2024-03-22T02:38:12.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:12 smithi120 ceph-mon[36803]: pgmap v1100: 129 pgs: 129 active+clean; 282 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2024-03-22T02:38:12.999 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:12 smithi083 ceph-mon[36340]: pgmap v1100: 129 pgs: 129 active+clean; 282 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2024-03-22T02:38:13.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:12 smithi028 ceph-mon[31916]: pgmap v1100: 129 pgs: 129 active+clean; 282 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.1 MiB/s wr, 502 op/s 2024-03-22T02:38:13.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 188482 15.36 MB/sec execute 533 sec latency 149.168 ms 2024-03-22T02:38:14.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 188733 15.35 MB/sec execute 534 sec latency 80.839 ms 2024-03-22T02:38:14.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:14 smithi120 ceph-mon[36803]: pgmap v1101: 129 pgs: 129 active+clean; 284 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-03-22T02:38:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:14 smithi083 ceph-mon[36340]: pgmap v1101: 129 pgs: 129 active+clean; 284 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-03-22T02:38:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:14 smithi028 ceph-mon[31916]: pgmap v1101: 129 pgs: 129 active+clean; 284 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 3.9 MiB/s wr, 482 op/s 2024-03-22T02:38:15.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 189017 15.35 MB/sec execute 535 sec latency 96.706 ms 2024-03-22T02:38:16.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 189317 15.35 MB/sec execute 536 sec latency 141.994 ms 2024-03-22T02:38:16.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:16 smithi120 ceph-mon[36803]: pgmap v1102: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-03-22T02:38:17.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:16 smithi083 ceph-mon[36340]: pgmap v1102: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-03-22T02:38:17.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:16 smithi028 ceph-mon[31916]: pgmap v1102: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2024-03-22T02:38:17.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 189609 15.34 MB/sec execute 537 sec latency 235.380 ms 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 -- 172.21.15.28:0/2524740092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 msgr2=0x7f4778104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 --2- 172.21.15.28:0/2524740092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f4778104a20 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f4774009900 tx=0x7f477402bc30 comp rx=0 tx=0).stop 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 -- 172.21.15.28:0/2524740092 shutdown_connections 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 --2- 172.21.15.28:0/2524740092 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778107890 0x7f4778109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 --2- 172.21.15.28:0/2524740092 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778104f60 0x7f4778107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 --2- 172.21.15.28:0/2524740092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f4778104a20 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 -- 172.21.15.28:0/2524740092 >> 172.21.15.28:0/2524740092 conn(0x7f47780fc020 msgr2=0x7f47780fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 -- 172.21.15.28:0/2524740092 shutdown_connections 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.063+0000 7f47802af640 1 -- 172.21.15.28:0/2524740092 wait complete. 2024-03-22T02:38:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.064+0000 7f47802af640 1 Processor -- start 2024-03-22T02:38:18.067 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.064+0000 7f47802af640 1 -- start start 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.064+0000 7f47802af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.064+0000 7f47802af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778104f60 0x7f477819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.064+0000 7f47802af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778107890 0x7f47781a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f47802af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f477810fa90 con 0x7f4778102630 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f47802af640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f477810f910 con 0x7f4778104f60 2024-03-22T02:38:18.068 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f47802af640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f477810fc10 con 0x7f4778107890 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45078/0 (socket says 172.21.15.28:45078) 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 -- 172.21.15.28:0/730784390 learned_addr learned my addr 172.21.15.28:0/730784390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477d84a640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778104f60 0x7f477819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 -- 172.21.15.28:0/730784390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778107890 msgr2=0x7f47781a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778107890 0x7f47781a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 -- 172.21.15.28:0/730784390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778104f60 msgr2=0x7f477819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778104f60 0x7f477819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 -- 172.21.15.28:0/730784390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47781a1c70 con 0x7f4778102630 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.065+0000 7f477e04b640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4774000c00 tx=0x7f477402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.066+0000 7f476f7fe640 1 -- 172.21.15.28:0/730784390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774043020 con 0x7f4778102630 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.066+0000 7f476f7fe640 1 -- 172.21.15.28:0/730784390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47740054e0 con 0x7f4778102630 2024-03-22T02:38:18.069 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.066+0000 7f476f7fe640 1 -- 172.21.15.28:0/730784390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774003c40 con 0x7f4778102630 2024-03-22T02:38:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.066+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47781a1f00 con 0x7f4778102630 2024-03-22T02:38:18.070 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.067+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781a22b0 con 0x7f4778102630 2024-03-22T02:38:18.073 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.070+0000 7f476f7fe640 1 -- 172.21.15.28:0/730784390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f4774046020 con 0x7f4778102630 2024-03-22T02:38:18.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.072+0000 7f4782f73740 1 Processor -- start 2024-03-22T02:38:18.074 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.072+0000 7f476f7fe640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 0x7f47640a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 -- start start 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f114e0 0x55b6b1f11900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f17040 0x55b6b1f194d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1e76040 con 0x55b6b1f11ed0 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1e1e9e0 con 0x55b6b1f17040 2024-03-22T02:38:18.075 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f4782f73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1e71ff0 con 0x55b6b1f114e0 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f476f7fe640 1 -- 172.21.15.28:0/730784390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f47740f1310 con 0x7f4778102630 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.073+0000 7f477e84c640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 0x7f47640a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.074+0000 7f477d84a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.074+0000 7f477d84a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45092/0 (socket says 172.21.15.28:45092) 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.074+0000 7f477d84a640 1 -- 172.21.15.28:0/1167519397 learned_addr learned my addr 172.21.15.28:0/1167519397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:18.076 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477e04b640 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f114e0 0x55b6b1f11900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 -- 172.21.15.28:0/1167519397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f114e0 msgr2=0x55b6b1f11900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f114e0 0x55b6b1f11900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 -- 172.21.15.28:0/1167519397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f17040 msgr2=0x55b6b1f194d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:38:18.077 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f17040 0x55b6b1f194d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 -- 172.21.15.28:0/1167519397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6b1b8a210 con 0x55b6b1f11ed0 2024-03-22T02:38:18.078 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.075+0000 7f477d84a640 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f4768011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727d82c3523394b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.076+0000 7f476cbfe640 1 -- 172.21.15.28:0/1167519397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476801e070 con 0x55b6b1f11ed0 2024-03-22T02:38:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.076+0000 7f476cbfe640 1 -- 172.21.15.28:0/1167519397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4768019b10 con 0x55b6b1f11ed0 2024-03-22T02:38:18.079 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.076+0000 7f476cbfe640 1 -- 172.21.15.28:0/1167519397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476800eb50 con 0x55b6b1f11ed0 2024-03-22T02:38:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.078+0000 7f4782f73740 1 -- 172.21.15.28:0/1167519397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 msgr2=0x55b6b1f169d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.080 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.078+0000 7f4782f73740 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f4768011810 comp rx=0 tx=0).stop 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f477e84c640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 0x7f47640a5500 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4770005c40 tx=0x7f4770005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f4782f73740 1 -- 172.21.15.28:0/1167519397 shutdown_connections 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f4782f73740 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f17040 0x55b6b1f194d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f4782f73740 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f11ed0 0x55b6b1f169d0 secure :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f4768009c60 tx=0x7f4768011810 comp rx=0 tx=0).stop 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f4782f73740 1 --2- 172.21.15.28:0/1167519397 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f114e0 0x55b6b1f11900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.080+0000 7f4782f73740 1 -- 172.21.15.28:0/1167519397 >> 172.21.15.28:0/1167519397 conn(0x55b6b1f0ca50 msgr2=0x55b6b1f0ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.082+0000 7f4782f73740 1 -- 172.21.15.28:0/1167519397 shutdown_connections 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.082+0000 7f4782f73740 1 -- 172.21.15.28:0/1167519397 wait complete. 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.082+0000 7f4782f73740 1 -- start start 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.082+0000 7f4782f73740 1 Processor -- start 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f34ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f3c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1f2b000 con 0x55b6b1f114e0 2024-03-22T02:38:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1e31db0 con 0x55b6b1f3a440 2024-03-22T02:38:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55b6b1f2b180 con 0x55b6b1f17040 2024-03-22T02:38:18.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f4782f73740 10 client.0 _ll_register_callbacks cb 0x55b6b1b89a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f477d84a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f477d84a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:42956/0 (socket says 172.21.15.28:42956) 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f477d84a640 1 -- 172.21.15.28:0/3495942898 learned_addr learned my addr 172.21.15.28:0/3495942898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f477e04b640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f34ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.083+0000 7f477e84c640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f3c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 -- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 msgr2=0x55b6b1f3c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f3c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 -- 172.21.15.28:0/3495942898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 msgr2=0x55b6b1f34ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f34ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 -- 172.21.15.28:0/3495942898 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6b1c56620 con 0x55b6b1f17040 2024-03-22T02:38:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f477d84a640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4768005890 tx=0x7f4768009990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cef7e7c72f47394f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.084+0000 7f4761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:38:18.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.085+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768029020 con 0x55b6b1f17040 2024-03-22T02:38:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.085+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f4768004340 con 0x55b6b1f17040 2024-03-22T02:38:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.085+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4768019b10 con 0x55b6b1f17040 2024-03-22T02:38:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.086+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f476801e040 con 0x55b6b1f17040 2024-03-22T02:38:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.086+0000 7f4782f73740 10 client.30205 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:18.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.086+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b6b1f2f1c0 con 0x55b6b1f17040 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.087+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4768019b10 con 0x55b6b1f17040 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f4768019430 con 0x55b6b1f17040 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 10 client.30205 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 10 client.30205 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 20 client.30205 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f30570 0x55b6b1f32960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 4 client.30205 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:38:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.088+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6b1e83580 con 0x55b6b1f30570 2024-03-22T02:38:18.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.092+0000 7f477e04b640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f30570 0x55b6b1f32960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.096 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.095+0000 7f477e04b640 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f30570 0x55b6b1f32960 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55f881ea0451aa6c server_cookie=bfde3fcab0e623ae in_seq=0 out_seq=0 2024-03-22T02:38:18.097 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.095+0000 7f4761ffb640 10 client.30205 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.099+0000 7f4761ffb640 1 -- 172.21.15.28:0/3495942898 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b6b1e83580 con 0x55b6b1f30570 2024-03-22T02:38:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.099+0000 7f4761ffb640 10 client.30205 handle_command_reply: tid=0 2024-03-22T02:38:18.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.099+0000 7f4782f73740 1 client.30205 shutdown 2024-03-22T02:38:18.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.101+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 msgr2=0x55b6b1f35410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.103 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.101+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4768005890 tx=0x7f4768009990 comp rx=0 tx=0).stop 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 shutdown_connections 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f30570 0x55b6b1f32960 crc :-1 s=READY pgs=450 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f3c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f17040 0x55b6b1f35410 secure :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4768005890 tx=0x7f4768009990 comp rx=0 tx=0).stop 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 --2- 172.21.15.28:0/3495942898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f34ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.102+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 >> 172.21.15.28:0/3495942898 conn(0x55b6b1f0ca50 msgr2=0x55b6b1f0bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.103+0000 7f4761ffb640 0 client.30205 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:18.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.104+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 shutdown_connections 2024-03-22T02:38:18.109 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.107+0000 7f4782f73740 1 -- 172.21.15.28:0/3495942898 wait complete. 2024-03-22T02:38:18.111 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.109+0000 7f4782f73740 20 client.30205 trim_cache size 0 max 16384 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 Processor -- start 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 -- start start 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f3a440 0x55b6b1f103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f13770 0x55b6b1f0b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47781a27f0 con 0x55b6b1f114e0 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4778071930 con 0x55b6b1f13770 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f4782f73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f477806f780 con 0x55b6b1f3a440 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.142+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45114/0 (socket says 172.21.15.28:45114) 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 -- 172.21.15.28:0/1613757339 learned_addr learned my addr 172.21.15.28:0/1613757339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e84c640 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f13770 0x55b6b1f0b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 -- 172.21.15.28:0/1613757339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f3a440 msgr2=0x55b6b1f103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f3a440 0x55b6b1f103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 -- 172.21.15.28:0/1613757339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f13770 msgr2=0x55b6b1f0b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f13770 0x55b6b1f0b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 -- 172.21.15.28:0/1613757339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6b1f2b7b0 con 0x55b6b1f114e0 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.143+0000 7f477e04b640 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f47740b9fa0 tx=0x7f47740ee870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37236f62123e142f server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.145+0000 7f4763fff640 1 -- 172.21.15.28:0/1613757339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740eee60 con 0x55b6b1f114e0 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.145+0000 7f4763fff640 1 -- 172.21.15.28:0/1613757339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47740d6070 con 0x55b6b1f114e0 2024-03-22T02:38:18.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.145+0000 7f4763fff640 1 -- 172.21.15.28:0/1613757339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740c1720 con 0x55b6b1f114e0 2024-03-22T02:38:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.147+0000 7f4782f73740 1 -- 172.21.15.28:0/1613757339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 msgr2=0x55b6b1f168f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.147+0000 7f4782f73740 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f47740b9fa0 tx=0x7f47740ee870 comp rx=0 tx=0).stop 2024-03-22T02:38:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.148+0000 7f4782f73740 1 -- 172.21.15.28:0/1613757339 shutdown_connections 2024-03-22T02:38:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.148+0000 7f4782f73740 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f13770 0x55b6b1f0b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.148+0000 7f4782f73740 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f3a440 0x55b6b1f103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.148+0000 7f4782f73740 1 --2- 172.21.15.28:0/1613757339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f168f0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.148+0000 7f4782f73740 1 -- 172.21.15.28:0/1613757339 >> 172.21.15.28:0/1613757339 conn(0x55b6b1f0ca50 msgr2=0x55b6b1f0fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.150+0000 7f4782f73740 1 -- 172.21.15.28:0/1613757339 shutdown_connections 2024-03-22T02:38:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.150+0000 7f4782f73740 1 -- 172.21.15.28:0/1613757339 wait complete. 2024-03-22T02:38:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 -- start start 2024-03-22T02:38:18.152 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 Processor -- start 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f13770 0x55b6b1f5b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f1dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f477806fdf0 con 0x55b6b1f114e0 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f4778071260 con 0x55b6b1f3a440 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f47780704c0 con 0x55b6b1f13770 2024-03-22T02:38:18.153 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.151+0000 7f4782f73740 10 client.0 _ll_register_callbacks cb 0x7f476801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:18.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45118/0 (socket says 172.21.15.28:45118) 2024-03-22T02:38:18.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 -- 172.21.15.28:0/3796828232 learned_addr learned my addr 172.21.15.28:0/3796828232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:18.154 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e84c640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f1dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 -- 172.21.15.28:0/3796828232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f13770 msgr2=0x55b6b1f5b660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f13770 0x55b6b1f5b660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 -- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 msgr2=0x55b6b1f1dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f1dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e04b640 1 -- 172.21.15.28:0/3796828232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6b1f5e730 con 0x55b6b1f114e0 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.152+0000 7f477e84c640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f1dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:18.155 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.153+0000 7f477e04b640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f47740b9a30 tx=0x7f47740bdd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cee0799eab3c8e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:18.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.154+0000 7f4761ffb640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:18.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.154+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47740eee60 con 0x55b6b1f114e0 2024-03-22T02:38:18.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.154+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f47740bb670 con 0x55b6b1f114e0 2024-03-22T02:38:18.156 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.154+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47740d6070 con 0x55b6b1f114e0 2024-03-22T02:38:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.155+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f4774005690 con 0x55b6b1f114e0 2024-03-22T02:38:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.155+0000 7f4782f73740 10 client.20448 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.155+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55b6b1f1f070 con 0x55b6b1f114e0 2024-03-22T02:38:18.157 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.156+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47740d6070 con 0x55b6b1f114e0 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f47740b9c40 con 0x55b6b1f114e0 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 10 client.20448 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 10 client.20448 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 20 client.20448 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f1fa10 0x55b6b1f21e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 4 client.20448 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b6b1e7d8d0 con 0x55b6b1f1fa10 2024-03-22T02:38:18.159 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.157+0000 7f477d84a640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f1fa10 0x55b6b1f21e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:18.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.158+0000 7f477d84a640 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f1fa10 0x55b6b1f21e30 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f1777aadf0776c0 server_cookie=84362d25f0c5d225 in_seq=0 out_seq=0 2024-03-22T02:38:18.160 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:18.158+0000 7f4761ffb640 10 client.20448 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:18.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 189750 15.34 MB/sec execute 538 sec latency 338.447 ms 2024-03-22T02:38:18.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:18 smithi120 ceph-mon[36803]: pgmap v1103: 129 pgs: 129 active+clean; 295 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 504 op/s 2024-03-22T02:38:19.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:18 smithi083 ceph-mon[36340]: pgmap v1103: 129 pgs: 129 active+clean; 295 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 504 op/s 2024-03-22T02:38:19.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:18 smithi028 ceph-mon[31916]: pgmap v1103: 129 pgs: 129 active+clean; 295 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.0 MiB/s wr, 504 op/s 2024-03-22T02:38:19.459 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4761ffb640 1 -- 172.21.15.28:0/3796828232 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b6b1e7d8d0 con 0x55b6b1f1fa10 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4761ffb640 10 client.20448 handle_command_reply: tid=0 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 client.20448 shutdown 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 msgr2=0x55b6b1f5b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f47740b9a30 tx=0x7f47740bdd40 comp rx=0 tx=0).stop 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 shutdown_connections 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55b6b1f1fa10 0x55b6b1f21e30 crc :-1 s=READY pgs=451 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55b6b1f3a440 0x55b6b1f1dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b6b1f13770 0x55b6b1f5b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 --2- 172.21.15.28:0/3796828232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b6b1f114e0 0x55b6b1f5b120 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 >> 172.21.15.28:0/3796828232 conn(0x55b6b1f0ca50 msgr2=0x55b6b1f34590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:19.460 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4761ffb640 0 client.20448 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 shutdown_connections 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 1 -- 172.21.15.28:0/3796828232 wait complete. 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.457+0000 7f4782f73740 20 client.20448 trim_cache size 0 max 16384 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:38:19.461 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:38:19.462 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 msgr2=0x7f47640a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 0x7f47640a5500 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4770005c40 tx=0x7f4770005bd0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 msgr2=0x7f477819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f4774000c00 tx=0x7f477402c650 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 shutdown_connections 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f47640a3040 0x7f47640a5500 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4778107890 0x7f47781a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f4778104f60 0x7f477819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 --2- 172.21.15.28:0/730784390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4778102630 0x7f477819a050 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 >> 172.21.15.28:0/730784390 conn(0x7f47780fc020 msgr2=0x7f4778102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 shutdown_connections 2024-03-22T02:38:19.463 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:19.458+0000 7f47802af640 1 -- 172.21.15.28:0/730784390 wait complete. 2024-03-22T02:38:19.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 189861 15.32 MB/sec execute 539 sec latency 1000.620 ms 2024-03-22T02:38:20.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 190334 15.34 MB/sec execute 540 sec latency 67.173 ms 2024-03-22T02:38:20.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:20 smithi120 ceph-mon[36803]: pgmap v1104: 129 pgs: 129 active+clean; 293 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-03-22T02:38:21.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:20 smithi083 ceph-mon[36340]: pgmap v1104: 129 pgs: 129 active+clean; 293 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-03-22T02:38:21.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:20 smithi028 ceph-mon[31916]: pgmap v1104: 129 pgs: 129 active+clean; 293 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 4.5 MiB/s wr, 442 op/s 2024-03-22T02:38:21.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 190723 15.35 MB/sec execute 541 sec latency 114.199 ms 2024-03-22T02:38:22.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 190949 15.34 MB/sec execute 542 sec latency 230.875 ms 2024-03-22T02:38:22.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:22 smithi120 ceph-mon[36803]: pgmap v1105: 129 pgs: 129 active+clean; 296 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 343 op/s 2024-03-22T02:38:23.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:22 smithi083 ceph-mon[36340]: pgmap v1105: 129 pgs: 129 active+clean; 296 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 343 op/s 2024-03-22T02:38:23.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:22 smithi028 ceph-mon[31916]: pgmap v1105: 129 pgs: 129 active+clean; 296 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 343 op/s 2024-03-22T02:38:23.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 191189 15.34 MB/sec execute 543 sec latency 168.796 ms 2024-03-22T02:38:24.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 191546 15.36 MB/sec execute 544 sec latency 49.675 ms 2024-03-22T02:38:24.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:24 smithi120 ceph-mon[36803]: pgmap v1106: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 411 op/s 2024-03-22T02:38:25.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:24 smithi083 ceph-mon[36340]: pgmap v1106: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 411 op/s 2024-03-22T02:38:25.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:24 smithi028 ceph-mon[31916]: pgmap v1106: 129 pgs: 129 active+clean; 292 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.2 MiB/s wr, 411 op/s 2024-03-22T02:38:25.700 INFO:tasks.workunit.client.0.smithi028.stdout: 10 191849 15.36 MB/sec execute 545 sec latency 131.894 ms 2024-03-22T02:38:26.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 192161 15.36 MB/sec execute 546 sec latency 94.786 ms 2024-03-22T02:38:26.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:26 smithi120 ceph-mon[36803]: pgmap v1107: 129 pgs: 129 active+clean; 287 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-03-22T02:38:27.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:26 smithi083 ceph-mon[36340]: pgmap v1107: 129 pgs: 129 active+clean; 287 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-03-22T02:38:27.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:26 smithi028 ceph-mon[31916]: pgmap v1107: 129 pgs: 129 active+clean; 287 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.9 MiB/s wr, 552 op/s 2024-03-22T02:38:27.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 192328 15.34 MB/sec execute 547 sec latency 327.300 ms 2024-03-22T02:38:28.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 192558 15.33 MB/sec execute 548 sec latency 169.441 ms 2024-03-22T02:38:28.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:28 smithi120 ceph-mon[36803]: pgmap v1108: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 506 op/s 2024-03-22T02:38:29.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:28 smithi083 ceph-mon[36340]: pgmap v1108: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 506 op/s 2024-03-22T02:38:29.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:28 smithi028 ceph-mon[31916]: pgmap v1108: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.8 MiB/s wr, 506 op/s 2024-03-22T02:38:29.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 192900 15.33 MB/sec execute 549 sec latency 138.155 ms 2024-03-22T02:38:30.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 193250 15.35 MB/sec execute 550 sec latency 117.243 ms 2024-03-22T02:38:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:30 smithi120 ceph-mon[36803]: pgmap v1109: 129 pgs: 129 active+clean; 291 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 496 op/s 2024-03-22T02:38:31.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:30 smithi083 ceph-mon[36340]: pgmap v1109: 129 pgs: 129 active+clean; 291 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 496 op/s 2024-03-22T02:38:31.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:30 smithi028 ceph-mon[31916]: pgmap v1109: 129 pgs: 129 active+clean; 291 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 4.3 MiB/s wr, 496 op/s 2024-03-22T02:38:31.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 193632 15.35 MB/sec execute 551 sec latency 61.285 ms 2024-03-22T02:38:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 -- 172.21.15.28:0/2985892883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0102640 msgr2=0x7f0ea0104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/2985892883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0102640 0x7f0ea0104a30 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009900 tx=0x7f0e9c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:38:32.557 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 -- 172.21.15.28:0/2985892883 shutdown_connections 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/2985892883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea01078a0 0x7f0ea0109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/2985892883 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea0104f70 0x7f0ea0107360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/2985892883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0102640 0x7f0ea0104a30 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 -- 172.21.15.28:0/2985892883 >> 172.21.15.28:0/2985892883 conn(0x7f0ea00fc050 msgr2=0x7f0ea00fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 -- 172.21.15.28:0/2985892883 shutdown_connections 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.554+0000 7f0ea7b13640 1 -- 172.21.15.28:0/2985892883 wait complete. 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 Processor -- start 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 -- start start 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 0x7f0ea0101fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.558 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 0x7f0ea0100b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea010d860 con 0x7f0ea0104f70 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea010d6e0 con 0x7f0ea01078a0 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea7b13640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea010d9e0 con 0x7f0ea0102640 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35752/0 (socket says 172.21.15.28:35752) 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea60b0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 0x7f0ea0100b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 0x7f0ea0101fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3408943700 learned_addr learned my addr 172.21.15.28:0/3408943700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3408943700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 msgr2=0x7f0ea0101fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 0x7f0ea0101fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3408943700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 msgr2=0x7f0ea0100b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 0x7f0ea0100b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.559 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3408943700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ea01013c0 con 0x7f0ea0104f70 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea58af640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 0x7f0ea0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.555+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 0x7f0ea0100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000e9c0 tx=0x7f0e9000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0e96ffd640 1 -- 172.21.15.28:0/3408943700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9000cd80 con 0x7f0ea0104f70 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0e96ffd640 1 -- 172.21.15.28:0/3408943700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e90005500 con 0x7f0ea0104f70 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ea01a1f80 con 0x7f0ea0104f70 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0e96ffd640 1 -- 172.21.15.28:0/3408943700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90010690 con 0x7f0ea0104f70 2024-03-22T02:38:32.560 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.556+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ea01a23e0 con 0x7f0ea0104f70 2024-03-22T02:38:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.559+0000 7f0e96ffd640 1 -- 172.21.15.28:0/3408943700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0e900046c0 con 0x7f0ea0104f70 2024-03-22T02:38:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.559+0000 7f0eaa7dc740 1 Processor -- start 2024-03-22T02:38:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.560+0000 7f0e96ffd640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 0x7f0e8c0a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.561 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.560+0000 7f0e96ffd640 1 -- 172.21.15.28:0/3408943700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f0e900cfb10 con 0x7f0ea0104f70 2024-03-22T02:38:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 -- start start 2024-03-22T02:38:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 0x7f0e8c0a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.563 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bb5040 0x55a0e3bb74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3b14040 con 0x55a0e3baf4e0 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3abc9e0 con 0x55a0e3bb5040 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3b0fff0 con 0x55a0e3bafed0 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35758/0 (socket says 172.21.15.28:35758) 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea58af640 1 -- 172.21.15.28:0/1288955746 learned_addr learned my addr 172.21.15.28:0/1288955746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.561+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bb5040 0x55a0e3bb74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 -- 172.21.15.28:0/1288955746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 msgr2=0x55a0e3bb49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 -- 172.21.15.28:0/1288955746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bb5040 msgr2=0x55a0e3bb74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.564 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bb5040 0x55a0e3bb74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 -- 172.21.15.28:0/1288955746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e3828210 con 0x55a0e3baf4e0 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c036040 tx=0x7f0e9c02cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f32fbc05b9073752 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 0x7f0e8c0a5480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0e98005fd0 tx=0x7f0e9800f3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0e8bfff640 1 -- 172.21.15.28:0/1288955746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c005d50 con 0x55a0e3baf4e0 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0e8bfff640 1 -- 172.21.15.28:0/1288955746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c005520 con 0x55a0e3baf4e0 2024-03-22T02:38:32.565 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.562+0000 7f0e8bfff640 1 -- 172.21.15.28:0/1288955746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c004c20 con 0x55a0e3baf4e0 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1288955746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 msgr2=0x55a0e3baf900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c036040 tx=0x7f0e9c02cd30 comp rx=0 tx=0).stop 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1288955746 shutdown_connections 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bb5040 0x55a0e3bb74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1288955746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3baf900 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.563+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1288955746 >> 172.21.15.28:0/1288955746 conn(0x55a0e3baaa50 msgr2=0x55a0e3baae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1288955746 shutdown_connections 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1288955746 wait complete. 2024-03-22T02:38:32.571 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- start start 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 Processor -- start 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bb9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bc09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3bb7f90 con 0x55a0e3bafed0 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3acfdb0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a0e3bb8110 con 0x55a0e3bb5040 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.564+0000 7f0eaa7dc740 10 client.0 _ll_register_callbacks cb 0x55a0e3827a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bb9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35774/0 (socket says 172.21.15.28:35774) 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3188269004 learned_addr learned my addr 172.21.15.28:0/3188269004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:32.572 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bc09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3188269004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 msgr2=0x55a0e3bc09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bc09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 msgr2=0x55a0e3bb9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bb9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 -- 172.21.15.28:0/3188269004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e38f4620 con 0x55a0e3bafed0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea58af640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bb9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cd200 tx=0x7f0e9009b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bca49746c2da48 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0e897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e900b4070 con 0x55a0e3bafed0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f0e900a45b0 con 0x55a0e3bafed0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e9c005520 con 0x55a0e3bafed0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0e9009ff80 con 0x55a0e3bafed0 2024-03-22T02:38:32.573 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0eaa7dc740 10 client.20460 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.565+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a0e3bc3e10 con 0x55a0e3bafed0 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c005520 con 0x55a0e3bafed0 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f0e9009b280 con 0x55a0e3bafed0 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 10 client.20460 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 10 client.20460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 20 client.20460 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3bc5500 0x55a0e3bc7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 4 client.20460 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0e3b21580 con 0x55a0e3bc5500 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0ea58af640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3bc5500 0x55a0e3bc7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.574 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0ea58af640 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3bc5500 0x55a0e3bc7930 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2f12afec2ba0184 server_cookie=f4d5040346a4c0c9 in_seq=0 out_seq=0 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.566+0000 7f0e897fa640 10 client.20460 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0e897fa640 1 -- 172.21.15.28:0/3188269004 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a0e3b21580 con 0x55a0e3bc5500 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0e897fa640 10 client.20460 handle_command_reply: tid=0 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 client.20460 shutdown 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 msgr2=0x55a0e3bb9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f0e900cd200 tx=0x7f0e9009b860 comp rx=0 tx=0).stop 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 shutdown_connections 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3bc5500 0x55a0e3bc7930 crc :-1 s=READY pgs=452 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bc09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bafed0 0x55a0e3bb9990 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/3188269004 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bb9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0e897fa640 0 client.20460 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:32.575 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 >> 172.21.15.28:0/3188269004 conn(0x55a0e3baaa50 msgr2=0x55a0e3ba9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 shutdown_connections 2024-03-22T02:38:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/3188269004 wait complete. 2024-03-22T02:38:32.576 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.569+0000 7f0eaa7dc740 20 client.20460 trim_cache size 0 max 16384 2024-03-22T02:38:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.584+0000 7f0eaa7dc740 1 Processor -- start 2024-03-22T02:38:32.586 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.584+0000 7f0eaa7dc740 1 -- start start 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bc0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01af410 con 0x55a0e3bb5040 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01acce0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01aab10 con 0x55a0e3bafed0 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea60b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bc0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bc0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:56842/0 (socket says 172.21.15.28:56842) 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea50ae640 1 -- 172.21.15.28:0/1132773331 learned_addr learned my addr 172.21.15.28:0/1132773331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:32.587 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea60b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35784/0 (socket says 172.21.15.28:35784) 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 -- 172.21.15.28:0/1132773331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 msgr2=0x55a0e3bc0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bc0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 -- 172.21.15.28:0/1132773331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 msgr2=0x55a0e3ba93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 -- 172.21.15.28:0/1132773331 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e3bb02b0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.585+0000 7f0ea58af640 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bae3c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c02c7b0 tx=0x7f0e9c005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c29ff9581eeb670 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0e8b7fe640 1 -- 172.21.15.28:0/1132773331 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c005030 con 0x55a0e3baf4e0 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0e8b7fe640 1 -- 172.21.15.28:0/1132773331 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c050080 con 0x55a0e3baf4e0 2024-03-22T02:38:32.588 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0e8b7fe640 1 -- 172.21.15.28:0/1132773331 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c004920 con 0x55a0e3baf4e0 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1132773331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 msgr2=0x55a0e3bae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bae3c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c02c7b0 tx=0x7f0e9c005e50 comp rx=0 tx=0).stop 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1132773331 shutdown_connections 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3bb5040 0x55a0e3ba93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bafed0 0x55a0e3bc0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/1132773331 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bae3c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1132773331 >> 172.21.15.28:0/1132773331 conn(0x55a0e3baaa50 msgr2=0x55a0e3badcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1132773331 shutdown_connections 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.586+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/1132773331 wait complete. 2024-03-22T02:38:32.592 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 -- start start 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 Processor -- start 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 0x55a0e3bbe3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bbf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01ab1a0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01ac610 con 0x55a0e3bafed0 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0ea01ab870 con 0x55a0e3bb5040 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0eaa7dc740 10 client.0 _ll_register_callbacks cb 0x7f0e9009bc00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:32.593 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.587+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea50ae640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 0x55a0e3bbe3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35796/0 (socket says 172.21.15.28:35796) 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 -- 172.21.15.28:0/500332286 learned_addr learned my addr 172.21.15.28:0/500332286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea60b0640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bbf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 -- 172.21.15.28:0/500332286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 msgr2=0x55a0e3bbf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bbf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 -- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 msgr2=0x55a0e3bbe3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 0x55a0e3bbe3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 -- 172.21.15.28:0/500332286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e3bbf9d0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 0x55a0e3bbe3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0ea58af640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c02cd70 tx=0x7f0e9c0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b0f56aa8bd1938 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:32.594 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0e897fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c050780 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f0e9c005040 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e9c050080 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f0e9c063050 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0eaa7dc740 10 client.20466 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.588+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a0e3bc0ce0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c050080 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f0e9c0035f0 con 0x55a0e3baf4e0 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 10 client.20466 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 10 client.20466 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 20 client.20466 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:32.595 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3c2bb70 0x55a0e3c2df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 4 client.20466 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:38:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a0e3b1b8d0 con 0x55a0e3c2bb70 2024-03-22T02:38:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3c2bb70 0x55a0e3c2df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0e897fa640 10 client.20466 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:32.596 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:32.589+0000 7f0ea50ae640 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3c2bb70 0x55a0e3c2df60 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1dbdfe0951eaab68 server_cookie=62b48dcf6cc39853 in_seq=0 out_seq=0 2024-03-22T02:38:32.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 193893 15.36 MB/sec execute 552 sec latency 354.981 ms 2024-03-22T02:38:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:32 smithi120 ceph-mon[36803]: pgmap v1110: 129 pgs: 129 active+clean; 292 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 431 op/s 2024-03-22T02:38:33.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:32 smithi083 ceph-mon[36340]: pgmap v1110: 129 pgs: 129 active+clean; 292 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 431 op/s 2024-03-22T02:38:33.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:32 smithi028 ceph-mon[31916]: pgmap v1110: 129 pgs: 129 active+clean; 292 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 3.4 MiB/s wr, 431 op/s 2024-03-22T02:38:33.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 193898 15.33 MB/sec execute 553 sec latency 1099.162 ms 2024-03-22T02:38:33.954 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.951+0000 7f0e897fa640 1 -- 172.21.15.28:0/500332286 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a0e3b1b8d0 con 0x55a0e3c2bb70 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.951+0000 7f0e897fa640 10 client.20466 handle_command_reply: tid=0 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.951+0000 7f0eaa7dc740 1 client.20466 shutdown 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.951+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 msgr2=0x55a0e3bbde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.951+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c02cd70 tx=0x7f0e9c0044b0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 shutdown_connections 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a0e3c2bb70 0x55a0e3c2df60 crc :-1 s=READY pgs=453 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.955 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0e897fa640 0 client.20466 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a0e3bb5040 0x55a0e3bbf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a0e3bafed0 0x55a0e3bbe3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 --2- 172.21.15.28:0/500332286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a0e3baf4e0 0x55a0e3bbde90 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 >> 172.21.15.28:0/500332286 conn(0x55a0e3baaa50 msgr2=0x55a0e3bae540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 shutdown_connections 2024-03-22T02:38:33.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 1 -- 172.21.15.28:0/500332286 wait complete. 2024-03-22T02:38:33.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.952+0000 7f0eaa7dc740 20 client.20466 trim_cache size 0 max 16384 2024-03-22T02:38:33.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:38:33.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:38:33.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:38:33.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:38:33.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 msgr2=0x7f0e8c0a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:33.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 0x7f0e8c0a5480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0e98005fd0 tx=0x7f0e9800f3c0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 msgr2=0x7f0ea01005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:33.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000e9c0 tx=0x7f0e9000ee80 comp rx=0 tx=0).stop 2024-03-22T02:38:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 shutdown_connections 2024-03-22T02:38:33.960 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0e8c0a2fc0 0x7f0e8c0a5480 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0ea01078a0 0x7f0ea0100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ea0104f70 0x7f0ea01005f0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 --2- 172.21.15.28:0/3408943700 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea0102640 0x7f0ea0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 >> 172.21.15.28:0/3408943700 conn(0x7f0ea00fc050 msgr2=0x7f0ea0102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 shutdown_connections 2024-03-22T02:38:33.961 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:33.953+0000 7f0ea7b13640 1 -- 172.21.15.28:0/3408943700 wait complete. 2024-03-22T02:38:34.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 194204 15.33 MB/sec execute 554 sec latency 1234.895 ms 2024-03-22T02:38:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:34 smithi120 ceph-mon[36803]: pgmap v1111: 129 pgs: 129 active+clean; 297 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-03-22T02:38:35.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:34 smithi083 ceph-mon[36340]: pgmap v1111: 129 pgs: 129 active+clean; 297 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-03-22T02:38:35.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:34 smithi028 ceph-mon[31916]: pgmap v1111: 129 pgs: 129 active+clean; 297 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 3.8 MiB/s wr, 414 op/s 2024-03-22T02:38:35.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 194537 15.33 MB/sec execute 555 sec latency 135.517 ms 2024-03-22T02:38:36.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 194861 15.34 MB/sec execute 556 sec latency 88.535 ms 2024-03-22T02:38:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:36 smithi028 ceph-mon[31916]: pgmap v1112: 129 pgs: 129 active+clean; 296 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.5 MiB/s wr, 502 op/s 2024-03-22T02:38:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:36 smithi120 ceph-mon[36803]: pgmap v1112: 129 pgs: 129 active+clean; 296 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.5 MiB/s wr, 502 op/s 2024-03-22T02:38:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:36 smithi083 ceph-mon[36340]: pgmap v1112: 129 pgs: 129 active+clean; 296 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.5 MiB/s wr, 502 op/s 2024-03-22T02:38:37.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 195061 15.34 MB/sec execute 557 sec latency 244.157 ms 2024-03-22T02:38:38.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 195349 15.34 MB/sec execute 558 sec latency 102.207 ms 2024-03-22T02:38:38.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:38 smithi120 ceph-mon[36803]: pgmap v1113: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 428 op/s 2024-03-22T02:38:39.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:38 smithi083 ceph-mon[36340]: pgmap v1113: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 428 op/s 2024-03-22T02:38:39.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:38 smithi028 ceph-mon[31916]: pgmap v1113: 129 pgs: 129 active+clean; 288 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 3.6 MiB/s wr, 428 op/s 2024-03-22T02:38:39.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 195653 15.34 MB/sec execute 559 sec latency 131.620 ms 2024-03-22T02:38:39.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:39 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:40.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:39 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:40.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:39 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:38:40.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 195935 15.33 MB/sec execute 560 sec latency 131.525 ms 2024-03-22T02:38:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:40 smithi120 ceph-mon[36803]: pgmap v1114: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 542 op/s 2024-03-22T02:38:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:40 smithi083 ceph-mon[36340]: pgmap v1114: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 542 op/s 2024-03-22T02:38:41.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:40 smithi028 ceph-mon[31916]: pgmap v1114: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.6 MiB/s wr, 542 op/s 2024-03-22T02:38:41.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 196282 15.33 MB/sec execute 561 sec latency 95.960 ms 2024-03-22T02:38:41.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:38:41.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:38:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:38:42.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:38:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:38:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:38:42.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:38:42.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 196461 15.32 MB/sec execute 562 sec latency 355.410 ms 2024-03-22T02:38:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:42 smithi120 ceph-mon[36803]: pgmap v1115: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 499 op/s 2024-03-22T02:38:43.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:42 smithi083 ceph-mon[36340]: pgmap v1115: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 499 op/s 2024-03-22T02:38:43.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:42 smithi028 ceph-mon[31916]: pgmap v1115: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.6 MiB/s wr, 499 op/s 2024-03-22T02:38:43.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 196738 15.32 MB/sec execute 563 sec latency 179.973 ms 2024-03-22T02:38:44.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 197131 15.34 MB/sec execute 564 sec latency 63.589 ms 2024-03-22T02:38:44.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:44 smithi120 ceph-mon[36803]: pgmap v1116: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 513 op/s 2024-03-22T02:38:45.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:44 smithi083 ceph-mon[36340]: pgmap v1116: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 513 op/s 2024-03-22T02:38:45.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:44 smithi028 ceph-mon[31916]: pgmap v1116: 129 pgs: 129 active+clean; 287 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.7 MiB/s wr, 513 op/s 2024-03-22T02:38:45.701 INFO:tasks.workunit.client.0.smithi028.stdout: 10 197527 15.35 MB/sec execute 565 sec latency 100.962 ms 2024-03-22T02:38:46.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 197877 15.35 MB/sec execute 566 sec latency 71.024 ms 2024-03-22T02:38:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:46 smithi120 ceph-mon[36803]: pgmap v1117: 129 pgs: 129 active+clean; 294 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2024-03-22T02:38:47.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:46 smithi083 ceph-mon[36340]: pgmap v1117: 129 pgs: 129 active+clean; 294 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2024-03-22T02:38:47.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:46 smithi028 ceph-mon[31916]: pgmap v1117: 129 pgs: 129 active+clean; 294 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.6 MiB/s wr, 540 op/s 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.079+0000 7fa23cc79640 1 -- 172.21.15.28:0/4290273370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234032340 con 0x7fa238100420 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 -- 172.21.15.28:0/4290273370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238100420 msgr2=0x7fa238102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 --2- 172.21.15.28:0/4290273370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238100420 0x7fa238102850 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fa234009860 tx=0x7fa23402bb70 comp rx=0 tx=0).stop 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 -- 172.21.15.28:0/4290273370 shutdown_connections 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 --2- 172.21.15.28:0/4290273370 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa238105740 0x7fa238105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 --2- 172.21.15.28:0/4290273370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238102d90 0x7fa238105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 --2- 172.21.15.28:0/4290273370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238100420 0x7fa238102850 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 -- 172.21.15.28:0/4290273370 >> 172.21.15.28:0/4290273370 conn(0x7fa2380fc020 msgr2=0x7fa2380fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 -- 172.21.15.28:0/4290273370 shutdown_connections 2024-03-22T02:38:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.080+0000 7fa23fedf640 1 -- 172.21.15.28:0/4290273370 wait complete. 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 Processor -- start 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 -- start start 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa238100420 0x7fa23810e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238102d90 0x7fa23810ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa238107010 con 0x7fa238102d90 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa238106e90 con 0x7fa238100420 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23fedf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa238107190 con 0x7fa238105740 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.081+0000 7fa23e47c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:33222/0 (socket says 172.21.15.28:33222) 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 -- 172.21.15.28:0/3671211706 learned_addr learned my addr 172.21.15.28:0/3671211706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23d47a640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238102d90 0x7fa23810ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 -- 172.21.15.28:0/3671211706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa238100420 msgr2=0x7fa23810e8c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa238100420 0x7fa23810e8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 -- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238102d90 msgr2=0x7fa23810ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238102d90 0x7fa23810ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 -- 172.21.15.28:0/3671211706 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2381104b0 con 0x7fa238105740 2024-03-22T02:38:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.082+0000 7fa23e47c640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa23000eb10 tx=0x7fa23000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.083+0000 7fa22effd640 1 -- 172.21.15.28:0/3671211706 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230004b20 con 0x7fa238105740 2024-03-22T02:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.083+0000 7fa22effd640 1 -- 172.21.15.28:0/3671211706 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa230004cc0 con 0x7fa238105740 2024-03-22T02:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.083+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2381107a0 con 0x7fa238105740 2024-03-22T02:38:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.083+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa238110cf0 con 0x7fa238105740 2024-03-22T02:38:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.085+0000 7fa22effd640 1 -- 172.21.15.28:0/3671211706 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230010760 con 0x7fa238105740 2024-03-22T02:38:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.087+0000 7fa22effd640 1 -- 172.21.15.28:0/3671211706 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa230005020 con 0x7fa238105740 2024-03-22T02:38:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.088+0000 7fa22effd640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 0x7fa2240a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.088+0000 7fa22effd640 1 -- 172.21.15.28:0/3671211706 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa2300cfd90 con 0x7fa238105740 2024-03-22T02:38:47.089 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.088+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 0x7fa2240a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.089+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 0x7fa2240a5590 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa2340095f0 tx=0x7fa234005760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 Processor -- start 2024-03-22T02:38:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 -- start start 2024-03-22T02:38:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 0x55a182be3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be9040 0x55a182beb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a182b48040 con 0x55a182be9040 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a182af09e0 con 0x55a182be3ed0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.091+0000 7fa242b9f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a182b43ff0 con 0x55a182be34e0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54232/0 (socket says 172.21.15.28:54232) 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 -- 172.21.15.28:0/1825934292 learned_addr learned my addr 172.21.15.28:0/1825934292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23e47c640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be9040 0x55a182beb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 0x55a182be3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 -- 172.21.15.28:0/1825934292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 msgr2=0x55a182be3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 0x55a182be3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 -- 172.21.15.28:0/1825934292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be9040 msgr2=0x55a182beb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be9040 0x55a182beb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23d47a640 1 -- 172.21.15.28:0/1825934292 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a18285c210 con 0x55a182be3ed0 2024-03-22T02:38:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.092+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 0x55a182be3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:38:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.093+0000 7fa23d47a640 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa228009c80 tx=0x7fa228011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=305b7c1dcaf5ef93 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.093+0000 7fa223fff640 1 -- 172.21.15.28:0/1825934292 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228004020 con 0x55a182be3ed0 2024-03-22T02:38:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.094+0000 7fa223fff640 1 -- 172.21.15.28:0/1825934292 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2280041c0 con 0x55a182be3ed0 2024-03-22T02:38:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.094+0000 7fa223fff640 1 -- 172.21.15.28:0/1825934292 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228019830 con 0x55a182be3ed0 2024-03-22T02:38:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 -- 172.21.15.28:0/1825934292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 msgr2=0x55a182be89d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa228009c80 tx=0x7fa228011810 comp rx=0 tx=0).stop 2024-03-22T02:38:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 -- 172.21.15.28:0/1825934292 shutdown_connections 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be9040 0x55a182beb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182be89d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 --2- 172.21.15.28:0/1825934292 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be34e0 0x55a182be3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.097+0000 7fa242b9f740 1 -- 172.21.15.28:0/1825934292 >> 172.21.15.28:0/1825934292 conn(0x55a182bdea50 msgr2=0x55a182bdee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.098+0000 7fa242b9f740 1 -- 172.21.15.28:0/1825934292 shutdown_connections 2024-03-22T02:38:47.099 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.098+0000 7fa242b9f740 1 -- 172.21.15.28:0/1825934292 wait complete. 2024-03-22T02:38:47.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.098+0000 7fa242b9f740 1 -- start start 2024-03-22T02:38:47.100 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.099+0000 7fa242b9f740 1 Processor -- start 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.099+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bed510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.099+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be3ed0 0x55a182beda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.099+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 0x55a182bf4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa242b9f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a182bebff0 con 0x55a182be34e0 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa242b9f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55a182b03db0 con 0x55a182be9040 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa242b9f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55a182bec170 con 0x55a182be3ed0 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa242b9f740 10 client.0 _ll_register_callbacks cb 0x55a18285ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23e47c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 0x55a182bf4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23e47c640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 0x55a182bf4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54234/0 (socket says 172.21.15.28:54234) 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23e47c640 1 -- 172.21.15.28:0/3130100580 learned_addr learned my addr 172.21.15.28:0/3130100580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:47.101 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23d47a640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be3ed0 0x55a182beda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bed510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 -- 172.21.15.28:0/3130100580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be3ed0 msgr2=0x55a182beda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be3ed0 0x55a182beda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 -- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 msgr2=0x55a182bf4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 0x55a182bf4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.100+0000 7fa23dc7b640 1 -- 172.21.15.28:0/3130100580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a182928620 con 0x55a182be34e0 2024-03-22T02:38:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.101+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bed510 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fa234009830 tx=0x7fa234005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d07bbea996f903 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.101+0000 7fa2217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.101+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234050060 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.101+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa234034640 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.101+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2280041c0 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.102+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa234039040 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.102+0000 7fa242b9f740 10 client.20478 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.102+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a182bf7c60 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.102+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2280041c0 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.102+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fa23403c030 con 0x55a182be34e0 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 10 client.20478 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 10 client.20478 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 20 client.20478 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182bfb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 4 client.20478 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:38:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a182b55580 con 0x55a182bf9330 2024-03-22T02:38:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.103+0000 7fa23d47a640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182bfb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.104+0000 7fa2217fa640 10 client.20478 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.104+0000 7fa23d47a640 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182bfb760 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5383d58c9a63fc60 server_cookie=35ac865e4ca1ed2c in_seq=0 out_seq=0 2024-03-22T02:38:47.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.112+0000 7fa2217fa640 1 -- 172.21.15.28:0/3130100580 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a182b55580 con 0x55a182bf9330 2024-03-22T02:38:47.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa2217fa640 10 client.20478 handle_command_reply: tid=0 2024-03-22T02:38:47.113 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 client.20478 shutdown 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 msgr2=0x55a182bed510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bed510 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fa234009830 tx=0x7fa234005f90 comp rx=0 tx=0).stop 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 shutdown_connections 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182bfb760 crc :-1 s=READY pgs=454 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa2217fa640 0 client.20478 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be9040 0x55a182bf4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.113+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be3ed0 0x55a182beda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.115 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.114+0000 7fa242b9f740 1 --2- 172.21.15.28:0/3130100580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bed510 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.114+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 >> 172.21.15.28:0/3130100580 conn(0x55a182bdea50 msgr2=0x55a182bddc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:47.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.114+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 shutdown_connections 2024-03-22T02:38:47.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.114+0000 7fa242b9f740 1 -- 172.21.15.28:0/3130100580 wait complete. 2024-03-22T02:38:47.116 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.114+0000 7fa242b9f740 20 client.20478 trim_cache size 0 max 16384 2024-03-22T02:38:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 Processor -- start 2024-03-22T02:38:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 -- start start 2024-03-22T02:38:47.140 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be3ed0 0x55a182bf4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bddad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c012700 con 0x55a182be3ed0 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c00ffd0 con 0x55a182be34e0 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.139+0000 7fa242b9f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c00de20 con 0x55a182be9040 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:54248/0 (socket says 172.21.15.28:54248) 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 -- 172.21.15.28:0/794756336 learned_addr learned my addr 172.21.15.28:0/794756336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:47.141 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23e47c640 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 -- 172.21.15.28:0/794756336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 msgr2=0x55a182bddad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bddad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 -- 172.21.15.28:0/794756336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be3ed0 msgr2=0x55a182bf4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be3ed0 0x55a182bf4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.140+0000 7fa23dc7b640 1 -- 172.21.15.28:0/794756336 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a182be8d50 con 0x55a182be34e0 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.141+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa234009780 tx=0x7fa234005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13406e81892c8b55 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.141+0000 7fa2237fe640 1 -- 172.21.15.28:0/794756336 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234050040 con 0x55a182be34e0 2024-03-22T02:38:47.142 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.141+0000 7fa2237fe640 1 -- 172.21.15.28:0/794756336 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2340044d0 con 0x55a182be34e0 2024-03-22T02:38:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.142+0000 7fa242b9f740 1 -- 172.21.15.28:0/794756336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 msgr2=0x55a182bed9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:47.143 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.142+0000 7fa242b9f740 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa234009780 tx=0x7fa234005970 comp rx=0 tx=0).stop 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.144+0000 7fa242b9f740 1 -- 172.21.15.28:0/794756336 shutdown_connections 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.144+0000 7fa242b9f740 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bddad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be3ed0 0x55a182bf4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 --2- 172.21.15.28:0/794756336 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be34e0 0x55a182bed9b0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 -- 172.21.15.28:0/794756336 >> 172.21.15.28:0/794756336 conn(0x55a182bdea50 msgr2=0x55a182bdd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 -- 172.21.15.28:0/794756336 shutdown_connections 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 -- 172.21.15.28:0/794756336 wait complete. 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 -- start start 2024-03-22T02:38:47.146 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.145+0000 7fa242b9f740 1 Processor -- start 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182bf24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bf3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c00e490 con 0x55a182be34e0 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c00f900 con 0x55a182be3ed0 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c00eb60 con 0x55a182be9040 2024-03-22T02:38:47.147 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa242b9f740 10 client.0 _ll_register_callbacks cb 0x7fa234037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:38:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa23d47a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182bf24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa23dc7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa23dc7b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38710/0 (socket says 172.21.15.28:38710) 2024-03-22T02:38:47.148 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.146+0000 7fa23dc7b640 1 -- 172.21.15.28:0/428575843 learned_addr learned my addr 172.21.15.28:0/428575843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 -- 172.21.15.28:0/428575843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 msgr2=0x55a182bf3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bf3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 -- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 msgr2=0x55a182bf24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182bf24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 -- 172.21.15.28:0/428575843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a182bf3a00 con 0x55a182be34e0 2024-03-22T02:38:47.149 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa23dc7b640 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fa2340038f0 tx=0x7fa234032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12bf3fab4917f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.147+0000 7fa2217fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa234043030 con 0x55a182be34e0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2340044d0 con 0x55a182be34e0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa234034650 con 0x55a182be34e0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fa23406c050 con 0x55a182be34e0 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa242b9f740 10 client.20481 fetch_fsmap learned FSMap version 46 2024-03-22T02:38:47.150 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.148+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55a182bf4d10 con 0x55a182be34e0 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.149+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2340044d0 con 0x55a182be34e0 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7fa23403c070 con 0x55a182be34e0 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 10 client.20481 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 10 client.20481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 20 client.20481 populate_metadata read hostname 'smithi028' 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182c61da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 4 client.20481 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:38:47.151 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.150+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a182b4f8d0 con 0x55a182bf9330 2024-03-22T02:38:47.158 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.158+0000 7fa23d47a640 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182c61da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:38:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.160+0000 7fa23d47a640 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182c61da0 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f7a69bb3d7ff1d4 server_cookie=f058fad702184fd2 in_seq=0 out_seq=0 2024-03-22T02:38:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:47.160+0000 7fa2217fa640 10 client.20481 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:47.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 198005 15.34 MB/sec execute 567 sec latency 413.260 ms 2024-03-22T02:38:48.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 198061 15.31 MB/sec execute 568 sec latency 1264.884 ms 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.823+0000 7fa2217fa640 1 -- 172.21.15.28:0/428575843 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a182b4f8d0 con 0x55a182bf9330 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.823+0000 7fa2217fa640 10 client.20481 handle_command_reply: tid=0 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.823+0000 7fa242b9f740 1 client.20481 shutdown 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 msgr2=0x55a182bf1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fa2340038f0 tx=0x7fa234032cb0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 shutdown_connections 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55a182bf9330 0x55a182c61da0 crc :-1 s=READY pgs=455 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a182be9040 0x55a182bf3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55a182be3ed0 0x55a182bf24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 --2- 172.21.15.28:0/428575843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a182be34e0 0x55a182bf1fa0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa2217fa640 0 client.20481 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 >> 172.21.15.28:0/428575843 conn(0x55a182bdea50 msgr2=0x55a182bddc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 shutdown_connections 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 1 -- 172.21.15.28:0/428575843 wait complete. 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa242b9f740 20 client.20481 trim_cache size 0 max 16384 2024-03-22T02:38:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:38:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:38:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:38:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:38:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 msgr2=0x7fa2240a5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 0x7fa2240a5590 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fa2340095f0 tx=0x7fa234005760 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 msgr2=0x7fa23810fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.824+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa23000eb10 tx=0x7fa23000c2d0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 shutdown_connections 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa2240a30d0 0x7fa2240a5590 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa238105740 0x7fa23810fe40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa238102d90 0x7fa23810ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 --2- 172.21.15.28:0/3671211706 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa238100420 0x7fa23810e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 >> 172.21.15.28:0/3671211706 conn(0x7fa2380fc020 msgr2=0x7fa2380fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 shutdown_connections 2024-03-22T02:38:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:38:48.825+0000 7fa23fedf640 1 -- 172.21.15.28:0/3671211706 wait complete. 2024-03-22T02:38:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:48 smithi120 ceph-mon[36803]: pgmap v1118: 129 pgs: 129 active+clean; 297 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-03-22T02:38:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:48 smithi083 ceph-mon[36340]: pgmap v1118: 129 pgs: 129 active+clean; 297 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-03-22T02:38:49.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:48 smithi028 ceph-mon[31916]: pgmap v1118: 129 pgs: 129 active+clean; 297 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.4 MiB/s wr, 438 op/s 2024-03-22T02:38:49.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 198354 15.32 MB/sec execute 569 sec latency 208.308 ms 2024-03-22T02:38:50.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 198683 15.33 MB/sec execute 570 sec latency 117.142 ms 2024-03-22T02:38:50.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:50 smithi120 ceph-mon[36803]: pgmap v1119: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-22T02:38:51.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:50 smithi083 ceph-mon[36340]: pgmap v1119: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-22T02:38:51.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:50 smithi028 ceph-mon[31916]: pgmap v1119: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 4.9 MiB/s wr, 496 op/s 2024-03-22T02:38:51.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 199049 15.34 MB/sec execute 571 sec latency 68.503 ms 2024-03-22T02:38:52.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 199326 15.33 MB/sec execute 572 sec latency 196.277 ms 2024-03-22T02:38:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:52 smithi120 ceph-mon[36803]: pgmap v1120: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-03-22T02:38:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:52 smithi083 ceph-mon[36340]: pgmap v1120: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-03-22T02:38:53.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:52 smithi028 ceph-mon[31916]: pgmap v1120: 129 pgs: 129 active+clean; 300 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.7 MiB/s wr, 381 op/s 2024-03-22T02:38:53.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 199509 15.32 MB/sec execute 573 sec latency 220.930 ms 2024-03-22T02:38:54.001 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:53 smithi120 ceph-mon[36803]: pgmap v1121: 129 pgs: 129 active+clean; 301 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 493 op/s 2024-03-22T02:38:54.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:53 smithi083 ceph-mon[36340]: pgmap v1121: 129 pgs: 129 active+clean; 301 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 493 op/s 2024-03-22T02:38:54.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:53 smithi028 ceph-mon[31916]: pgmap v1121: 129 pgs: 129 active+clean; 301 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 493 op/s 2024-03-22T02:38:54.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 199795 15.32 MB/sec execute 574 sec latency 121.096 ms 2024-03-22T02:38:55.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 200114 15.32 MB/sec execute 575 sec latency 89.279 ms 2024-03-22T02:38:56.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:56 smithi028 ceph-mon[31916]: pgmap v1122: 129 pgs: 129 active+clean; 293 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-03-22T02:38:56.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:56 smithi120 ceph-mon[36803]: pgmap v1122: 129 pgs: 129 active+clean; 293 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-03-22T02:38:56.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:56 smithi083 ceph-mon[36340]: pgmap v1122: 129 pgs: 129 active+clean; 293 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 5.1 MiB/s wr, 566 op/s 2024-03-22T02:38:56.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 200445 15.33 MB/sec execute 576 sec latency 70.042 ms 2024-03-22T02:38:57.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 200747 15.33 MB/sec execute 577 sec latency 192.719 ms 2024-03-22T02:38:58.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:38:58 smithi120 ceph-mon[36803]: pgmap v1123: 129 pgs: 129 active+clean; 293 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-22T02:38:58.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:38:58 smithi083 ceph-mon[36340]: pgmap v1123: 129 pgs: 129 active+clean; 293 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-22T02:38:58.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:38:58 smithi028 ceph-mon[31916]: pgmap v1123: 129 pgs: 129 active+clean; 293 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 512 op/s 2024-03-22T02:38:58.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 200959 15.33 MB/sec execute 578 sec latency 273.882 ms 2024-03-22T02:38:59.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 201344 15.33 MB/sec execute 579 sec latency 95.973 ms 2024-03-22T02:39:00.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:00 smithi120 ceph-mon[36803]: pgmap v1124: 129 pgs: 129 active+clean; 296 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 535 op/s 2024-03-22T02:39:00.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:00 smithi083 ceph-mon[36340]: pgmap v1124: 129 pgs: 129 active+clean; 296 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 535 op/s 2024-03-22T02:39:00.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:00 smithi028 ceph-mon[31916]: pgmap v1124: 129 pgs: 129 active+clean; 296 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.0 MiB/s wr, 535 op/s 2024-03-22T02:39:00.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 201592 15.32 MB/sec execute 580 sec latency 94.813 ms 2024-03-22T02:39:01.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 201945 15.33 MB/sec execute 581 sec latency 54.824 ms 2024-03-22T02:39:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 -- 172.21.15.28:0/2451965000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978101720 msgr2=0x7f6978101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 --2- 172.21.15.28:0/2451965000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978101720 0x7f6978101ba0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f6970009ca0 tx=0x7f697002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 -- 172.21.15.28:0/2451965000 shutdown_connections 2024-03-22T02:39:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 --2- 172.21.15.28:0/2451965000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978101720 0x7f6978101ba0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 --2- 172.21.15.28:0/2451965000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 0x7f69781011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 --2- 172.21.15.28:0/2451965000 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978100420 0x7f6978100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 -- 172.21.15.28:0/2451965000 >> 172.21.15.28:0/2451965000 conn(0x7f69780fc020 msgr2=0x7f69780fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 -- 172.21.15.28:0/2451965000 shutdown_connections 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.921+0000 7f6980233640 1 -- 172.21.15.28:0/2451965000 wait complete. 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 Processor -- start 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 -- start start 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 0x7f697819a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978101720 0x7f69781a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f697810fa00 con 0x7f6978100420 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f697810f880 con 0x7f6978101720 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f6980233640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f697810fb80 con 0x7f6978100d80 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52978/0 (socket says 172.21.15.28:52978) 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 -- 172.21.15.28:0/2695561482 learned_addr learned my addr 172.21.15.28:0/2695561482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 0x7f697819a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697e7d0640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978101720 0x7f69781a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 -- 172.21.15.28:0/2695561482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 msgr2=0x7f697819a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 0x7f697819a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 -- 172.21.15.28:0/2695561482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978101720 msgr2=0x7f69781a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978101720 0x7f69781a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.922+0000 7f697dfcf640 1 -- 172.21.15.28:0/2695561482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69781a1ba0 con 0x7f6978100420 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f696800e9f0 tx=0x7f696800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f696effd640 1 -- 172.21.15.28:0/2695561482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f696800cd80 con 0x7f6978100420 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f696effd640 1 -- 172.21.15.28:0/2695561482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6968010910 con 0x7f6978100420 2024-03-22T02:39:01.926 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69781a1e90 con 0x7f6978100420 2024-03-22T02:39:01.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f696effd640 1 -- 172.21.15.28:0/2695561482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968018990 con 0x7f6978100420 2024-03-22T02:39:01.927 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.923+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69781a2350 con 0x7f6978100420 2024-03-22T02:39:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f696effd640 1 -- 172.21.15.28:0/2695561482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f6968010ac0 con 0x7f6978100420 2024-03-22T02:39:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f6982e16740 1 Processor -- start 2024-03-22T02:39:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f6982e16740 1 -- start start 2024-03-22T02:39:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f6982e16740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.932 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f6982e16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baeda09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.925+0000 7f6982e16740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baeda34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f6982e16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558baed00040 con 0x558baed9bed0 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f6982e16740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558baeca89e0 con 0x558baeda1040 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f6982e16740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558baecfbff0 con 0x558baed9b4e0 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f696effd640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 0x7f69640a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baeda09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baeda34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44334/0 (socket says 172.21.15.28:44334) 2024-03-22T02:39:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 -- 172.21.15.28:0/2697453828 learned_addr learned my addr 172.21.15.28:0/2697453828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 0x7f69640a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f696effd640 1 -- 172.21.15.28:0/2695561482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f69680cfa60 con 0x7f6978100420 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 -- 172.21.15.28:0/2697453828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 msgr2=0x558baed9b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 -- 172.21.15.28:0/2697453828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 msgr2=0x558baeda09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baeda09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 -- 172.21.15.28:0/2697453828 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558baea14210 con 0x558baeda1040 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697e7d0640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 0x7f69640a5500 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x558baed9c730 tx=0x7f6970032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baeda34d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6974009bf0 tx=0x7f6974011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=208e5eb8c3a06f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.926+0000 7f6963fff640 1 -- 172.21.15.28:0/2697453828 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974004050 con 0x558baeda1040 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6963fff640 1 -- 172.21.15.28:0/2697453828 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6974005500 con 0x558baeda1040 2024-03-22T02:39:01.934 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6963fff640 1 -- 172.21.15.28:0/2697453828 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974004e40 con 0x558baeda1040 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 -- 172.21.15.28:0/2697453828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 msgr2=0x558baeda34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baeda34d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6974009bf0 tx=0x7f6974011810 comp rx=0 tx=0).stop 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 -- 172.21.15.28:0/2697453828 shutdown_connections 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baeda34d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baeda09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 --2- 172.21.15.28:0/2697453828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 -- 172.21.15.28:0/2697453828 >> 172.21.15.28:0/2697453828 conn(0x558baed96a50 msgr2=0x558baed96e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 -- 172.21.15.28:0/2697453828 shutdown_connections 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.927+0000 7f6982e16740 1 -- 172.21.15.28:0/2697453828 wait complete. 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 -- start start 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 Processor -- start 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baed9b4e0 0x558baeda5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.935 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baeda59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558baeda3ff0 con 0x558baeda1040 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x558baecbbdb0 con 0x558baed9b4e0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x558baeda4170 con 0x558baed9bed0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f6982e16740 10 client.0 _ll_register_callbacks cb 0x558baea13a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f697e7d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baed9b4e0 0x558baeda5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baeda59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f697e7d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53004/0 (socket says 172.21.15.28:53004) 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.928+0000 7f697e7d0640 1 -- 172.21.15.28:0/549997448 learned_addr learned my addr 172.21.15.28:0/549997448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 -- 172.21.15.28:0/549997448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 msgr2=0x558baeda59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baeda59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.936 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 -- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baed9b4e0 msgr2=0x558baeda5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baed9b4e0 0x558baeda5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 -- 172.21.15.28:0/549997448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558baeae0620 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f697e7d0640 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f6970005e30 tx=0x7f697002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6fd48f461aa9eb server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6970053060 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f6970004040 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6974005500 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f6970039040 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f6982e16740 10 client.20493 fetch_fsmap learned FSMap version 46 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x558baedafde0 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6974005500 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.929+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f697004e470 con 0x558baeda1040 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 10 client.20493 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:39:01.937 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 10 client.20493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 20 client.20493 populate_metadata read hostname 'smithi028' 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baedb1500 0x558baedb3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 4 client.20493 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558baed0d580 con 0x558baedb1500 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f697dfcf640 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baedb1500 0x558baedb3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f697dfcf640 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baedb1500 0x558baedb3990 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f923abbb454f972 server_cookie=3d33ead3422cddb7 in_seq=0 out_seq=0 2024-03-22T02:39:01.938 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.930+0000 7f69617fa640 10 client.20493 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f69617fa640 1 -- 172.21.15.28:0/549997448 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558baed0d580 con 0x558baedb1500 2024-03-22T02:39:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f69617fa640 10 client.20493 handle_command_reply: tid=0 2024-03-22T02:39:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 client.20493 shutdown 2024-03-22T02:39:01.939 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 msgr2=0x558baedaca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f6970005e30 tx=0x7f697002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 shutdown_connections 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baedb1500 0x558baedb3990 crc :-1 s=READY pgs=456 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baeda1040 0x558baedaca00 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baeda59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f69617fa640 0 client.20493 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 --2- 172.21.15.28:0/549997448 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baed9b4e0 0x558baeda5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 >> 172.21.15.28:0/549997448 conn(0x558baed96a50 msgr2=0x558baed95c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 shutdown_connections 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 1 -- 172.21.15.28:0/549997448 wait complete. 2024-03-22T02:39:01.940 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.933+0000 7f6982e16740 20 client.20493 trim_cache size 0 max 16384 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 Processor -- start 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 -- start start 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baed9cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69781af430 con 0x558baed9bed0 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69781acd00 con 0x558baeda1040 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f6982e16740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69781aab50 con 0x558baed9b4e0 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53014/0 (socket says 172.21.15.28:53014) 2024-03-22T02:39:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f697d7ce640 1 -- 172.21.15.28:0/906100583 learned_addr learned my addr 172.21.15.28:0/906100583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.948+0000 7f697dfcf640 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697e7d0640 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baed9cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 -- 172.21.15.28:0/906100583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 msgr2=0x558baed9c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 -- 172.21.15.28:0/906100583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 msgr2=0x558baed9cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baed9cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 -- 172.21.15.28:0/906100583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558baed977c0 con 0x558baed9bed0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f697d7ce640 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x558baed979c0 tx=0x7f6974009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d8daf6b0bf82d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f69637fe640 1 -- 172.21.15.28:0/906100583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974005040 con 0x558baed9bed0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f69637fe640 1 -- 172.21.15.28:0/906100583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69740035f0 con 0x558baed9bed0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.949+0000 7f69637fe640 1 -- 172.21.15.28:0/906100583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974019650 con 0x558baed9bed0 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- 172.21.15.28:0/906100583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 msgr2=0x558baedac9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x558baed979c0 tx=0x7f6974009930 comp rx=0 tx=0).stop 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- 172.21.15.28:0/906100583 shutdown_connections 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baed9cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9bed0 0x558baedac9c0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- 172.21.15.28:0/906100583 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9b4e0 0x558baed9c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- 172.21.15.28:0/906100583 >> 172.21.15.28:0/906100583 conn(0x558baed96a50 msgr2=0x558baedad450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- 172.21.15.28:0/906100583 shutdown_connections 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- 172.21.15.28:0/906100583 wait complete. 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- start start 2024-03-22T02:39:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 Processor -- start 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baedab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f69781ab1c0 con 0x558baed9b4e0 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f69781ac630 con 0x558baeda1040 2024-03-22T02:39:01.956 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.950+0000 7f6982e16740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f69781ab890 con 0x558baed9bed0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f6982e16740 10 client.0 _ll_register_callbacks cb 0x7f6974005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697e7d0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baedab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697d7ce640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:44366/0 (socket says 172.21.15.28:44366) 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697d7ce640 1 -- 172.21.15.28:0/2439441060 learned_addr learned my addr 172.21.15.28:0/2439441060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53024/0 (socket says 172.21.15.28:53024) 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 -- 172.21.15.28:0/2439441060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 msgr2=0x558baedaa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 -- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 msgr2=0x558baedab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baedab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 -- 172.21.15.28:0/2439441060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558baedabea0 con 0x558baed9b4e0 2024-03-22T02:39:01.957 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f697dfcf640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f69680cd1a0 tx=0x7f69680c6a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548a77874b3240fc server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f69617fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6968026040 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f69680984a0 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69740035f0 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f69680a4790 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f6982e16740 10 client.20499 fetch_fsmap learned FSMap version 46 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.951+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x558baee17b50 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f69680b8080 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f69680baaf0 con 0x558baed9b4e0 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 10 client.20499 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 10 client.20499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:39:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 20 client.20499 populate_metadata read hostname 'smithi028' 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baee18520 0x558baee1a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 4 client.20499 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558baed078d0 con 0x558baee18520 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baee18520 0x558baee1a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f697d7ce640 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baee18520 0x558baee1a940 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9285eb5ce22901b9 server_cookie=368301c6cbc813bd in_seq=0 out_seq=0 2024-03-22T02:39:01.959 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:01.952+0000 7f69617fa640 10 client.20499 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:02.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:02 smithi120 ceph-mon[36803]: pgmap v1125: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-22T02:39:02.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:02 smithi083 ceph-mon[36340]: pgmap v1125: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-22T02:39:02.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:02 smithi028 ceph-mon[31916]: pgmap v1125: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 414 op/s 2024-03-22T02:39:02.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 202042 15.31 MB/sec execute 582 sec latency 558.693 ms 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f69617fa640 1 -- 172.21.15.28:0/2439441060 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558baed078d0 con 0x558baee18520 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f69617fa640 10 client.20499 handle_command_reply: tid=0 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 client.20499 shutdown 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 msgr2=0x558baedaa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f69680cd1a0 tx=0x7f69680c6a40 comp rx=0 tx=0).stop 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 shutdown_connections 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x558baee18520 0x558baee1a940 crc :-1 s=READY pgs=457 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x558baeda1040 0x558baedab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f69617fa640 0 client.20499 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x558baed9bed0 0x558baedaa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 --2- 172.21.15.28:0/2439441060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558baed9b4e0 0x558baedaa3b0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.599+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 >> 172.21.15.28:0/2439441060 conn(0x558baed96a50 msgr2=0x558baed953c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.600+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 shutdown_connections 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.600+0000 7f6982e16740 1 -- 172.21.15.28:0/2439441060 wait complete. 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.600+0000 7f6982e16740 20 client.20499 trim_cache size 0 max 16384 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:39:03.602 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 msgr2=0x7f69640a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 0x7f69640a5500 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x558baed9c730 tx=0x7f6970032590 comp rx=0 tx=0).stop 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 msgr2=0x7f6978199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f696800e9f0 tx=0x7f696800eeb0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 shutdown_connections 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f69640a3040 0x7f69640a5500 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f6978101720 0x7f69781a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978100d80 0x7f697819a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 --2- 172.21.15.28:0/2695561482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6978100420 0x7f6978199f80 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.601+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 >> 172.21.15.28:0/2695561482 conn(0x7f69780fc020 msgr2=0x7f69780fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.602+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 shutdown_connections 2024-03-22T02:39:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:03.602+0000 7f6980233640 1 -- 172.21.15.28:0/2695561482 wait complete. 2024-03-22T02:39:03.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 202190 15.31 MB/sec execute 583 sec latency 1263.958 ms 2024-03-22T02:39:04.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:04 smithi120 ceph-mon[36803]: pgmap v1126: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 424 op/s 2024-03-22T02:39:04.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:04 smithi083 ceph-mon[36340]: pgmap v1126: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 424 op/s 2024-03-22T02:39:04.573 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:04 smithi028 ceph-mon[31916]: pgmap v1126: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.1 MiB/s wr, 424 op/s 2024-03-22T02:39:04.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 202493 15.31 MB/sec execute 584 sec latency 122.848 ms 2024-03-22T02:39:05.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 202863 15.31 MB/sec execute 585 sec latency 95.175 ms 2024-03-22T02:39:06.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:06 smithi120 ceph-mon[36803]: pgmap v1127: 129 pgs: 129 active+clean; 296 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 475 op/s 2024-03-22T02:39:06.518 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:06 smithi083 ceph-mon[36340]: pgmap v1127: 129 pgs: 129 active+clean; 296 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 475 op/s 2024-03-22T02:39:06.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:06 smithi028 ceph-mon[31916]: pgmap v1127: 129 pgs: 129 active+clean; 296 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.4 MiB/s wr, 475 op/s 2024-03-22T02:39:06.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 203172 15.32 MB/sec execute 586 sec latency 72.768 ms 2024-03-22T02:39:07.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 203499 15.32 MB/sec execute 587 sec latency 91.743 ms 2024-03-22T02:39:08.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:08 smithi120 ceph-mon[36803]: pgmap v1128: 129 pgs: 129 active+clean; 295 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.0 MiB/s wr, 501 op/s 2024-03-22T02:39:08.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:08 smithi083 ceph-mon[36340]: pgmap v1128: 129 pgs: 129 active+clean; 295 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.0 MiB/s wr, 501 op/s 2024-03-22T02:39:08.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:08 smithi028 ceph-mon[31916]: pgmap v1128: 129 pgs: 129 active+clean; 295 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.0 MiB/s wr, 501 op/s 2024-03-22T02:39:08.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 203703 15.31 MB/sec execute 588 sec latency 266.073 ms 2024-03-22T02:39:09.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 203961 15.31 MB/sec execute 589 sec latency 83.000 ms 2024-03-22T02:39:10.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:10 smithi120 ceph-mon[36803]: pgmap v1129: 129 pgs: 129 active+clean; 298 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.1 MiB/s wr, 565 op/s 2024-03-22T02:39:10.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:10 smithi083 ceph-mon[36340]: pgmap v1129: 129 pgs: 129 active+clean; 298 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.1 MiB/s wr, 565 op/s 2024-03-22T02:39:10.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:10 smithi028 ceph-mon[31916]: pgmap v1129: 129 pgs: 129 active+clean; 298 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.1 MiB/s wr, 565 op/s 2024-03-22T02:39:10.702 INFO:tasks.workunit.client.0.smithi028.stdout: 10 204243 15.31 MB/sec execute 590 sec latency 151.477 ms 2024-03-22T02:39:11.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 204558 15.32 MB/sec execute 591 sec latency 107.842 ms 2024-03-22T02:39:12.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:12 smithi120 ceph-mon[36803]: pgmap v1130: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-03-22T02:39:12.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:12 smithi083 ceph-mon[36340]: pgmap v1130: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-03-22T02:39:12.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:12 smithi028 ceph-mon[31916]: pgmap v1130: 129 pgs: 129 active+clean; 299 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.3 MiB/s wr, 499 op/s 2024-03-22T02:39:12.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 204907 15.32 MB/sec execute 592 sec latency 75.411 ms 2024-03-22T02:39:13.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 205077 15.30 MB/sec execute 593 sec latency 241.937 ms 2024-03-22T02:39:14.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:14 smithi120 ceph-mon[36803]: pgmap v1131: 129 pgs: 129 active+clean; 300 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.2 MiB/s wr, 497 op/s 2024-03-22T02:39:14.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:14 smithi083 ceph-mon[36340]: pgmap v1131: 129 pgs: 129 active+clean; 300 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.2 MiB/s wr, 497 op/s 2024-03-22T02:39:14.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:14 smithi028 ceph-mon[31916]: pgmap v1131: 129 pgs: 129 active+clean; 300 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 4.2 MiB/s wr, 497 op/s 2024-03-22T02:39:14.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 205314 15.30 MB/sec execute 594 sec latency 106.999 ms 2024-03-22T02:39:15.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 205648 15.31 MB/sec execute 595 sec latency 88.251 ms 2024-03-22T02:39:16.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:16 smithi120 ceph-mon[36803]: pgmap v1132: 129 pgs: 129 active+clean; 297 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 513 op/s 2024-03-22T02:39:16.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:16 smithi083 ceph-mon[36340]: pgmap v1132: 129 pgs: 129 active+clean; 297 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 513 op/s 2024-03-22T02:39:16.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:16 smithi028 ceph-mon[31916]: pgmap v1132: 129 pgs: 129 active+clean; 297 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.6 MiB/s wr, 513 op/s 2024-03-22T02:39:16.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 205958 15.31 MB/sec execute 596 sec latency 105.666 ms 2024-03-22T02:39:16.705 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 -- 172.21.15.28:0/2653050387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 msgr2=0x7f651c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 --2- 172.21.15.28:0/2653050387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c100ed0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f650c009ca0 tx=0x7f650c02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 -- 172.21.15.28:0/2653050387 shutdown_connections 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 --2- 172.21.15.28:0/2653050387 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 0x7f651c108ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 --2- 172.21.15.28:0/2653050387 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 0x7f651c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 --2- 172.21.15.28:0/2653050387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c100ed0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 -- 172.21.15.28:0/2653050387 >> 172.21.15.28:0/2653050387 conn(0x7f651c0fc020 msgr2=0x7f651c0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 -- 172.21.15.28:0/2653050387 shutdown_connections 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.702+0000 7f6521d93640 1 -- 172.21.15.28:0/2653050387 wait complete. 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 Processor -- start 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 -- start start 2024-03-22T02:39:16.706 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 0x7f651c19cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 0x7f651c19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c730 con 0x7f651c100ab0 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c5b0 con 0x7f651c101ca0 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f6521d93640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f651c10c8b0 con 0x7f651c102660 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44938/0 (socket says 172.21.15.28:44938) 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f651b7fe640 1 -- 172.21.15.28:0/1911943529 learned_addr learned my addr 172.21.15.28:0/1911943529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f651affd640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 0x7f651c19cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.703+0000 7f651bfff640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 0x7f651c19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 -- 172.21.15.28:0/1911943529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 msgr2=0x7f651c19f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 0x7f651c19f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.707 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 -- 172.21.15.28:0/1911943529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 msgr2=0x7f651c19cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 0x7f651c19cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 -- 172.21.15.28:0/1911943529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f651c19fa80 con 0x7f651c100ab0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f650c0324c0 tx=0x7f650c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f6518ff9640 1 -- 172.21.15.28:0/1911943529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c041020 con 0x7f651c100ab0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f6518ff9640 1 -- 172.21.15.28:0/1911943529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f650c034cd0 con 0x7f651c100ab0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f651c19fd10 con 0x7f651c100ab0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f6518ff9640 1 -- 172.21.15.28:0/1911943529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650c039040 con 0x7f651c100ab0 2024-03-22T02:39:16.708 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.704+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f651c1a01b0 con 0x7f651c100ab0 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.706+0000 7f6518ff9640 1 -- 172.21.15.28:0/1911943529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f650c044020 con 0x7f651c100ab0 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 Processor -- start 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 -- start start 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4ea900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.714 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e85a44f040 con 0x55e85a4eaed0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e85a3f79e0 con 0x55e85a4ea4e0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6524977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e85a44aff0 con 0x55e85a4f0040 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6518ff9640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 0x7f65040a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4ef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44954/0 (socket says 172.21.15.28:44954) 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651b7fe640 1 -- 172.21.15.28:0/3483580268 learned_addr learned my addr 172.21.15.28:0/3483580268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 0x7f65040a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651affd640 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f6518ff9640 1 -- 172.21.15.28:0/1911943529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f650c0f1690 con 0x7f651c100ab0 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 -- 172.21.15.28:0/3483580268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 msgr2=0x55e85a4f24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.715 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4f24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 -- 172.21.15.28:0/3483580268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 msgr2=0x55e85a4ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.707+0000 7f651bfff640 1 -- 172.21.15.28:0/3483580268 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e85a163210 con 0x55e85a4ea4e0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.708+0000 7f651bfff640 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4ea900 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6514010a00 tx=0x7f6514010ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c09184f2212c86d server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.708+0000 7f651bfff640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 0x7f65040a5480 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f651400f790 tx=0x7f6514018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.708+0000 7f6501ffb640 1 -- 172.21.15.28:0/3483580268 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514019040 con 0x55e85a4ea4e0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.708+0000 7f6501ffb640 1 -- 172.21.15.28:0/3483580268 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6514005500 con 0x55e85a4ea4e0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.708+0000 7f6501ffb640 1 -- 172.21.15.28:0/3483580268 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65140047a0 con 0x55e85a4ea4e0 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- 172.21.15.28:0/3483580268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 msgr2=0x55e85a4ea900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4ea900 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6514010a00 tx=0x7f6514010ec0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- 172.21.15.28:0/3483580268 shutdown_connections 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4f24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.716 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 --2- 172.21.15.28:0/3483580268 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4ea900 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- 172.21.15.28:0/3483580268 >> 172.21.15.28:0/3483580268 conn(0x55e85a4e5a50 msgr2=0x55e85a4e5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- 172.21.15.28:0/3483580268 shutdown_connections 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- 172.21.15.28:0/3483580268 wait complete. 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.709+0000 7f6524977740 1 -- start start 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 Processor -- start 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e85a4f2f90 con 0x55e85a4f0040 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x55e85a40adb0 con 0x55e85a4eaed0 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x55e85a4f3110 con 0x55e85a4ea4e0 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f6524977740 10 client.0 _ll_register_callbacks cb 0x55e85a162a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:39:16.717 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44970/0 (socket says 172.21.15.28:44970) 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:38008/0 (socket says 172.21.15.28:38008) 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651bfff640 1 -- 172.21.15.28:0/1084829168 learned_addr learned my addr 172.21.15.28:0/1084829168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.710+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.28:36726/0 (socket says 172.21.15.28:36726) 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 -- 172.21.15.28:0/1084829168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 msgr2=0x55e85a4f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 -- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 msgr2=0x55e85a4f49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4f49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 -- 172.21.15.28:0/1084829168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e85a22f620 con 0x55e85a4f0040 2024-03-22T02:39:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f651bfff640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f6514006010 tx=0x7f6514026af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e45c08f61b34357c server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f64faffd640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65140056b0 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f651400e790 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6514005500 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f651403c050 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f6524977740 10 client.20511 fetch_fsmap learned FSMap version 46 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e85a4fedc0 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.711+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6514005500 con 0x55e85a4f0040 2024-03-22T02:39:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f651402eb60 con 0x55e85a4f0040 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 10 client.20511 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 10 client.20511 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 20 client.20511 populate_metadata read hostname 'smithi028' 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a500280 0x55e85a502670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 4 client.20511 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e85a45c580 con 0x55e85a500280 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a500280 0x55e85a502670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a500280 0x55e85a502670 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0bb4c768e4813f0 server_cookie=29675cc5dc5ad101 in_seq=0 out_seq=0 2024-03-22T02:39:16.720 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.712+0000 7f64faffd640 10 client.20511 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:16.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f64faffd640 1 -- 172.21.15.28:0/1084829168 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e85a45c580 con 0x55e85a500280 2024-03-22T02:39:16.723 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f64faffd640 10 client.20511 handle_command_reply: tid=0 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 client.20511 shutdown 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 msgr2=0x55e85a4fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f6514006010 tx=0x7f6514026af0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 shutdown_connections 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a500280 0x55e85a502670 crc :-1 s=READY pgs=458 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4f0040 0x55e85a4fb9e0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4f49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f64faffd640 0 client.20511 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 --2- 172.21.15.28:0/1084829168 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 >> 172.21.15.28:0/1084829168 conn(0x55e85a4e5a50 msgr2=0x55e85a4e4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 shutdown_connections 2024-03-22T02:39:16.724 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 1 -- 172.21.15.28:0/1084829168 wait complete. 2024-03-22T02:39:16.725 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.714+0000 7f6524977740 20 client.20511 trim_cache size 0 max 16384 2024-03-22T02:39:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 Processor -- start 2024-03-22T02:39:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 -- start start 2024-03-22T02:39:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1ad210 con 0x55e85a4ea4e0 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1aaae0 con 0x55e85a4eaed0 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.735+0000 7f6524977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1a8930 con 0x55e85a4f0040 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44974/0 (socket says 172.21.15.28:44974) 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4fb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651b7fe640 1 -- 172.21.15.28:0/981736345 learned_addr learned my addr 172.21.15.28:0/981736345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651bfff640 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 -- 172.21.15.28:0/981736345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 msgr2=0x55e85a4e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 -- 172.21.15.28:0/981736345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 msgr2=0x55e85a4e8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 -- 172.21.15.28:0/981736345 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e85a4efd80 con 0x55e85a4eaed0 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651b7fe640 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.736+0000 7f651affd640 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4fb9a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55e85a4f5940 tx=0x7f6510005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab5cb6ae7a4ad6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f65017fa640 1 -- 172.21.15.28:0/981736345 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510022070 con 0x55e85a4eaed0 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f65017fa640 1 -- 172.21.15.28:0/981736345 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65100120d0 con 0x55e85a4eaed0 2024-03-22T02:39:16.739 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f65017fa640 1 -- 172.21.15.28:0/981736345 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651001d460 con 0x55e85a4eaed0 2024-03-22T02:39:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 -- 172.21.15.28:0/981736345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 msgr2=0x55e85a4fb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4fb9a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55e85a4f5940 tx=0x7f6510005f90 comp rx=0 tx=0).stop 2024-03-22T02:39:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 -- 172.21.15.28:0/981736345 shutdown_connections 2024-03-22T02:39:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4eaed0 0x55e85a4fb9a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 --2- 172.21.15.28:0/981736345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4ea4e0 0x55e85a4e8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.737+0000 7f6524977740 1 -- 172.21.15.28:0/981736345 >> 172.21.15.28:0/981736345 conn(0x55e85a4e5a50 msgr2=0x55e85a4f4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 -- 172.21.15.28:0/981736345 shutdown_connections 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 -- 172.21.15.28:0/981736345 wait complete. 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 -- start start 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 Processor -- start 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.738+0000 7f6524977740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f6524977740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1a8fa0 con 0x55e85a4eaed0 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f6524977740 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1aa410 con 0x55e85a4ea4e0 2024-03-22T02:39:16.744 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f6524977740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f651c1a9670 con 0x55e85a4f0040 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f6524977740 10 client.0 _ll_register_callbacks cb 0x7f650c0b9d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651b7fe640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.83:3300/0 says I am v2:172.21.15.28:38026/0 (socket says 172.21.15.28:38026) 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44988/0 (socket says 172.21.15.28:44988) 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651b7fe640 1 -- 172.21.15.28:0/1590571059 learned_addr learned my addr 172.21.15.28:0/1590571059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 -- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 msgr2=0x55e85a4f8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 -- 172.21.15.28:0/1590571059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 msgr2=0x55e85a4f91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 -- 172.21.15.28:0/1590571059 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e85a4fa7c0 con 0x55e85a4f0040 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651affd640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f651bfff640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4fa1d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6514009c00 tx=0x7f65140097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c896b7609b6ac1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.739+0000 7f64faffd640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6514045060 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f651401e930 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65100120d0 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f651404c050 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f6524977740 10 client.30274 fetch_fsmap learned FSMap version 46 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55e85a4fbad0 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65100120d0 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.740+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 46) ==== 8346+0+0 (secure 0 0 0) 0x7f651401ebe0 con 0x55e85a4f0040 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 10 client.30274 fetch_fsmap finished waiting for FSMap version 46 2024-03-22T02:39:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 10 client.30274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 20 client.30274 populate_metadata read hostname 'smithi028' 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a56f370 0x55e85a571790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 4 client.30274 mds_command: new command op to 24509 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 --> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e85a4568d0 con 0x55e85a56f370 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.741+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a56f370 0x55e85a571790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.742+0000 7f64faffd640 10 client.30274 ms_handle_connect on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:16.742+0000 7f651b7fe640 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a56f370 0x55e85a571790 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24a5768f43801b39 server_cookie=dc7d47ed5808c5cf in_seq=0 out_seq=0 2024-03-22T02:39:17.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 206008 15.30 MB/sec execute 597 sec latency 862.535 ms 2024-03-22T02:39:17.990 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f64faffd640 1 -- 172.21.15.28:0/1590571059 <== mds.0 v2:172.21.15.83:6838/2818900206 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e85a4568d0 con 0x55e85a56f370 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f64faffd640 10 client.30274 handle_command_reply: tid=0 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 client.30274 shutdown 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 msgr2=0x55e85a4fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4fa1d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6514009c00 tx=0x7f65140097f0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 shutdown_connections 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:6838/2818900206,v1:172.21.15.83:6839/2818900206] conn(0x55e85a56f370 0x55e85a571790 crc :-1 s=READY pgs=459 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55e85a4f0040 0x55e85a4fa1d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.991 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e85a4eaed0 0x55e85a4f91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 --2- 172.21.15.28:0/1590571059 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x55e85a4ea4e0 0x55e85a4f8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f64faffd640 0 client.30274 ms_handle_reset on v2:172.21.15.83:6838/2818900206 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 >> 172.21.15.28:0/1590571059 conn(0x55e85a4e5a50 msgr2=0x55e85a4e8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 shutdown_connections 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.987+0000 7f6524977740 1 -- 172.21.15.28:0/1590571059 wait complete. 2024-03-22T02:39:17.992 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6524977740 20 client.30274 trim_cache size 0 max 16384 2024-03-22T02:39:17.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:{ 2024-03-22T02:39:17.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "message": "", 2024-03-22T02:39:17.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout: "return_code": 0 2024-03-22T02:39:17.993 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stdout:} 2024-03-22T02:39:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 msgr2=0x7f65040a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 0x7f65040a5480 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f651400f790 tx=0x7f6514018000 comp rx=0 tx=0).stop 2024-03-22T02:39:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 msgr2=0x7f651c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:17.995 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f650c0324c0 tx=0x7f650c032f40 comp rx=0 tx=0).stop 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 shutdown_connections 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f65040a2fc0 0x7f65040a5480 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f651c102660 0x7f651c19f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f651c101ca0 0x7f651c19cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 --2- 172.21.15.28:0/1911943529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f651c100ab0 0x7f651c069da0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 >> 172.21.15.28:0/1911943529 conn(0x7f651c0fc020 msgr2=0x7f651c104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 shutdown_connections 2024-03-22T02:39:17.996 INFO:teuthology.task.background_exec.ubuntu@smithi028.front.sepia.ceph.com.smithi028.stderr:2024-03-22T02:39:17.988+0000 7f6521d93640 1 -- 172.21.15.28:0/1911943529 wait complete. 2024-03-22T02:39:18.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:18 smithi120 ceph-mon[36803]: pgmap v1133: 129 pgs: 129 active+clean; 300 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:39:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:18 smithi083 ceph-mon[36340]: pgmap v1133: 129 pgs: 129 active+clean; 300 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:39:18.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:18 smithi028 ceph-mon[31916]: pgmap v1133: 129 pgs: 129 active+clean; 300 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 4.4 MiB/s wr, 394 op/s 2024-03-22T02:39:18.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 206185 15.28 MB/sec execute 598 sec latency 1009.040 ms 2024-03-22T02:39:19.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 206452 15.27 MB/sec execute 599 sec latency 130.333 ms 2024-03-22T02:39:20.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:20 smithi120 ceph-mon[36803]: pgmap v1134: 129 pgs: 129 active+clean; 298 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.6 MiB/s wr, 441 op/s 2024-03-22T02:39:20.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:20 smithi083 ceph-mon[36340]: pgmap v1134: 129 pgs: 129 active+clean; 298 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.6 MiB/s wr, 441 op/s 2024-03-22T02:39:20.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:20 smithi028 ceph-mon[31916]: pgmap v1134: 129 pgs: 129 active+clean; 298 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 4.6 MiB/s wr, 441 op/s 2024-03-22T02:39:20.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 cleanup 600 sec 2024-03-22T02:39:21.703 INFO:tasks.workunit.client.0.smithi028.stdout: 10 cleanup 601 sec 2024-03-22T02:39:22.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:22 smithi028 ceph-mon[31916]: pgmap v1135: 129 pgs: 129 active+clean; 297 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 425 op/s 2024-03-22T02:39:22.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:22 smithi120 ceph-mon[36803]: pgmap v1135: 129 pgs: 129 active+clean; 297 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 425 op/s 2024-03-22T02:39:22.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:22 smithi083 ceph-mon[36340]: pgmap v1135: 129 pgs: 129 active+clean; 297 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 3.6 MiB/s wr, 425 op/s 2024-03-22T02:39:22.703 INFO:tasks.workunit.client.0.smithi028.stdout: 2 cleanup 602 sec 2024-03-22T02:39:22.944 INFO:tasks.workunit.client.0.smithi028.stdout: 0 cleanup 602 sec 2024-03-22T02:39:22.944 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:39:22.944 INFO:tasks.workunit.client.0.smithi028.stdout: Operation Count AvgLat MaxLat 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: ---------------------------------------- 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: NTCreateX 293974 8.402 1175.116 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Close 215951 0.003 0.227 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Rename 12467 34.667 1079.456 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Unlink 59359 13.196 287.811 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Qpathinfo 266825 6.464 1263.952 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Qfileinfo 46552 0.002 0.041 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Qfsinfo 48868 0.212 41.345 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Sfileinfo 23972 1.222 627.608 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: Find 103078 0.857 1154.380 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: WriteX 145350 0.222 1094.989 2024-03-22T02:39:22.945 INFO:tasks.workunit.client.0.smithi028.stdout: ReadX 461499 0.118 1049.704 2024-03-22T02:39:22.946 INFO:tasks.workunit.client.0.smithi028.stdout: LockX 960 13.253 112.026 2024-03-22T02:39:22.946 INFO:tasks.workunit.client.0.smithi028.stdout: UnlockX 960 11.347 163.068 2024-03-22T02:39:22.946 INFO:tasks.workunit.client.0.smithi028.stdout: Flush 20593 16.886 1264.880 2024-03-22T02:39:22.946 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-03-22T02:39:22.946 INFO:tasks.workunit.client.0.smithi028.stdout:Throughput 15.275 MB/sec 10 clients 10 procs max_latency=1264.884 ms 2024-03-22T02:39:22.946 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-22T02:39:22.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-22T02:39:24.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:24 smithi120 ceph-mon[36803]: pgmap v1136: 129 pgs: 129 active+clean; 294 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-22T02:39:24.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:24 smithi083 ceph-mon[36340]: pgmap v1136: 129 pgs: 129 active+clean; 294 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-22T02:39:24.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:24 smithi028 ceph-mon[31916]: pgmap v1136: 129 pgs: 129 active+clean; 294 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 4.2 MiB/s wr, 512 op/s 2024-03-22T02:39:25.405 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-03-22T02:39:25.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-22T02:39:25.889 DEBUG:teuthology.parallel:result is None 2024-03-22T02:39:25.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T02:39:25.925 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-22T02:39:25.925 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-22T02:39:25.936 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:39:25.974 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-22T02:39:25.985 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-22T02:39:25.985 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-22T02:39:25.986 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-22T02:39:26.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:39:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.115+0000 7fd12cf81640 1 -- 172.21.15.28:0/2280716791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128103790 msgr2=0x7fd12810a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.115+0000 7fd12cf81640 1 --2- 172.21.15.28:0/2280716791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128103790 0x7fd12810a020 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fd118009ca0 tx=0x7fd11802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 -- 172.21.15.28:0/2280716791 shutdown_connections 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 --2- 172.21.15.28:0/2280716791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128103790 0x7fd12810a020 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 --2- 172.21.15.28:0/2280716791 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128102dd0 0x7fd128103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 --2- 172.21.15.28:0/2280716791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd128102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 -- 172.21.15.28:0/2280716791 >> 172.21.15.28:0/2280716791 conn(0x7fd1280fd130 msgr2=0x7fd1280ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 -- 172.21.15.28:0/2280716791 shutdown_connections 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 -- 172.21.15.28:0/2280716791 wait complete. 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 Processor -- start 2024-03-22T02:39:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.116+0000 7fd12cf81640 1 -- start start 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd12819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128103790 0x7fd1281a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810d840 con 0x7fd128102dd0 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810d6c0 con 0x7fd128103790 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12cf81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd12810d9c0 con 0x7fd128101be0 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37628/0 (socket says 172.21.15.28:37628) 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 -- 172.21.15.28:0/403655234 learned_addr learned my addr 172.21.15.28:0/403655234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12659c640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd12819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd126d9d640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128103790 0x7fd1281a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 -- 172.21.15.28:0/403655234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 msgr2=0x7fd12819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd12819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 -- 172.21.15.28:0/403655234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128103790 msgr2=0x7fd1281a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128103790 0x7fd1281a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 -- 172.21.15.28:0/403655234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1281a1ca0 con 0x7fd128102dd0 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd12659c640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd12819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.117+0000 7fd125d9b640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e9f0 tx=0x7fd11c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.120+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c00cd80 con 0x7fd128102dd0 2024-03-22T02:39:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.120+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd11c010910 con 0x7fd128102dd0 2024-03-22T02:39:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.120+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1281a1f90 con 0x7fd128102dd0 2024-03-22T02:39:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.120+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c018990 con 0x7fd128102dd0 2024-03-22T02:39:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.120+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1281a24a0 con 0x7fd128102dd0 2024-03-22T02:39:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.122+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd11c010ac0 con 0x7fd128102dd0 2024-03-22T02:39:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.123+0000 7fd1177fe640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 0x7fd1100a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.123+0000 7fd12659c640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 0x7fd1100a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.123+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd11c0cfa30 con 0x7fd128102dd0 2024-03-22T02:39:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.121+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1281a26d0 con 0x7fd128102dd0 2024-03-22T02:39:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.138+0000 7fd12659c640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 0x7fd1100a5460 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd128102c30 tx=0x7fd120005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.143+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd11c09be00 con 0x7fd128102dd0 2024-03-22T02:39:26.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:26 smithi028 ceph-mon[31916]: pgmap v1137: 129 pgs: 129 active+clean; 270 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-03-22T02:39:26.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.353+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1281a2a60 con 0x7fd128102dd0 2024-03-22T02:39:26.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.355+0000 7fd1177fe640 1 -- 172.21.15.28:0/403655234 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24903 (secure 0 0 0) 0x7fd11c09fea0 con 0x7fd128102dd0 2024-03-22T02:39:26.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:39:26.357 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":98,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:29:42.392608+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 msgr2=0x7fd1100a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 0x7fd1100a5460 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd128102c30 tx=0x7fd120005f90 comp rx=0 tx=0).stop 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 msgr2=0x7fd12819a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fd11c00e9f0 tx=0x7fd11c00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 shutdown_connections 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1100a2fa0 0x7fd1100a5460 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd128103790 0x7fd1281a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128102dd0 0x7fd12819a5c0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 --2- 172.21.15.28:0/403655234 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd128101be0 0x7fd12819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 >> 172.21.15.28:0/403655234 conn(0x7fd1280fd130 msgr2=0x7fd128073dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 shutdown_connections 2024-03-22T02:39:26.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.357+0000 7fd12cf81640 1 -- 172.21.15.28:0/403655234 wait complete. 2024-03-22T02:39:26.370 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:39:26.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:39:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- 172.21.15.28:0/1158844310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4107880 msgr2=0x7f17d4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 --2- 172.21.15.28:0/1158844310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4107880 0x7f17d4109cd0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f17cc009ca0 tx=0x7f17cc02c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- 172.21.15.28:0/1158844310 shutdown_connections 2024-03-22T02:39:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 --2- 172.21.15.28:0/1158844310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4107880 0x7f17d4109cd0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 --2- 172.21.15.28:0/1158844310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4104f50 0x7f17d4107340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 --2- 172.21.15.28:0/1158844310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4102620 0x7f17d4104a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- 172.21.15.28:0/1158844310 >> 172.21.15.28:0/1158844310 conn(0x7f17d40fc030 msgr2=0x7f17d40fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- 172.21.15.28:0/1158844310 shutdown_connections 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- 172.21.15.28:0/1158844310 wait complete. 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 Processor -- start 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.475+0000 7f17da8d4640 1 -- start start 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4104f50 0x7f17d41005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4107880 0x7f17d4100b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410d840 con 0x7f17d4102620 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410d6c0 con 0x7f17d4107880 2024-03-22T02:39:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f17d410d9c0 con 0x7f17d4104f50 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37652/0 (socket says 172.21.15.28:37652) 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 -- 172.21.15.28:0/3597846322 learned_addr learned my addr 172.21.15.28:0/3597846322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d37fe640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4104f50 0x7f17d41005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d8e71640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4107880 0x7f17d4100b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 -- 172.21.15.28:0/3597846322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4104f50 msgr2=0x7f17d41005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4104f50 0x7f17d41005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 -- 172.21.15.28:0/3597846322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4107880 msgr2=0x7f17d4100b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4107880 0x7f17d4100b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 -- 172.21.15.28:0/3597846322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d41013a0 con 0x7f17d4102620 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d3fff640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f17c400e9f0 tx=0x7f17c400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c400cd80 con 0x7f17d4102620 2024-03-22T02:39:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17c4005500 con 0x7f17d4102620 2024-03-22T02:39:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d41a1fa0 con 0x7f17d4102620 2024-03-22T02:39:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.476+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c4010690 con 0x7f17d4102620 2024-03-22T02:39:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.477+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d41a23b0 con 0x7f17d4102620 2024-03-22T02:39:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.477+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d40fdb20 con 0x7f17d4102620 2024-03-22T02:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.479+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f17c40046c0 con 0x7f17d4102620 2024-03-22T02:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.479+0000 7f17d17fa640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 0x7f17bc0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.479+0000 7f17d37fe640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 0x7f17bc0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.479+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f17c40cfd70 con 0x7f17d4102620 2024-03-22T02:39:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.480+0000 7f17d37fe640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 0x7f17bc0a53e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f17d4101d00 tx=0x7f17c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.483+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17c4053740 con 0x7f17d4102620 2024-03-22T02:39:26.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:26 smithi120 ceph-mon[36803]: pgmap v1137: 129 pgs: 129 active+clean; 270 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-03-22T02:39:26.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:26 smithi083 ceph-mon[36340]: pgmap v1137: 129 pgs: 129 active+clean; 270 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 4.6 MiB/s wr, 537 op/s 2024-03-22T02:39:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.668+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f17d402d3b0 con 0x7f17d4102620 2024-03-22T02:39:26.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.668+0000 7f17d17fa640 1 -- 172.21.15.28:0/3597846322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f17c409c790 con 0x7f17d4102620 2024-03-22T02:39:26.670 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:39:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 msgr2=0x7f17bc0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 0x7f17bc0a53e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f17d4101d00 tx=0x7f17c800b410 comp rx=0 tx=0).stop 2024-03-22T02:39:26.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 msgr2=0x7f17d4101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f17c400e9f0 tx=0x7f17c400edf0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 shutdown_connections 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f17bc0a2f20 0x7f17bc0a53e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f17d4107880 0x7f17d4100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17d4104f50 0x7f17d41005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 --2- 172.21.15.28:0/3597846322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17d4102620 0x7f17d4101f90 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.670+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 >> 172.21.15.28:0/3597846322 conn(0x7f17d40fc030 msgr2=0x7f17d4102f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.671+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 shutdown_connections 2024-03-22T02:39:26.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.671+0000 7f17da8d4640 1 -- 172.21.15.28:0/3597846322 wait complete. 2024-03-22T02:39:26.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:39:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 -- 172.21.15.28:0/1215650828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760100a90 msgr2=0x7f1760100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 --2- 172.21.15.28:0/1215650828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760100a90 0x7f1760100eb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f1750009900 tx=0x7f175002bc30 comp rx=0 tx=0).stop 2024-03-22T02:39:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 -- 172.21.15.28:0/1215650828 shutdown_connections 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 --2- 172.21.15.28:0/1215650828 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 0x7f1760108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 --2- 172.21.15.28:0/1215650828 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760101c80 0x7f1760102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 --2- 172.21.15.28:0/1215650828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760100a90 0x7f1760100eb0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 -- 172.21.15.28:0/1215650828 >> 172.21.15.28:0/1215650828 conn(0x7f17600fc020 msgr2=0x7f17600fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 -- 172.21.15.28:0/1215650828 shutdown_connections 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.789+0000 7f1766815640 1 -- 172.21.15.28:0/1215650828 wait complete. 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 Processor -- start 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 -- start start 2024-03-22T02:39:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760100a90 0x7f1760197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 0x7f176019f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f176010c6e0 con 0x7f1760101c80 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f176010c560 con 0x7f1760102640 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1766815640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f176010c860 con 0x7f1760100a90 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f175f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f175f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37666/0 (socket says 172.21.15.28:37666) 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f175f7fe640 1 -- 172.21.15.28:0/72943555 learned_addr learned my addr 172.21.15.28:0/72943555 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f175ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760100a90 0x7f1760197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.790+0000 7f1764db2640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 0x7f176019f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 -- 172.21.15.28:0/72943555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760100a90 msgr2=0x7f1760197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760100a90 0x7f1760197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 -- 172.21.15.28:0/72943555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 msgr2=0x7f176019f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 0x7f176019f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 -- 172.21.15.28:0/72943555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176019fb20 con 0x7f1760101c80 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175f7fe640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f174800d950 tx=0x7f174800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748017070 con 0x7f1760101c80 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1748009c30 con 0x7f1760101c80 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176019fe10 con 0x7f1760101c80 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174800f6b0 con 0x7f1760101c80 2024-03-22T02:39:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.791+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17601a02d0 con 0x7f1760101c80 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.792+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17601a0500 con 0x7f1760101c80 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.793+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f17480109d0 con 0x7f1760101c80 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.794+0000 7f175d7fa640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 0x7f174c0a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.794+0000 7f175ffff640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 0x7f174c0a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.794+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f17480cf330 con 0x7f1760101c80 2024-03-22T02:39:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.794+0000 7f175ffff640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 0x7f174c0a53e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1750000c00 tx=0x7f1750005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:26.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.798+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1748009de0 con 0x7f1760101c80 2024-03-22T02:39:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.983+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f17601a06d0 con 0x7f1760101c80 2024-03-22T02:39:26.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.983+0000 7f175d7fa640 1 -- 172.21.15.28:0/72943555 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7f174809b6a0 con 0x7f1760101c80 2024-03-22T02:39:26.985 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:39:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 msgr2=0x7f174c0a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 0x7f174c0a53e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1750000c00 tx=0x7f1750005fd0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 msgr2=0x7f1760198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f174800d950 tx=0x7f174800de10 comp rx=0 tx=0).stop 2024-03-22T02:39:26.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 shutdown_connections 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f174c0a2f20 0x7f174c0a53e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f1760102640 0x7f176019f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1760101c80 0x7f1760198440 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 --2- 172.21.15.28:0/72943555 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1760100a90 0x7f1760197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 >> 172.21.15.28:0/72943555 conn(0x7f17600fc020 msgr2=0x7f17600fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 shutdown_connections 2024-03-22T02:39:26.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:26.985+0000 7f1766815640 1 -- 172.21.15.28:0/72943555 wait complete. 2024-03-22T02:39:26.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:39:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 -- 172.21.15.28:0/1382623151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10100d70 msgr2=0x7f8d101011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/1382623151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10100d70 0x7f8d101011f0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009900 tx=0x7f8cf802bc30 comp rx=0 tx=0).stop 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 -- 172.21.15.28:0/1382623151 shutdown_connections 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/1382623151 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 0x7f8d10108fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/1382623151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10100d70 0x7f8d101011f0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/1382623151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10102040 0x7f8d10102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 -- 172.21.15.28:0/1382623151 >> 172.21.15.28:0/1382623151 conn(0x7f8d100fb780 msgr2=0x7f8d100fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 -- 172.21.15.28:0/1382623151 shutdown_connections 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.102+0000 7f8d14bf7640 1 -- 172.21.15.28:0/1382623151 wait complete. 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 Processor -- start 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 -- start start 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10100d70 0x7f8d1019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 0x7f8d1019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d860 con 0x7f8d10102040 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d6e0 con 0x7f8d10101730 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d14bf7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1010d9e0 con 0x7f8d10100d70 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d0ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d0ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37680/0 (socket says 172.21.15.28:37680) 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d0ed9d640 1 -- 172.21.15.28:0/469979183 learned_addr learned my addr 172.21.15.28:0/469979183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d0e59c640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10100d70 0x7f8d1019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.103+0000 7f8d07fff640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 0x7f8d1019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 -- 172.21.15.28:0/469979183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10100d70 msgr2=0x7f8d1019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10100d70 0x7f8d1019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 -- 172.21.15.28:0/469979183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 msgr2=0x7f8d1019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 0x7f8d1019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 -- 172.21.15.28:0/469979183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d101a1d50 con 0x7f8d10102040 2024-03-22T02:39:27.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d0ed9d640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800eb10 tx=0x7f8d0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08004b20 con 0x7f8d10102040 2024-03-22T02:39:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d08004cc0 con 0x7f8d10102040 2024-03-22T02:39:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d08010760 con 0x7f8d10102040 2024-03-22T02:39:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d101a1fe0 con 0x7f8d10102040 2024-03-22T02:39:27.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.104+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d101a24b0 con 0x7f8d10102040 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.105+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d101a2740 con 0x7f8d10102040 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.106+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f8d08005020 con 0x7f8d10102040 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.107+0000 7f8d077fe640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 0x7f8cfc0a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.107+0000 7f8d0e59c640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 0x7f8cfc0a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.107+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f8d080cfd50 con 0x7f8d10102040 2024-03-22T02:39:27.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.107+0000 7f8d0e59c640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 0x7f8cfc0a5460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8d10069ee0 tx=0x7f8d00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.111+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8d0809c710 con 0x7f8d10102040 2024-03-22T02:39:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.303+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d101a2910 con 0x7f8d10102040 2024-03-22T02:39:27.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.303+0000 7f8d077fe640 1 -- 172.21.15.28:0/469979183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f8d0809c010 con 0x7f8d10102040 2024-03-22T02:39:27.305 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:39:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 msgr2=0x7f8cfc0a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 0x7f8cfc0a5460 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f8d10069ee0 tx=0x7f8d00005f90 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 msgr2=0x7f8d101a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800eb10 tx=0x7f8d0800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 shutdown_connections 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f8cfc0a2fa0 0x7f8cfc0a5460 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d10102040 0x7f8d101a16e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f8d10101730 0x7f8d1019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 --2- 172.21.15.28:0/469979183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8d10100d70 0x7f8d1019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.305+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 >> 172.21.15.28:0/469979183 conn(0x7f8d100fb780 msgr2=0x7f8d10109890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.306+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 shutdown_connections 2024-03-22T02:39:27.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.306+0000 7f8d14bf7640 1 -- 172.21.15.28:0/469979183 wait complete. 2024-03-22T02:39:27.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:27 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:27.317 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:27 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/403655234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:39:27.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:27 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3597846322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:27 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/72943555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:39:27.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 -- 172.21.15.28:0/2385064347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824103750 msgr2=0x7fa824109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 --2- 172.21.15.28:0/2385064347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824103750 0x7fa824109fe0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fa814009ca0 tx=0x7fa81402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 -- 172.21.15.28:0/2385064347 shutdown_connections 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 --2- 172.21.15.28:0/2385064347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824103750 0x7fa824109fe0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 --2- 172.21.15.28:0/2385064347 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824102d90 0x7fa824103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 --2- 172.21.15.28:0/2385064347 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 -- 172.21.15.28:0/2385064347 >> 172.21.15.28:0/2385064347 conn(0x7fa8240fd130 msgr2=0x7fa8240ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 -- 172.21.15.28:0/2385064347 shutdown_connections 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.422+0000 7fa829902640 1 -- 172.21.15.28:0/2385064347 wait complete. 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 Processor -- start 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 -- start start 2024-03-22T02:39:27.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824103750 0x7fa82419f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa82410d820 con 0x7fa824102d90 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fa82410d6a0 con 0x7fa824101ba0 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa829902640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fa82410d9a0 con 0x7fa824103750 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37692/0 (socket says 172.21.15.28:37692) 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 -- 172.21.15.28:0/3442733355 learned_addr learned my addr 172.21.15.28:0/3442733355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa822ffd640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8237fe640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824103750 0x7fa82419f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 -- 172.21.15.28:0/3442733355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824103750 msgr2=0x7fa82419f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824103750 0x7fa82419f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 -- 172.21.15.28:0/3442733355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 msgr2=0x7fa824197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa8227fc640 1 -- 172.21.15.28:0/3442733355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82419fa80 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.423+0000 7fa822ffd640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa8227fc640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e9f0 tx=0x7fa80c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c00cd80 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa80c005500 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82419fd70 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c010690 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.424+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8241a0210 con 0x7fa824102d90 2024-03-22T02:39:27.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.425+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa824108ec0 con 0x7fa824102d90 2024-03-22T02:39:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.426+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fa80c0046c0 con 0x7fa824102d90 2024-03-22T02:39:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.426+0000 7fa828900640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 0x7fa8100a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.427+0000 7fa822ffd640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 0x7fa8100a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.427+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fa80c0cfbc0 con 0x7fa824102d90 2024-03-22T02:39:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.427+0000 7fa822ffd640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 0x7fa8100a53e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa824102bf0 tx=0x7fa81c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.431+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa80c09c5e0 con 0x7fa824102d90 2024-03-22T02:39:27.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:27 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:27.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:27 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/403655234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:39:27.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:27 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3597846322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:27 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/72943555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:27 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:27 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/403655234' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:39:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:27 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3597846322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:27 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/72943555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:39:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.614+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa824062830 con 0x7fa824102d90 2024-03-22T02:39:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.614+0000 7fa828900640 1 -- 172.21.15.28:0/3442733355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v98) ==== 103+0+11 (secure 0 0 0) 0x7fa80c0bc0d0 con 0x7fa824102d90 2024-03-22T02:39:27.616 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 msgr2=0x7fa8100a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 0x7fa8100a53e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa824102bf0 tx=0x7fa81c005f90 comp rx=0 tx=0).stop 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 msgr2=0x7fa8241983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00e9f0 tx=0x7fa80c00edf0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 shutdown_connections 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fa8100a2f20 0x7fa8100a53e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa824103750 0x7fa82419f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa824102d90 0x7fa8241983a0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 --2- 172.21.15.28:0/3442733355 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fa824101ba0 0x7fa824197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 >> 172.21.15.28:0/3442733355 conn(0x7fa8240fd130 msgr2=0x7fa824106090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 shutdown_connections 2024-03-22T02:39:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.616+0000 7fa829902640 1 -- 172.21.15.28:0/3442733355 wait complete. 2024-03-22T02:39:27.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 -- 172.21.15.28:0/2306040317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 msgr2=0x7fad1c100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 --2- 172.21.15.28:0/2306040317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c100290 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7fad14009930 tx=0x7fad1402bc30 comp rx=0 tx=0).stop 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 -- 172.21.15.28:0/2306040317 shutdown_connections 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 --2- 172.21.15.28:0/2306040317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 0x7fad1c10a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 --2- 172.21.15.28:0/2306040317 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 0x7fad1c1032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 --2- 172.21.15.28:0/2306040317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c100290 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 -- 172.21.15.28:0/2306040317 >> 172.21.15.28:0/2306040317 conn(0x7fad1c0fb7c0 msgr2=0x7fad1c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 -- 172.21.15.28:0/2306040317 shutdown_connections 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.730+0000 7fad21655640 1 -- 172.21.15.28:0/2306040317 wait complete. 2024-03-22T02:39:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 Processor -- start 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 -- start start 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 0x7fad1c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 0x7fad1c19f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c10d820 con 0x7fad1c0ffe70 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c10d6a0 con 0x7fad1c102e40 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad21655640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c10d9a0 con 0x7fad1c103800 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad1affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37704/0 (socket says 172.21.15.28:37704) 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.731+0000 7fad1affd640 1 -- 172.21.15.28:0/403489900 learned_addr learned my addr 172.21.15.28:0/403489900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1a7fc640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 0x7fad1c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1b7fe640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 0x7fad1c19f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 -- 172.21.15.28:0/403489900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 msgr2=0x7fad1c19f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 0x7fad1c19f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 -- 172.21.15.28:0/403489900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 msgr2=0x7fad1c198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 0x7fad1c198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 -- 172.21.15.28:0/403489900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c19fb40 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad1affd640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fad1402c7b0 tx=0x7fad1402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14041070 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad140054e0 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c19fdd0 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14034440 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.732+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c1a0200 con 0x7fad1c0ffe70 2024-03-22T02:39:27.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.733+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c109040 con 0x7fad1c0ffe70 2024-03-22T02:39:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.734+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fad14044070 con 0x7fad1c0ffe70 2024-03-22T02:39:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.735+0000 7fad13fff640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 0x7fad080a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:39:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.735+0000 7fad1a7fc640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 0x7fad080a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:39:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.735+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fad140f15e0 con 0x7fad1c0ffe70 2024-03-22T02:39:27.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.735+0000 7fad1a7fc640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 0x7fad080a53e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fad040095f0 tx=0x7fad04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:39:27.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.739+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad140bd9b0 con 0x7fad1c0ffe70 2024-03-22T02:39:27.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.971+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fad1c062790 con 0x7fad1c0ffe70 2024-03-22T02:39:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.980+0000 7fad13fff640 1 -- 172.21.15.28:0/403489900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+741842 (secure 0 0 0) 0x7fad140c1a50 con 0x7fad1c0ffe70 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 20, 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 14152, 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-22T02:39:27.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4205492780 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4205492780 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:27.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/4205492780", 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-03-22T02:04:26.467876+0000", 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24118, 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:27.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:27.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:27.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:27.999 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.002 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-03-22T02:39:28.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.012 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.015 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-03-22T02:39:28.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T02:39:28.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:28.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-03-22T02:39:28.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-03-22T02:39:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-03-22T02:39:28.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-03-22T02:39:28.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-03-22T02:39:28.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.049 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-03-22T02:39:28.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-03-22T02:39:28.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T02:39:28.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-03-22T02:39:28.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-03-22T02:39:28.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-03-22T02:39:28.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.066 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-03-22T02:39:28.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T02:39:28.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-03-22T02:39:28.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T02:39:28.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-03-22T02:39:28.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.084 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.092 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.093 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.094 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-03-22T02:39:28.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-03-22T02:39:28.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T02:39:28.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-03-22T02:39:28.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-03-22T02:39:28.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.115 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.122 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.125 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-03-22T02:39:28.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T02:39:28.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.130 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T02:39:28.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.141 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.146 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T02:39:28.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T02:39:28.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T02:39:28.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T02:39:28.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T02:39:28.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T02:39:28.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T02:39:28.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T02:39:28.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T02:39:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T02:39:28.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.194 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.205 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-03-22T02:39:28.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.210 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-03-22T02:39:28.211 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:28.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.213 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-03-22T02:39:28.223 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.224 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.226 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:28.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.231 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.234 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.235 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.240 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.243 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-03-22T02:39:28.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.251 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.254 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.255 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-03-22T02:39:28.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.262 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.263 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.265 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.268 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.270 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.272 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.275 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-03-22T02:39:28.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.281 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.284 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-03-22T02:39:28.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.289 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.292 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.293 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.295 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-03-22T02:39:28.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.302 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.303 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.307 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.308 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.309 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.315 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.316 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.317 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-03-22T02:39:28.322 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.324 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.345 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.346 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.349 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.350 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.354 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T02:39:28.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.361 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-03-22T02:39:28.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-03-22T02:39:28.366 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.370 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.371 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.373 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:28.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.387 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.388 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.391 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.392 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-03-22T02:39:28.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.402 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.408 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-03-22T02:39:28.416 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.420 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.422 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-03-22T02:39:28.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-03-22T02:39:28.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-03-22T02:39:28.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-03-22T02:39:28.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-03-22T02:39:28.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-03-22T02:39:28.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-03-22T02:39:28.455 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.457 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.458 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.460 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.464 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.467 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.471 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.473 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.474 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.476 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-03-22T02:39:28.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.482 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.485 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.491 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-03-22T02:39:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.494 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.496 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.503 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-03-22T02:39:28.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.505 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.506 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.507 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.510 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.514 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.517 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-03-22T02:39:28.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.520 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.521 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.524 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.528 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.529 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.530 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.532 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.533 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-03-22T02:39:28.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 24124, 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-03-22T02:39:28.538 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.544 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-03-22T02:39:28.546 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.547 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-03-22T02:39:28.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-03-22T02:39:28.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T02:39:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.555 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.556 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.559 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-03-22T02:39:28.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-03-22T02:39:28.563 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.567 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.570 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.573 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.574 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-03-22T02:39:28.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-03-22T02:39:28.583 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T02:39:28.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-03-22T02:39:28.587 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.590 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-03-22T02:39:28.593 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T02:39:28.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.601 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.604 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T02:39:28.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-03-22T02:39:28.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-03-22T02:39:28.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.614 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-03-22T02:39:28.618 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-03-22T02:39:28.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-03-22T02:39:28.623 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.630 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.631 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.632 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T02:39:28.635 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-03-22T02:39:28.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.641 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-03-22T02:39:28.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T02:39:28.644 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-03-22T02:39:28.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-03-22T02:39:28.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T02:39:28.654 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.658 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.660 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.661 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.664 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.666 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.668 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.669 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.671 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T02:39:28.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.675 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.676 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.680 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.684 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.686 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T02:39:28.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T02:39:28.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.696 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.698 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.699 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T02:39:28.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.703 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T02:39:28.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.706 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T02:39:28.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.711 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T02:39:28.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.714 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T02:39:28.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T02:39:28.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.717 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.718 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.720 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.723 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.725 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-03-22T02:39:28.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.729 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.731 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.733 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.734 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T02:39:28.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.738 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.743 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.747 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.748 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.749 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-03-22T02:39:28.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-03-22T02:39:28.755 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.759 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.763 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.768 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:28.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-03-22T02:39:28.770 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-03-22T02:39:28.783 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-03-22T02:39:28.786 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.789 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.790 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.796 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.797 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.805 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.806 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-03-22T02:39:28.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.812 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-03-22T02:39:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.816 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-03-22T02:39:28.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.822 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.824 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.827 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.830 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.831 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.833 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.836 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.837 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.842 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.844 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.854 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.856 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.857 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.858 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.860 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.863 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.864 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.866 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.869 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:28.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.874 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.875 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.878 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-03-22T02:39:28.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.881 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T02:39:28.883 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.884 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.885 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-03-22T02:39:28.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:28.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.893 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:28.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.902 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-03-22T02:39:28.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.905 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.912 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:28.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.915 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.920 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.921 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.926 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.927 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.934 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.940 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-03-22T02:39:28.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-03-22T02:39:28.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-03-22T02:39:28.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.948 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.954 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.957 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.959 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.960 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.963 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.970 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.971 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.976 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.977 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T02:39:28.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T02:39:28.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:28.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.991 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:28.998 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:28.999 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.000 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.002 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.003 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.008 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-03-22T02:39:29.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-03-22T02:39:29.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.012 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-03-22T02:39:29.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-03-22T02:39:29.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.019 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.024 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-03-22T02:39:29.027 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "restful" 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.030 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.041 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-03-22T02:39:29.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.048 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-03-22T02:39:29.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-03-22T02:39:29.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-03-22T02:39:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-03-22T02:39:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-22T02:39:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-22T02:39:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-03-22T02:39:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-03-22T02:39:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.071 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-03-22T02:39:29.073 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.078 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-03-22T02:39:29.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-03-22T02:39:29.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-03-22T02:39:29.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-03-22T02:39:29.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-22T02:39:29.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.102 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.104 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.106 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.109 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-22T02:39:29.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-03-22T02:39:29.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-03-22T02:39:29.117 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-22T02:39:29.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.122 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-03-22T02:39:29.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-03-22T02:39:29.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-03-22T02:39:29.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-22T02:39:29.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.134 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.136 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.138 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.139 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-22T02:39:29.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-22T02:39:29.147 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.148 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.149 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.159 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.168 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-22T02:39:29.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-22T02:39:29.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-22T02:39:29.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-22T02:39:29.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.180 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.191 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.193 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.196 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.197 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.199 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.204 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.205 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.206 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.209 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.210 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.212 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.214 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.215 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-03-22T02:39:29.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-22T02:39:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.225 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.226 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.227 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.230 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-03-22T02:39:29.231 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.232 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.233 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.236 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.237 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-03-22T02:39:29.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-03-22T02:39:29.243 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.249 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.250 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.252 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.254 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.255 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.258 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.260 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.261 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-03-22T02:39:29.262 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-03-22T02:39:29.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.265 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.267 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.269 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.271 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.272 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.273 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.274 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.277 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-03-22T02:39:29.279 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-03-22T02:39:29.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.281 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.282 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.283 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.284 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.285 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.288 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.289 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.291 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.294 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.295 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.298 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.304 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.305 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-03-22T02:39:29.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.312 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.316 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.317 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-03-22T02:39:29.318 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.319 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.320 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.326 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.328 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.329 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.331 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.334 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.336 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-22T02:39:29.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.339 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.340 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.343 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.345 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.347 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-03-22T02:39:29.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-03-22T02:39:29.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.352 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.353 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.357 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-03-22T02:39:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.362 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.364 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.366 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.367 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-03-22T02:39:29.371 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.372 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.376 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.377 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.378 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.380 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.381 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.387 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.390 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-03-22T02:39:29.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-03-22T02:39:29.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-03-22T02:39:29.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-03-22T02:39:29.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-03-22T02:39:29.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.397 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.398 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.400 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.403 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.405 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.408 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.410 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.411 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.413 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.416 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.419 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.421 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.422 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-03-22T02:39:29.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.425 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.427 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.430 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.432 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.434 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.436 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.438 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.441 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.445 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.448 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.451 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.454 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-03-22T02:39:29.454 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.454 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.455 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.459 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.460 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.461 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.463 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.467 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-03-22T02:39:29.469 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.470 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.473 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-03-22T02:39:29.474 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-03-22T02:39:29.479 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-22T02:39:29.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-03-22T02:39:29.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.482 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-03-22T02:39:29.483 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-03-22T02:39:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-03-22T02:39:29.485 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 3, 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-22T02:39:29.486 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 526964316 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3351322426 2024-03-22T02:39:29.487 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 2120269619 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:29.488 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3236218180 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-03-22T02:39:29.489 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:28 smithi083 ceph-mon[36340]: pgmap v1138: 129 pgs: 129 active+clean; 229 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 522 op/s 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:28 smithi083 ceph-mon[36340]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:28 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/469979183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:28 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3442733355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:28 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/403489900' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:28 smithi120 ceph-mon[36803]: pgmap v1138: 129 pgs: 129 active+clean; 229 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 522 op/s 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:28 smithi120 ceph-mon[36803]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:29.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:28 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/469979183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.492 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:28 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3442733355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.492 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:28 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/403489900' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 msgr2=0x7fad080a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 0x7fad080a53e0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fad040095f0 tx=0x7fad04005f90 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 msgr2=0x7fad1c197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7fad1402c7b0 tx=0x7fad1402c650 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 shutdown_connections 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fad080a2f20 0x7fad080a53e0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fad1c103800 0x7fad1c19f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fad1c102e40 0x7fad1c198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 --2- 172.21.15.28:0/403489900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1c0ffe70 0x7fad1c197f20 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 >> 172.21.15.28:0/403489900 conn(0x7fad1c0fb7c0 msgr2=0x7fad1c106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:39:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 shutdown_connections 2024-03-22T02:39:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:39:27.985+0000 7fad21655640 1 -- 172.21.15.28:0/403489900 wait complete. 2024-03-22T02:39:29.499 DEBUG:tasks.check_counter:Getting stats from a 2024-03-22T02:39:29.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-22T02:39:29.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:28 smithi028 ceph-mon[31916]: pgmap v1138: 129 pgs: 129 active+clean; 229 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.1 MiB/s wr, 522 op/s 2024-03-22T02:39:29.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:28 smithi028 ceph-mon[31916]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2024-03-22T02:39:29.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:28 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/469979183' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:28 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3442733355' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:39:29.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:28 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/403489900' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-22T02:39:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:39:30.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:30 smithi083 ceph-mon[36340]: pgmap v1139: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2024-03-22T02:39:30.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:30 smithi028 ceph-mon[31916]: pgmap v1139: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2024-03-22T02:39:30.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:30 smithi120 ceph-mon[36803]: pgmap v1139: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 4.0 MiB/s wr, 551 op/s 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3072, 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:31.853 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001699064, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000037183, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000611831, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000827130, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000275710 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.854 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 341, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 216016, 2024-03-22T02:39:31.855 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.021624178, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.017209536, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001617201, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 339, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.016440980, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000048498 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.856 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 216016 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 451, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 446, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 820416, 2024-03-22T02:39:31.857 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 276592, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.056183405, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.023077975, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.023963416, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.008239016, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 445, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.015634945, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000035134 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.858 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 820416, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 276592 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-03-22T02:39:31.859 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:31.861 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148704, 2024-03-22T02:39:31.862 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 29, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:31.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-03-22T02:39:31.864 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 453, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 756646, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 453, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 453, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 756646, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:31.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:31.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:31.866 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:39:32.528 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-22T02:39:32.529 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-22T02:39:32.529 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-03-22T02:39:32.529 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-03-22T02:39:32.529 DEBUG:tasks.check_counter:Getting stats from d 2024-03-22T02:39:32.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-22T02:39:32.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:32 smithi083 ceph-mon[36340]: pgmap v1140: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 374 op/s 2024-03-22T02:39:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:39:32.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:32 smithi028 ceph-mon[31916]: pgmap v1140: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 374 op/s 2024-03-22T02:39:32.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:32 smithi120 ceph-mon[36803]: pgmap v1140: 129 pgs: 129 active+clean; 142 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 3.1 MiB/s wr, 374 op/s 2024-03-22T02:39:34.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:34 smithi028 ceph-mon[31916]: pgmap v1141: 129 pgs: 129 active+clean; 140 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-03-22T02:39:34.633 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001411911, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000394725, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:34.634 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 458, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 453, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 822384, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 280592, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-03-22T02:39:34.635 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.059230354, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.024007332, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.026038994, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.008363078, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 452, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.018174667, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000040209 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:34.636 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 822384, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 280592 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 348, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3392, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 217344, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.023197112, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.018427792, 2024-03-22T02:39:34.637 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.002000719, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 347, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.019217100, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000055380 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.638 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 217344 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:34.639 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:34.640 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:34.641 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:34.642 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149775, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:34.643 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:34.644 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-03-22T02:39:34.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 460, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 757620, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 460, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 460, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 757620, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:34.646 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:34.647 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:34.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:34.647 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:34.647 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:39:34.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:34 smithi120 ceph-mon[36803]: pgmap v1141: 129 pgs: 129 active+clean; 140 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-03-22T02:39:35.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:34 smithi083 ceph-mon[36340]: pgmap v1141: 129 pgs: 129 active+clean; 140 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-03-22T02:39:35.620 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-22T02:39:35.620 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-22T02:39:35.620 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-22T02:39:35.621 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-22T02:39:35.621 DEBUG:tasks.check_counter:Getting stats from g 2024-03-22T02:39:35.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-22T02:39:35.872 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:39:36.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:36 smithi028 ceph-mon[31916]: pgmap v1142: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.7 MiB/s wr, 251 op/s 2024-03-22T02:39:36.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:36 smithi120 ceph-mon[36803]: pgmap v1142: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.7 MiB/s wr, 251 op/s 2024-03-22T02:39:37.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:36 smithi083 ceph-mon[36340]: pgmap v1142: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.7 MiB/s wr, 251 op/s 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3072, 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.001494435, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000041954, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000579496, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000753542, 2024-03-22T02:39:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000251180 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-03-22T02:39:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 343, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 216368, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.022154643, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.017804768, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001604289, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 341, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.021376020, 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000062686 2024-03-22T02:39:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 216368 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 454, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 449, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 821296, 2024-03-22T02:39:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 278032, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.058962511, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.024873592, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.026079321, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.008041606, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 448, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017017613, 2024-03-22T02:39:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000037985 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 821296, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 278032 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-03-22T02:39:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:37.564 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:37.565 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:37.566 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148704, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 29, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:37.567 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-03-22T02:39:37.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 456, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 757100, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:37.569 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 456, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 456, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 757100, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:37.570 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:39:38.269 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-22T02:39:38.269 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-22T02:39:38.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-22T02:39:38.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-22T02:39:38.269 DEBUG:tasks.check_counter:Getting stats from j 2024-03-22T02:39:38.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-22T02:39:38.691 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T02:39:38.800 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:38 smithi120 ceph-mon[36803]: pgmap v1143: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-03-22T02:39:38.817 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:38 smithi083 ceph-mon[36340]: pgmap v1143: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-03-22T02:39:38.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:38 smithi028 ceph-mon[31916]: pgmap v1143: 129 pgs: 129 active+clean; 130 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 2.2 MiB/s wr, 211 op/s 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 460, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 455, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 822928, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 281872, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-03-22T02:39:40.570 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.059795521, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.024437873, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.026670531, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.008446235, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 454, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.020005107, 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000044064 2024-03-22T02:39:40.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 822928, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 281872 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 348, 2024-03-22T02:39:40.572 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 217248, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 2, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.021334997, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.017075237, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001384362, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 346, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.015371633, 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000044426 2024-03-22T02:39:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 217248 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3072, 2024-03-22T02:39:40.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.000651239, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000036155, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000590949, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000717953, 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000239317 2024-03-22T02:39:40.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-03-22T02:39:40.576 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:40.577 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:40.578 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:40.579 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148704, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 29, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-03-22T02:39:40.580 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:40.581 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 462, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 757880, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 462, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-03-22T02:39:40.582 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 462, 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 757880, 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-03-22T02:39:40.583 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-03-22T02:39:40.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:40 smithi028 ceph-mon[31916]: pgmap v1144: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 2.6 MiB/s wr, 197 op/s 2024-03-22T02:39:40.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:40 smithi120 ceph-mon[36803]: pgmap v1144: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 2.6 MiB/s wr, 197 op/s 2024-03-22T02:39:41.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:40 smithi083 ceph-mon[36340]: pgmap v1144: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 2.6 MiB/s wr, 197 op/s 2024-03-22T02:39:41.269 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-22T02:39:41.269 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-22T02:39:41.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-22T02:39:41.269 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-22T02:39:41.269 DEBUG:tasks.check_counter:Getting stats from b 2024-03-22T02:39:41.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-22T02:39:41.723 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:39:41.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:41 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:39:41.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:41 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:39:41.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:41 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T02:39:42.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:42 smithi028 ceph-mon[31916]: pgmap v1145: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-22T02:39:42.950 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:42 smithi083 ceph-mon[36340]: pgmap v1145: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-22T02:39:42.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:42 smithi120 ceph-mon[36803]: pgmap v1145: 129 pgs: 129 active+clean; 123 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-03-22T02:39:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:39:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:39:43.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:43 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:39:43.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:39:43.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:39:43.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:43 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:39:43.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T02:39:43.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T02:39:43.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:43 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' 2024-03-22T02:39:44.685 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-22T02:39:44.685 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 373320, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 588844, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 78026780, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 614954736, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 416, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 4, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 29.264645376, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 11.902178742, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 11.406722690, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 5.638388165, 2024-03-22T02:39:44.686 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 588846, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 136.300510488, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000231470 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 664323, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.103550818, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000155 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 1007488, 2024-03-22T02:39:44.687 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 321680 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 428954, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 629914, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 96340654, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 571389859, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 29, 2024-03-22T02:39:44.688 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 5, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 29.159395607, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 10.562197698, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 12.462074665, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 6.059386150, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 629916, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 177.557373604, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000281874 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 705072, 2024-03-22T02:39:44.689 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.101077506, 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000143 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 3232, 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 701360 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 1179589, 2024-03-22T02:39:44.690 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1734672, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 359672227, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 1838334965, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 18, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 8, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 71.166538947, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 31.317506785, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 37.957585536, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 8.387343625, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1734664, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 288.969723485, 2024-03-22T02:39:44.691 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000166585 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2341554, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.355199855, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000151 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 58624, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 29792 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-03-22T02:39:44.692 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 253706, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 532.257515656, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.002097930 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 195476, 2024-03-22T02:39:44.693 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 69.806611511, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000357110 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 722678, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 718534, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 718534, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 6818.833355993, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.009489924 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.694 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 4144, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 25, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 23, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 10514, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 33, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 50, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 3102, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 1148, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 1943, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 11, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 1462, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 127, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 448, 2024-03-22T02:39:44.695 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 448, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 3, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 1091013, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 629132, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 4143, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 0, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 7555, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-03-22T02:39:44.696 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 10, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 67, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 10, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 7, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 2946, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 197, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 43131217, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 2946, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-03-22T02:39:44.697 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 606, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 26, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 580, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 2340, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 18, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 44782, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 430418, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:39:44.698 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 207908, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 88067, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 36519, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 444312 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 211, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 0, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 0, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 0, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 1376, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 466, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-03-22T02:39:44.699 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 114491, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 114025, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:39:44.700 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 26, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 10, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 83, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 548737, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 543804, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 543804, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 4933, 2024-03-22T02:39:44.701 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 6500, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 681, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 675, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 675, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 6, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 1, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 8, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 1168517280, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 1177774393, 2024-03-22T02:39:44.702 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 0, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 144857, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 2040.211386021, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.014084313 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 0 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 692, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 114645, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 113953, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 88, 2024-03-22T02:39:44.703 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 473, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 385, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 3085, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 285359, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 282274, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 254, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 126372, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 126118, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 98884, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 231800 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 918337, 2024-03-22T02:39:44.704 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 16268, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 722678, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 16020, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 32695, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 114434, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 935.252684186, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008172856 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 11262, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 339.853172198, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.030176982 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.705 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 268722, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1861.338387645, 2024-03-22T02:39:44.706 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.006926631 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000226806, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000226806 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.707 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.708 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.709 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 119, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1.037639016, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008719655 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.710 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 10, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.022743730, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.002274373 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 180806, 2024-03-22T02:39:44.711 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 2448.758331865, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.013543567 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 24189, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 204.366523010, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008448737 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-03-22T02:39:44.712 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 124, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 1.995507301, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.016092800 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.713 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 576, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 25.559368737, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.044373904 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.714 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 3904, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 28.602690320, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.007326508 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.715 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 9, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.020288498, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.002254277 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 114378, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 972.025792681, 2024-03-22T02:39:44.716 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008498363 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 4, 2024-03-22T02:39:44.717 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 3, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 1, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 46168, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 46168, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 1550, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:44.718 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:44.719 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:44.720 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 4076777, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 1864, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-03-22T02:39:44.721 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9280, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 179, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 5262329, 2024-03-22T02:39:44.722 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 171325, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 507031, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 1279435048, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 32, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 507031, 2024-03-22T02:39:44.723 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 507031, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 42312.137129925, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.083450789 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 507031, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 704501 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 507031, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 6013, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 501018, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-03-22T02:39:44.724 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 10074, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 110029, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 2993, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 147824, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 1720, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 32, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 229265, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-03-22T02:39:44.725 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 3020, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 167887, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-03-22T02:39:44.726 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-03-22T02:39:44.727 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 98, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 13, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 12, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 12, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 0, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 22689, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 96, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 8872 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-03-22T02:39:44.728 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 1, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 0 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 17, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 16, 2024-03-22T02:39:44.729 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 227957, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 114025, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 1981859, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 355630207, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 1981859, 2024-03-22T02:39:44.730 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 1981859, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 355630207, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:44.731 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 507031, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 1291862537, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 507031, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 1291862537, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.732 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 507031, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 507031, 2024-03-22T02:39:44.733 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 507031, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 507031, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:44.734 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 114025, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 12427981, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 114025, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 2967, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 12427981, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.735 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x5643b1884500": { 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 525351, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 548737, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 1173580089, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 548737, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 144857, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 1173054738, 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:44.736 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.737 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:44.737 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-22T02:39:44.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:44 smithi083 ceph-mon[36340]: pgmap v1146: 129 pgs: 129 active+clean; 121 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-03-22T02:39:44.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:44 smithi028 ceph-mon[31916]: pgmap v1146: 129 pgs: 129 active+clean; 121 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-03-22T02:39:44.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:44 smithi120 ceph-mon[36803]: pgmap v1146: 129 pgs: 129 active+clean; 121 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 2.2 MiB/s wr, 134 op/s 2024-03-22T02:39:45.472 INFO:tasks.check_counter:Daemon mds.b mds.exported=10 2024-03-22T02:39:45.472 INFO:tasks.check_counter:Daemon mds.b mds.imported=10 2024-03-22T02:39:45.472 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=211 2024-03-22T02:39:45.472 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=0 2024-03-22T02:39:45.472 DEBUG:tasks.check_counter:Getting stats from e 2024-03-22T02:39:45.472 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-22T02:39:45.799 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:39:46.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:46 smithi028 ceph-mon[31916]: pgmap v1147: 129 pgs: 129 active+clean; 118 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-22T02:39:46.879 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:46 smithi083 ceph-mon[36340]: pgmap v1147: 129 pgs: 129 active+clean; 118 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-22T02:39:46.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:46 smithi120 ceph-mon[36803]: pgmap v1147: 129 pgs: 129 active+clean; 118 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 2.6 MiB/s wr, 145 op/s 2024-03-22T02:39:48.538 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 1239, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1241, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 12899259, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 394066, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 8, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 3, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.084980988, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.024953379, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.027639908, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.025227566, 2024-03-22T02:39:48.539 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1239, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.028618569, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000023098 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:39:48.540 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 2402, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 2371, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 22396890, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 889083, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 5, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 4, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.187771680, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.050835118, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.076400194, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.052528676, 2024-03-22T02:39:48.541 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2371, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.077104195, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000032519 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 946496, 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 284560 2024-03-22T02:39:48.542 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 2917, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 3263, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 14704252, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 1616824, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 6, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 5, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.214486252, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.066500693, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.059943162, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.070043380, 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:48.543 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 3261, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.106153777, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000032552 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 689536 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-03-22T02:39:48.544 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 10451, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.195004265, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000018658 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:48.545 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 0, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 0, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.546 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 5775, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 5775, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 5775, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 0, 2024-03-22T02:39:48.547 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 1, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 0, 2024-03-22T02:39:48.548 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 0, 2024-03-22T02:39:48.549 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 0, 2024-03-22T02:39:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 0 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 0, 2024-03-22T02:39:48.551 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 0, 2024-03-22T02:39:48.552 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 37251, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 0, 2024-03-22T02:39:48.553 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 0, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 47, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 5, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 0, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 4194304, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 51577470, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 51576332, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.554 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 37251 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 45, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 5816, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 5771, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 11, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 54, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 43, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 5775, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 5874, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 99, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 0, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 0, 2024-03-22T02:39:48.555 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 58112, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 215416 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.556 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.557 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-03-22T02:39:48.558 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.559 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.560 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.561 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-03-22T02:39:48.563 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.564 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.565 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.566 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.567 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.568 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 2, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 1, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 2, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 0, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 0, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 4261, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-03-22T02:39:48.569 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:48.570 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:48.571 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 227673, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 132, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:48.572 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9280, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 179, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 4246161, 2024-03-22T02:39:48.573 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 30653, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 6056, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 21, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 3, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 6056, 2024-03-22T02:39:48.574 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 6056, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 22.432460850, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.003704171 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 6056, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 6057 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 6056, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 6056, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 0, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-03-22T02:39:48.575 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 3191, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 2863, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-03-22T02:39:48.576 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 1, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-03-22T02:39:48.577 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-03-22T02:39:48.578 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 98, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 8, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 9, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 10, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 1, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 0, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 2, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 0 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-03-22T02:39:48.579 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 0 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 0, 2024-03-22T02:39:48.580 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 6554, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 49382066, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 6554, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:48.581 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 6554, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 49382066, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:48.582 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 6056, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 47387087, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 6056, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 47387087, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.583 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 6056, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 6056, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 6056, 2024-03-22T02:39:48.584 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 6056, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:48.585 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.586 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x55c3d5ed66e0": { 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-03-22T02:39:48.587 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:48.588 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-22T02:39:48.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:48 smithi028 ceph-mon[31916]: pgmap v1148: 129 pgs: 129 active+clean; 115 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.4 MiB/s wr, 131 op/s 2024-03-22T02:39:48.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:48 smithi120 ceph-mon[36803]: pgmap v1148: 129 pgs: 129 active+clean; 115 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.4 MiB/s wr, 131 op/s 2024-03-22T02:39:49.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:48 smithi083 ceph-mon[36340]: pgmap v1148: 129 pgs: 129 active+clean; 115 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.4 MiB/s wr, 131 op/s 2024-03-22T02:39:49.446 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-22T02:39:49.447 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-22T02:39:49.447 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-22T02:39:49.447 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-22T02:39:49.447 DEBUG:tasks.check_counter:Getting stats from h 2024-03-22T02:39:49.447 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-22T02:39:49.707 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:39:50.793 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:50 smithi083 ceph-mon[36340]: pgmap v1149: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-03-22T02:39:50.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:50 smithi028 ceph-mon[31916]: pgmap v1149: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-03-22T02:39:50.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:50 smithi120 ceph-mon[36803]: pgmap v1149: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-03-22T02:39:51.409 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-22T02:39:51.409 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 1424, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 1426, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 218160263, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 452842, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 8, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 4, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.216273257, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.031544653, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.140277620, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.035037596, 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:51.410 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 1424, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.076417708, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000053664 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.411 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 719, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 843, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 180544809, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 533021, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 7, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 6, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.160172687, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.017412203, 2024-03-22T02:39:51.412 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.119912083, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.017629741, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 841, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.043407864, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000051614 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-22T02:39:51.413 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 304640 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 845, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 839, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 168059077, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 314406, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 5, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 4, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.171037569, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.018517323, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.129238593, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.018574393, 2024-03-22T02:39:51.414 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 839, 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.034591507, 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000041229 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.415 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 218288, 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 102608 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 1, 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-MDSRank": { 2024-03-22T02:39:51.416 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 4415, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.744399695, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000168606 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "finisher-PurgeQueue": { 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "queue_len": 0, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "complete_latency": { 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.417 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "mds": { 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "request": 0, 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "reply": 0, 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "reply_latency": { 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.418 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_reply": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "forward": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_complete": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_fetch_keys": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_commit": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_split": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_merge": 0, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes": 55764, 2024-03-22T02:39:51.419 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_top": 37905, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_bottom": 17859, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pin_tail": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_pinned": 2286, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_expired": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "inodes_with_caps": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "caps": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "subtrees": 2, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_hit": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_forward": 0, 2024-03-22T02:39:51.420 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_discover": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_remote_ino": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "traverse_lock": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "load_cent": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "q": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "exported": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "exported_inodes": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "imported": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "imported_inodes": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_dir_fetch": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_backtrace_fetch": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "openino_peer_discover": 0, 2024-03-22T02:39:51.421 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rfiles": 197, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rbytes": 43131217, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "root_rsnaps": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_fetch": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_set_tag": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_inodes": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "scrub_file_inodes": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_inode_file_caps": 0, 2024-03-22T02:39:51.422 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_revoke": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_grant": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_caps_dirty": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_cap_release": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "process_request_cap_release": 0 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_cache": { 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_update_receipt": 0, 2024-03-22T02:39:51.423 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_try_discover": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_send_discover": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "dir_handle_discover": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_delayed": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_created": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_enqueued": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_reintegrated": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "strays_migrated": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_processing": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:39:51.424 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_started": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "recovery_completed": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_exportdir": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_flush": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragmentdir": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_fragstats": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "ireq_inodestats": 0 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_log": { 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "evlrg": 0, 2024-03-22T02:39:51.425 INFO:teuthology.orchestra.run.smithi083.stdout: "evadd": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "evex": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "evtrm": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "ev": 264729, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "evexg": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "evexd": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segadd": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segex": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segtrm": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "seg": 6, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segmjr": 41, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segexg": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "segexd": 0, 2024-03-22T02:39:51.426 INFO:teuthology.orchestra.run.smithi083.stdout: "expos": 1168517280, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "wrpos": 1178243472, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "rdpos": 1178241380, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "jlat": { 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "replayed": 264729 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_mem": { 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "ino": 289, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "ino+": 55506, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "ino-": 55217, 2024-03-22T02:39:51.427 INFO:teuthology.orchestra.run.smithi083.stdout: "dir": 82, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dir+": 327, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dir-": 245, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dn": 55764, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dn+": 56980, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dn-": 1216, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "cap": 0, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "cap+": 0, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "cap-": 0, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "rss": 101680, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "heap": 215392 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_server": { 2024-03-22T02:39:51.428 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_client_request": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "dispatch_server_request": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_request": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_client_session": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "handle_peer_request": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "req_create_latency": { 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.429 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getattr_latency": { 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getfilelock_latency": { 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "req_link_latency": { 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.430 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookup_latency": { 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookuphash_latency": { 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.431 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupino_latency": { 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupname_latency": { 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupparent_latency": { 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.432 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lookupsnap_latency": { 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "req_lssnap_latency": { 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.433 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.434 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mkdir_latency": { 2024-03-22T02:39:51.434 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.434 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.434 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.434 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mknod_latency": { 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "req_mksnap_latency": { 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "req_open_latency": { 2024-03-22T02:39:51.435 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "req_readdir_latency": { 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rename_latency": { 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.436 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "req_renamesnap_latency": { 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "req_snapdiff_latency": { 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmdir_latency": { 2024-03-22T02:39:51.437 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmsnap_latency": { 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "req_rmxattr_latency": { 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.438 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setattr_latency": { 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setdirlayout_latency": { 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.439 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setfilelock_latency": { 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setlayout_latency": { 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.440 INFO:teuthology.orchestra.run.smithi083.stdout: "req_setxattr_latency": { 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "req_symlink_latency": { 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "req_unlink_latency": { 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.441 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "req_getvxattr_latency": { 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_sessions": { 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "session_count": 1, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "session_add": 1, 2024-03-22T02:39:51.442 INFO:teuthology.orchestra.run.smithi083.stdout: "session_remove": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_open": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "sessions_stale": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "total_load": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "average_load": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "avg_session_uptime": 2909, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "mdthresh_evicted": 0 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:51.443 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:51.444 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:51.445 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:51.446 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 255889, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 542, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 9280, 2024-03-22T02:39:51.447 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 179, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 40548455, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 284780, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "objecter": { 2024-03-22T02:39:51.448 INFO:teuthology.orchestra.run.smithi083.stdout: "op_active": 0, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_laggy": 0, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send": 2808, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_send_bytes": 26, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_resend": 4, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_reply": 2808, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_latency": { 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2808, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 23.057446679, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.008211341 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "op_inflight": 0, 2024-03-22T02:39:51.449 INFO:teuthology.orchestra.run.smithi083.stdout: "oplen_avg": { 2024-03-22T02:39:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 2808, 2024-03-22T02:39:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 2809 2024-03-22T02:39:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: "op": 2808, 2024-03-22T02:39:51.450 INFO:teuthology.orchestra.run.smithi083.stdout: "op_r": 2808, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "op_w": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "op_rmw": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "op_pg": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_stat": 1580, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_create": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_read": 1225, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_write": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writefull": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_writesame": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_append": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_zero": 0, 2024-03-22T02:39:51.451 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_truncate": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_delete": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_mapext": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_sparse_read": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_clonerange": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_getxattr": 2, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_setxattr": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_rmxattr": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_call": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_watch": 0, 2024-03-22T02:39:51.452 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_notify": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "osdop_other": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_active": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_send": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_resend": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "linger_ping": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_active": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_send": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "poolop_resend": 0, 2024-03-22T02:39:51.453 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_active": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_send": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "poolstat_resend": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_active": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_send": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "statfs_resend": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "command_active": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "command_send": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "command_resend": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "map_epoch": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "map_full": 0, 2024-03-22T02:39:51.454 INFO:teuthology.orchestra.run.smithi083.stdout: "map_inc": 0, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_sessions": 11, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_open": 11, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_session_close": 0, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_laggy": 0, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_wr": 0, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_rd": 2, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_del": 0 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "oft": { 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_objs": 0, 2024-03-22T02:39:51.455 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_updates": 0, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "omap_total_removes": 0 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "osd-slow-ops": { 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "slow_ops_count": 0 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "purge_queue": { 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops": 0, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_ops_high_water": 0, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing": 0, 2024-03-22T02:39:51.456 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executing_high_water": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed_ops": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_executed": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "pq_item_in_journal": 0 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 2984, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 566483526, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 2984, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:51.457 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 2984, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 566483526, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_bytes": { 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:51.458 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 2808, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 565777357, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 2808, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 565777357, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.459 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-objecter_ops": { 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 1024, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 2808, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 2808, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 2808, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 2808, 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:51.460 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:51.461 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.462 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-write_buf_throttle-0x55d36091e6e0": { 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 3758096384, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 0, 2024-03-22T02:39:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 0, 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:51.464 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-22T02:39:52.629 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-22T02:39:52.629 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-22T02:39:52.630 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-22T02:39:52.630 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-22T02:39:52.630 DEBUG:tasks.check_counter:Getting stats from k 2024-03-22T02:39:52.630 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-22T02:39:52.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:52 smithi028 ceph-mon[31916]: pgmap v1150: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.2 MiB/s wr, 121 op/s 2024-03-22T02:39:52.918 INFO:teuthology.orchestra.run.smithi083.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.b/config 2024-03-22T02:39:52.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:52 smithi120 ceph-mon[36803]: pgmap v1150: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.2 MiB/s wr, 121 op/s 2024-03-22T02:39:53.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:52 smithi083 ceph-mon[36340]: pgmap v1150: 129 pgs: 129 active+clean; 110 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 2.2 MiB/s wr, 121 op/s 2024-03-22T02:39:54.652 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:54 smithi083 ceph-mon[36340]: pgmap v1151: 129 pgs: 129 active+clean; 107 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-03-22T02:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout:{ 2024-03-22T02:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 0, 2024-03-22T02:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 2, 2024-03-22T02:39:54.764 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 0, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 0, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 0, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.001124475, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.000312999, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:54.765 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 2, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 351, 2024-03-22T02:39:54.766 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 320, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 217776, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 4, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 2, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.013876923, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.010478724, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.001075051, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 349, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.015182558, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000043503 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:54.767 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 217776 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_messages": 468, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages": 461, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_bytes": 827088, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_bytes": 284880, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_created_connections": 2, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_active_connections": 1, 2024-03-22T02:39:54.768 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_total_time": 0.051895537, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_send_time": 0.018907044, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_recv_time": 0.024133371, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_running_fast_dispatch_time": 0.008978124, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 461, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.021597276, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000046848 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:54.769 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_recv_encrypted_bytes": 827088, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "msgr_send_encrypted_bytes": 284880 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "cct": { 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "total_workers": 0, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "unhealthy_workers": 0 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "mempool": { 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:54.770 INFO:teuthology.orchestra.run.smithi083.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:54.771 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:54.772 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:54.773 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_items": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_bytes": 148704, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_anon_items": 29, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "buffer_meta_items": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_bytes": 0, 2024-03-22T02:39:54.774 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_bytes": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "pgmap_items": 0, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_bytes": 928, 2024-03-22T02:39:54.775 INFO:teuthology.orchestra.run.smithi083.stdout: "mds_co_items": 2, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_1_items": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "unittest_2_items": 0 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: }, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "val": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "max": 104857600, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "get_started": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "get": 466, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "get_sum": 758400, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:54.776 INFO:teuthology.orchestra.run.smithi083.stdout: "get_or_fail_success": 466, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "take": 0, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "take_sum": 0, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "put": 466, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "put_sum": 758400, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "wait": { 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "avgcount": 0, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "sum": 0.000000000, 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: "avgtime": 0.000000000 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout: } 2024-03-22T02:39:54.777 INFO:teuthology.orchestra.run.smithi083.stdout:} 2024-03-22T02:39:54.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:54 smithi028 ceph-mon[31916]: pgmap v1151: 129 pgs: 129 active+clean; 107 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-03-22T02:39:54.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:54 smithi120 ceph-mon[36803]: pgmap v1151: 129 pgs: 129 active+clean; 107 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 2.5 MiB/s wr, 131 op/s 2024-03-22T02:39:55.451 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-22T02:39:55.451 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-22T02:39:55.452 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-22T02:39:55.452 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-22T02:39:55.452 DEBUG:tasks.check_counter:Getting stats from c 2024-03-22T02:39:55.452 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-22T02:39:55.820 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:39:56.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:56 smithi028 ceph-mon[31916]: pgmap v1152: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-22T02:39:56.989 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:56 smithi120 ceph-mon[36803]: pgmap v1152: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-22T02:39:57.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:56 smithi083 ceph-mon[36340]: pgmap v1152: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 2.4 MiB/s wr, 135 op/s 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 469189, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 268563, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 126896848, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 72433507, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 57, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 18.859275414, 2024-03-22T02:39:57.808 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 6.984071039, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 15.673969094, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.305178624, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 268562, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 7.268367902, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000027064 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 712470, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.091091108, 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000127 2024-03-22T02:39:57.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3072, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 448 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 11304, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 11454, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 3675240, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 21490815, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 10, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 6, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.954879078, 2024-03-22T02:39:57.810 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.377146306, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.271200017, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.275095421, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 11453, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.684127368, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000059733 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3623, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000671556, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000185 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.811 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 961680, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 321984 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 129549, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 156712, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 37109885, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 63214805, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 16, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 7.362406543, 2024-03-22T02:39:57.812 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 2.858062432, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 5.416001851, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.194896208, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 156710, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5.666381397, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000036158 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 262180, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.042621375, 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000162 2024-03-22T02:39:57.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 781600 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 22138, 2024-03-22T02:39:57.814 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 14.408266742, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000650838 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 12124, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.818140007, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000067481 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-22T02:39:57.815 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 103668, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 28267, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 28267, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 134.699289219, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004765248 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 75401, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 11, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-22T02:39:57.816 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 48, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 2, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 36, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 10, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 45, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 244, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 1, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 1, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 3, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 145200, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 29663, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 75400, 2024-03-22T02:39:57.817 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 121, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 937, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 7, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 7, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 7, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 33, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 112, 2024-03-22T02:39:57.818 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 112, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 112, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 4, 2024-03-22T02:39:57.819 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 1104, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 22379, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 13255, 2024-03-22T02:39:57.820 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 7532, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 2679, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 27558 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 737, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 720, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 414, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 639, 2024-03-22T02:39:57.821 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 741, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 5765, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 5765, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-22T02:39:57.822 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 10, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 37316, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-22T02:39:57.823 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 37316, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 47, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 47, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 5, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 51665601, 2024-03-22T02:39:57.824 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 10447, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 52.496776991, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.005025057 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 31, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 5912, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 5881, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 32, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 373, 2024-03-22T02:39:57.825 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 341, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 27, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 7434, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 7407, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 1, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 5831, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 5830, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 62640, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 223608 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 134289, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 129, 2024-03-22T02:39:57.826 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 103668, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 841, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 12868, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5747, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 67.644220424, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.011770353 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 472, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 3.060570277, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.006484259 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-22T02:39:57.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 7882, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 12.732960691, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001615447 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.828 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000188798, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000188798 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.830 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.831 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.831 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.832 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.833 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 7382, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 4.913746546, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000665638 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 940, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 17.122904514, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.018215855 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.834 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.835 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.836 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 78, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 1.199944445, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.015383903 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.837 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.838 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5765, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 28.024753524, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004861188 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:39:57.839 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 2, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 1, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-03-22T02:39:57.840 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 2025, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 2025, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 1566, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:39:57.841 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:39:57.842 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-22T02:39:57.843 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 601619, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 223, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 176, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 2, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:39:57.844 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9280, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 179, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 235560, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 6544, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.845 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 26158, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 48887005, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 33, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 26158, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 26158, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 231.713170320, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.008858214 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-22T02:39:57.846 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 26158, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 27988 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 26158, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 1172, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 24986, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 0, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 10, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 1060, 2024-03-22T02:39:57.847 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 11507, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 645, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 11553, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 112, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 20, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-22T02:39:57.848 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-22T02:39:57.849 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 98, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-22T02:39:57.850 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 8, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 12, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 12, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 2447, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 634 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-22T02:39:57.851 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 16, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 8, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 11530, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 5765, 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-22T02:39:57.852 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 610038, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 112746487, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 610038, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:39:57.853 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 610038, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 112746487, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:39:57.854 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 26158, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 49515390, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 26158, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 49515390, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.855 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 26158, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 26158, 2024-03-22T02:39:57.856 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 26158, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 26158, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 5765, 2024-03-22T02:39:57.857 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 628385, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 5765, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 1060, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 628385, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:39:57.858 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55fdc9c06500": { 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 6589, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 37316, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 47471297, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 37316, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 10447, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 47464708, 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:39:57.859 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:39:57.860 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:39:57.860 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:39:57.860 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.860 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:39:57.860 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-22T02:39:58.653 INFO:tasks.check_counter:Daemon mds.c mds.exported=7 2024-03-22T02:39:58.653 INFO:tasks.check_counter:Daemon mds.c mds.imported=7 2024-03-22T02:39:58.654 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=737 2024-03-22T02:39:58.654 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=720 2024-03-22T02:39:58.654 DEBUG:tasks.check_counter:Getting stats from f 2024-03-22T02:39:58.654 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-22T02:39:58.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:39:58 smithi028 ceph-mon[31916]: pgmap v1153: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-03-22T02:39:58.913 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:39:58.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:39:58 smithi120 ceph-mon[36803]: pgmap v1153: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-03-22T02:39:59.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:39:58 smithi083 ceph-mon[36340]: pgmap v1153: 129 pgs: 129 active+clean; 92 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-03-22T02:40:00.322 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:39:59.999+0000 7fa3d2c81700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:40:00.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:00 smithi120 ceph-mon[36803]: pgmap v1154: 129 pgs: 129 active+clean; 80 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-22T02:40:00.758 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:00 smithi120 ceph-mon[36803]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 4801, 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 4803, 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 13964958, 2024-03-22T02:40:00.783 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 1526725, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.306215166, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.086316697, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.082668047, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.117111206, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 4801, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.120516377, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000025102 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:00.784 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 463, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 809, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 4314190, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 926588, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-03-22T02:40:00.785 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.049360815, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.019180468, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.013498854, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.011944422, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 808, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.034816722, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000043090 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.786 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 781200 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 512, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 483, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 947304, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 289690, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 5, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.053626812, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.018833220, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.026967880, 2024-03-22T02:40:00.787 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.009037286, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 482, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.014602642, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000030295 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 944576, 2024-03-22T02:40:00.788 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 286192 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8887, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.186976881, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000021039 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.789 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.790 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 1679, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 1679, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-22T02:40:00.791 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 1679, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-03-22T02:40:00.792 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-22T02:40:00.793 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-03-22T02:40:00.794 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-03-22T02:40:00.795 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:40:00.796 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 12674, 2024-03-22T02:40:00.797 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 18, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 3, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 20997681, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 20995601, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.798 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 12674 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 16, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 1700, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 1684, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 11, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 31, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 20, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 1679, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 1756, 2024-03-22T02:40:00.799 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 77, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 54944, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215416 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-03-22T02:40:00.800 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.801 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.802 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.803 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-22T02:40:00.803 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.803 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.803 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.803 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.804 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-22T02:40:00.804 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.805 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.806 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.807 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-22T02:40:00.808 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.809 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.810 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.811 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.812 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.813 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-03-22T02:40:00.814 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 2, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 1, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 2923, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:40:00.815 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-22T02:40:00.816 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-22T02:40:00.817 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 199160, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 62, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-22T02:40:00.818 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9280, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 179, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 1232631, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 7232, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-22T02:40:00.819 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 5269, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 21, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 3, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 5269, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5269, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 21.671548769, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004113028 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-22T02:40:00.820 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5269, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5270 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 5269, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 5269, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 3244, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 2023, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-03-22T02:40:00.821 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 1, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:40:00.822 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-22T02:40:00.823 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 98, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 8, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 6, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 7, 2024-03-22T02:40:00.824 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 1, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-22T02:40:00.825 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:40:00.826 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 5772, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 18678246, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 5772, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 5772, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 18678246, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.827 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 5269, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 16804174, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 5269, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 16804174, 2024-03-22T02:40:00.828 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:00.829 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 5269, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 5269, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 5269, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 5269, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:40:00.830 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.831 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55b2f514a6e0": { 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:00.832 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:00.833 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:00.833 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:00.833 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.833 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:00.833 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-22T02:40:00.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:00 smithi028 ceph-mon[31916]: pgmap v1154: 129 pgs: 129 active+clean; 80 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-22T02:40:00.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:00 smithi028 ceph-mon[31916]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:40:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:00 smithi083 ceph-mon[36340]: pgmap v1154: 129 pgs: 129 active+clean; 80 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-03-22T02:40:01.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:00 smithi083 ceph-mon[36340]: overall HEALTH_ERR 1 MDSs report damaged metadata 2024-03-22T02:40:01.802 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-22T02:40:01.802 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-22T02:40:01.802 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-22T02:40:01.802 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-22T02:40:01.805 DEBUG:tasks.check_counter:Getting stats from i 2024-03-22T02:40:01.805 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-22T02:40:02.061 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:40:02.749 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:02 smithi120 ceph-mon[36803]: pgmap v1155: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1004 KiB/s wr, 63 op/s 2024-03-22T02:40:02.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:02 smithi028 ceph-mon[31916]: pgmap v1155: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1004 KiB/s wr, 63 op/s 2024-03-22T02:40:03.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:02 smithi083 ceph-mon[36340]: pgmap v1155: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 1004 KiB/s wr, 63 op/s 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 451230, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 235606, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 121796630, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 58626489, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 56, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-22T02:40:03.874 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 18.015181414, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 6.628231146, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 15.352200327, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.173683561, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 235604, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5.085822607, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000021586 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 671324, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.091678841, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000136 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.875 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 94189, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 102971, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 26615219, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 39400466, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 20, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 5.222727883, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 1.948092421, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 4.191282118, 2024-03-22T02:40:03.876 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.038276828, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 102970, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 3.527743485, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000034259 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 190055, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.027406968, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000144 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 960288, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 292960 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.877 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 3499, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 3670, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1030303, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 1735829, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 12, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 6, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.257726429, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.106097244, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.117530605, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.043320743, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3669, 2024-03-22T02:40:03.878 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.255763776, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000069709 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3612, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000598173, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000165 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3392, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 782672 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-22T02:40:03.879 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 7009, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 4.549906427, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000649152 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3765, 2024-03-22T02:40:03.880 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.236801061, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000062895 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 84232, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 9175, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 9175, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 56.937856082, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.006205760 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.881 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 75057, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 11, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 19, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 2, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 17, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 16, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 226, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-22T02:40:03.882 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 3, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 98684, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 9929, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 75055, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 107, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 290, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 3, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-22T02:40:03.883 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 3, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 34, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 38, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 38, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-22T02:40:03.884 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 38, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 733, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 9198, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 5386, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 1819, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 1080, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 8716 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.885 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 198, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 1782, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 942, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 1689, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 211, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 1669, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 1669, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-22T02:40:03.886 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_path": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_quiesce_inode": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 3, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-22T02:40:03.887 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 12678, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 12678, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 18, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 18, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 3, 2024-03-22T02:40:03.888 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 21000616, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3370, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 28.889671529, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.008572602 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.889 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 20, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 1802, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 1782, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 25, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 881, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 856, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 16, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 3246, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 3230, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 2378, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 2378, 2024-03-22T02:40:03.890 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 59696, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 223608 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 94015, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 10, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 84232, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 125, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 3678, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1635, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 29.579122677, 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.018091206 2024-03-22T02:40:03.891 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.009922820, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001984564 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.892 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2595, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 3.296208475, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001270215 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.893 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.894 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-22T02:40:03.895 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.896 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2977, 2024-03-22T02:40:03.897 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 3.084923159, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001036252 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 294, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 5.349817686, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.018196658 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.898 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-22T02:40:03.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.901 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1669, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 15.617861265, 2024-03-22T02:40:03.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.009357616 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 2, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 1, 2024-03-22T02:40:03.903 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 1096, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 1096, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 1482, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:40:03.904 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-22T02:40:03.905 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:40:03.906 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 329465, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 134, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 264, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 3, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:40:03.907 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9280, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 179, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 100180, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 852, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-22T02:40:03.908 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 8165, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 17293760, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 33, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 8165, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8165, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 103.924837204, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.012728087 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.909 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 8165, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 8768 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 8165, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 386, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 7779, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 0, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 10, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 348, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 3718, 2024-03-22T02:40:03.910 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 244, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 3354, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 38, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 20, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-22T02:40:03.911 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-22T02:40:03.912 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 98, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 8, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 12, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 17, 2024-03-22T02:40:03.913 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 5, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 798, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 238 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-22T02:40:03.914 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 16, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 8, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 3338, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 1669, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:40:03.915 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 548914, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 100009316, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 548914, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 548914, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 100009316, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.916 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 8165, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 17475681, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 8165, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 17475681, 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:03.917 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 8165, 2024-03-22T02:40:03.918 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 8165, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 8165, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 8165, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 1669, 2024-03-22T02:40:03.919 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 181921, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 1669, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 348, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 181921, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x556e7e28a500": { 2024-03-22T02:40:03.920 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 12678, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 16806312, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 12678, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 3370, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 16806312, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:03.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:03.922 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.922 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:03.922 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-22T02:40:04.660 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:04 smithi120 ceph-mon[36803]: pgmap v1156: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 946 KiB/s wr, 59 op/s 2024-03-22T02:40:04.687 INFO:tasks.check_counter:Daemon mds.i mds.exported=3 2024-03-22T02:40:04.687 INFO:tasks.check_counter:Daemon mds.i mds.imported=3 2024-03-22T02:40:04.687 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=198 2024-03-22T02:40:04.687 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1782 2024-03-22T02:40:04.687 DEBUG:tasks.check_counter:Getting stats from l 2024-03-22T02:40:04.687 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e shell --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-22T02:40:04.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:04 smithi028 ceph-mon[31916]: pgmap v1156: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 946 KiB/s wr, 59 op/s 2024-03-22T02:40:04.944 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/mon.c/config 2024-03-22T02:40:05.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:04 smithi083 ceph-mon[36340]: pgmap v1156: 129 pgs: 129 active+clean; 78 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 946 KiB/s wr, 59 op/s 2024-03-22T02:40:06.792 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:06 smithi120 ceph-mon[36803]: pgmap v1157: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 643 KiB/s wr, 47 op/s 2024-03-22T02:40:06.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:06 smithi028 ceph-mon[31916]: pgmap v1157: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 643 KiB/s wr, 47 op/s 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 0, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 2, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 0, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 0, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 0, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.000944048, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.000000000, 2024-03-22T02:40:06.823 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.000158653, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-22T02:40:06.824 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 2, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 354, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 320, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 218304, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.013819440, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.010493815, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.000964887, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:06.825 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 352, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.012722963, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000036144 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 218304 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-22T02:40:06.826 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 473, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 466, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 828512, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 287600, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 2, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 1, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.052585254, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.018943180, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.024795077, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.009181633, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 466, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.013485362, 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000028938 2024-03-22T02:40:06.827 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 828512, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 287600 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 0, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.828 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-22T02:40:06.829 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-22T02:40:06.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-22T02:40:06.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-22T02:40:06.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-22T02:40:06.830 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-22T02:40:06.831 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 148744, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 30, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-22T02:40:06.832 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 928, 2024-03-22T02:40:06.833 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 2, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 471, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 759114, 2024-03-22T02:40:06.834 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 471, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 471, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 759114, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-22T02:40:06.835 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-22T02:40:07.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:06 smithi083 ceph-mon[36340]: pgmap v1157: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 643 KiB/s wr, 47 op/s 2024-03-22T02:40:07.705 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-22T02:40:07.705 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-22T02:40:07.705 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-22T02:40:07.705 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-22T02:40:07.705 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-22T02:40:07.719 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-22T02:40:07.719 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:40:07.719 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:40:07.749 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:40:07.749 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:40:07.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:40:07.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 -- 172.21.15.28:0/967073275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc101ad0 msgr2=0x7faafc101f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 --2- 172.21.15.28:0/967073275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc101ad0 0x7faafc101f50 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7faaec009900 tx=0x7faaec02bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 -- 172.21.15.28:0/967073275 shutdown_connections 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 --2- 172.21.15.28:0/967073275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc102490 0x7faafc108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 --2- 172.21.15.28:0/967073275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc101ad0 0x7faafc101f50 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 --2- 172.21.15.28:0/967073275 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc1008e0 0x7faafc100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 -- 172.21.15.28:0/967073275 >> 172.21.15.28:0/967073275 conn(0x7faafc0fbe70 msgr2=0x7faafc0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 -- 172.21.15.28:0/967073275 shutdown_connections 2024-03-22T02:40:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.962+0000 7fab02da2640 1 -- 172.21.15.28:0/967073275 wait complete. 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 Processor -- start 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 -- start start 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc1008e0 0x7faafc197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 0x7faafc198200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faafc10c540 con 0x7faafc102490 2024-03-22T02:40:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7faafc10c3c0 con 0x7faafc101ad0 2024-03-22T02:40:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab02da2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7faafc10c6c0 con 0x7faafc1008e0 2024-03-22T02:40:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab025a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab025a1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56484/0 (socket says 172.21.15.28:56484) 2024-03-22T02:40:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab025a1640 1 -- 172.21.15.28:0/2214850914 learned_addr learned my addr 172.21.15.28:0/2214850914 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab0159f640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 0x7faafc198200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.963+0000 7fab01da0640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc1008e0 0x7faafc197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 -- 172.21.15.28:0/2214850914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc1008e0 msgr2=0x7faafc197cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc1008e0 0x7faafc197cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 -- 172.21.15.28:0/2214850914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 msgr2=0x7faafc198200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 0x7faafc198200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 -- 172.21.15.28:0/2214850914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faafc19f900 con 0x7faafc102490 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab0159f640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 0x7faafc198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab025a1640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7faaf400eb10 tx=0x7faaf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4004b20 con 0x7faafc102490 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf4004cc0 con 0x7faafc102490 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faafc19fbf0 con 0x7faafc102490 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4010760 con 0x7faafc102490 2024-03-22T02:40:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.964+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faafc1a0110 con 0x7faafc102490 2024-03-22T02:40:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.965+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faafc1a0340 con 0x7faafc102490 2024-03-22T02:40:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.966+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7faaf4005020 con 0x7faafc102490 2024-03-22T02:40:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.966+0000 7faaf2ffd640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 0x7faae80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.967+0000 7fab01da0640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 0x7faae80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.967+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7faaf40cfd50 con 0x7faafc102490 2024-03-22T02:40:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.967+0000 7fab01da0640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 0x7faae80a5460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faafc198b90 tx=0x7faaf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:07.971+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faaf409c710 con 0x7faafc102490 2024-03-22T02:40:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.154+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faafc1a05b0 con 0x7faafc102490 2024-03-22T02:40:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.156+0000 7faaf2ffd640 1 -- 172.21.15.28:0/2214850914 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24903 (secure 0 0 0) 0x7faaf409c010 con 0x7faafc102490 2024-03-22T02:40:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:40:08.158 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":98,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:29:42.392608+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 msgr2=0x7faae80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 0x7faae80a5460 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7faafc198b90 tx=0x7faaf8005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 msgr2=0x7faafc19f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7faaf400eb10 tx=0x7faaf400c2d0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 shutdown_connections 2024-03-22T02:40:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7faae80a2fa0 0x7faae80a5460 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faafc102490 0x7faafc19f260 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7faafc101ad0 0x7faafc198200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 --2- 172.21.15.28:0/2214850914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faafc1008e0 0x7faafc197cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 >> 172.21.15.28:0/2214850914 conn(0x7faafc0fbe70 msgr2=0x7faafc068cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 shutdown_connections 2024-03-22T02:40:08.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.158+0000 7fab02da2640 1 -- 172.21.15.28:0/2214850914 wait complete. 2024-03-22T02:40:08.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T02:06:54.565769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-22T02:11:31.490485+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-22T02:11:32.219129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-22T02:11:35.917141+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-22T02:40:08.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 -- 172.21.15.28:0/2267326310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0102640 msgr2=0x7f24b0108ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/2267326310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0102640 0x7f24b0108ed0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ca0 tx=0x7f24a002c0f0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 -- 172.21.15.28:0/2267326310 shutdown_connections 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/2267326310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0102640 0x7f24b0108ed0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/2267326310 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0101c80 0x7f24b0102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/2267326310 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 -- 172.21.15.28:0/2267326310 >> 172.21.15.28:0/2267326310 conn(0x7f24b00fc020 msgr2=0x7f24b00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 -- 172.21.15.28:0/2267326310 shutdown_connections 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.276+0000 7f24b5a0e640 1 -- 172.21.15.28:0/2267326310 wait complete. 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 Processor -- start 2024-03-22T02:40:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 -- start start 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0102640 0x7f24b019f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c6f0 con 0x7f24b0101c80 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c570 con 0x7f24b0100a90 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24b5a0e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f24b010c870 con 0x7f24b0102640 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24ae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24ae7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56502/0 (socket says 172.21.15.28:56502) 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24ae7fc640 1 -- 172.21.15.28:0/977521369 learned_addr learned my addr 172.21.15.28:0/977521369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.277+0000 7f24aeffd640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24af7fe640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0102640 0x7f24b019f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 -- 172.21.15.28:0/977521369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0102640 msgr2=0x7f24b019f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0102640 0x7f24b019f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 -- 172.21.15.28:0/977521369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 msgr2=0x7f24b0197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 -- 172.21.15.28:0/977521369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b019fad0 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24aeffd640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24ae7fc640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f24a400ea60 tx=0x7f24a400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a400cde0 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b019fdc0 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24a40035f0 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a4010670 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.278+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b01a0290 con 0x7f24b0101c80 2024-03-22T02:40:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.279+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b01a0550 con 0x7f24b0101c80 2024-03-22T02:40:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.280+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f24a40037a0 con 0x7f24b0101c80 2024-03-22T02:40:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.281+0000 7f24b4a0c640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 0x7f24980a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.281+0000 7f24aeffd640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 0x7f24980a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.281+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f24a40cfb10 con 0x7f24b0101c80 2024-03-22T02:40:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.281+0000 7f24aeffd640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 0x7f24980a53e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f24b0101ae0 tx=0x7f24a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.285+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24a409c4d0 con 0x7f24b0101c80 2024-03-22T02:40:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.468+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f24b01a07c0 con 0x7f24b0101c80 2024-03-22T02:40:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.468+0000 7f24b4a0c640 1 -- 172.21.15.28:0/977521369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7f24a409bdd0 con 0x7f24b0101c80 2024-03-22T02:40:08.470 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 msgr2=0x7f24980a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 0x7f24980a53e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f24b0101ae0 tx=0x7f24a8005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 msgr2=0x7f24b01983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f24a400ea60 tx=0x7f24a400ef20 comp rx=0 tx=0).stop 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 shutdown_connections 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f24980a2f20 0x7f24980a53e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f24b0102640 0x7f24b019f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24b0101c80 0x7f24b01983f0 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 --2- 172.21.15.28:0/977521369 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f24b0100a90 0x7f24b0197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 >> 172.21.15.28:0/977521369 conn(0x7f24b00fc020 msgr2=0x7f24b00fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 shutdown_connections 2024-03-22T02:40:08.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.470+0000 7f24b5a0e640 1 -- 172.21.15.28:0/977521369 wait complete. 2024-03-22T02:40:08.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- 172.21.15.28:0/1289793180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 msgr2=0x7f2abc100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 --2- 172.21.15.28:0/1289793180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc100eb0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f2aac009900 tx=0x7f2aac02bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- 172.21.15.28:0/1289793180 shutdown_connections 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 --2- 172.21.15.28:0/1289793180 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 0x7f2abc108ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 --2- 172.21.15.28:0/1289793180 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 0x7f2abc102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 --2- 172.21.15.28:0/1289793180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc100eb0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- 172.21.15.28:0/1289793180 >> 172.21.15.28:0/1289793180 conn(0x7f2abc0fc020 msgr2=0x7f2abc0fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- 172.21.15.28:0/1289793180 shutdown_connections 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- 172.21.15.28:0/1289793180 wait complete. 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 Processor -- start 2024-03-22T02:40:08.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.588+0000 7f2ac2993640 1 -- start start 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 0x7f2abc19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 0x7f2abc19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10c760 con 0x7f2abc100a90 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10c5e0 con 0x7f2abc101c80 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc10c8e0 con 0x7f2abc102640 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56510/0 (socket says 172.21.15.28:56510) 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 -- 172.21.15.28:0/106556496 learned_addr learned my addr 172.21.15.28:0/106556496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abb7fe640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 0x7f2abc19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac0f30640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 0x7f2abc19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 -- 172.21.15.28:0/106556496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 msgr2=0x7f2abc19f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 0x7f2abc19f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 -- 172.21.15.28:0/106556496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 msgr2=0x7f2abc19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 0x7f2abc19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 -- 172.21.15.28:0/106556496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2abc19fb40 con 0x7f2abc100a90 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2abbfff640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f2aac000c00 tx=0x7f2aac02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac043020 con 0x7f2abc100a90 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aac0054e0 con 0x7f2abc100a90 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2abc19fdd0 con 0x7f2abc100a90 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac003ce0 con 0x7f2abc100a90 2024-03-22T02:40:08.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.589+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2abc1a0290 con 0x7f2abc100a90 2024-03-22T02:40:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.590+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2abc107db0 con 0x7f2abc100a90 2024-03-22T02:40:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.592+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2aac046020 con 0x7f2abc100a90 2024-03-22T02:40:08.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.593+0000 7f2ab97fa640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 0x7f2aa40a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.593+0000 7f2abb7fe640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 0x7f2aa40a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.593+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f2aac0f1c70 con 0x7f2abc100a90 2024-03-22T02:40:08.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.593+0000 7f2abb7fe640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 0x7f2aa40a53e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2abc06ab00 tx=0x7f2ab000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.596+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2aac04d020 con 0x7f2abc100a90 2024-03-22T02:40:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.778+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2abc02d450 con 0x7f2abc100a90 2024-03-22T02:40:08.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.779+0000 7f2ab97fa640 1 -- 172.21.15.28:0/106556496 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7f2aac0be690 con 0x7f2abc100a90 2024-03-22T02:40:08.781 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 msgr2=0x7f2aa40a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 0x7f2aa40a53e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2abc06ab00 tx=0x7f2ab000b410 comp rx=0 tx=0).stop 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 msgr2=0x7f2abc06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f2aac000c00 tx=0x7f2aac02c6c0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 shutdown_connections 2024-03-22T02:40:08.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2aa40a2f20 0x7f2aa40a53e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2abc102640 0x7f2abc19f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2abc101c80 0x7f2abc19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 --2- 172.21.15.28:0/106556496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2abc100a90 0x7f2abc06ad90 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 >> 172.21.15.28:0/106556496 conn(0x7f2abc0fc020 msgr2=0x7f2abc104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 shutdown_connections 2024-03-22T02:40:08.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.781+0000 7f2ac2993640 1 -- 172.21.15.28:0/106556496 wait complete. 2024-03-22T02:40:08.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:08 smithi028 ceph-mon[31916]: pgmap v1158: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 361 KiB/s wr, 23 op/s 2024-03-22T02:40:08.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:08 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2214850914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:08.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:08 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/977521369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:40:08.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-22T02:40:08.817 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:08 smithi083 ceph-mon[36340]: pgmap v1158: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 361 KiB/s wr, 23 op/s 2024-03-22T02:40:08.817 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:08 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2214850914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:08.817 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:08 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/977521369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:40:08.899 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:08 smithi120 ceph-mon[36803]: pgmap v1158: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 361 KiB/s wr, 23 op/s 2024-03-22T02:40:08.899 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:08 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2214850914' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:08.899 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:08 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/977521369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T02:40:08.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 -- 172.21.15.28:0/2690834027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc104fb0 msgr2=0x7f2bdc1073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 --2- 172.21.15.28:0/2690834027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc1073a0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009900 tx=0x7f2bd002bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:08.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 -- 172.21.15.28:0/2690834027 shutdown_connections 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 --2- 172.21.15.28:0/2690834027 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 --2- 172.21.15.28:0/2690834027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc1073a0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 --2- 172.21.15.28:0/2690834027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 -- 172.21.15.28:0/2690834027 >> 172.21.15.28:0/2690834027 conn(0x7f2bdc0fc030 msgr2=0x7f2bdc0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 -- 172.21.15.28:0/2690834027 shutdown_connections 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 -- 172.21.15.28:0/2690834027 wait complete. 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.898+0000 7f2be21d7640 1 Processor -- start 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 -- start start 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc101ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10d860 con 0x7f2bdc1078e0 2024-03-22T02:40:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10d6e0 con 0x7f2bdc104fb0 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2be21d7640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10d9e0 con 0x7f2bdc102680 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56530/0 (socket says 172.21.15.28:56530) 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdb7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc101ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 -- 172.21.15.28:0/117062258 learned_addr learned my addr 172.21.15.28:0/117062258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdaffd640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 -- 172.21.15.28:0/117062258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 msgr2=0x7f2bdc101ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc101ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 -- 172.21.15.28:0/117062258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc104fb0 msgr2=0x7f2bdc100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 -- 172.21.15.28:0/117062258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bdc101400 con 0x7f2bdc1078e0 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdb7fe640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:40:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bdbfff640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009fd0 tx=0x7f2bd400b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.899+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd4017070 con 0x7f2bdc1078e0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.900+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bd4003e50 con 0x7f2bdc1078e0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.900+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bdc1a1fc0 con 0x7f2bdc1078e0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.900+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd40106c0 con 0x7f2bdc1078e0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.900+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bdc1a2480 con 0x7f2bdc1078e0 2024-03-22T02:40:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.900+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bdc0fdb20 con 0x7f2bdc1078e0 2024-03-22T02:40:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.902+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f2bd4003970 con 0x7f2bdc1078e0 2024-03-22T02:40:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.902+0000 7f2bd8ff9640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 0x7f2bc40a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.902+0000 7f2bdb7fe640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 0x7f2bc40a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.902+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f2bd40cf390 con 0x7f2bdc1078e0 2024-03-22T02:40:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.903+0000 7f2bdb7fe640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 0x7f2bc40a5460 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009fc0 tx=0x7f2bcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:08.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:08.906+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2bd409b7e0 con 0x7f2bdc1078e0 2024-03-22T02:40:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.091+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2bdc02d5b0 con 0x7f2bdc1078e0 2024-03-22T02:40:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.091+0000 7f2bd8ff9640 1 -- 172.21.15.28:0/117062258 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f2bd409f880 con 0x7f2bdc1078e0 2024-03-22T02:40:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-03-22T02:40:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 msgr2=0x7f2bc40a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 0x7f2bc40a5460 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009fc0 tx=0x7f2bcc005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 msgr2=0x7f2bdc100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4009fd0 tx=0x7f2bd400b260 comp rx=0 tx=0).stop 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 shutdown_connections 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f2bc40a2fa0 0x7f2bc40a5460 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc1078e0 0x7f2bdc100b70 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f2bdc104fb0 0x7f2bdc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 --2- 172.21.15.28:0/117062258 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2bdc102680 0x7f2bdc101ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 >> 172.21.15.28:0/117062258 conn(0x7f2bdc0fc030 msgr2=0x7f2bdc103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 shutdown_connections 2024-03-22T02:40:09.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.093+0000 7f2be21d7640 1 -- 172.21.15.28:0/117062258 wait complete. 2024-03-22T02:40:09.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- 172.21.15.28:0/1135643117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c1031f0 msgr2=0x7f990c103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- 172.21.15.28:0/1135643117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c1031f0 0x7f990c103610 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f9908009900 tx=0x7f990802bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- 172.21.15.28:0/1135643117 shutdown_connections 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- 172.21.15.28:0/1135643117 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c100620 0x7f990c10a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- 172.21.15.28:0/1135643117 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c0ffc60 0x7f990c1000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- 172.21.15.28:0/1135643117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c1031f0 0x7f990c103610 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- 172.21.15.28:0/1135643117 >> 172.21.15.28:0/1135643117 conn(0x7f990c0fb7c0 msgr2=0x7f990c0fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- 172.21.15.28:0/1135643117 shutdown_connections 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- 172.21.15.28:0/1135643117 wait complete. 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 Processor -- start 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- start start 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c0ffc60 0x7f990c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c1031f0 0x7f990c19f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f990c10d840 con 0x7f990c100620 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f990c10d6c0 con 0x7f990c1031f0 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.210+0000 7f9913ee1640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f990c10d9c0 con 0x7f990c0ffc60 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56538/0 (socket says 172.21.15.28:56538) 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9911c7d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c0ffc60 0x7f990c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 -- 172.21.15.28:0/3738516285 learned_addr learned my addr 172.21.15.28:0/3738516285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991247e640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c1031f0 0x7f990c19f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 -- 172.21.15.28:0/3738516285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c0ffc60 msgr2=0x7f990c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c0ffc60 0x7f990c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 -- 172.21.15.28:0/3738516285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c1031f0 msgr2=0x7f990c19f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c1031f0 0x7f990c19f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 -- 172.21.15.28:0/3738516285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c19fb50 con 0x7f990c100620 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f991147c640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9f0 tx=0x7f98fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc00cd80 con 0x7f990c100620 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc010910 con 0x7f990c100620 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c19fe40 con 0x7f990c100620 2024-03-22T02:40:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98fc018990 con 0x7f990c100620 2024-03-22T02:40:09.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.211+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c1a0300 con 0x7f990c100620 2024-03-22T02:40:09.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.212+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990c1091a0 con 0x7f990c100620 2024-03-22T02:40:09.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.213+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f98fc010ac0 con 0x7f990c100620 2024-03-22T02:40:09.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.214+0000 7f9902ffd640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 0x7f98f80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.214+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f98fc0cfef0 con 0x7f990c100620 2024-03-22T02:40:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.214+0000 7f9911c7d640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 0x7f98f80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.214+0000 7f9911c7d640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 0x7f98f80a53e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9908000c00 tx=0x7f9908005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.218+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98fc09c2c0 con 0x7f990c100620 2024-03-22T02:40:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.400+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f990c02d600 con 0x7f990c100620 2024-03-22T02:40:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.401+0000 7f9902ffd640 1 -- 172.21.15.28:0/3738516285 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v98) ==== 103+0+11 (secure 0 0 0) 0x7f98fc0a0360 con 0x7f990c100620 2024-03-22T02:40:09.403 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 msgr2=0x7f98f80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 0x7f98f80a53e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9908000c00 tx=0x7f9908005fd0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 msgr2=0x7f990c198470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f98fc00e9f0 tx=0x7f98fc00eeb0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 shutdown_connections 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f98f80a2f20 0x7f98f80a53e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f990c1031f0 0x7f990c19f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f990c100620 0x7f990c198470 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 --2- 172.21.15.28:0/3738516285 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f990c0ffc60 0x7f990c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 >> 172.21.15.28:0/3738516285 conn(0x7f990c0fb7c0 msgr2=0x7f990c106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 shutdown_connections 2024-03-22T02:40:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.403+0000 7f9913ee1640 1 -- 172.21.15.28:0/3738516285 wait complete. 2024-03-22T02:40:09.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-22T02:40:09.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 -- 172.21.15.28:0/3181150247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 msgr2=0x7f0318107350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 --2- 172.21.15.28:0/3181150247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318107350 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f0308009900 tx=0x7f030802bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 -- 172.21.15.28:0/3181150247 shutdown_connections 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 --2- 172.21.15.28:0/3181150247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318107890 0x7f0318109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 --2- 172.21.15.28:0/3181150247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318107350 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 --2- 172.21.15.28:0/3181150247 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318102630 0x7f0318104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 -- 172.21.15.28:0/3181150247 >> 172.21.15.28:0/3181150247 conn(0x7f03180fc040 msgr2=0x7f03180fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 -- 172.21.15.28:0/3181150247 shutdown_connections 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.518+0000 7f031f895640 1 -- 172.21.15.28:0/3181150247 wait complete. 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 Processor -- start 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 -- start start 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318102630 0x7f0318197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318107890 0x7f031819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f031810d830 con 0x7f0318104f60 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f031810d6b0 con 0x7f0318107890 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031f895640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f031810d9b0 con 0x7f0318102630 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56552/0 (socket says 172.21.15.28:56552) 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 -- 172.21.15.28:0/2246283538 learned_addr learned my addr 172.21.15.28:0/2246283538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031d631640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318102630 0x7f0318197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031de32640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318107890 0x7f031819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 -- 172.21.15.28:0/2246283538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318102630 msgr2=0x7f0318197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318102630 0x7f0318197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 -- 172.21.15.28:0/2246283538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318107890 msgr2=0x7f031819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318107890 0x7f031819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.519+0000 7f031ce30640 1 -- 172.21.15.28:0/2246283538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031819fb00 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f031ce30640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f030802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308041070 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0308004450 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031819fd90 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308034440 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.520+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03181a0220 con 0x7f0318104f60 2024-03-22T02:40:09.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.521+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03180fdb30 con 0x7f0318104f60 2024-03-22T02:40:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.522+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0308044070 con 0x7f0318104f60 2024-03-22T02:40:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.523+0000 7f030e7fc640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 0x7f03040a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.523+0000 7f031d631640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 0x7f03040a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.523+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f03080f1400 con 0x7f0318104f60 2024-03-22T02:40:09.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.523+0000 7f031d631640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 0x7f03040a53e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0314009bd0 tx=0x7f0314005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:09.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.526+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f03080bd860 con 0x7f0318104f60 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.725+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f031802d570 con 0x7f0318104f60 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.725+0000 7f030e7fc640 1 -- 172.21.15.28:0/2246283538 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v98) ==== 81+0+587 (secure 0 0 0) 0x7f03080c1900 con 0x7f0318104f60 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2329372296 2024-03-23T02:04:26.467767+0000 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3429985942 2024-03-23T02:04:26.467767+0000 2024-03-22T02:40:09.727 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1957817657 2024-03-23T02:04:26.467767+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/857886266 2024-03-23T02:03:59.710695+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/548967036 2024-03-23T02:03:59.710695+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/30661343 2024-03-23T02:03:59.710695+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2074246765 2024-03-23T02:04:26.467767+0000 2024-03-22T02:40:09.728 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4059471246 2024-03-23T02:03:59.710695+0000 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:listed 10 entries 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 msgr2=0x7f03040a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 0x7f03040a53e0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0314009bd0 tx=0x7f0314005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 msgr2=0x7f0318198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0308000c00 tx=0x7f030802c640 comp rx=0 tx=0).stop 2024-03-22T02:40:09.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 shutdown_connections 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f03040a2f20 0x7f03040a53e0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0318107890 0x7f031819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0318104f60 0x7f0318198420 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 --2- 172.21.15.28:0/2246283538 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0318102630 0x7f0318197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 >> 172.21.15.28:0/2246283538 conn(0x7f03180fc040 msgr2=0x7f03181030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 shutdown_connections 2024-03-22T02:40:09.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:09.727+0000 7f031f895640 1 -- 172.21.15.28:0/2246283538 wait complete. 2024-03-22T02:40:09.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:09 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/106556496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:09 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/117062258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:09 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3738516285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.740 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-22T02:40:09.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:09.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-22T02:40:09.822 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T02:40:09.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:09.822 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-22T02:40:09.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:09 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/106556496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:09 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/117062258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:09 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3738516285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:40:09.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:09.878 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-22T02:40:09.941 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-22T02:40:09.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:09.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-22T02:40:10.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:10.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-03-22T02:40:10.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:09 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/106556496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-22T02:40:10.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:09 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/117062258' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-22T02:40:10.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:09 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3738516285' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-22T02:40:10.091 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-22T02:40:10.092 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-03-22T02:40:10.092 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-22T02:40:10.130 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4057146,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17911 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/6f1caabab526e71b3fc428fbf2e847f95f3dde5730d40da67115880d895825dc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/6f1caabab526e71b3fc428fbf2e847f95f3dde5730d40da67115880d895825dc/diff,workdir=/var/lib/containers/storage/overlay/6f1caabab526e71b3fc428fbf2e847f95f3dde5730d40da67115880d895825dc/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1f601a3aed248d10090c89074447595338ece00a44501be8a770af69586f3022/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/1f601a3aed248d10090c89074447595338ece00a44501be8a770af69586f3022/diff,workdir=/var/lib/containers/storage/overlay/1f601a3aed248d10090c89074447595338ece00a44501be8a770af69586f3022/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/ff6409fabf3eeadb514e0809d8400c496eca09218d4d3b24e677d8f831018895/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/ff6409fabf3eeadb514e0809d8400c496eca09218d4d3b24e677d8f831018895/diff,workdir=/var/lib/containers/storage/overlay/ff6409fabf3eeadb514e0809d8400c496eca09218d4d3b24e677d8f831018895/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4ea0d29f2e044dec497e550077e3b231af55636157d5ced8515d1ca2ff367335/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/4ea0d29f2e044dec497e550077e3b231af55636157d5ced8515d1ca2ff367335/diff,workdir=/var/lib/containers/storage/overlay/4ea0d29f2e044dec497e550077e3b231af55636157d5ced8515d1ca2ff367335/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/1f2206dcdf6d05f030c4ff98ba6dba843111cbf44d62cdf36d9ce698f716a756/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/1f2206dcdf6d05f030c4ff98ba6dba843111cbf44d62cdf36d9ce698f716a756/diff,workdir=/var/lib/containers/storage/overlay/1f2206dcdf6d05f030c4ff98ba6dba843111cbf44d62cdf36d9ce698f716a756/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c9be67f14815f1f4be29ea762661ed804e97dd9a18c9863fd239c620806ec518/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/c9be67f14815f1f4be29ea762661ed804e97dd9a18c9863fd239c620806ec518/diff,workdir=/var/lib/containers/storage/overlay/c9be67f14815f1f4be29ea762661ed804e97dd9a18c9863fd239c620806ec518/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/604bf38168f6f834924c53f9d4dde71c47f0d02c49e7801d1361483c1acaeb5f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/604bf38168f6f834924c53f9d4dde71c47f0d02c49e7801d1361483c1acaeb5f/diff,workdir=/var/lib/containers/storage/overlay/604bf38168f6f834924c53f9d4dde71c47f0d02c49e7801d1361483c1acaeb5f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5d860371553f85407d63138275a687de7b40963672ce6a3f408ade69697d3f59/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/5d860371553f85407d63138275a687de7b40963672ce6a3f408ade69697d3f59/diff,workdir=/var/lib/containers/storage/overlay/5d860371553f85407d63138275a687de7b40963672ce6a3f408ade69697d3f59/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c24b8adf61dca058702e3e89092f493bc7f390836c9680ed40b4d0be41b4415c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/c24b8adf61dca058702e3e89092f493bc7f390836c9680ed40b4d0be41b4415c/diff,workdir=/var/lib/containers/storage/overlay/c24b8adf61dca058702e3e89092f493bc7f390836c9680ed40b4d0be41b4415c/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/0f299028b0672383609856c5a8350976bc903784a105eccfd538c7cf75d03dab/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/7PJ7A2WR4B2GBX5X6GYEZMLCI6:/var/lib/containers/storage/overlay/l/EOWD2X63F7JAATDNKSDBD7BPGE,upperdir=/var/lib/containers/storage/overlay/0f299028b0672383609856c5a8350976bc903784a105eccfd538c7cf75d03dab/diff,workdir=/var/lib/containers/storage/overlay/0f299028b0672383609856c5a8350976bc903784a105eccfd538c7cf75d03dab/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 2024-03-22T02:40:10.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:10.130 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-03-22T02:40:10.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-22T02:40:10.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-03-22T02:40:10.328 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-03-22T02:40:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T02:40:10.330 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-22T02:40:10.342 INFO:tasks.cephadm:Teardown begin 2024-03-22T02:40:10.342 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-22T02:40:10.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 -- 172.21.15.28:0/1875400827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c103760 msgr2=0x7f341c109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 --2- 172.21.15.28:0/1875400827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c103760 0x7f341c109ff0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f3414009ca0 tx=0x7f341402c0f0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 -- 172.21.15.28:0/1875400827 shutdown_connections 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 --2- 172.21.15.28:0/1875400827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c103760 0x7f341c109ff0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 --2- 172.21.15.28:0/1875400827 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c102da0 0x7f341c103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 --2- 172.21.15.28:0/1875400827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c101bb0 0x7f341c101fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 -- 172.21.15.28:0/1875400827 >> 172.21.15.28:0/1875400827 conn(0x7f341c0fd140 msgr2=0x7f341c0ff560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 -- 172.21.15.28:0/1875400827 shutdown_connections 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.448+0000 7f342333a640 1 -- 172.21.15.28:0/1875400827 wait complete. 2024-03-22T02:40:10.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 Processor -- start 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 -- start start 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c101bb0 0x7f341c197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c103760 0x7f341c19f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d830 con 0x7f341c102da0 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d6b0 con 0x7f341c101bb0 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f342333a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d9b0 con 0x7f341c103760 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56574/0 (socket says 172.21.15.28:56574) 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 -- 172.21.15.28:0/1236420204 learned_addr learned my addr 172.21.15.28:0/1236420204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34210d6640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c101bb0 0x7f341c197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34218d7640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c103760 0x7f341c19f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 -- 172.21.15.28:0/1236420204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c103760 msgr2=0x7f341c19f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c103760 0x7f341c19f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 -- 172.21.15.28:0/1236420204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c101bb0 msgr2=0x7f341c197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c101bb0 0x7f341c197e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 -- 172.21.15.28:0/1236420204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c19fa60 con 0x7f341c102da0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.449+0000 7f34208d5640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d950 tx=0x7f340c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.450+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c017030 con 0x7f341c102da0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.450+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f340c004020 con 0x7f341c102da0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.450+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c19fd50 con 0x7f341c102da0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.450+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c0047a0 con 0x7f341c102da0 2024-03-22T02:40:10.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.450+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c1a0240 con 0x7f341c102da0 2024-03-22T02:40:10.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.451+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c108ed0 con 0x7f341c102da0 2024-03-22T02:40:10.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.452+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f340c0041d0 con 0x7f341c102da0 2024-03-22T02:40:10.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.452+0000 7f34127fc640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 0x7f34080a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.453+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7f340c0cebe0 con 0x7f341c102da0 2024-03-22T02:40:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.453+0000 7f34210d6640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 0x7f34080a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.453+0000 7f34210d6640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 0x7f34080a53e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f341c102c00 tx=0x7f3418005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:10.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.457+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f340c09b5a0 con 0x7f341c102da0 2024-03-22T02:40:10.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:10 smithi028 ceph-mon[31916]: pgmap v1159: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 290 KiB/s wr, 20 op/s 2024-03-22T02:40:10.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:10 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/2246283538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:40:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.681+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f341c02d3b0 con 0x7f341c102da0 2024-03-22T02:40:10.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.681+0000 7f34127fc640 1 -- 172.21.15.28:0/1236420204 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11869 (secure 0 0 0) 0x7f340c09aea0 con 0x7f341c102da0 2024-03-22T02:40:10.682 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:40:10.683 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":48,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:39:26.237225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":246,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29624":{"gid":29624,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2257967634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2257967634},{"type":"v1","addr":"172.21.15.83:6837","nonce":2257967634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:40:10.683 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 48 2024-03-22T02:40:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 msgr2=0x7f34080a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 0x7f34080a53e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f341c102c00 tx=0x7f3418005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 msgr2=0x7f341c198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d950 tx=0x7f340c00de10 comp rx=0 tx=0).stop 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 shutdown_connections 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f34080a2f20 0x7f34080a53e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f341c103760 0x7f341c19f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f341c102da0 0x7f341c198380 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 --2- 172.21.15.28:0/1236420204 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f341c101bb0 0x7f341c197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 >> 172.21.15.28:0/1236420204 conn(0x7f341c0fd140 msgr2=0x7f341c1060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 shutdown_connections 2024-03-22T02:40:10.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.683+0000 7f342333a640 1 -- 172.21.15.28:0/1236420204 wait complete. 2024-03-22T02:40:10.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-22T02:40:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:10 smithi083 ceph-mon[36340]: pgmap v1159: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 290 KiB/s wr, 20 op/s 2024-03-22T02:40:10.765 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:10 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/2246283538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:40:10.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 -- 172.21.15.28:0/528408962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 msgr2=0x7fb298100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 --2- 172.21.15.28:0/528408962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb298100ef0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fb294009900 tx=0x7fb29402bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 -- 172.21.15.28:0/528408962 shutdown_connections 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 --2- 172.21.15.28:0/528408962 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 0x7fb298108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 --2- 172.21.15.28:0/528408962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 0x7fb298102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 --2- 172.21.15.28:0/528408962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb298100ef0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 -- 172.21.15.28:0/528408962 >> 172.21.15.28:0/528408962 conn(0x7fb2980fc040 msgr2=0x7fb2980fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 -- 172.21.15.28:0/528408962 shutdown_connections 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.797+0000 7fb2a079a640 1 -- 172.21.15.28:0/528408962 wait complete. 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 Processor -- start 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 -- start start 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 0x7fb29819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 0x7fb29819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c740 con 0x7fb298100ad0 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c5c0 con 0x7fb298102680 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb2a079a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c8c0 con 0x7fb298101cc0 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56596/0 (socket says 172.21.15.28:56596) 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 -- 172.21.15.28:0/3394955900 learned_addr learned my addr 172.21.15.28:0/3394955900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29dd35640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 0x7fb29819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29ed37640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 0x7fb29819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 -- 172.21.15.28:0/3394955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 msgr2=0x7fb29819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 0x7fb29819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 -- 172.21.15.28:0/3394955900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 msgr2=0x7fb29819f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 0x7fb29819f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 -- 172.21.15.28:0/3394955900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29819fb40 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.798+0000 7fb29e536640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb294000c00 tx=0x7fb29402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.799+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294041070 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.799+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb294004450 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.799+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29819fdd0 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.799+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb294034440 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.799+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2981a0280 con 0x7fb298100ad0 2024-03-22T02:40:10.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.800+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb298107e40 con 0x7fb298100ad0 2024-03-22T02:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.801+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb294044070 con 0x7fb298100ad0 2024-03-22T02:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.801+0000 7fb28f7fe640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 0x7fb2840a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.801+0000 7fb29dd35640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 0x7fb2840a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.801+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fb2940f1200 con 0x7fb298100ad0 2024-03-22T02:40:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.802+0000 7fb29dd35640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 0x7fb2840a5460 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb29806aac0 tx=0x7fb288005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:10.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:10.806+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb2940bd660 con 0x7fb298100ad0 2024-03-22T02:40:10.931 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:10 smithi120 ceph-mon[36803]: pgmap v1159: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 290 KiB/s wr, 20 op/s 2024-03-22T02:40:10.931 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:10 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/2246283538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T02:40:11.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.028+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2980628a0 con 0x7fb298100ad0 2024-03-22T02:40:11.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.028+0000 7fb28f7fe640 1 -- 172.21.15.28:0/3394955900 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11869 (secure 0 0 0) 0x7fb2940c1700 con 0x7fb298100ad0 2024-03-22T02:40:11.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:40:11.029 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/4228376531","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4228376531},{"type":"v1","addr":"172.21.15.28:6835","nonce":4228376531}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1410938784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1410938784},{"type":"v1","addr":"172.21.15.28:6837","nonce":1410938784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14637,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/678476485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":678476485},{"type":"v1","addr":"172.21.15.28:6839","nonce":678476485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/2956844816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":2956844816},{"type":"v1","addr":"172.21.15.28:6841","nonce":2956844816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.83:6833/675405105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6832","nonce":675405105},{"type":"v1","addr":"172.21.15.83:6833","nonce":675405105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24424,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/2414441153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":2414441153},{"type":"v1","addr":"172.21.15.120:6833","nonce":2414441153}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":48,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-22T02:11:33.259569+0000","modified":"2024-03-22T02:39:26.237225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24509,"mds_1":24508,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24443":{"gid":24443,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6835/3494340980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6834","nonce":3494340980},{"type":"v1","addr":"172.21.15.83:6835","nonce":3494340980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24466":{"gid":24466,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1917291995","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1917291995},{"type":"v1","addr":"172.21.15.120:6835","nonce":1917291995}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":246,"addr":"172.21.15.120:6837/2149554336","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2149554336},{"type":"v1","addr":"172.21.15.120:6837","nonce":2149554336}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.120:6839/990751976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":990751976},{"type":"v1","addr":"172.21.15.120:6839","nonce":990751976}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24509":{"gid":24509,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.83:6839/2818900206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6838","nonce":2818900206},{"type":"v1","addr":"172.21.15.83:6839","nonce":2818900206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_29624":{"gid":29624,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.83:6837/2257967634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6836","nonce":2257967634},{"type":"v1","addr":"172.21.15.83:6837","nonce":2257967634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24509,"qdb_cluster":[24478,24508,24509]},"id":1}]} 2024-03-22T02:40:11.030 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 48 2024-03-22T02:40:11.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 msgr2=0x7fb2840a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 0x7fb2840a5460 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb29806aac0 tx=0x7fb288005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 msgr2=0x7fb29806ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fb294000c00 tx=0x7fb29402c640 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 shutdown_connections 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb2840a2fa0 0x7fb2840a5460 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb298102680 0x7fb29819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb298101cc0 0x7fb29819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 --2- 172.21.15.28:0/3394955900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb298100ad0 0x7fb29806ad50 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 >> 172.21.15.28:0/3394955900 conn(0x7fb2980fc040 msgr2=0x7fb298105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 shutdown_connections 2024-03-22T02:40:11.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.030+0000 7fb2a079a640 1 -- 172.21.15.28:0/3394955900 wait complete. 2024-03-22T02:40:11.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 -- 172.21.15.28:0/1977035425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac102630 msgr2=0x7fecac104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 --2- 172.21.15.28:0/1977035425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac102630 0x7fecac104a20 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7feca4009900 tx=0x7feca402bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 -- 172.21.15.28:0/1977035425 shutdown_connections 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 --2- 172.21.15.28:0/1977035425 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac107890 0x7fecac109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 --2- 172.21.15.28:0/1977035425 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 0x7fecac107350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 --2- 172.21.15.28:0/1977035425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac102630 0x7fecac104a20 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 -- 172.21.15.28:0/1977035425 >> 172.21.15.28:0/1977035425 conn(0x7fecac0fc020 msgr2=0x7fecac0fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 -- 172.21.15.28:0/1977035425 shutdown_connections 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.144+0000 7fecb4779640 1 -- 172.21.15.28:0/1977035425 wait complete. 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 Processor -- start 2024-03-22T02:40:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 -- start start 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac102630 0x7fecac197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 0x7fecac198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecac10d850 con 0x7fecac107890 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fecac10d6d0 con 0x7fecac104f60 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb4779640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fecac10d9d0 con 0x7fecac102630 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb2d16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb2d16640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56622/0 (socket says 172.21.15.28:56622) 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb2d16640 1 -- 172.21.15.28:0/1641724449 learned_addr learned my addr 172.21.15.28:0/1641724449 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb2515640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac102630 0x7fecac197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.145+0000 7fecb1d14640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 0x7fecac198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 -- 172.21.15.28:0/1641724449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac102630 msgr2=0x7fecac197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac102630 0x7fecac197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 -- 172.21.15.28:0/1641724449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 msgr2=0x7fecac198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 0x7fecac198410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 -- 172.21.15.28:0/1641724449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecac19faf0 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb2d16640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7feca800eb10 tx=0x7feca800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8004b20 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feca8004cc0 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecac19fde0 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8010760 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.146+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecac1a02a0 con 0x7fecac107890 2024-03-22T02:40:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.147+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecac0fdb10 con 0x7fecac107890 2024-03-22T02:40:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.148+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7feca8005020 con 0x7fecac107890 2024-03-22T02:40:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.148+0000 7feca37fe640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 0x7fec980a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.149+0000 7fecb2515640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 0x7fec980a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.149+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7feca80cfef0 con 0x7fecac107890 2024-03-22T02:40:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.149+0000 7fecb2515640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 0x7fec980a53e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7feca402c7b0 tx=0x7feca4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.153+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feca809c8b0 con 0x7fecac107890 2024-03-22T02:40:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.337+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fecac062790 con 0x7fecac107890 2024-03-22T02:40:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.339+0000 7feca37fe640 1 -- 172.21.15.28:0/1641724449 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24903 (secure 0 0 0) 0x7feca809c1b0 con 0x7fecac107890 2024-03-22T02:40:11.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-03-22T02:40:11.340 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":98,"fsid":"426c3602-e7f0-11ee-95cd-87774f69a715","created":"2024-03-22T02:03:23.538926+0000","modified":"2024-03-22T02:29:42.392608+0000","last_up_change":"2024-03-22T02:10:42.153553+0000","last_in_change":"2024-03-22T02:10:21.778330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T02:06:54.565769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-22T02:11:31.490485+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-22T02:11:32.219129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-22T02:11:35.917141+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"edd58ef4-5561-4043-85e2-d205cb7c9396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6803","nonce":2828048761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6805","nonce":2828048761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6809","nonce":2828048761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":2828048761},{"type":"v1","addr":"172.21.15.28:6807","nonce":2828048761}]},"public_addr":"172.21.15.28:6803/2828048761","cluster_addr":"172.21.15.28:6805/2828048761","heartbeat_back_addr":"172.21.15.28:6809/2828048761","heartbeat_front_addr":"172.21.15.28:6807/2828048761","state":["exists","up"]},{"osd":1,"uuid":"dd32d180-5e18-4813-8e6d-74ed124ee68f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6801","nonce":1302915006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6803","nonce":1302915006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6806","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6807","nonce":1302915006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":1302915006},{"type":"v1","addr":"172.21.15.83:6805","nonce":1302915006}]},"public_addr":"172.21.15.83:6801/1302915006","cluster_addr":"172.21.15.83:6803/1302915006","heartbeat_back_addr":"172.21.15.83:6807/1302915006","heartbeat_front_addr":"172.21.15.83:6805/1302915006","state":["exists","up"]},{"osd":2,"uuid":"e9244c70-b1b3-42f4-a4e9-285e6933cc9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6801","nonce":2212833348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6803","nonce":2212833348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6807","nonce":2212833348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":2212833348},{"type":"v1","addr":"172.21.15.120:6805","nonce":2212833348}]},"public_addr":"172.21.15.120:6801/2212833348","cluster_addr":"172.21.15.120:6803/2212833348","heartbeat_back_addr":"172.21.15.120:6807/2212833348","heartbeat_front_addr":"172.21.15.120:6805/2212833348","state":["exists","up"]},{"osd":3,"uuid":"242938b2-bb3e-4a38-8c8e-7491101ee0bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6811","nonce":3156433849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6813","nonce":3156433849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6817","nonce":3156433849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3156433849},{"type":"v1","addr":"172.21.15.28:6815","nonce":3156433849}]},"public_addr":"172.21.15.28:6811/3156433849","cluster_addr":"172.21.15.28:6813/3156433849","heartbeat_back_addr":"172.21.15.28:6817/3156433849","heartbeat_front_addr":"172.21.15.28:6815/3156433849","state":["exists","up"]},{"osd":4,"uuid":"39ccc6c8-4269-4d9e-b30f-40311d5558b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6809","nonce":3880312837}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6811","nonce":3880312837}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6815","nonce":3880312837}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6812","nonce":3880312837},{"type":"v1","addr":"172.21.15.83:6813","nonce":3880312837}]},"public_addr":"172.21.15.83:6809/3880312837","cluster_addr":"172.21.15.83:6811/3880312837","heartbeat_back_addr":"172.21.15.83:6815/3880312837","heartbeat_front_addr":"172.21.15.83:6813/3880312837","state":["exists","up"]},{"osd":5,"uuid":"17fac3fc-aa99-48a3-8ce4-565522ae800d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6809","nonce":1653723326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6811","nonce":1653723326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6815","nonce":1653723326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":1653723326},{"type":"v1","addr":"172.21.15.120:6813","nonce":1653723326}]},"public_addr":"172.21.15.120:6809/1653723326","cluster_addr":"172.21.15.120:6811/1653723326","heartbeat_back_addr":"172.21.15.120:6815/1653723326","heartbeat_front_addr":"172.21.15.120:6813/1653723326","state":["exists","up"]},{"osd":6,"uuid":"1b4222ad-6096-499f-ba76-609404830d4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6819","nonce":1589283942}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6821","nonce":1589283942}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6825","nonce":1589283942}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1589283942},{"type":"v1","addr":"172.21.15.28:6823","nonce":1589283942}]},"public_addr":"172.21.15.28:6819/1589283942","cluster_addr":"172.21.15.28:6821/1589283942","heartbeat_back_addr":"172.21.15.28:6825/1589283942","heartbeat_front_addr":"172.21.15.28:6823/1589283942","state":["exists","up"]},{"osd":7,"uuid":"5d99f5cf-caaa-4c34-96da-37dd87bc1491","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6816","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6817","nonce":1165824815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6818","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6819","nonce":1165824815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6822","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6823","nonce":1165824815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6820","nonce":1165824815},{"type":"v1","addr":"172.21.15.83:6821","nonce":1165824815}]},"public_addr":"172.21.15.83:6817/1165824815","cluster_addr":"172.21.15.83:6819/1165824815","heartbeat_back_addr":"172.21.15.83:6823/1165824815","heartbeat_front_addr":"172.21.15.83:6821/1165824815","state":["exists","up"]},{"osd":8,"uuid":"574e783a-b2d6-439e-8bde-68e2d4b577b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6817","nonce":739561916}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6819","nonce":739561916}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6823","nonce":739561916}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":739561916},{"type":"v1","addr":"172.21.15.120:6821","nonce":739561916}]},"public_addr":"172.21.15.120:6817/739561916","cluster_addr":"172.21.15.120:6819/739561916","heartbeat_back_addr":"172.21.15.120:6823/739561916","heartbeat_front_addr":"172.21.15.120:6821/739561916","state":["exists","up"]},{"osd":9,"uuid":"f26642a2-c7eb-495a-979a-6002cf9f9438","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6827","nonce":855775168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6829","nonce":855775168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6833","nonce":855775168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":855775168},{"type":"v1","addr":"172.21.15.28:6831","nonce":855775168}]},"public_addr":"172.21.15.28:6827/855775168","cluster_addr":"172.21.15.28:6829/855775168","heartbeat_back_addr":"172.21.15.28:6833/855775168","heartbeat_front_addr":"172.21.15.28:6831/855775168","state":["exists","up"]},{"osd":10,"uuid":"9eafc799-33ff-4ef4-b4ec-235fcd7b9366","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6824","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6825","nonce":3932179263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6826","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6827","nonce":3932179263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6830","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6831","nonce":3932179263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6828","nonce":3932179263},{"type":"v1","addr":"172.21.15.83:6829","nonce":3932179263}]},"public_addr":"172.21.15.83:6825/3932179263","cluster_addr":"172.21.15.83:6827/3932179263","heartbeat_back_addr":"172.21.15.83:6831/3932179263","heartbeat_front_addr":"172.21.15.83:6829/3932179263","state":["exists","up"]},{"osd":11,"uuid":"ed2af09c-3a9c-4b54-a40b-ac1f2fbb6e7e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6825","nonce":511858328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6827","nonce":511858328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6831","nonce":511858328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":511858328},{"type":"v1","addr":"172.21.15.120:6829","nonce":511858328}]},"public_addr":"172.21.15.120:6825/511858328","cluster_addr":"172.21.15.120:6827/511858328","heartbeat_back_addr":"172.21.15.120:6831/511858328","heartbeat_front_addr":"172.21.15.120:6829/511858328","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:03.403285+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:27.297355+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:06:50.686939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:15.556887+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:07:40.526316+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:05.108086+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:30.895072+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:08:56.282055+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:21.696313+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:09:47.589476+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:13.794537+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T02:10:40.130612+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":3,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6},{"from":5,"to":7}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":5,"to":6}]},{"pgid":"3.2a","mappings":[{"from":1,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/2329372296":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/3429985942":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6801/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/1957817657":"2024-03-23T02:04:26.467767+0000","172.21.15.28:6800/857886266":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/548967036":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/30661343":"2024-03-23T02:03:59.710695+0000","172.21.15.28:0/2074246765":"2024-03-23T02:04:26.467767+0000","172.21.15.28:0/4059471246":"2024-03-23T02:03:59.710695+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T02:40:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.341+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 msgr2=0x7fec980a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.341+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 0x7fec980a53e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7feca402c7b0 tx=0x7feca4005fd0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.341+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 msgr2=0x7fecac19f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.341+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7feca800eb10 tx=0x7feca800c2d0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 shutdown_connections 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fec980a2f20 0x7fec980a53e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecac107890 0x7fecac19f480 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fecac104f60 0x7fecac198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 --2- 172.21.15.28:0/1641724449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fecac102630 0x7fecac197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 >> 172.21.15.28:0/1641724449 conn(0x7fecac0fc020 msgr2=0x7fecac103050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 shutdown_connections 2024-03-22T02:40:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.342+0000 7fecb4779640 1 -- 172.21.15.28:0/1641724449 wait complete. 2024-03-22T02:40:11.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 -- 172.21.15.28:0/2561018898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 msgr2=0x7fd1c0107330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 --2- 172.21.15.28:0/2561018898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c0107330 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009860 tx=0x7fd1b402bb70 comp rx=0 tx=0).stop 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 -- 172.21.15.28:0/2561018898 shutdown_connections 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 --2- 172.21.15.28:0/2561018898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0107870 0x7fd1c0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 --2- 172.21.15.28:0/2561018898 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c0107330 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 --2- 172.21.15.28:0/2561018898 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0102610 0x7fd1c0104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 -- 172.21.15.28:0/2561018898 >> 172.21.15.28:0/2561018898 conn(0x7fd1c00fc020 msgr2=0x7fd1c00fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 -- 172.21.15.28:0/2561018898 shutdown_connections 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.457+0000 7fd1c695e640 1 -- 172.21.15.28:0/2561018898 wait complete. 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 Processor -- start 2024-03-22T02:40:11.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 -- start start 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0107870 0x7fd1c01a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010d820 con 0x7fd1c0102610 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010d6a0 con 0x7fd1c0104f40 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c695e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c010d9a0 con 0x7fd1c0107870 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56644/0 (socket says 172.21.15.28:56644) 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1bffff640 1 -- 172.21.15.28:0/4278612989 learned_addr learned my addr 172.21.15.28:0/4278612989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1c4efb640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0107870 0x7fd1c01a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.458+0000 7fd1bf7fe640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 -- 172.21.15.28:0/4278612989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0107870 msgr2=0x7fd1c01a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0107870 0x7fd1c01a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 -- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 msgr2=0x7fd1c019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 -- 172.21.15.28:0/4278612989 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c01a1d90 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bffff640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bf7fe640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c019a6b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd1b402c0b0 tx=0x7fd1b4032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4041020 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1b402c9f0 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c01a2020 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4039040 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.459+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c01a2560 con 0x7fd1c0104f40 2024-03-22T02:40:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.460+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c01093c0 con 0x7fd1c0104f40 2024-03-22T02:40:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.461+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fd1b4044020 con 0x7fd1c0104f40 2024-03-22T02:40:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.462+0000 7fd1bd7fa640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 0x7fd1a80a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.462+0000 7fd1bffff640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 0x7fd1a80a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.462+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9274+0+0 (secure 0 0 0) 0x7fd1b40f13e0 con 0x7fd1c0104f40 2024-03-22T02:40:11.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.462+0000 7fd1bffff640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 0x7fd1a80a5460 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0011500 tx=0x7fd1b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.466+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd1b404b020 con 0x7fd1c0104f40 2024-03-22T02:40:11.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.688+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd1c002d600 con 0x7fd1c0104f40 2024-03-22T02:40:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1236420204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3394955900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:11 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1641724449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.699+0000 7fd1bd7fa640 1 -- 172.21.15.28:0/4278612989 <== mon.2 v2:172.21.15.83:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) ==== 161+0+0 (secure 0 0 0) 0x7fd1b40bde80 con 0x7fd1c0104f40 2024-03-22T02:40:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 msgr2=0x7fd1a80a5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 0x7fd1a80a5460 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd1b0011500 tx=0x7fd1b0005f90 comp rx=0 tx=0).stop 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 msgr2=0x7fd1c019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c019a6b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd1b402c0b0 tx=0x7fd1b4032430 comp rx=0 tx=0).stop 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 shutdown_connections 2024-03-22T02:40:11.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fd1a80a2fa0 0x7fd1a80a5460 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd1c0107870 0x7fd1c01a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fd1c0104f40 0x7fd1c019a6b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 --2- 172.21.15.28:0/4278612989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1c0102610 0x7fd1c019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 >> 172.21.15.28:0/4278612989 conn(0x7fd1c00fc020 msgr2=0x7fd1c0101d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 shutdown_connections 2024-03-22T02:40:11.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.701+0000 7fd1c695e640 1 -- 172.21.15.28:0/4278612989 wait complete. 2024-03-22T02:40:11.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-22T02:40:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 -- 172.21.15.28:0/109588781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 msgr2=0x7ff4d010a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/109588781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d010a190 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009ca0 tx=0x7ff4c802c0f0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 -- 172.21.15.28:0/109588781 shutdown_connections 2024-03-22T02:40:11.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/109588781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d010a190 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/109588781 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 0x7ff4d01000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/109588781 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d01035d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 -- 172.21.15.28:0/109588781 >> 172.21.15.28:0/109588781 conn(0x7ff4d00fb780 msgr2=0x7ff4d00fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 -- 172.21.15.28:0/109588781 shutdown_connections 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.818+0000 7ff4d62f4640 1 -- 172.21.15.28:0/109588781 wait complete. 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 Processor -- start 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 -- start start 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 0x7ff4d0197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d019f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d840 con 0x7ff4d01005e0 2024-03-22T02:40:11.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d6c0 con 0x7ff4d01031b0 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d62f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d9c0 con 0x7ff4d00ffc20 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 0x7ff4d0197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47816/0 (socket says 172.21.15.28:47816) 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 -- 172.21.15.28:0/1284176820 learned_addr learned my addr 172.21.15.28:0/1284176820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d4891640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d019f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 -- 172.21.15.28:0/1284176820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 msgr2=0x7ff4d0197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 0x7ff4d0197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 -- 172.21.15.28:0/1284176820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 msgr2=0x7ff4d019f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d019f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 -- 172.21.15.28:0/1284176820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d019fa70 con 0x7ff4d01005e0 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4d4891640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-22T02:40:11.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.819+0000 7ff4cf7fe640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400d770 tx=0x7ff4c400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.820+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4017070 con 0x7ff4d01005e0 2024-03-22T02:40:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.820+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d019fd60 con 0x7ff4d01005e0 2024-03-22T02:40:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.820+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c4005500 con 0x7ff4d01005e0 2024-03-22T02:40:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.820+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c4004e00 con 0x7ff4d01005e0 2024-03-22T02:40:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.820+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d01a0250 con 0x7ff4d01005e0 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.821+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d0109010 con 0x7ff4d01005e0 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.822+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7ff4c4004fa0 con 0x7ff4d01005e0 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.822+0000 7ff4cd7fa640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 0x7ff4b80a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.823+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 0x7ff4b80a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.823+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9532+0+0 (secure 0 0 0) 0x7ff4c40cf7b0 con 0x7ff4d01005e0 2024-03-22T02:40:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.823+0000 7ff4cffff640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 0x7ff4b80a53e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0069e60 tx=0x7ff4c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:11.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:11.826+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff4c409c070 con 0x7ff4d01005e0 2024-03-22T02:40:11.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1236420204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3394955900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1641724449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mds[67822]: -- 172.21.15.120:0/2742641558 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563dda189a40 con 0x563dda15d000 2024-03-22T02:40:11.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: starting mds.i at 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mds[67515]: -- 172.21.15.120:0/2480204154 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561dae173a40 con 0x561dae146800 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: starting mds.c at 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-mds[67211]: -- 172.21.15.120:0/1053302803 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a09d451a40 con 0x55a09d424800 2024-03-22T02:40:12.000 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:40:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: starting mds.f at 2024-03-22T02:40:12.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1236420204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:12.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3394955900' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-22T02:40:12.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1641724449' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mds[66500]: -- 172.21.15.83:0/667065271 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f7e1955a40 con 0x55f7e1928800 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: starting mds.h at 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:12.016 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mds[66191]: -- 172.21.15.83:0/837066959 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ae72cdfa40 con 0x55ae72cb2c00 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: starting mds.e at 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: ignoring --setuser ceph since I am not root 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: ignoring --setgroup ceph since I am not root 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-mds[66809]: -- 172.21.15.83:0/3682290301 <== mon.2 v2:172.21.15.83:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5583f252ba40 con 0x5583f24fe800 2024-03-22T02:40:12.017 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:40:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: starting mds.b at 2024-03-22T02:40:12.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.054+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff4d002d450 con 0x7ff4d01005e0 2024-03-22T02:40:12.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:40:11.689+0000 7fa3d047c700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:40:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.692+0000 7ff4cd7fa640 1 -- 172.21.15.28:0/1284176820 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) ==== 104+0+0 (secure 0 0 0) 0x7ff4c409b970 con 0x7ff4d01005e0 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 msgr2=0x7ff4b80a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 0x7ff4b80a53e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0069e60 tx=0x7ff4c000b390 comp rx=0 tx=0).stop 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 msgr2=0x7ff4d0198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7ff4c400d770 tx=0x7ff4c400dc30 comp rx=0 tx=0).stop 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 shutdown_connections 2024-03-22T02:40:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7ff4b80a2f20 0x7ff4b80a53e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7ff4d01031b0 0x7ff4d019f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d01005e0 0x7ff4d0198390 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 --2- 172.21.15.28:0/1284176820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff4d00ffc20 0x7ff4d0197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 >> 172.21.15.28:0/1284176820 conn(0x7ff4d00fb780 msgr2=0x7ff4d01061e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 shutdown_connections 2024-03-22T02:40:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.694+0000 7ff4d62f4640 1 -- 172.21.15.28:0/1284176820 wait complete. 2024-03-22T02:40:12.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.819+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/748131963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8101bc0 msgr2=0x7fb7f8101fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.819+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/748131963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8101fe0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009900 tx=0x7fb7e802bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.819+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/748131963 shutdown_connections 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/748131963 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8103770 0x7fb7f810a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/748131963 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 0x7fb7f8103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/748131963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8101fe0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/748131963 >> 172.21.15.28:0/748131963 conn(0x7fb7f80fd130 msgr2=0x7fb7f80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:12.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/748131963 shutdown_connections 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/748131963 wait complete. 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 Processor -- start 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- start start 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 0x7fb7f81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f810d840 con 0x7fb7f8103770 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f810d6c0 con 0x7fb7f8102db0 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.820+0000 7fb7fe8e0640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f810d9c0 con 0x7fb7f8101bc0 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47834/0 (socket says 172.21.15.28:47834) 2024-03-22T02:40:12.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 -- 172.21.15.28:0/935888491 learned_addr learned my addr 172.21.15.28:0/935888491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f7fff640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f77fe640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 0x7fb7f81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 -- 172.21.15.28:0/935888491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 msgr2=0x7fb7f8197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 -- 172.21.15.28:0/935888491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 msgr2=0x7fb7f81983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 0x7fb7f81983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 -- 172.21.15.28:0/935888491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f819faa0 con 0x7fb7f8103770 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f7fff640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fce7d640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000eaa0 tx=0x7fb7f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f000cdb0 con 0x7fb7f8103770 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7f0005020 con 0x7fb7f8103770 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f819fd90 con 0x7fb7f8103770 2024-03-22T02:40:12.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00116b0 con 0x7fb7f8103770 2024-03-22T02:40:12.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.821+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f81a0280 con 0x7fb7f8103770 2024-03-22T02:40:12.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.822+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f8108ee0 con 0x7fb7f8103770 2024-03-22T02:40:12.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.823+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7fb7f0005560 con 0x7fb7f8103770 2024-03-22T02:40:12.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.824+0000 7fb7f57fa640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 0x7fb7e00a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:12.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.824+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9532+0+0 (secure 0 0 0) 0x7fb7f00cff80 con 0x7fb7f8103770 2024-03-22T02:40:12.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.824+0000 7fb7f7fff640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 0x7fb7e00a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:12.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.824+0000 7fb7f7fff640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 0x7fb7e00a53e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7e802c7b0 tx=0x7fb7e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:12.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:12.828+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7f0016030 con 0x7fb7f8103770 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: pgmap v1160: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/4278612989' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-22T02:40:12.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:12 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:40:13.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.011+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb7f802d3b0 con 0x7fb7f8103770 2024-03-22T02:40:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: pgmap v1160: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-03-22T02:40:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/4278612989' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:13.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-22T02:40:13.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:12 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: pgmap v1160: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/4278612989' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-22T02:40:13.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: osdmap e99: 12 total, 12 up, 12 in 2024-03-22T02:40:13.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-22T02:40:13.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:12 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T02:40:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.699+0000 7fb7f57fa640 1 -- 172.21.15.28:0/935888491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) ==== 181+0+0 (secure 0 0 0) 0x7fb7f009c8a0 con 0x7fb7f8103770 2024-03-22T02:40:13.700 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-03-22T02:40:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 msgr2=0x7fb7e00a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 0x7fb7e00a53e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7e802c7b0 tx=0x7fb7e8005fd0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 msgr2=0x7fb7f819f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000eaa0 tx=0x7fb7f000ef60 comp rx=0 tx=0).stop 2024-03-22T02:40:13.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 shutdown_connections 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7fb7e00a2f20 0x7fb7e00a53e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7f8103770 0x7fb7f819f430 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7fb7f8102db0 0x7fb7f81983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 --2- 172.21.15.28:0/935888491 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7f8101bc0 0x7fb7f8197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.703+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 >> 172.21.15.28:0/935888491 conn(0x7fb7f80fd130 msgr2=0x7fb7f81060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:13.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.704+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 shutdown_connections 2024-03-22T02:40:13.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.704+0000 7fb7fe8e0640 1 -- 172.21.15.28:0/935888491 wait complete. 2024-03-22T02:40:13.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-22T02:40:13.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 -- 172.21.15.28:0/2713132695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c100ad0 msgr2=0x7f0a9c100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 --2- 172.21.15.28:0/2713132695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c100ad0 0x7f0a9c100ef0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009900 tx=0x7f0a8c02bc60 comp rx=0 tx=0).stop 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 -- 172.21.15.28:0/2713132695 shutdown_connections 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 --2- 172.21.15.28:0/2713132695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 0x7f0a9c108f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 --2- 172.21.15.28:0/2713132695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 --2- 172.21.15.28:0/2713132695 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c100ad0 0x7f0a9c100ef0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c009900 tx=0x7f0a8c02bc60 comp rx=0 tx=0).stop 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 -- 172.21.15.28:0/2713132695 >> 172.21.15.28:0/2713132695 conn(0x7f0a9c0fc040 msgr2=0x7f0a9c0fe480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 -- 172.21.15.28:0/2713132695 shutdown_connections 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.868+0000 7f0aa173e640 1 -- 172.21.15.28:0/2713132695 wait complete. 2024-03-22T02:40:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 Processor -- start 2024-03-22T02:40:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 -- start start 2024-03-22T02:40:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 0x7f0a9c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 0x7f0a9c1a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c10a050 con 0x7f0a9c101cc0 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c109ed0 con 0x7f0a9c19f990 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.869+0000 7f0aa173e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c10a1d0 con 0x7f0a9c102680 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47862/0 (socket says 172.21.15.28:47862) 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 -- 172.21.15.28:0/1304168107 learned_addr learned my addr 172.21.15.28:0/1304168107 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9b7fe640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 0x7f0a9c1a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9a7fc640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 0x7f0a9c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 -- 172.21.15.28:0/1304168107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 msgr2=0x7f0a9c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 0x7f0a9c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 -- 172.21.15.28:0/1304168107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 msgr2=0x7f0a9c1a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 0x7f0a9c1a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 -- 172.21.15.28:0/1304168107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c1a23f0 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9b7fe640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 0x7f0a9c1a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.870+0000 7f0a9affd640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c02c1f0 tx=0x7f0a8c02ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.871+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c041020 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.871+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c1a2680 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.871+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c1a2b90 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.871+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a8c02ee20 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.871+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c02c600 con 0x7f0a9c101cc0 2024-03-22T02:40:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.872+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a880052c0 con 0x7f0a9c101cc0 2024-03-22T02:40:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.876+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f0a8c044020 con 0x7f0a9c101cc0 2024-03-22T02:40:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.876+0000 7f0a83fff640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 0x7f0a840a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.877+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8978+0+0 (secure 0 0 0) 0x7f0a8c0f1380 con 0x7f0a9c101cc0 2024-03-22T02:40:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.877+0000 7f0a9a7fc640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 0x7f0a840a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.878+0000 7f0a9a7fc640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 0x7f0a840a5530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c19b900 tx=0x7f0a90009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:13.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:13.880+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a8c04b020 con 0x7f0a9c101cc0 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: Cluster is now healthy 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6836/14769166,v1:172.21.15.120:6837/14769166] up:boot 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6834/674643369,v1:172.21.15.120:6835/674643369] up:boot 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.120:6838/3884455737,v1:172.21.15.120:6839/3884455737] up:boot 2024-03-22T02:40:13.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6836/3901321083,v1:172.21.15.83:6839/3901321083] up:boot 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6835/2665125026,v1:172.21.15.83:6838/2665125026] up:boot 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: mds.? [v2:172.21.15.83:6834/2194484552,v1:172.21.15.83:6837/2194484552] up:boot 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: fsmap 12 up:standby 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:40:14.000 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:13 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:40:14.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: Cluster is now healthy 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6836/14769166,v1:172.21.15.120:6837/14769166] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6834/674643369,v1:172.21.15.120:6835/674643369] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.120:6838/3884455737,v1:172.21.15.120:6839/3884455737] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6836/3901321083,v1:172.21.15.83:6839/3901321083] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6835/2665125026,v1:172.21.15.83:6838/2665125026] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: mds.? [v2:172.21.15.83:6834/2194484552,v1:172.21.15.83:6837/2194484552] up:boot 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: fsmap 12 up:standby 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:40:14.016 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:40:14.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:40:14.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:40:14.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:40:14.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:40:14.017 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:13 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-22T02:40:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-22T02:40:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: Cluster is now healthy 2024-03-22T02:40:14.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1284176820' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6836/14769166,v1:172.21.15.120:6837/14769166] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6834/674643369,v1:172.21.15.120:6835/674643369] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.120:6838/3884455737,v1:172.21.15.120:6839/3884455737] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6836/3901321083,v1:172.21.15.83:6839/3901321083] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6835/2665125026,v1:172.21.15.83:6838/2665125026] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: mds.? [v2:172.21.15.83:6834/2194484552,v1:172.21.15.83:6837/2194484552] up:boot 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: fsmap 12 up:standby 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-22T02:40:14.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-22T02:40:14.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='mgr.14152 172.21.15.28:0/1877203768' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-22T02:40:14.074 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:13 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:14.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.078+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f0a880055b0 con 0x7f0a9c101cc0 2024-03-22T02:40:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.700+0000 7f0a83fff640 1 -- 172.21.15.28:0/1304168107 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) ==== 169+0+0 (secure 0 0 0) 0x7f0a8c0bde60 con 0x7f0a9c101cc0 2024-03-22T02:40:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 msgr2=0x7f0a840a5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 0x7f0a840a5530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c19b900 tx=0x7f0a90009110 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 msgr2=0x7f0a9c19a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c02c1f0 tx=0x7f0a8c02ea20 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 shutdown_connections 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f0a840a3070 0x7f0a840a5530 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f0a9c19f990 0x7f0a9c1a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102680 0x7f0a9c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.705+0000 7f0aa173e640 1 --2- 172.21.15.28:0/1304168107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a9c101cc0 0x7f0a9c19a3f0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.706+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 >> 172.21.15.28:0/1304168107 conn(0x7f0a9c0fc040 msgr2=0x7f0a9c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:14.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.706+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 shutdown_connections 2024-03-22T02:40:14.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.706+0000 7f0aa173e640 1 -- 172.21.15.28:0/1304168107 wait complete. 2024-03-22T02:40:14.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-22T02:40:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- 172.21.15.28:0/1749500306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c103770 msgr2=0x7f5c3c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:14.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- 172.21.15.28:0/1749500306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c103770 0x7f5c3c10a000 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009900 tx=0x7f5c2c02bc30 comp rx=0 tx=0).stop 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- 172.21.15.28:0/1749500306 shutdown_connections 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- 172.21.15.28:0/1749500306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c103770 0x7f5c3c10a000 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- 172.21.15.28:0/1749500306 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c102db0 0x7f5c3c103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- 172.21.15.28:0/1749500306 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c101fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- 172.21.15.28:0/1749500306 >> 172.21.15.28:0/1749500306 conn(0x7f5c3c0fd130 msgr2=0x7f5c3c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- 172.21.15.28:0/1749500306 shutdown_connections 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- 172.21.15.28:0/1749500306 wait complete. 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 Processor -- start 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- start start 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:14.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 0x7f5c3c1983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 0x7f5c3c19f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d840 con 0x7f5c3c101bc0 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- --> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d6c0 con 0x7f5c3c102db0 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.861+0000 7f5c4392b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d9c0 con 0x7f5c3c103770 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47880/0 (socket says 172.21.15.28:47880) 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 -- 172.21.15.28:0/3998200723 learned_addr learned my addr 172.21.15.28:0/3998200723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c40ec6640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 0x7f5c3c1983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c41ec8640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 0x7f5c3c19f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 -- 172.21.15.28:0/3998200723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 msgr2=0x7f5c3c19f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 0x7f5c3c19f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 -- 172.21.15.28:0/3998200723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 msgr2=0x7f5c3c1983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 0x7f5c3c1983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 -- 172.21.15.28:0/3998200723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c19fab0 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c41ec8640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 0x7f5c3c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c40ec6640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 0x7f5c3c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c416c7640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400ebc0 tx=0x7f5c3400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34003b70 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c34003d10 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c19fda0 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34019980 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.862+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c3c1a0210 con 0x7f5c3c101bc0 2024-03-22T02:40:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.864+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3c108e80 con 0x7f5c3c101bc0 2024-03-22T02:40:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.864+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 20) ==== 137854+0+0 (secure 0 0 0) 0x7f5c34019b20 con 0x7f5c3c101bc0 2024-03-22T02:40:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.865+0000 7f5c3a7fc640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 0x7f5c240a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-22T02:40:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.865+0000 7f5c40ec6640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 0x7f5c240a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-22T02:40:14.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.865+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8324+0+0 (secure 0 0 0) 0x7f5c340cfdc0 con 0x7f5c3c101bc0 2024-03-22T02:40:14.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.866+0000 7f5c40ec6640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 0x7f5c240a53e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f5c300058c0 tx=0x7f5c30005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-22T02:40:14.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:14.870+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c3409cb30 con 0x7f5c3c101bc0 2024-03-22T02:40:14.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:14 smithi120 ceph-mon[36803]: pgmap v1162: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-03-22T02:40:14.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:14 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:14.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:14 smithi120 ceph-mon[36803]: osdmap e100: 12 total, 12 up, 12 in 2024-03-22T02:40:14.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:14 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:14 smithi083 ceph-mon[36340]: pgmap v1162: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-03-22T02:40:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:14 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:14 smithi083 ceph-mon[36340]: osdmap e100: 12 total, 12 up, 12 in 2024-03-22T02:40:15.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:14 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.057+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5c3c02d3b0 con 0x7f5c3c101bc0 2024-03-22T02:40:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:14 smithi028 ceph-mon[31916]: pgmap v1162: 129 pgs: 129 active+clean; 79 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-03-22T02:40:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:14 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/935888491' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:14 smithi028 ceph-mon[31916]: osdmap e100: 12 total, 12 up, 12 in 2024-03-22T02:40:15.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:14 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:15.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.713+0000 7f5c3a7fc640 1 -- 172.21.15.28:0/3998200723 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v102) ==== 178+0+0 (secure 0 0 0) 0x7f5c3c02d3b0 con 0x7f5c3c101bc0 2024-03-22T02:40:15.714 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-03-22T02:40:15.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.716+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 msgr2=0x7f5c240a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:15.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.716+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 0x7f5c240a53e0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f5c300058c0 tx=0x7f5c30005850 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.716+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 msgr2=0x7f5c3c197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.716+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400ebc0 tx=0x7f5c3400c6a0 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 shutdown_connections 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:6800/4205492780,v1:172.21.15.28:6801/4205492780] conn(0x7f5c240a2f20 0x7f5c240a53e0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5c3c103770 0x7f5c3c19f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] conn(0x7f5c3c102db0 0x7f5c3c1983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 --2- 172.21.15.28:0/3998200723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3c101bc0 0x7f5c3c197e90 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 >> 172.21.15.28:0/3998200723 conn(0x7f5c3c0fd130 msgr2=0x7f5c3c106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-22T02:40:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 shutdown_connections 2024-03-22T02:40:15.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-03-22T02:40:15.717+0000 7f5c4392b640 1 -- 172.21.15.28:0/3998200723 wait complete. 2024-03-22T02:40:15.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:40:15.776 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:40:15.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:40:15.847 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-22T02:40:15.847 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T02:40:15.871 DEBUG:teuthology.orchestra.run.smithi083:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T02:40:15.885 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T02:40:15.902 INFO:tasks.cephadm:Stopping all daemons... 2024-03-22T02:40:15.903 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-22T02:40:15.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a 2024-03-22T02:40:15.998 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:15.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:15 smithi120 ceph-mon[36803]: osdmap e101: 12 total, 12 up, 12 in 2024-03-22T02:40:15.999 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:15 smithi120 ceph-mon[36803]: from='client.? 172.21.15.28:0/3998200723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:16.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:16.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:15 smithi083 ceph-mon[36340]: osdmap e101: 12 total, 12 up, 12 in 2024-03-22T02:40:16.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:15 smithi083 ceph-mon[36340]: from='client.? 172.21.15.28:0/3998200723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/1304168107' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-22T02:40:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:15 smithi028 ceph-mon[31916]: osdmap e101: 12 total, 12 up, 12 in 2024-03-22T02:40:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:15 smithi028 ceph-mon[31916]: from='client.? 172.21.15.28:0/3998200723' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-22T02:40:16.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:15 smithi028 systemd[1]: Stopping Ceph mon.a for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:16.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:40:16.247+0000 7fa3d5c87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:16.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a[31912]: 2024-03-22T02:40:16.247+0000 7fa3d5c87700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-22T02:40:16.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 podman[110469]: 2024-03-22 02:40:16.289027544 +0000 UTC m=+0.147301551 container died 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:40:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 podman[110469]: 2024-03-22 02:40:16.696951409 +0000 UTC m=+0.555225420 container cleanup 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-22T02:40:16.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 bash[110469]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a 2024-03-22T02:40:17.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Mar 22 02:40:16 smithi028 podman[110483]: 2024-03-22 02:40:16.97345017 +0000 UTC m=+0.677647960 container remove 9bbba678bd3fc3960d908a6d72ae9c3107263ac4923535db8e2cbdb4e7c2c780 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-22T02:40:17.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.a.service' 2024-03-22T02:40:17.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:17.351 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-22T02:40:17.352 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-22T02:40:17.352 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.b 2024-03-22T02:40:17.703 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:17 smithi083 systemd[1]: Stopping Ceph mon.b for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b[36336]: 2024-03-22T02:40:17.702+0000 7fd793f95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b[36336]: 2024-03-22T02:40:17.702+0000 7fd793f95700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-22T02:40:18.015 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:17 smithi083 podman[76299]: 2024-03-22 02:40:17.875603248 +0000 UTC m=+0.296880589 container died a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T02:40:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:18 smithi083 podman[76299]: 2024-03-22 02:40:18.244183384 +0000 UTC m=+0.665460704 container cleanup a41eea566d9b4674b2ac08f85fb5a3aeae34540543c89a76c455e51368f75c81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-22T02:40:18.515 INFO:journalctl@ceph.mon.b.smithi083.stdout:Mar 22 02:40:18 smithi083 bash[76299]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-b 2024-03-22T02:40:18.672 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.b.service' 2024-03-22T02:40:18.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:18.725 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-22T02:40:18.725 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-22T02:40:18.726 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.c 2024-03-22T02:40:19.102 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:18 smithi120 systemd[1]: Stopping Ceph mon.c for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:19.498 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c[36798]: 2024-03-22T02:40:19.100+0000 7f738931b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c[36798]: 2024-03-22T02:40:19.100+0000 7f738931b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-22T02:40:19.499 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:19 smithi120 podman[74953]: 2024-03-22 02:40:19.280649915 +0000 UTC m=+0.312329249 container died 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T02:40:19.938 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:19 smithi120 podman[74953]: 2024-03-22 02:40:19.685714297 +0000 UTC m=+0.717393640 container cleanup 94c70714e4f06fa9c23595d6e09565e93b66660671072ebc53fc5ed7685d9245 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:19.938 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 22 02:40:19 smithi120 bash[74953]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mon-c 2024-03-22T02:40:20.041 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mon.c.service' 2024-03-22T02:40:20.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:20.085 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-22T02:40:20.085 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-22T02:40:20.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x 2024-03-22T02:40:20.572 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:40:20 smithi028 systemd[1]: Stopping Ceph mgr.x for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:21.322 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:40:20 smithi028 podman[113478]: 2024-03-22 02:40:20.941825479 +0000 UTC m=+0.132471078 container died 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:21.822 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:40:21 smithi028 podman[113478]: 2024-03-22 02:40:21.512725537 +0000 UTC m=+0.703371131 container cleanup 035fb819d0bdd9356760e83fa0c86a1f1efd9dda75fc77cf76f22bc673fc85bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:c3a04c0e75f6e20a134d0f5e6ebc95d5f534c43e, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-22T02:40:21.822 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Mar 22 02:40:21 smithi028 bash[113478]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-x 2024-03-22T02:40:21.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-22T02:40:22.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:22.005 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-22T02:40:22.005 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-22T02:40:22.005 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.y 2024-03-22T02:40:22.337 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:22 smithi083 systemd[1]: Stopping Ceph mgr.y for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:22.643 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:22 smithi083 podman[76421]: 2024-03-22 02:40:22.336949382 +0000 UTC m=+0.146134120 container died 9bff5d61bdce22019adf8282c2e61ef6b8a7b32ef0f8740b4fcdcf91709aeb66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:40:23.515 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:23 smithi083 podman[76421]: 2024-03-22 02:40:23.067321081 +0000 UTC m=+0.876505820 container cleanup 9bff5d61bdce22019adf8282c2e61ef6b8a7b32ef0f8740b4fcdcf91709aeb66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-22T02:40:23.515 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:23 smithi083 bash[76421]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y 2024-03-22T02:40:23.515 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:23 smithi083 podman[76435]: 2024-03-22 02:40:23.302300828 +0000 UTC m=+0.963192006 container remove 9bff5d61bdce22019adf8282c2e61ef6b8a7b32ef0f8740b4fcdcf91709aeb66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.license=GPLv2) 2024-03-22T02:40:23.515 INFO:journalctl@ceph.mgr.y.smithi083.stdout:Mar 22 02:40:23 smithi083 systemd[1]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-22T02:40:23.535 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-22T02:40:23.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:23.573 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-22T02:40:23.573 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-22T02:40:23.573 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.z 2024-03-22T02:40:23.887 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:40:23 smithi120 systemd[1]: Stopping Ceph mgr.z for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:24.184 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:40:23 smithi120 podman[75075]: 2024-03-22 02:40:23.886587134 +0000 UTC m=+0.122866504 container died 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:40:24.499 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:40:24 smithi120 podman[75075]: 2024-03-22 02:40:24.291897851 +0000 UTC m=+0.528177213 container cleanup 481400a94c66d83644a8cb8729021c8f78639c5c3a4ebb4d45b4b7d81d1e4b02 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:40:24.499 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 22 02:40:24 smithi120 bash[75075]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mgr-z 2024-03-22T02:40:24.704 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-22T02:40:24.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:24.742 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-22T02:40:24.742 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-22T02:40:24.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.0 2024-03-22T02:40:25.059 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:24 smithi028 systemd[1]: Stopping Ceph osd.0 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:25.322 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0[46820]: 2024-03-22T02:40:25.057+0000 7f894d98d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:25.322 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0[46820]: 2024-03-22T02:40:25.057+0000 7f894d98d700 -1 osd.0 102 *** Got signal Terminated *** 2024-03-22T02:40:25.322 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0[46820]: 2024-03-22T02:40:25.057+0000 7f894d98d700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:40:31.020 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:30 smithi028 podman[113612]: 2024-03-22 02:40:30.700341558 +0000 UTC m=+5.767941784 container died 06051385962ddecba36e90a232e72084fc41c75d36e52794d785e80eeadb4a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:40:31.322 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:31 smithi028 podman[113612]: 2024-03-22 02:40:31.141520819 +0000 UTC m=+6.209121045 container cleanup 06051385962ddecba36e90a232e72084fc41c75d36e52794d785e80eeadb4a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-22T02:40:31.322 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:31 smithi028 bash[113612]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0 2024-03-22T02:40:31.761 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:31 smithi028 podman[113629]: 2024-03-22 02:40:31.418198348 +0000 UTC m=+0.705188926 container remove 06051385962ddecba36e90a232e72084fc41c75d36e52794d785e80eeadb4a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319) 2024-03-22T02:40:32.072 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:31 smithi028 podman[113718]: 2024-03-22 02:40:31.664135488 +0000 UTC m=+0.018261760 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:40:32.072 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:31 smithi028 podman[113718]: 2024-03-22 02:40:31.827453774 +0000 UTC m=+0.181580064 container create dd430e1eb23a463e6214544114a225dc26d80a277d3d032eee38e907ac883faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:40:32.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:32 smithi028 podman[113718]: 2024-03-22 02:40:32.130200145 +0000 UTC m=+0.484326397 container init dd430e1eb23a463e6214544114a225dc26d80a277d3d032eee38e907ac883faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-22T02:40:32.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:32 smithi028 podman[113718]: 2024-03-22 02:40:32.133359058 +0000 UTC m=+0.487485311 container start dd430e1eb23a463e6214544114a225dc26d80a277d3d032eee38e907ac883faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:40:32.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:32 smithi028 podman[113718]: 2024-03-22 02:40:32.221723065 +0000 UTC m=+0.575849332 container attach dd430e1eb23a463e6214544114a225dc26d80a277d3d032eee38e907ac883faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-22T02:40:32.783 INFO:journalctl@ceph.osd.0.smithi028.stdout:Mar 22 02:40:32 smithi028 podman[113718]: 2024-03-22 02:40:32.430271049 +0000 UTC m=+0.784397308 container died dd430e1eb23a463e6214544114a225dc26d80a277d3d032eee38e907ac883faf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-22T02:40:33.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.0.service' 2024-03-22T02:40:33.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:33.174 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-22T02:40:33.174 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-22T02:40:33.174 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.1 2024-03-22T02:40:33.515 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:33 smithi083 systemd[1]: Stopping Ceph osd.1 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:34.015 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1[40978]: 2024-03-22T02:40:33.714+0000 7f3cb99e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:34.015 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1[40978]: 2024-03-22T02:40:33.714+0000 7f3cb99e7700 -1 osd.1 102 *** Got signal Terminated *** 2024-03-22T02:40:34.016 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1[40978]: 2024-03-22T02:40:33.714+0000 7f3cb99e7700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:40:39.422 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:39 smithi083 podman[76544]: 2024-03-22 02:40:39.161528845 +0000 UTC m=+5.790507104 container died ec430ad646e2b90875c2aa7ea96d06fce742233b64df0db6781408d099ea2c08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:39.765 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:39 smithi083 podman[76544]: 2024-03-22 02:40:39.514104719 +0000 UTC m=+6.143082965 container cleanup ec430ad646e2b90875c2aa7ea96d06fce742233b64df0db6781408d099ea2c08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:39.765 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:39 smithi083 bash[76544]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1 2024-03-22T02:40:40.174 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:39 smithi083 podman[76561]: 2024-03-22 02:40:39.824406891 +0000 UTC m=+0.666938003 container remove ec430ad646e2b90875c2aa7ea96d06fce742233b64df0db6781408d099ea2c08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-22T02:40:40.436 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:40 smithi083 podman[76655]: 2024-03-22 02:40:40.07675037 +0000 UTC m=+0.017793834 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:40:40.437 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:40 smithi083 podman[76655]: 2024-03-22 02:40:40.191927466 +0000 UTC m=+0.132970921 container create 350edc4a41a247d5a3f864e4cca5eba8047211fc6d78627f11899c4c50772a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:40:40.765 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:40 smithi083 podman[76655]: 2024-03-22 02:40:40.436649515 +0000 UTC m=+0.377692984 container init 350edc4a41a247d5a3f864e4cca5eba8047211fc6d78627f11899c4c50772a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD) 2024-03-22T02:40:40.765 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:40 smithi083 podman[76655]: 2024-03-22 02:40:40.439714258 +0000 UTC m=+0.380757729 container start 350edc4a41a247d5a3f864e4cca5eba8047211fc6d78627f11899c4c50772a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:40.766 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:40 smithi083 podman[76655]: 2024-03-22 02:40:40.570301921 +0000 UTC m=+0.511345413 container attach 350edc4a41a247d5a3f864e4cca5eba8047211fc6d78627f11899c4c50772a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-22T02:40:41.305 INFO:journalctl@ceph.osd.1.smithi083.stdout:Mar 22 02:40:41 smithi083 podman[76655]: 2024-03-22 02:40:41.039322316 +0000 UTC m=+0.980365780 container died 350edc4a41a247d5a3f864e4cca5eba8047211fc6d78627f11899c4c50772a4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-22T02:40:41.694 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.1.service' 2024-03-22T02:40:41.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:41.735 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-22T02:40:41.735 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-22T02:40:41.735 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.2 2024-03-22T02:40:42.038 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:41 smithi120 systemd[1]: Stopping Ceph osd.2 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:42.377 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2[41989]: 2024-03-22T02:40:42.036+0000 7fe92f920700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:42.377 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2[41989]: 2024-03-22T02:40:42.036+0000 7fe92f920700 -1 osd.2 102 *** Got signal Terminated *** 2024-03-22T02:40:42.377 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2[41989]: 2024-03-22T02:40:42.036+0000 7fe92f920700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:40:47.899 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:47 smithi120 podman[75197]: 2024-03-22 02:40:47.601363766 +0000 UTC m=+5.674955573 container died a0b7d7f521e75f0ba078ef539a17e9dea5c816666b48b8b29265fdeb865095df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-22T02:40:48.249 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:47 smithi120 podman[75197]: 2024-03-22 02:40:47.996722662 +0000 UTC m=+6.070314467 container cleanup a0b7d7f521e75f0ba078ef539a17e9dea5c816666b48b8b29265fdeb865095df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-22T02:40:48.249 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:47 smithi120 bash[75197]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2 2024-03-22T02:40:48.564 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75228]: 2024-03-22 02:40:48.273451065 +0000 UTC m=+0.675815103 container remove a0b7d7f521e75f0ba078ef539a17e9dea5c816666b48b8b29265fdeb865095df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-22T02:40:48.853 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75318]: 2024-03-22 02:40:48.466309117 +0000 UTC m=+0.017571081 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:40:48.853 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75318]: 2024-03-22 02:40:48.591197167 +0000 UTC m=+0.142459117 container create 47e7b925e3ee6568433e722e5bfcef5f5d32f4ed766f5c8a371c487e004b84dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T02:40:48.853 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75318]: 2024-03-22 02:40:48.852586518 +0000 UTC m=+0.403848483 container init 47e7b925e3ee6568433e722e5bfcef5f5d32f4ed766f5c8a371c487e004b84dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:40:49.249 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75318]: 2024-03-22 02:40:48.855652164 +0000 UTC m=+0.406914114 container start 47e7b925e3ee6568433e722e5bfcef5f5d32f4ed766f5c8a371c487e004b84dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T02:40:49.249 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:48 smithi120 podman[75318]: 2024-03-22 02:40:48.961353768 +0000 UTC m=+0.512615722 container attach 47e7b925e3ee6568433e722e5bfcef5f5d32f4ed766f5c8a371c487e004b84dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:40:49.615 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 22 02:40:49 smithi120 podman[75318]: 2024-03-22 02:40:49.257663556 +0000 UTC m=+0.808925518 container died 47e7b925e3ee6568433e722e5bfcef5f5d32f4ed766f5c8a371c487e004b84dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:40:49.952 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.2.service' 2024-03-22T02:40:49.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:49.994 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-22T02:40:49.994 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-22T02:40:49.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.3 2024-03-22T02:40:50.320 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:50 smithi028 systemd[1]: Stopping Ceph osd.3 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:50.572 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:50 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:40:50.319+0000 7f65cba02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:50.572 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:50 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:40:50.319+0000 7f65cba02700 -1 osd.3 102 *** Got signal Terminated *** 2024-03-22T02:40:50.572 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:50 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:40:50.319+0000 7f65cba02700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:40:51.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:50.922+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:52.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:51.972+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:52.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:51 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:51.921+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:53.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:52.971+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:53.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:52 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:52.916+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:53.822 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:53 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:40:53.422+0000 7f65c3fee700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.594497+0000 front 2024-03-22T02:40:27.594526+0000 (oldest deadline 2024-03-22T02:40:52.894686+0000) 2024-03-22T02:40:54.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:53.953+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:53.793+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:54.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:53 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:53.910+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:54.822 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3[53512]: 2024-03-22T02:40:54.440+0000 7f65c3fee700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.594497+0000 front 2024-03-22T02:40:27.594526+0000 (oldest deadline 2024-03-22T02:40:52.894686+0000) 2024-03-22T02:40:55.248 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:54.806+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:55.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:54.926+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:55.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:54 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:54.943+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:55.786+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:56.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:55.933+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:56.279 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:55 smithi028 podman[113821]: 2024-03-22 02:40:55.990974969 +0000 UTC m=+5.798439019 container died 745ecd519fdf16c8313d36323edd3ce98ed428639f60c6d5cb02f42fc92e5fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-22T02:40:56.279 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:55 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:55.939+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:56.572 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:56 smithi028 podman[113821]: 2024-03-22 02:40:56.369771205 +0000 UTC m=+6.177235256 container cleanup 745ecd519fdf16c8313d36323edd3ce98ed428639f60c6d5cb02f42fc92e5fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-03-22T02:40:56.572 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:56 smithi028 bash[113821]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3 2024-03-22T02:40:56.915 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:56 smithi028 podman[113852]: 2024-03-22 02:40:56.643807484 +0000 UTC m=+0.651214582 container remove 745ecd519fdf16c8313d36323edd3ce98ed428639f60c6d5cb02f42fc92e5fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-22T02:40:57.172 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:56 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:56.914+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:57.172 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:56 smithi028 podman[113943]: 2024-03-22 02:40:56.927849317 +0000 UTC m=+0.107654698 container create 4dea5f22f496a73a959e9c11b22b3bff57d74204ac56f2bb1ce0263d2e2b9fd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:40:57.172 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:56 smithi028 podman[113943]: 2024-03-22 02:40:56.838408648 +0000 UTC m=+0.018214051 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:40:57.172 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:57 smithi028 podman[113943]: 2024-03-22 02:40:57.17208583 +0000 UTC m=+0.351891215 container init 4dea5f22f496a73a959e9c11b22b3bff57d74204ac56f2bb1ce0263d2e2b9fd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T02:40:57.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:56.932+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:56.775+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:57.475 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:57 smithi028 podman[113943]: 2024-03-22 02:40:57.175393791 +0000 UTC m=+0.355199176 container start 4dea5f22f496a73a959e9c11b22b3bff57d74204ac56f2bb1ce0263d2e2b9fd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:40:57.475 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:57 smithi028 podman[113943]: 2024-03-22 02:40:57.280738408 +0000 UTC m=+0.460543799 container attach 4dea5f22f496a73a959e9c11b22b3bff57d74204ac56f2bb1ce0263d2e2b9fd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:40:57.822 INFO:journalctl@ceph.osd.3.smithi028.stdout:Mar 22 02:40:57 smithi028 podman[113943]: 2024-03-22 02:40:57.475200135 +0000 UTC m=+0.655005529 container died 4dea5f22f496a73a959e9c11b22b3bff57d74204ac56f2bb1ce0263d2e2b9fd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-22T02:40:58.118 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:57 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:57.893+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:40:58.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.3.service' 2024-03-22T02:40:58.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:40:58.178 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-22T02:40:58.178 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-22T02:40:58.178 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.4 2024-03-22T02:40:58.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:57.972+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:57.762+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:58.503 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:40:58 smithi083 systemd[1]: Stopping Ceph osd.4 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:40:58.765 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:40:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4[47694]: 2024-03-22T02:40:58.502+0000 7f90fe605700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:40:58.765 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:40:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4[47694]: 2024-03-22T02:40:58.502+0000 7f90fe605700 -1 osd.4 102 *** Got signal Terminated *** 2024-03-22T02:40:58.765 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:40:58 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4[47694]: 2024-03-22T02:40:58.502+0000 7f90fe605700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:40:58.999 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:58.973+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:40:58.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:58.735+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:40:59.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:58 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:58.870+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:00.248 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:40:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:40:59.783+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:00.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:40:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:40:59.982+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:00.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:40:59 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:40:59.915+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:00.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:00 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:00.319+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:00.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:00 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:00.735+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:00.999 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:00 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:00.966+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:00.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:00 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:00.736+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:01.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:00.889+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:01.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:00 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:00.889+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:01.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:01 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:01.322+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:01.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:01 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:01.692+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:01.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:01 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:01.688+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:01.999 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:01 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:01.928+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:01.999 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:01 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:01.928+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:02.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:01 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:01.855+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:02.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:01 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:01.917+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:02.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:01 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:01.917+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:02.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:02 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:02.325+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:02.962 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:02 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:02.643+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:02.962 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:02 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:02.723+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:03.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:02 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:02.960+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:03.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:02 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:02.960+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:03.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:02 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:02.853+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:03.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:02 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:02.870+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:03.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:02 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:02.870+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:03.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:03 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:03.358+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:03.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:03 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:03.619+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:03.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:03 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:03.753+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:04.265 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:03 smithi083 podman[76772]: 2024-03-22 02:41:03.978831511 +0000 UTC m=+5.607135959 container died f7e492e9f1a106738a544be3ccaf00e8926f07689b664e30867042ab9d162a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-22T02:41:04.266 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:03 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:03.843+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:04.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:04.030+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:04.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:03 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:03.906+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:04.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:03 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:03.906+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:04.498 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:04.003+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:04.499 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:04.004+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:04.630 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:04 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:04.397+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:04.630 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:04 smithi083 podman[76772]: 2024-03-22 02:41:04.386328335 +0000 UTC m=+6.014632782 container cleanup f7e492e9f1a106738a544be3ccaf00e8926f07689b664e30867042ab9d162a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-22T02:41:04.630 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:04 smithi083 bash[76772]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4 2024-03-22T02:41:04.984 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:04.651+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:04.984 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:04.759+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:05.015 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:04 smithi083 podman[76789]: 2024-03-22 02:41:04.629519991 +0000 UTC m=+0.649872659 container remove f7e492e9f1a106738a544be3ccaf00e8926f07689b664e30867042ab9d162a6e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-22T02:41:05.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:04 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:04.809+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:05.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:04.982+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:05.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:04 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:04.982+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:05.317 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:05.038750158 +0000 UTC m=+0.104986649 container create 84b747607e53bd9fe91a9236040ee54470a95c34901122a063b04bd9fb716a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-22T02:41:05.317 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:04.951315565 +0000 UTC m=+0.017552080 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:05.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:05 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:05.026+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:05.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:04.878+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:05.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:04 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:04.878+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:05.608 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:05 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:05.351+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:05.609 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:05.316458345 +0000 UTC m=+0.382694850 container init 84b747607e53bd9fe91a9236040ee54470a95c34901122a063b04bd9fb716a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T02:41:05.609 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:05.31999002 +0000 UTC m=+0.386226522 container start 84b747607e53bd9fe91a9236040ee54470a95c34901122a063b04bd9fb716a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T02:41:05.609 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:05.399686607 +0000 UTC m=+0.465923129 container attach 84b747607e53bd9fe91a9236040ee54470a95c34901122a063b04bd9fb716a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T02:41:05.945 INFO:journalctl@ceph.osd.4.smithi083.stdout:Mar 22 02:41:05 smithi083 podman[76880]: 2024-03-22 02:41:05.608603548 +0000 UTC m=+0.674840059 container died 84b747607e53bd9fe91a9236040ee54470a95c34901122a063b04bd9fb716a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:41:05.945 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:05 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:05.843+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:05.960 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:05.623+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:05.961 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:05.736+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:06.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:05.959+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:06.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:05 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:05.959+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:06.269 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.4.service' 2024-03-22T02:41:06.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:06.311 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-22T02:41:06.311 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-22T02:41:06.311 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.5 2024-03-22T02:41:06.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:06 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:06.075+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:06.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:05 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:05.881+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:06.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:05 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:05.881+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:06.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:06 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:06.324+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:06.611 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 systemd[1]: Stopping Ceph osd.5 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:06.929 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:06.629+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:06.786+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:06.610+0000 7ffb2a077700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:06.610+0000 7ffb2a077700 -1 osd.5 102 *** Got signal Terminated *** 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:06.610+0000 7ffb2a077700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:06.928+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:06.930 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:06 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:06.928+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:07.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:06 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:06.814+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:07.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:07 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:07.041+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:07.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:06 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:06.904+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:07.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:06 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:06.904+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:07.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:07 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:07.371+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:07.919 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:07 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:07.648+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:07.919 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:07 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:07.792+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:08.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:07 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:07.917+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:08.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:07 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:07.917+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:08.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:07 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:07.917+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.399592+0000 front 2024-03-22T02:40:45.399612+0000 (oldest deadline 2024-03-22T02:41:07.699771+0000) 2024-03-22T02:41:08.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:07 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:07.807+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:08.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:08 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:08.044+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:08.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:07 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:07.868+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:08.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:07 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:07.868+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:08.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:08 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:08.413+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:08.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:08 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:08.413+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:08.959 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:08 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:08.687+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:08.960 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:08 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:08.768+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:09.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:08 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:08.958+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:09.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:08 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:08.958+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:09.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:08 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:08.958+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.399592+0000 front 2024-03-22T02:40:45.399612+0000 (oldest deadline 2024-03-22T02:41:07.699771+0000) 2024-03-22T02:41:09.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:08 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:08.847+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:09.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:08 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:08.847+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:09.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:09 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:09.000+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:09.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:08 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:08.895+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:09.323 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:08 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:08.895+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:09.323 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:08 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:08.895+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:09.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:09 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:09.366+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:09.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:09 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:09.366+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:09.917 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:09 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:09.657+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:09.917 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:09 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:09.749+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:10.248 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:09 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:09.916+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:10.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:09 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:09.916+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:10.249 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:09 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:09.916+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.399592+0000 front 2024-03-22T02:40:45.399612+0000 (oldest deadline 2024-03-22T02:41:07.699771+0000) 2024-03-22T02:41:10.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:09 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:09.803+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:10.265 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:09 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:09.803+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:10.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:09 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:09.989+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:10.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:09 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:09.853+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:10.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:09 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:09.853+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:10.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:09 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:09.853+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:10.541 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:10 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:10.321+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:10.542 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:10 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:10.321+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:10.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:10.706+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:10.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:10.714+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:10.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:10.714+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:11.000 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:10.904+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.096549+0000 front 2024-03-22T02:40:29.096576+0000 (oldest deadline 2024-03-22T02:40:50.796730+0000) 2024-03-22T02:41:11.000 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:10.904+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:38.398655+0000 front 2024-03-22T02:40:38.398707+0000 (oldest deadline 2024-03-22T02:41:01.898644+0000) 2024-03-22T02:41:11.000 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:10 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5[48678]: 2024-03-22T02:41:10.904+0000 7ffb2666b700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.399592+0000 front 2024-03-22T02:40:45.399612+0000 (oldest deadline 2024-03-22T02:41:07.699771+0000) 2024-03-22T02:41:11.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:10 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:10.780+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:11.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:10 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:10.780+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:11.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:10 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:10.990+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:11.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:10 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:10.990+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:11.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:10 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:10.830+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:11.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:10 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:10.830+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:11.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:10 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:10.830+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:11.732 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:11.276+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:11.733 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:11.276+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:11.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:11.723+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:11.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:11.748+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:11.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:11 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:11.748+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:12.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:11.731+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:12.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:11 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:11.731+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:12.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:12 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:12.020+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:12.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:12 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:12.020+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:12.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:11.831+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:12.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:11.831+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:12.323 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:11.831+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:12.479 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:12 smithi120 podman[75422]: 2024-03-22 02:41:12.19894512 +0000 UTC m=+5.686835362 container died 3759941b4ac8f43a1d512cedb232d0ca4b52831f9fe31466f7b7d1b21c2156c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-22T02:41:12.703 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:12 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:12.308+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:12.704 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:12 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:12.308+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:12.748 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:12 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:12.716+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:12.749 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:12 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:12.716+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:12.749 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:12 smithi120 podman[75422]: 2024-03-22 02:41:12.577663369 +0000 UTC m=+6.065553612 container cleanup 3759941b4ac8f43a1d512cedb232d0ca4b52831f9fe31466f7b7d1b21c2156c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T02:41:12.749 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:12 smithi120 bash[75422]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5 2024-03-22T02:41:13.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:12 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:12.701+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:13.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:12 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:12.701+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:13.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:13.043+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:13.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:13.043+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:13.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:12 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:12.811+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:13.073 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:12 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:12.811+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:13.073 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:12 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:12.811+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:13.210 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:12 smithi120 podman[75439]: 2024-03-22 02:41:12.921139393 +0000 UTC m=+0.717367322 container remove 3759941b4ac8f43a1d512cedb232d0ca4b52831f9fe31466f7b7d1b21c2156c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-22T02:41:13.210 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:12 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:12.757+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:13.210 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:12 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:12.757+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:13.492 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.112351866 +0000 UTC m=+0.016927346 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:13.492 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.239276532 +0000 UTC m=+0.143852050 container create 8e3abfd7699754f3d0feb1b3e81acc2725f22271698ba1b93ed9fcb1aae02ba6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-22T02:41:13.700 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:13 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:13.277+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:13.700 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:13 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:13.277+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:13.749 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:13 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:13.675+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:13.749 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:13 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:13.675+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:13.749 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.492248263 +0000 UTC m=+0.396823738 container init 8e3abfd7699754f3d0feb1b3e81acc2725f22271698ba1b93ed9fcb1aae02ba6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T02:41:13.749 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.49532906 +0000 UTC m=+0.399904527 container start 8e3abfd7699754f3d0feb1b3e81acc2725f22271698ba1b93ed9fcb1aae02ba6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-22T02:41:13.749 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.600692742 +0000 UTC m=+0.505268228 container attach 8e3abfd7699754f3d0feb1b3e81acc2725f22271698ba1b93ed9fcb1aae02ba6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T02:41:14.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:13 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:13.698+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:14.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:13 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:13.698+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:14.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:13.998+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:14.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:13.998+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:14.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:13.781+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:14.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:13.781+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:14.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:13 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:13.781+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:14.145 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 22 02:41:13 smithi120 podman[75527]: 2024-03-22 02:41:13.785756409 +0000 UTC m=+0.690331890 container died 8e3abfd7699754f3d0feb1b3e81acc2725f22271698ba1b93ed9fcb1aae02ba6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-22T02:41:14.146 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:13 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:13.764+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:14.146 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:13 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:13.764+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:14.485 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.5.service' 2024-03-22T02:41:14.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:14 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:14.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:14.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:14 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:14.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:14.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:14.528 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-22T02:41:14.529 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-22T02:41:14.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.6 2024-03-22T02:41:14.740 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:14 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:14.662+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:14.741 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:14 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:14.662+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:14.822 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:14 smithi028 systemd[1]: Stopping Ceph osd.6 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:14.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:14.759+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:14.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:14.759+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:14.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:14.759+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:14.998 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:14 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:14.747+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:14.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:14 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:14.747+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:15.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:14 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:14.662+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:15.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:14 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:14.662+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:15.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:14.843+0000 7f9febbd2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:15.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:14.843+0000 7f9febbd2700 -1 osd.6 102 *** Got signal Terminated *** 2024-03-22T02:41:15.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:14 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:14.843+0000 7f9febbd2700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:15.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:15 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:15.012+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:15.323 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:15 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:15.012+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:15.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:15.188+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:15.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:15.188+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:15.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:15 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:15.796+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:15.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:15 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:15.796+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:15.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:15 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:15.705+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:15.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:15 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:15.705+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:16.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:15.636+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:16.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:15 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:15.636+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:16.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:16.000+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:16.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:16.000+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:16.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:15 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:15.762+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:16.073 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:15 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:15.762+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:16.073 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:15 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:15.762+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:16.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:16 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:16.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:16.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:16 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:16.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:16.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:16 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:16.749+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:16.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:16 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:16.749+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:16.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:16 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:16.728+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:16.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:16 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:16.728+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:17.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:16 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:16.636+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:17.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:16 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:16.636+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:17.042 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:16.718+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:17.043 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:16.718+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:17.043 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:16.718+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:17.043 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:16.718+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:17.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:17.041+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:17.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:17.041+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:17.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:17.232+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:17.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:17.232+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:17.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:17 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:17.778+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:17.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:17 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:17.778+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:17.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:17 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:17.728+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:17.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:17 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:17.728+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:18.006 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:17.690+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:18.006 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:17.690+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:18.006 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:17.690+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:18.006 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:17 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:17.690+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:18.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:17.588+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:18.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:17 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:17.588+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:18.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:18.005+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:18.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:18.005+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:18.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:18 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:18.221+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:18.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:18 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:18.221+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:18.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:18 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:18.221+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:18.994 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:18.695+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:18.995 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:18.695+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:18.995 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:18.695+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:18.995 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:18.695+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:18.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:18 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:18.788+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:18.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:18 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:18.788+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:18.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:18 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:18.741+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:18.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:18 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:18.741+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:19.017 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:18 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:18.585+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:19.018 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:18 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:18.585+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:19.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:18.992+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.195967+0000 front 2024-03-22T02:40:37.196040+0000 (oldest deadline 2024-03-22T02:41:03.596694+0000) 2024-03-22T02:41:19.322 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:18 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6[60353]: 2024-03-22T02:41:18.992+0000 7f9fe81c6700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.798544+0000 front 2024-03-22T02:40:45.798670+0000 (oldest deadline 2024-03-22T02:41:10.498727+0000) 2024-03-22T02:41:19.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:19 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:19.173+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:19.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:19 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:19.173+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:19.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:19 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:19.173+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:19.998 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:19.717+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:19.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:19.717+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:19.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:19.812+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:19.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:19.812+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:19.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:19 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:19.812+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:20.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:19 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:19.558+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:20.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:19 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:19.558+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:20.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:19.700+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:20.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:19.700+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:20.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:19.700+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:20.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:19 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:19.700+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:20.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:20.128+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:20.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:20.128+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:20.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:20.128+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:20.572 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:20 smithi028 podman[114051]: 2024-03-22 02:41:20.275150473 +0000 UTC m=+5.546666109 container died b9c672cd5d6cd7e3fb752b1414e463280fef15a31892d96ed2c226dc3c7a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:41:20.966 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:20.582+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:20.966 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:20.582+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:20.966 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:20.582+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:20.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:20 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:20.739+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:20.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:20 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:20.739+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:20.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:20 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:20.835+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:20.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:20 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:20.835+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:20.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:20 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:20.835+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:21.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:20.710+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:21.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:20.710+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:21.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:20.710+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:21.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:20 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:20.710+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:21.072 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:20 smithi028 podman[114051]: 2024-03-22 02:41:20.603248637 +0000 UTC m=+5.874764282 container cleanup b9c672cd5d6cd7e3fb752b1414e463280fef15a31892d96ed2c226dc3c7a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-22T02:41:21.073 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:20 smithi028 bash[114051]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6 2024-03-22T02:41:21.073 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:20 smithi028 podman[114068]: 2024-03-22 02:41:20.821273141 +0000 UTC m=+0.546019179 container remove b9c672cd5d6cd7e3fb752b1414e463280fef15a31892d96ed2c226dc3c7a4478 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T02:41:21.267 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:21.117+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:21.267 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:21.117+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:21.267 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:21.117+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:21.558 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.161530788 +0000 UTC m=+0.018651625 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:21.558 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.272203902 +0000 UTC m=+0.129324734 container create fdc13f2becd264db2466b3950fc6daae3e99aa4328d65a515f2d7bbecda4497d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:41:21.558 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.483442595 +0000 UTC m=+0.340563438 container init fdc13f2becd264db2466b3950fc6daae3e99aa4328d65a515f2d7bbecda4497d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:41:21.559 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.486794365 +0000 UTC m=+0.343915206 container start fdc13f2becd264db2466b3950fc6daae3e99aa4328d65a515f2d7bbecda4497d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0) 2024-03-22T02:41:21.559 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.557970654 +0000 UTC m=+0.415091501 container attach fdc13f2becd264db2466b3950fc6daae3e99aa4328d65a515f2d7bbecda4497d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T02:41:21.822 INFO:journalctl@ceph.osd.6.smithi028.stdout:Mar 22 02:41:21 smithi028 podman[114157]: 2024-03-22 02:41:21.776029279 +0000 UTC m=+0.633150122 container died fdc13f2becd264db2466b3950fc6daae3e99aa4328d65a515f2d7bbecda4497d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T02:41:21.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:21.663+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:21.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:21.663+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:21.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:21.663+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:21.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:21 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:21.663+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:22.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:21.554+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:22.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:21.554+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:22.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:21 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:21.554+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:21.783+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:21.783+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:21.783+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:21.824+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:21.824+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:22.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:21 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:21.824+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:22.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:22.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:22.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:22.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:22.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:22.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:22.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.6.service' 2024-03-22T02:41:22.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:22.593 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-22T02:41:22.593 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-22T02:41:22.594 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.7 2024-03-22T02:41:22.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:22 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:22.661+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:22 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:22.661+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:22 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:22.661+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:22.932 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:22 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:22.661+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:23.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:22.597+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:23.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:22.597+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:23.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:22 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:22.597+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:23.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:22 smithi083 systemd[1]: Stopping Ceph osd.7 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:22.853+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:22.853+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:22.853+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:22.772+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:22.772+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:23.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:22 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:22.772+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:23.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:23.138+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:23.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:23.138+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:23.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:23.138+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:23.516 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.022+0000 7fb524649700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:23.516 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.022+0000 7fb524649700 -1 osd.7 102 *** Got signal Terminated *** 2024-03-22T02:41:23.516 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.022+0000 7fb524649700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:23.920 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:23.623+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:23.920 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:23.623+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:23.920 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:23.623+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:23.920 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:23 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:23.623+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:24.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.618+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:24.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.618+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:24.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:23 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:23.618+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:24.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:23.854+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:24.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:23.854+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:24.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:23.854+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:24.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:23.803+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:24.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:23.803+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:24.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:23 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:23.803+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:24.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:24.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:24.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:24.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:24.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:24.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:24.863 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:24.597+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:24.863 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:24.597+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:24.863 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:24.597+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:24.864 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:24 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:24.597+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:25.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:24.667+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:25.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:24.667+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:25.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:24.667+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:24.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:24.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:24.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:24.797+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:24.797+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:24.797+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:25.169 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:24 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:24.797+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:25.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:25.168+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:25.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:25.168+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:25.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:25.168+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:25.895+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:25.895+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:25.895+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:25.895+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:25.788+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:25.999 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:25.788+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:26.000 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:25.788+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:26.000 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:25 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:25.788+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:26.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:25.687+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:26.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:25.687+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:26.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:25 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:25.687+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:26.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:25.578+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:26.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:25.578+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:26.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:25.578+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:26.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:25.578+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:26.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:25 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:25.578+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:26.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:26.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:26.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:26.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:26.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:26.145+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:26.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:26.562+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:26.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:26.562+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:26.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:26.562+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:26.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:26.562+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:26.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:26 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:26.562+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:27.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:26.712+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:27.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:26.712+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:27.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:26.712+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:27.016 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:26 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:26.712+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:00.313864+0000 front 2024-03-22T02:41:00.313836+0000 (oldest deadline 2024-03-22T02:41:26.214079+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:26.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:26.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:26.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:26.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:26.835+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:26.835+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:27.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:26.835+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:27.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:26 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:26.835+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:27.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:27.166+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:27.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:27.166+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:27.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:27.166+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:27.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:27.166+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:28.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:27.700+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:36.409432+0000 front 2024-03-22T02:40:36.409459+0000 (oldest deadline 2024-03-22T02:41:01.709594+0000) 2024-03-22T02:41:28.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:27.700+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.911724+0000 front 2024-03-22T02:40:46.911713+0000 (oldest deadline 2024-03-22T02:41:08.611624+0000) 2024-03-22T02:41:28.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:27.700+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.413480+0000 front 2024-03-22T02:40:54.413365+0000 (oldest deadline 2024-03-22T02:41:20.313402+0000) 2024-03-22T02:41:28.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:27 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7[54741]: 2024-03-22T02:41:27.700+0000 7fb520c3d700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:00.313864+0000 front 2024-03-22T02:41:00.313836+0000 (oldest deadline 2024-03-22T02:41:26.214079+0000) 2024-03-22T02:41:28.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:27.591+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:28.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:27.591+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:28.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:27.591+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:28.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:27.591+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:28.072 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:27 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:27.591+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:28.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:27.926+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:28.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:27.926+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:28.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:27.926+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:28.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:27.926+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:28.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:27.873+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:28.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:27.873+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:28.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:27.873+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:28.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:27 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:27.873+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:28.437 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:28.132+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:28.437 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:28.132+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:28.437 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:28.132+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:28.437 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:28.132+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:28.765 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:28 smithi083 podman[76997]: 2024-03-22 02:41:28.436345277 +0000 UTC m=+5.649799056 container died b3d2882814f70c7c6a0657ce933ad537579285f2f0881462fdc4c724610806f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-22T02:41:28.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:28 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:28.554+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:28.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:28 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:28.554+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:28.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:28 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:28.554+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:28.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:28 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:28.554+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:28.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:28 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:28.554+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:29.149 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:28 smithi083 podman[76997]: 2024-03-22 02:41:28.881516637 +0000 UTC m=+6.094970417 container cleanup b3d2882814f70c7c6a0657ce933ad537579285f2f0881462fdc4c724610806f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-22T02:41:29.149 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:28 smithi083 bash[76997]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:28.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:28.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:28.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:28.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:28.895+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:29.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:28.895+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:29.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:28.895+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:29.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:28 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:28.895+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:29.443 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77014]: 2024-03-22 02:41:29.149350715 +0000 UTC m=+0.714053550 container remove b3d2882814f70c7c6a0657ce933ad537579285f2f0881462fdc4c724610806f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:41:29.443 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:29 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:29.176+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:29.443 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:29 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:29.176+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:29.443 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:29 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:29.176+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:29.444 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:29 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:29.176+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:29.736 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77108]: 2024-03-22 02:41:29.34544571 +0000 UTC m=+0.018268496 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:29.737 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77108]: 2024-03-22 02:41:29.508854037 +0000 UTC m=+0.181676805 container create 41ec6ed3eb55e8a7f7d5c31b11ef85a3901d144a09efb7c61a394e673ea7c715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-22T02:41:29.737 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77108]: 2024-03-22 02:41:29.736259047 +0000 UTC m=+0.409081822 container init 41ec6ed3eb55e8a7f7d5c31b11ef85a3901d144a09efb7c61a394e673ea7c715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-22T02:41:29.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:29 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:29.506+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:29.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:29 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:29.506+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:29.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:29 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:29.506+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:29.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:29 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:29.506+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:29.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:29 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:29.506+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:30.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77108]: 2024-03-22 02:41:29.739465013 +0000 UTC m=+0.412287785 container start 41ec6ed3eb55e8a7f7d5c31b11ef85a3901d144a09efb7c61a394e673ea7c715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-22T02:41:30.015 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:29 smithi083 podman[77108]: 2024-03-22 02:41:29.836310143 +0000 UTC m=+0.509132928 container attach 41ec6ed3eb55e8a7f7d5c31b11ef85a3901d144a09efb7c61a394e673ea7c715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:29.880+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:29.880+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:29.880+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:29.880+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:29.846+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:30.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:29.846+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:30.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:29.846+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:30.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:29 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:29.846+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:30.298 INFO:journalctl@ceph.osd.7.smithi083.stdout:Mar 22 02:41:30 smithi083 podman[77108]: 2024-03-22 02:41:30.035640116 +0000 UTC m=+0.708462892 container died 41ec6ed3eb55e8a7f7d5c31b11ef85a3901d144a09efb7c61a394e673ea7c715 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-22T02:41:30.298 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:30 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:30.206+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:30.298 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:30 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:30.206+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:30.298 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:30 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:30.206+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:30.298 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:30 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:30.206+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:30.605 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.7.service' 2024-03-22T02:41:30.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:30.647 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-22T02:41:30.648 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-22T02:41:30.648 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.8 2024-03-22T02:41:30.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:30.464+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:30.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:30.464+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:30.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:30.464+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:30.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:30.464+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:30.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:30 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:30.464+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:30.966 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:30.819+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:30.966 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:30.819+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:30.966 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:30.819+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:30.819+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 systemd[1]: Stopping Ceph osd.8 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.871+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.871+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.871+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:30.967 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.871+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:31.248 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.965+0000 7f88de3a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:31.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.965+0000 7f88de3a5700 -1 osd.8 102 *** Got signal Terminated *** 2024-03-22T02:41:31.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:30 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:30.965+0000 7f88de3a5700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:31.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:31 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:31.184+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:31.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:31 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:31.184+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:31.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:31 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:31.184+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:31.517 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:31 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:31.184+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:31.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:31.433+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:31.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:31.433+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:31.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:31.433+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:31.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:31.433+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:31.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:31 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:31.433+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:31.872+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:31.872+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:31.872+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:31.872+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:31.806+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:31.806+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:31.806+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:32.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:31 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:31.806+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:32.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:32.186+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:32.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:32.186+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:32.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:32.187+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:32.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:32.187+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:32.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:32.459+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:32.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:32.459+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:32.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:32.459+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:32.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:32.459+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:32.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:32 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:32.459+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:33.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:32.919+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:33.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:32.919+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:33.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:32.919+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:33.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:32.919+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:33.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:32.855+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:33.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:32.855+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:33.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:32.855+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:33.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:32 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:32.855+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:33.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:33.149+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:33.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:33.149+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:33.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:33.149+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:33.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:33.149+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:33.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:33 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:33.149+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:33.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:33.412+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:33.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:33.412+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:33.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:33.412+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:33.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:33.412+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:33.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:33 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:33.412+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:33.885+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:33.885+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:33.885+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:33.885+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:33.967+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:34.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:33.967+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:34.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:33.967+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:34.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:33.967+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:34.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:33 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:33.967+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:08.156188+0000 front 2024-03-22T02:41:08.156245+0000 (oldest deadline 2024-03-22T02:41:33.456213+0000) 2024-03-22T02:41:34.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:34 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:34.171+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:34.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:34 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:34.171+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:34.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:34 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:34.171+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:34.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:34 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:34.171+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:34.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:34 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:34.171+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:34.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:34.384+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:34.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:34.384+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:34.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:34.384+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:34.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:34.384+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:34.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:34 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:34.384+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:34.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:34.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:34.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:34.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:34.962+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:35.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:34.962+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:35.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:34.962+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:35.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:34.962+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:35.250 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:34 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:34.962+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:08.156188+0000 front 2024-03-22T02:41:08.156245+0000 (oldest deadline 2024-03-22T02:41:33.456213+0000) 2024-03-22T02:41:35.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:35 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:35.144+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:35.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:35 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:35.144+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:35.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:35 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:35.144+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:35.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:35 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:35.144+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:35.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:35 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:35.144+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:35.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:35.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:35.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:35.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:35.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:35.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:35 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:35.389+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:35.931+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.449730+0000 front 2024-03-22T02:40:37.449688+0000 (oldest deadline 2024-03-22T02:41:00.349664+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:35.931+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.651985+0000 front 2024-03-22T02:40:46.652018+0000 (oldest deadline 2024-03-22T02:41:11.952192+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:35.931+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:51.952735+0000 front 2024-03-22T02:40:51.952929+0000 (oldest deadline 2024-03-22T02:41:20.753427+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:35.931+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.454606+0000 front 2024-03-22T02:41:02.454746+0000 (oldest deadline 2024-03-22T02:41:24.754587+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8[55295]: 2024-03-22T02:41:35.931+0000 7f88da999700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:08.156188+0000 front 2024-03-22T02:41:08.156245+0000 (oldest deadline 2024-03-22T02:41:33.456213+0000) 2024-03-22T02:41:36.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:35.917+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:36.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:35.917+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:36.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:35.917+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:36.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:35.917+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:36.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:35 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:35.917+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:36.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:36 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:36.159+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:36.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:36 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:36.159+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:36.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:36 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:36.159+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:36.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:36 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:36.159+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:36.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:36 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:36.159+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:36.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:36.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:36.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:36.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:36.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:36.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:36 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:36.431+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:36.877 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:36 smithi120 podman[75644]: 2024-03-22 02:41:36.564235634 +0000 UTC m=+5.720116421 container died f82ae8dd55bb9e08c647b4c36d622f6db4ad4728e51d5074e28a53e02e84e018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:36 smithi120 podman[75644]: 2024-03-22 02:41:36.974185821 +0000 UTC m=+6.130066606 container cleanup f82ae8dd55bb9e08c647b4c36d622f6db4ad4728e51d5074e28a53e02e84e018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:36 smithi120 bash[75644]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:36 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:36.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:36 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:36.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:36 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:36.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:36 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:36.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:37.226 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:36 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:36.898+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:37.499 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75661]: 2024-03-22 02:41:37.225509702 +0000 UTC m=+0.658852337 container remove f82ae8dd55bb9e08c647b4c36d622f6db4ad4728e51d5074e28a53e02e84e018 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS) 2024-03-22T02:41:37.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:37 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:37.157+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:37.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:37 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:37.157+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:37.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:37 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:37.157+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:37.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:37 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:37.157+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:37.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:37 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:37.157+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:37.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:37 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:37.452+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:37.898 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75753]: 2024-03-22 02:41:37.527669807 +0000 UTC m=+0.017665860 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:37.899 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75753]: 2024-03-22 02:41:37.640848681 +0000 UTC m=+0.130844744 container create e639676c86f5e2a20545633ac3b15b0e1fa384e32b47eeea25e5dcfc2f86e491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T02:41:37.899 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75753]: 2024-03-22 02:41:37.846165635 +0000 UTC m=+0.336161684 container init e639676c86f5e2a20545633ac3b15b0e1fa384e32b47eeea25e5dcfc2f86e491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T02:41:37.899 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75753]: 2024-03-22 02:41:37.849169674 +0000 UTC m=+0.339165716 container start e639676c86f5e2a20545633ac3b15b0e1fa384e32b47eeea25e5dcfc2f86e491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-22T02:41:38.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:37 smithi120 podman[75753]: 2024-03-22 02:41:37.930244463 +0000 UTC m=+0.420240520 container attach e639676c86f5e2a20545633ac3b15b0e1fa384e32b47eeea25e5dcfc2f86e491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5) 2024-03-22T02:41:38.249 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 22 02:41:38 smithi120 podman[75753]: 2024-03-22 02:41:38.132365341 +0000 UTC m=+0.622361394 container died e639676c86f5e2a20545633ac3b15b0e1fa384e32b47eeea25e5dcfc2f86e491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-22T02:41:38.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:37 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:37.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:38.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:37 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:37.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:38.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:37 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:37.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:38.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:37 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:37.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:38.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:37 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:37.897+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:38.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:38 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:38.169+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:38.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:38 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:38.169+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:38.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:38 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:38.169+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:38.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:38 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:38.169+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:38.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:38 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:38.169+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:38.773 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.8.service' 2024-03-22T02:41:38.815 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:38.815 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-22T02:41:38.815 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-22T02:41:38.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.9 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:38.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:38.471+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:39.107 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:38 smithi028 systemd[1]: Stopping Ceph osd.9 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:39.248 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:38.943+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:39.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:38.943+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:39.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:38.943+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:39.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:38.943+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:39.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:38.943+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:39.430 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.106+0000 7f1e9b032700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:39.430 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.106+0000 7f1e9b032700 -1 osd.9 102 *** Got signal Terminated *** 2024-03-22T02:41:39.430 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.106+0000 7f1e9b032700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:39.430 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:39.430 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:39.431 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:39.431 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:39.431 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:39.431 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:39 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:39.429+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:39.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:39.183+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:39.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:39.183+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:39.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:39.183+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:39.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:39.183+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:39.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:39 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:39.183+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:40.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:39.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:40.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:39.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:40.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:39.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:40.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:39.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:40.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:39 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:39.899+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:40.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:40 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:40.204+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:40.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:40 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:40.204+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:40.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:40 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:40.204+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:40.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:40 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:40.204+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:40.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:40 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:40.204+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:40.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:40 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:40.430+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:41.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:40.936+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:41.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:41.210+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:41.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:41.210+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:41.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:41.210+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:41.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:41.210+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:41.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:41 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:41.210+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:41.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:41 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:41.388+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.904+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.905+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.905+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.905+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.905+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:42.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:41 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:41.905+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:42.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:42.245+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:42.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:42.245+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:42.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:42.245+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:42.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:42.245+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:42.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:42 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:42.245+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:42.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:42.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:42.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:42.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:42.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:42.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:42 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:42.340+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:43.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:42 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:42.861+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:43.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:43.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:43.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:43.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:43.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:43.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:43.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:43.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:43.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:43 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:43.229+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:43.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:27.678327+0000 front 2024-03-22T02:40:27.678294+0000 (oldest deadline 2024-03-22T02:40:51.778534+0000) 2024-03-22T02:41:43.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:34.079707+0000 front 2024-03-22T02:40:34.079772+0000 (oldest deadline 2024-03-22T02:40:59.979590+0000) 2024-03-22T02:41:43.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:44.680825+0000 front 2024-03-22T02:40:44.680729+0000 (oldest deadline 2024-03-22T02:41:08.780839+0000) 2024-03-22T02:41:43.822 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.182493+0000 front 2024-03-22T02:40:55.182471+0000 (oldest deadline 2024-03-22T02:41:16.282573+0000) 2024-03-22T02:41:43.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:01.885276+0000 front 2024-03-22T02:41:01.885334+0000 (oldest deadline 2024-03-22T02:41:24.785142+0000) 2024-03-22T02:41:43.823 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:43 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9[67331]: 2024-03-22T02:41:43.334+0000 7f1e97626700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:10.086333+0000 front 2024-03-22T02:41:10.086296+0000 (oldest deadline 2024-03-22T02:41:34.785996+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:44.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:43 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:43.843+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:44.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:44.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:44.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:44.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:44.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:44.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:44 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:44.251+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:45.034 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:44 smithi028 podman[114274]: 2024-03-22 02:41:44.750002278 +0000 UTC m=+5.741362945 container died 7a5f252af47d0dd08487554d7e9133ef3aeb7f221c60eb5b08e2794ab94f7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:45.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:44.821+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:45.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114274]: 2024-03-22 02:41:45.122717695 +0000 UTC m=+6.114078363 container cleanup 7a5f252af47d0dd08487554d7e9133ef3aeb7f221c60eb5b08e2794ab94f7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:41:45.322 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 bash[114274]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9 2024-03-22T02:41:45.733 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114293]: 2024-03-22 02:41:45.449278534 +0000 UTC m=+0.692386813 container remove 7a5f252af47d0dd08487554d7e9133ef3aeb7f221c60eb5b08e2794ab94f7e8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T02:41:45.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:45.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:45.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:45.766 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:45.766 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:45.766 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:45 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:45.269+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:46.027 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114387]: 2024-03-22 02:41:45.733506144 +0000 UTC m=+0.099305694 container create 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-22T02:41:46.028 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114387]: 2024-03-22 02:41:45.651957196 +0000 UTC m=+0.017756697 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:46.028 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114387]: 2024-03-22 02:41:45.944133499 +0000 UTC m=+0.309933002 container init 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-22T02:41:46.028 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:45 smithi028 podman[114387]: 2024-03-22 02:41:45.947160538 +0000 UTC m=+0.312960030 container start 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T02:41:46.028 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:46 smithi028 podman[114387]: 2024-03-22 02:41:46.027323385 +0000 UTC m=+0.393122892 container attach 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:46.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:45 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:45.833+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:46.307 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:46 smithi028 podman[114387]: 2024-03-22 02:41:46.257876317 +0000 UTC m=+0.623675803 container died 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T02:41:46.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:46.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:46.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:46.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:46.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:46.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:46.254+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:46.864 INFO:journalctl@ceph.osd.9.smithi028.stdout:Mar 22 02:41:46 smithi028 podman[114414]: 2024-03-22 02:41:46.84772373 +0000 UTC m=+0.584444485 container remove 9e09988ddefb930a942232cce4e6f7fda5756e37fee0db72edf318840a4afa7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:41:46.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.9.service' 2024-03-22T02:41:46.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:46.949 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-22T02:41:46.949 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-22T02:41:46.949 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.10 2024-03-22T02:41:47.239 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:46 smithi083 systemd[1]: Stopping Ceph osd.10 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:47.248 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:47.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:47.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:47.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:47.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:47.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:46 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:46.851+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:47.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.238+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.277+0000 7f8fe1019700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.277+0000 7f8fe1019700 -1 osd.10 102 *** Got signal Terminated *** 2024-03-22T02:41:47.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:47 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:47.277+0000 7f8fe1019700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:48.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:47 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:47.873+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:48.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:48.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:48.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:48.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:48.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:48.766 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:48 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:48.278+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:49.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:48.884+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:49.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:49.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:49.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:49.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:49.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:49.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:49 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:49.235+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:50.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:49 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:49.848+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:50.482 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:50.482 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:50.483 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:50.483 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:50.483 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:50.483 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:50 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:50.189+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:51.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:50 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:50.882+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:51.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:51.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:51.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:51.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:51.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:51.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:51 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:51.182+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:52.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:51 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:51.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:52.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6804 osd.1 since back 2024-03-22T02:40:37.890984+0000 front 2024-03-22T02:40:37.891010+0000 (oldest deadline 2024-03-22T02:41:00.191116+0000) 2024-03-22T02:41:52.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:46.092452+0000 front 2024-03-22T02:40:46.092469+0000 (oldest deadline 2024-03-22T02:41:08.392507+0000) 2024-03-22T02:41:52.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:54.194432+0000 front 2024-03-22T02:40:54.194492+0000 (oldest deadline 2024-03-22T02:41:17.694554+0000) 2024-03-22T02:41:52.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:41:02.197167+0000 front 2024-03-22T02:41:02.197143+0000 (oldest deadline 2024-03-22T02:41:26.897266+0000) 2024-03-22T02:41:52.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:06.897812+0000 front 2024-03-22T02:41:06.897778+0000 (oldest deadline 2024-03-22T02:41:32.797883+0000) 2024-03-22T02:41:52.516 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10[61544]: 2024-03-22T02:41:52.137+0000 7f8fd9605700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:18.599538+0000 front 2024-03-22T02:41:18.599565+0000 (oldest deadline 2024-03-22T02:41:43.299544+0000) 2024-03-22T02:41:53.129 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:52 smithi083 podman[77224]: 2024-03-22 02:41:52.859037902 +0000 UTC m=+5.721490330 container died b86a0921af6a43be4f948fd0eb89047b7b8e3c9a20031a8175395b1515525faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:53.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:52 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:52.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:53.487 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:53 smithi083 podman[77224]: 2024-03-22 02:41:53.227818219 +0000 UTC m=+6.090270640 container cleanup b86a0921af6a43be4f948fd0eb89047b7b8e3c9a20031a8175395b1515525faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T02:41:53.487 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:53 smithi083 bash[77224]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10 2024-03-22T02:41:53.765 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:53 smithi083 podman[77241]: 2024-03-22 02:41:53.48720835 +0000 UTC m=+0.623767524 container remove b86a0921af6a43be4f948fd0eb89047b7b8e3c9a20031a8175395b1515525faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-22T02:41:54.091 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:53 smithi083 podman[77332]: 2024-03-22 02:41:53.697548919 +0000 UTC m=+0.017681590 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:41:54.091 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:53 smithi083 podman[77332]: 2024-03-22 02:41:53.804889565 +0000 UTC m=+0.125022216 container create 26492b70fc4b4296c8e85e6c11e8152bd127c8c5740411f0cb178175b8a51c4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10-deactivate, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-22T02:41:54.091 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:54 smithi083 podman[77332]: 2024-03-22 02:41:54.007425352 +0000 UTC m=+0.327557993 container init 26492b70fc4b4296c8e85e6c11e8152bd127c8c5740411f0cb178175b8a51c4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True) 2024-03-22T02:41:54.091 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:54 smithi083 podman[77332]: 2024-03-22 02:41:54.010378322 +0000 UTC m=+0.330510966 container start 26492b70fc4b4296c8e85e6c11e8152bd127c8c5740411f0cb178175b8a51c4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T02:41:54.091 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:54 smithi083 podman[77332]: 2024-03-22 02:41:54.090731694 +0000 UTC m=+0.410864347 container attach 26492b70fc4b4296c8e85e6c11e8152bd127c8c5740411f0cb178175b8a51c4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:54.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:53 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:53.870+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:54.515 INFO:journalctl@ceph.osd.10.smithi083.stdout:Mar 22 02:41:54 smithi083 podman[77332]: 2024-03-22 02:41:54.297193835 +0000 UTC m=+0.617326488 container died 26492b70fc4b4296c8e85e6c11e8152bd127c8c5740411f0cb178175b8a51c4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:41:54.841 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.10.service' 2024-03-22T02:41:54.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:41:54.883 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-22T02:41:54.883 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-22T02:41:54.883 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.11 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:55.195 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:55.196 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:54.903+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:55.196 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:54 smithi120 systemd[1]: Stopping Ceph osd.11 for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:41:55.196 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.193+0000 7f8f15b1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:41:55.196 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.193+0000 7f8f15b1a700 -1 osd.11 102 *** Got signal Terminated *** 2024-03-22T02:41:55.196 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.193+0000 7f8f15b1a700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:56.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:55 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:55.893+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:57.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:56 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:56.874+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:58.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:57 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:57.923+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:41:59.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:41:59.250 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:58 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:58.889+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6820 osd.8 since back 2024-03-22T02:41:34.910302+0000 front 2024-03-22T02:41:34.910326+0000 (oldest deadline 2024-03-22T02:41:58.410371+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-03-22T02:40:29.000001+0000 front 2024-03-22T02:40:28.999811+0000 (oldest deadline 2024-03-22T02:40:53.699927+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-22T02:40:45.402112+0000 front 2024-03-22T02:40:45.402082+0000 (oldest deadline 2024-03-22T02:41:10.702411+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-03-22T02:40:55.204885+0000 front 2024-03-22T02:40:55.204823+0000 (oldest deadline 2024-03-22T02:41:19.304790+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6812 osd.4 since back 2024-03-22T02:40:59.305594+0000 front 2024-03-22T02:40:59.305789+0000 (oldest deadline 2024-03-22T02:41:25.205502+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-22T02:41:09.906484+0000 front 2024-03-22T02:41:09.906452+0000 (oldest deadline 2024-03-22T02:41:35.806591+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.28:6822 osd.6 since back 2024-03-22T02:41:19.208186+0000 front 2024-03-22T02:41:19.208424+0000 (oldest deadline 2024-03-22T02:41:40.307954+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.83:6820 osd.7 since back 2024-03-22T02:41:26.709557+0000 front 2024-03-22T02:41:26.709539+0000 (oldest deadline 2024-03-22T02:41:50.209381+0000) 2024-03-22T02:42:00.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:41:59 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11[62199]: 2024-03-22T02:41:59.868+0000 7f8f0e106700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.120:6820 osd.8 since back 2024-03-22T02:41:34.910302+0000 front 2024-03-22T02:41:34.910326+0000 (oldest deadline 2024-03-22T02:41:58.410371+0000) 2024-03-22T02:42:00.882 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:00 smithi120 podman[75873]: 2024-03-22 02:42:00.620023944 +0000 UTC m=+5.543243645 container died 191406d335a97fd13c2fc82ad3e85f631a9d8844a73876e391fa2adc66cc38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:42:01.221 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:00 smithi120 podman[75873]: 2024-03-22 02:42:00.96071284 +0000 UTC m=+5.883932533 container cleanup 191406d335a97fd13c2fc82ad3e85f631a9d8844a73876e391fa2adc66cc38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-22T02:42:01.221 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:00 smithi120 bash[75873]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11 2024-03-22T02:42:01.498 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75891]: 2024-03-22 02:42:01.220450389 +0000 UTC m=+0.599492653 container remove 191406d335a97fd13c2fc82ad3e85f631a9d8844a73876e391fa2adc66cc38b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-22T02:42:01.874 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75981]: 2024-03-22 02:42:01.613195791 +0000 UTC m=+0.111373988 container create 975de3ff7a6f38f1763bb458948133ff2d1a0178ae750e56a28bd2fdbb617b06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-22T02:42:01.874 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75981]: 2024-03-22 02:42:01.519591999 +0000 UTC m=+0.017770167 image pull 9d92c22378ed84fc2c811df10df04a727e8692298ba89c4c0ff8ee0294070ac0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0 2024-03-22T02:42:01.874 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75981]: 2024-03-22 02:42:01.798945345 +0000 UTC m=+0.297123490 container init 975de3ff7a6f38f1763bb458948133ff2d1a0178ae750e56a28bd2fdbb617b06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-22T02:42:01.874 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75981]: 2024-03-22 02:42:01.802290845 +0000 UTC m=+0.300469007 container start 975de3ff7a6f38f1763bb458948133ff2d1a0178ae750e56a28bd2fdbb617b06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-22T02:42:02.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:01 smithi120 podman[75981]: 2024-03-22 02:42:01.873853202 +0000 UTC m=+0.372031371 container attach 975de3ff7a6f38f1763bb458948133ff2d1a0178ae750e56a28bd2fdbb617b06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T02:42:02.249 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 22 02:42:02 smithi120 podman[75981]: 2024-03-22 02:42:02.092620592 +0000 UTC m=+0.590798740 container died 975de3ff7a6f38f1763bb458948133ff2d1a0178ae750e56a28bd2fdbb617b06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-22T02:42:02.655 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@osd.11.service' 2024-03-22T02:42:02.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:02.695 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-22T02:42:02.695 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-22T02:42:02.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.a 2024-03-22T02:42:02.993 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:02 smithi028 systemd[1]: Stopping Ceph mds.a for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:03.322 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:02 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a[74330]: 2024-03-22T02:42:02.991+0000 7f9ab87ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:03.322 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:02 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a[74330]: 2024-03-22T02:42:02.991+0000 7f9ab87ef700 -1 mds.a *** got signal Terminated *** 2024-03-22T02:42:07.043 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:06 smithi028 podman[114504]: 2024-03-22 02:42:06.776035755 +0000 UTC m=+3.885196778 container died ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-22T02:42:07.322 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 podman[114504]: 2024-03-22 02:42:07.124751836 +0000 UTC m=+4.233912871 container cleanup ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:42:07.322 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 bash[114504]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a 2024-03-22T02:42:07.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.a.service' 2024-03-22T02:42:07.580 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 podman[114518]: 2024-03-22 02:42:07.325792559 +0000 UTC m=+0.537452618 container remove ff56731f8d2d4ec30648a11a7eff7829201177ec2ed3ad484f119305bd4cbeee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:42:07.580 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 systemd[1]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.a.service: Deactivated successfully. 2024-03-22T02:42:07.580 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 systemd[1]: Stopped Ceph mds.a for 426c3602-e7f0-11ee-95cd-87774f69a715. 2024-03-22T02:42:07.580 INFO:journalctl@ceph.mds.a.smithi028.stdout:Mar 22 02:42:07 smithi028 systemd[1]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.a.service: Consumed 1.207s CPU time. 2024-03-22T02:42:07.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:07.593 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-22T02:42:07.593 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-22T02:42:07.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.d 2024-03-22T02:42:07.912 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:07 smithi028 systemd[1]: Stopping Ceph mds.d for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:08.322 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:07 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d[73373]: 2024-03-22T02:42:07.911+0000 7f65a71b2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:08.322 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:07 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d[73373]: 2024-03-22T02:42:07.911+0000 7f65a71b2700 -1 mds.d *** got signal Terminated *** 2024-03-22T02:42:10.903 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:10 smithi028 podman[114625]: 2024-03-22 02:42:10.589066771 +0000 UTC m=+2.784010948 container died 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:10.903 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:10 smithi028 podman[114625]: 2024-03-22 02:42:10.903066471 +0000 UTC m=+3.098010649 container cleanup 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T02:42:11.299 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:10 smithi028 bash[114625]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d 2024-03-22T02:42:11.299 INFO:journalctl@ceph.mds.d.smithi028.stdout:Mar 22 02:42:11 smithi028 podman[114639]: 2024-03-22 02:42:11.088001163 +0000 UTC m=+0.496966006 container remove 86d0234f3b3ae0cbd92c5b8554c814a488d1c95b08a6e597df602c00d9068b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:11.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.d.service' 2024-03-22T02:42:11.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:11.344 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-22T02:42:11.344 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-22T02:42:11.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.g 2024-03-22T02:42:11.572 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:11 smithi028 systemd[1]: Stopping Ceph mds.g for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:12.072 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g[73979]: 2024-03-22T02:42:11.664+0000 7f6d9fd92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:12.072 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:11 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g[73979]: 2024-03-22T02:42:11.664+0000 7f6d9fd92700 -1 mds.g *** got signal Terminated *** 2024-03-22T02:42:15.950 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:15 smithi028 podman[114746]: 2024-03-22 02:42:15.675169088 +0000 UTC m=+4.110101189 container died 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-22T02:42:16.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:16 smithi028 podman[114746]: 2024-03-22 02:42:16.048241271 +0000 UTC m=+4.483173374 container cleanup 13d3f9505bd612c77cb10a2b3443350b8ca1ac9940c1cb86b0a315f4ab7b9b12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:42:16.322 INFO:journalctl@ceph.mds.g.smithi028.stdout:Mar 22 02:42:16 smithi028 bash[114746]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-g 2024-03-22T02:42:16.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.g.service' 2024-03-22T02:42:16.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:16.512 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-22T02:42:16.513 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-22T02:42:16.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.j 2024-03-22T02:42:17.072 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:16 smithi028 systemd[1]: Stopping Ceph mds.j for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:17.073 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j[72985]: 2024-03-22T02:42:16.843+0000 7f52ced58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:17.073 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:16 smithi028 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j[72985]: 2024-03-22T02:42:16.843+0000 7f52ced58700 -1 mds.j *** got signal Terminated *** 2024-03-22T02:42:20.072 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:19 smithi028 podman[114865]: 2024-03-22 02:42:19.808051519 +0000 UTC m=+3.066253538 container died 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:20.572 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:20 smithi028 podman[114865]: 2024-03-22 02:42:20.135598953 +0000 UTC m=+3.393800989 container cleanup 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=) 2024-03-22T02:42:20.572 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:20 smithi028 bash[114865]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j 2024-03-22T02:42:20.572 INFO:journalctl@ceph.mds.j.smithi028.stdout:Mar 22 02:42:20 smithi028 podman[114879]: 2024-03-22 02:42:20.353483253 +0000 UTC m=+0.542193453 container remove 731ee3bdf87141e6b48b424c467c57107d78db0b5ed7b3947eec10ad56da5fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-j, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:42:20.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.j.service' 2024-03-22T02:42:20.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:20.628 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-22T02:42:20.628 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-22T02:42:20.628 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.b 2024-03-22T02:42:20.945 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:20 smithi083 systemd[1]: Stopping Ceph mds.b for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:21.265 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: 2024-03-22T02:42:20.944+0000 7f3715ab3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:21.265 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:20 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b[66805]: 2024-03-22T02:42:20.944+0000 7f3715ab3700 -1 mds.b *** got signal Terminated *** 2024-03-22T02:42:24.112 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:23 smithi083 podman[77439]: 2024-03-22 02:42:23.814137582 +0000 UTC m=+2.995213428 container died 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-22T02:42:24.445 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:24 smithi083 podman[77439]: 2024-03-22 02:42:24.210899252 +0000 UTC m=+3.391975091 container cleanup 9e6641025827f43280d114a069afba71edb8bcfd63b674d298a886167e7185ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T02:42:24.446 INFO:journalctl@ceph.mds.b.smithi083.stdout:Mar 22 02:42:24 smithi083 bash[77439]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-b 2024-03-22T02:42:24.552 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.b.service' 2024-03-22T02:42:24.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:24.591 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-22T02:42:24.591 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-22T02:42:24.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.e 2024-03-22T02:42:24.766 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:24 smithi083 systemd[1]: Stopping Ceph mds.e for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:25.265 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: 2024-03-22T02:42:24.906+0000 7effb2b71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:25.265 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:24 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e[66187]: 2024-03-22T02:42:24.906+0000 7effb2b71700 -1 mds.e *** got signal Terminated *** 2024-03-22T02:42:28.114 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:27 smithi083 podman[77569]: 2024-03-22 02:42:27.818486588 +0000 UTC m=+3.010184774 container died bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5) 2024-03-22T02:42:28.115 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:28 smithi083 podman[77569]: 2024-03-22 02:42:28.114430945 +0000 UTC m=+3.306129117 container cleanup bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:28.515 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:28 smithi083 bash[77569]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e 2024-03-22T02:42:28.516 INFO:journalctl@ceph.mds.e.smithi083.stdout:Mar 22 02:42:28 smithi083 podman[77582]: 2024-03-22 02:42:28.299086925 +0000 UTC m=+0.477683858 container remove bf52c6033e90ffc9aa2df1e6c6a33f02eb4671d54aa4344614eb54d93333d349 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-e, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-22T02:42:28.534 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.e.service' 2024-03-22T02:42:28.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:28.570 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-22T02:42:28.570 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-22T02:42:28.571 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.h 2024-03-22T02:42:28.877 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:28 smithi083 systemd[1]: Stopping Ceph mds.h for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:29.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: 2024-03-22T02:42:28.876+0000 7f7985757700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:29.265 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:28 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h[66496]: 2024-03-22T02:42:28.876+0000 7f7985757700 -1 mds.h *** got signal Terminated *** 2024-03-22T02:42:32.086 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:31 smithi083 podman[77689]: 2024-03-22 02:42:31.811121716 +0000 UTC m=+3.025959242 container died 174bfe21ccc35ae9f24a71db21de3da227d0f6057b1fac13943276f1c9c12206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-22T02:42:32.444 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:32 smithi083 podman[77689]: 2024-03-22 02:42:32.168306797 +0000 UTC m=+3.383144321 container cleanup 174bfe21ccc35ae9f24a71db21de3da227d0f6057b1fac13943276f1c9c12206 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0) 2024-03-22T02:42:32.445 INFO:journalctl@ceph.mds.h.smithi083.stdout:Mar 22 02:42:32 smithi083 bash[77689]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-h 2024-03-22T02:42:32.593 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.h.service' 2024-03-22T02:42:32.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:32.631 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-22T02:42:32.631 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-22T02:42:32.631 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.k 2024-03-22T02:42:32.964 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:32 smithi083 systemd[1]: Stopping Ceph mds.k for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:33.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k[65866]: 2024-03-22T02:42:32.963+0000 7f13683c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:33.265 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:32 smithi083 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k[65866]: 2024-03-22T02:42:32.963+0000 7f13683c8700 -1 mds.k *** got signal Terminated *** 2024-03-22T02:42:37.164 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:36 smithi083 podman[77812]: 2024-03-22 02:42:36.898977784 +0000 UTC m=+4.049626743 container died e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:37.440 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:37 smithi083 podman[77812]: 2024-03-22 02:42:37.24720576 +0000 UTC m=+4.397854750 container cleanup e2cc96f1bf02e8c01c30af51cf2dbe4543776c1c9de614db240c8797efbd864d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-22T02:42:37.440 INFO:journalctl@ceph.mds.k.smithi083.stdout:Mar 22 02:42:37 smithi083 bash[77812]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-k 2024-03-22T02:42:37.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.k.service' 2024-03-22T02:42:37.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:37.696 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-22T02:42:37.696 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-22T02:42:37.696 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.c 2024-03-22T02:42:37.999 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:37 smithi120 systemd[1]: Stopping Ceph mds.c for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:38.499 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:42:38.076+0000 7f2bf553a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:38.499 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:38 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c[67511]: 2024-03-22T02:42:38.076+0000 7f2bf553a700 -1 mds.c *** got signal Terminated *** 2024-03-22T02:42:40.120 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:39 smithi120 podman[76086]: 2024-03-22 02:42:39.762964247 +0000 UTC m=+1.872868108 container died 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-03-22T02:42:40.499 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:40 smithi120 podman[76086]: 2024-03-22 02:42:40.210340666 +0000 UTC m=+2.320244529 container cleanup 78642f6109684892837a05b2b7e72bbf24ea7253c5fa6faa9f5b0a30ffb16c73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T02:42:40.499 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 22 02:42:40 smithi120 bash[76086]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-c 2024-03-22T02:42:40.626 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.c.service' 2024-03-22T02:42:40.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:40.664 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-22T02:42:40.664 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-22T02:42:40.664 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.f 2024-03-22T02:42:40.981 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:40 smithi120 systemd[1]: Stopping Ceph mds.f for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:41.248 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: 2024-03-22T02:42:40.979+0000 7f6a181e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:41.249 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:40 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f[67207]: 2024-03-22T02:42:40.979+0000 7f6a181e6700 -1 mds.f *** got signal Terminated *** 2024-03-22T02:42:44.056 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:43 smithi120 podman[76220]: 2024-03-22 02:42:43.768825728 +0000 UTC m=+2.894408174 container died 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T02:42:44.056 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:44 smithi120 podman[76220]: 2024-03-22 02:42:44.05563367 +0000 UTC m=+3.181216115 container cleanup 52a4e0b4566cb8f9b2b2bb3bc6114e8b8739c96282f202e6b63d4c77625cdba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T02:42:44.315 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 22 02:42:44 smithi120 bash[76220]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-f 2024-03-22T02:42:44.461 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.f.service' 2024-03-22T02:42:44.497 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:44.497 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-22T02:42:44.497 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-22T02:42:44.497 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.i 2024-03-22T02:42:44.998 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:44 smithi120 systemd[1]: Stopping Ceph mds.i for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:44.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: 2024-03-22T02:42:44.817+0000 7f3c6ab77700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:44.999 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:44 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i[67818]: 2024-03-22T02:42:44.817+0000 7f3c6ab77700 -1 mds.i *** got signal Terminated *** 2024-03-22T02:42:48.052 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:47 smithi120 podman[76342]: 2024-03-22 02:42:47.778501038 +0000 UTC m=+3.066316584 container died 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True) 2024-03-22T02:42:48.427 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:48 smithi120 podman[76342]: 2024-03-22 02:42:48.142798859 +0000 UTC m=+3.430614470 container cleanup 102ba44f91bbdcf63d4a39f4de14c7f46d148429eb104388d47288728da1795d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T02:42:48.428 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 22 02:42:48 smithi120 bash[76342]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-i 2024-03-22T02:42:48.566 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.i.service' 2024-03-22T02:42:48.601 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:48.601 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-22T02:42:48.601 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-22T02:42:48.601 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.l 2024-03-22T02:42:48.998 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:48 smithi120 systemd[1]: Stopping Ceph mds.l for 426c3602-e7f0-11ee-95cd-87774f69a715... 2024-03-22T02:42:48.999 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l[66888]: 2024-03-22T02:42:48.920+0000 7ff12b25e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T02:42:48.999 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:48 smithi120 ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l[66888]: 2024-03-22T02:42:48.920+0000 7ff12b25e700 -1 mds.l *** got signal Terminated *** 2024-03-22T02:42:50.231 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:49 smithi120 podman[76462]: 2024-03-22 02:42:49.97563495 +0000 UTC m=+1.151103814 container died 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T02:42:50.499 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:50 smithi120 podman[76462]: 2024-03-22 02:42:50.313363179 +0000 UTC m=+1.488832046 container cleanup 118c41b9f894d0358e72c2ccae40e20c167983aa6b0a61933cd39939521935bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:a06af61f3caaea4418abf5c5438a99b24e0163a5c209d73a1bde86d2658df0d0, name=ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240321.172515-c3a04c0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-22T02:42:50.499 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 22 02:42:50 smithi120 bash[76462]: ceph-426c3602-e7f0-11ee-95cd-87774f69a715-mds-l 2024-03-22T02:42:50.735 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-426c3602-e7f0-11ee-95cd-87774f69a715@mds.l.service' 2024-03-22T02:42:50.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T02:42:50.769 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-22T02:42:50.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T02:42:50.970 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:42:53.611 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T02:42:53.801 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:42:56.359 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T02:42:56.548 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T02:42:58.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:42:59.024 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:42:59.058 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T02:42:59.086 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-22T02:42:59.089 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi028/crash 2024-03-22T02:42:59.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash -- . 2024-03-22T02:42:59.142 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T02:42:59.142 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-03-22T02:42:59.144 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi083/crash 2024-03-22T02:42:59.145 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash -- . 2024-03-22T02:42:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:tar: /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T02:42:59.196 INFO:teuthology.orchestra.run.smithi083.stderr:tar: Error is not recoverable: exiting now 2024-03-22T02:42:59.198 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi120/crash 2024-03-22T02:42:59.199 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash -- . 2024-03-22T02:42:59.232 INFO:teuthology.orchestra.run.smithi120.stderr:tar: /var/lib/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T02:42:59.232 INFO:teuthology.orchestra.run.smithi120.stderr:tar: Error is not recoverable: exiting now 2024-03-22T02:42:59.234 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-22T02:42:59.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-22T02:42:59.287 INFO:teuthology.orchestra.run.smithi028.stdout:2024-03-22T02:27:28.746916+0000 mds.c (mds.2) 69 : cluster [WRN] Scrub error on inode 0x1000000b87e (/volumes/qa/sv_1/33b08d6b-38d5-41a8-9440-fb5f0ce7dfa4/client.0/tmp/clients/client9/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.c log and `damage ls` output for details 2024-03-22T02:42:59.287 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-22T02:42:59.288 INFO:tasks.cephadm:Compressing logs... 2024-03-22T02:42:59.288 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T02:42:59.290 DEBUG:teuthology.orchestra.run.smithi083:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T02:42:59.292 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T02:42:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-22T02:42:59.332 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T02:42:59.336 INFO:teuthology.orchestra.run.smithi120.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-22T02:42:59.336 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T02:42:59.337 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-22T02:42:59.337 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T02:42:59.337 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-22T02:42:59.337 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.l.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-22T02:42:59.340 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T02:42:59.341 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-22T02:42:59.341 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.c.log: 94.8% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-22T02:42:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-22T02:42:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-22T02:42:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-22T02:42:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T02:42:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-22T02:42:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-22T02:42:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-22T02:42:59.344 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.j.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.x.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T02:42:59.345 INFO:teuthology.orchestra.run.smithi120.stderr: 93.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-22T02:42:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-22T02:42:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-22T02:42:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-22T02:42:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.d.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.a.log: 87.0% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T02:42:59.349 INFO:teuthology.orchestra.run.smithi083.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-22T02:42:59.350 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-22T02:42:59.353 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-22T02:42:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-22T02:42:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T02:42:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log 2024-03-22T02:42:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T02:42:59.354 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-22T02:42:59.357 INFO:teuthology.orchestra.run.smithi083.stderr:/var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.b.log: 94.6% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-22T02:42:59.359 INFO:teuthology.orchestra.run.smithi083.stderr: 93.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-22T02:42:59.405 INFO:teuthology.orchestra.run.smithi028.stderr: 90.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T02:42:59.412 INFO:teuthology.orchestra.run.smithi120.stderr: 83.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T02:42:59.446 INFO:teuthology.orchestra.run.smithi120.stderr: 90.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T02:42:59.448 INFO:teuthology.orchestra.run.smithi120.stderr: 87.0% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T02:42:59.482 INFO:teuthology.orchestra.run.smithi083.stderr: 83.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T02:42:59.487 INFO:teuthology.orchestra.run.smithi028.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T02:42:59.545 INFO:teuthology.orchestra.run.smithi120.stderr: 90.9% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T02:42:59.579 INFO:teuthology.orchestra.run.smithi083.stderr: 87.0% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T02:42:59.579 INFO:teuthology.orchestra.run.smithi083.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T02:42:59.600 INFO:teuthology.orchestra.run.smithi083.stderr: 90.9% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T02:42:59.618 INFO:teuthology.orchestra.run.smithi028.stderr: 83.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T02:42:59.699 INFO:teuthology.orchestra.run.smithi083.stderr: 90.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T02:42:59.725 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T02:42:59.778 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-22T02:42:59.789 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-22T02:43:00.054 INFO:teuthology.orchestra.run.smithi028.stderr: 94.7% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-22T02:43:00.660 INFO:teuthology.orchestra.run.smithi028.stderr: 94.8% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-22T02:43:00.705 INFO:teuthology.orchestra.run.smithi028.stderr: 90.6% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-22T02:43:01.479 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-22T02:43:05.557 INFO:teuthology.orchestra.run.smithi120.stderr: 94.8% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-22T02:43:09.229 INFO:teuthology.orchestra.run.smithi083.stderr: 92.0% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-22T02:43:10.130 INFO:teuthology.orchestra.run.smithi120.stderr: 93.3% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-22T02:44:01.631 INFO:teuthology.orchestra.run.smithi028.stderr: 91.6% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-22T02:44:02.048 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-22T02:44:04.843 INFO:teuthology.orchestra.run.smithi083.stderr: 94.7% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-22T02:44:26.066 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-22T02:44:26.403 INFO:teuthology.orchestra.run.smithi028.stderr: 91.3% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-22T02:44:26.406 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-03-22T02:44:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:real 1m27.103s 2024-03-22T02:44:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:user 3m56.531s 2024-03-22T02:44:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m7.505s 2024-03-22T02:45:39.972 INFO:teuthology.orchestra.run.smithi120.stderr: 94.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-22T02:45:53.222 INFO:teuthology.orchestra.run.smithi120.stderr: 94.1% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-22T02:46:13.519 INFO:teuthology.orchestra.run.smithi120.stderr: 91.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-22T02:46:43.944 INFO:teuthology.orchestra.run.smithi120.stderr: 91.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-22T02:46:49.195 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-22T02:47:11.481 INFO:teuthology.orchestra.run.smithi120.stderr: 91.1% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-22T02:47:11.574 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-22T02:47:11.574 INFO:teuthology.orchestra.run.smithi120.stderr:real 4m12.177s 2024-03-22T02:47:11.574 INFO:teuthology.orchestra.run.smithi120.stderr:user 4m40.265s 2024-03-22T02:47:11.575 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m10.513s 2024-03-22T02:51:02.475 INFO:teuthology.orchestra.run.smithi083.stderr: 94.5% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-22T02:51:38.804 INFO:teuthology.orchestra.run.smithi083.stderr: 91.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-22T02:52:06.529 INFO:teuthology.orchestra.run.smithi083.stderr: 91.6% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-22T02:54:28.679 INFO:teuthology.orchestra.run.smithi083.stderr: 91.3% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-22T02:55:24.874 INFO:teuthology.orchestra.run.smithi083.stderr: 91.4% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-22T03:04:40.932 INFO:teuthology.orchestra.run.smithi083.stderr: 93.2% -- replaced with /var/log/ceph/426c3602-e7f0-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-22T03:04:40.954 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-03-22T03:04:40.955 INFO:teuthology.orchestra.run.smithi083.stderr:real 21m41.638s 2024-03-22T03:04:40.955 INFO:teuthology.orchestra.run.smithi083.stderr:user 13m1.504s 2024-03-22T03:04:40.955 INFO:teuthology.orchestra.run.smithi083.stderr:sys 0m32.112s 2024-03-22T03:04:40.955 INFO:tasks.cephadm:Archiving logs... 2024-03-22T03:04:40.957 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi028/log 2024-03-22T03:04:40.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T03:05:22.653 DEBUG:teuthology.misc:Transferring archived files from smithi083:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi083/log 2024-03-22T03:05:22.654 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T03:07:34.385 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi120/log 2024-03-22T03:07:34.390 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T03:08:23.619 INFO:tasks.cephadm:Removing cluster... 2024-03-22T03:08:23.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force 2024-03-22T03:08:23.818 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T03:08:24.408 DEBUG:teuthology.orchestra.run.smithi083:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force 2024-03-22T03:08:24.935 INFO:teuthology.orchestra.run.smithi083.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T03:08:25.780 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 426c3602-e7f0-11ee-95cd-87774f69a715 --force 2024-03-22T03:08:26.088 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: 426c3602-e7f0-11ee-95cd-87774f69a715 2024-03-22T03:08:26.711 INFO:tasks.cephadm:Removing cephadm ... 2024-03-22T03:08:26.712 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T03:08:26.733 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T03:08:26.755 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T03:08:26.772 INFO:tasks.cephadm:Teardown complete 2024-03-22T03:08:26.773 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-22T03:08:26.841 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-22T03:08:26.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T03:08:26.844 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T03:08:26.847 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-22T03:08:27.192 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-22T03:08:27.213 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-22T03:08:27.213 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T03:08:27.220 DEBUG:teuthology.orchestra.run.smithi083:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T03:08:27.223 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T03:08:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +2405us[+2405us] +/- 90ms 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -1898us[-1909us] +/- 74ms 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -252us[ -262us] +/- 88ms 2024-03-22T03:08:27.236 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T03:08:27.237 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-03-22T03:08:27.238 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-22T03:08:27.311 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T03:08:27.311 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-22T03:08:27.311 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +2434us[+2434us] +/- 90ms 2024-03-22T03:08:27.311 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -1867us[-1870us] +/- 74ms 2024-03-22T03:08:27.312 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -226us[ -228us] +/- 88ms 2024-03-22T03:08:27.312 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T03:08:27.354 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T03:08:27.354 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-03-22T03:08:27.354 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 +2406us[+2409us] +/- 90ms 2024-03-22T03:08:27.354 INFO:teuthology.orchestra.run.smithi083.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -1889us[-1886us] +/- 74ms 2024-03-22T03:08:27.355 INFO:teuthology.orchestra.run.smithi083.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -244us[ -241us] +/- 88ms 2024-03-22T03:08:27.355 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T03:08:27.356 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-22T03:08:27.369 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-22T03:08:27.370 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-22T03:08:27.383 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T03:08:27.524 DEBUG:teuthology.orchestra.run.smithi083:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T03:08:27.691 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T03:08:27.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T03:08:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:27.831 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T03:08:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:27.865 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-22T03:08:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:27.896 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-22T03:08:27.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-22T03:08:27.916 INFO:teuthology.task.internal:Duration was 4394.116963 seconds 2024-03-22T03:08:27.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-22T03:08:27.927 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-22T03:08:27.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T03:08:27.930 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T03:08:27.932 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T03:08:28.000 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T03:08:28.009 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T03:08:28.018 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T03:08:28.584 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-22T03:08:28.584 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-03-22T03:08:28.585 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T03:08:28.609 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi083.front.sepia.ceph.com 2024-03-22T03:08:28.609 DEBUG:teuthology.orchestra.run.smithi083:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T03:08:28.649 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-03-22T03:08:28.649 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T03:08:28.679 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-22T03:08:28.679 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T03:08:28.682 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T03:08:28.691 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T03:08:28.750 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-22T03:08:28.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T03:08:28.755 DEBUG:teuthology.orchestra.run.smithi083:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T03:08:28.776 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T03:08:29.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-22T03:08:29.185 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-22T03:08:29.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T03:08:29.226 DEBUG:teuthology.orchestra.run.smithi083:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T03:08:29.267 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T03:08:29.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-22T03:08:29.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.312 DEBUG:teuthology.orchestra.run.smithi083:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.315 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.353 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-03-22T03:08:29.354 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = core 2024-03-22T03:08:29.379 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-03-22T03:08:29.423 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:29.437 DEBUG:teuthology.orchestra.run.smithi083:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:29.451 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T03:08:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T03:08:29.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-22T03:08:29.490 INFO:teuthology.task.internal:Transferring archived files... 2024-03-22T03:08:29.492 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi028 2024-03-22T03:08:29.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T03:08:29.540 DEBUG:teuthology.misc:Transferring archived files from smithi083:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi083 2024-03-22T03:08:29.541 DEBUG:teuthology.orchestra.run.smithi083:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T03:08:29.582 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780/remote/smithi120 2024-03-22T03:08:29.582 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T03:08:29.623 INFO:teuthology.task.internal:Removing archive directory... 2024-03-22T03:08:29.623 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T03:08:29.626 DEBUG:teuthology.orchestra.run.smithi083:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T03:08:29.629 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T03:08:29.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-22T03:08:29.688 INFO:teuthology.task.internal:Not uploading archives. 2024-03-22T03:08:29.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-22T03:08:29.700 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-22T03:08:29.700 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T03:08:29.702 DEBUG:teuthology.orchestra.run.smithi083:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T03:08:29.705 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T03:08:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: 265575 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 03:08 /home/ubuntu/cephtest 2024-03-22T03:08:29.717 INFO:teuthology.orchestra.run.smithi083.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 03:08 /home/ubuntu/cephtest 2024-03-22T03:08:29.725 INFO:teuthology.orchestra.run.smithi120.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 03:08 /home/ubuntu/cephtest 2024-03-22T03:08:29.740 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-22T03:08:29.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-22T03:08:29.778 INFO:teuthology.nuke:Checking targets against current locks 2024-03-22T03:08:29.805 DEBUG:teuthology.nuke:shortname: smithi028 2024-03-22T03:08:29.806 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T03:08:29.829 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.866723', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBABBEJJ12xVeU9VVaU7NVr9a5HHvBVgeLXII/Azl/ExMNqZ9vKDXvj6JO64DOTTW2+8Q/3XW1ZbVtXg9Jq8DBy0='} 2024-03-22T03:08:29.834 DEBUG:teuthology.nuke:shortname: smithi083 2024-03-22T03:08:29.835 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T03:08:29.860 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.865487', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJcvtlWFYaAxAV7dno+NRUQgPrLuxHpRzJclzqb8XzmZ47IN8waAaMBXKI6oJCaTkx5G96PlyI5zFOO/ure4baM='} 2024-03-22T03:08:29.861 DEBUG:teuthology.nuke:shortname: smithi120 2024-03-22T03:08:29.861 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T03:08:29.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-22_01:06:30-fs-wip-batrick-testing-20240321.172515-distro-default-smithi/7615780', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 01:47:09.864015', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBYr+YhhK5n9sH0EXt+XDMAMa+El1iIL7sgm3Fd58rBkOpLPMTCoB97q5tRyIZsQbSzCcJFqVOC3AuLbsggY1gk='} 2024-03-22T03:08:29.895 INFO:teuthology.orchestra.console.smithi028:Power off 2024-03-22T03:08:29.895 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T03:08:29.925 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-03-22T03:08:29.925 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:29.942 INFO:teuthology.orchestra.console.smithi083:Power off 2024-03-22T03:08:29.943 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T03:08:29.959 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-22T03:08:29.969 DEBUG:teuthology.orchestra.console.smithi083:power off output: Chassis Power Control: Down/Off 2024-03-22T03:08:29.969 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:29.989 INFO:teuthology.orchestra.console.smithi120:Power off 2024-03-22T03:08:29.989 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T03:08:29.999 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-22T03:08:30.013 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-03-22T03:08:30.013 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:30.041 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-22T03:08:33.960 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:34.000 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:34.042 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:34.071 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-03-22T03:08:34.110 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-22T03:08:34.152 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-22T03:08:38.074 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:38.111 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:38.153 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:38.185 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-03-22T03:08:38.221 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is on 2024-03-22T03:08:38.266 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-22T03:08:38.286 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-03-22T03:08:42.226 DEBUG:teuthology.orchestra.console.smithi083:pexpect command: ipmitool -H smithi083.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:42.267 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T03:08:42.337 DEBUG:teuthology.orchestra.console.smithi083:check power output: Chassis Power is off 2024-03-22T03:08:42.377 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-03-22T03:08:42.438 INFO:teuthology.orchestra.console.smithi083:Power off completed 2024-03-22T03:08:42.479 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-03-22T03:08:42.587 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} duration: 4394.1169629096985 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_pdonnell@teuthology status: fail success: false 2024-03-22T03:08:42.588 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T03:08:42.677 INFO:teuthology.run:FAIL